When you get a "follower", you can choose to add them. That way you can choose who you want to be adding to the twitter string.
From the Twitter "Help" page: Suggested reading
Do you follow me? (What are followers?)
Followers are people who receive other peoples' Twitter updates. (Wait, isn't that everyone? Most of the time, yes.) When you post an update to your Twitter account, your followers will get it on their home page and/or phone. You don't have to follow everyone who follows you, and unless an account is private, you can follow and un-follow who ever you want without them following back. Mutual followers can send each other private messages, and you can even choose to get notified by email when someone new follows you or sends you a private message. Your follower/following stats are listed on your profile page.