I would like that as well. There are some people in there that shouldn't even be following you to begin with. By giving you the option to remove them , gives you a bit more privacy. Allowing people to freely follow anyone is stalking or an easy bookmark to your profile page with little effort.