They are supposedly removing that penalty.
You realize that if you played for about 4 hours a day, and did about 2 hours of content, and had to switch something every time, this would be 1440 switches a day? That's assuming it takes 5 seconds to switch something. It could get really tedious after awhile. Especially when it's something that should simply be a base ability to begin with.