We could swing this the other way though in terms of immersion.
In real life, I have many talents.
I'm a wizard in the kitchen, you name it I can bake it.
I can ride horses, play instruments and even a mean game of tennis.
Why does it ruin immersion to have many talents ingame too? o_o.
It would suck if my life was defined by one of the things I do well, hell I'd be stuck in the kitchen forever. (inb4 sexist 'well you shouldn't be out of it jokes' lol)
If anything I find it flattering to your character's prowess if they can master many things without being forced into one little pidgeonhole D:
As for changing hats or whatever and suddenly having a different skillset:
Do you ride horses with scuba gear on?
Do you bake cookies while wearing plate armor? D:
Dressing for the occasion never hurt!