I'm surprised that more of you don't have a problem with how unrealistic the gear swapping is. If you really want gear swap, I'd say swapping weapons would be alright, but armor, too? I am not at all okay with how unrealistic it feels in a game.
I would rather have the skill in playing revolve around timing(battle regimens and weaponskills), wearing the right armor for the abilities I have on my bar/role I play, and in putting together a well-balanced party. Instead of placing all "skill" in how many macros you can write or pressing one more button before an attack, it is more fun (imho) to have to use strategy in order to get what you want. Having to pick and choose between what you are able to do, and another party member filling roles that do what you can't.
If you need to do more dmg, don't rely on gear swapping, have a party member spec himself to do buffs, or battle regimens to pick up that slack. Use the tools already in game to achieve those stat bonuses.