We all love the wardrobe, we really do, and we do understand your worries about server load and zoning-time etc etc, I have a suggestion that may have been suggested before, but, I strongly believe in it, if we have the capability to do such.
Make the wardrobe job-specific, (Or preferably create a second, job-specific wardrobe), this new wardrobe doesn't even need to load if you're not on a particular job, and it shouldn't accept gear that's not job specific, for example:
I'm on Blue Mage, I have a wardrobe that I can only add items to that are equippable by Blue Mage, and Blue Mage only, I can equip directly from it, like our current wardrobe.
When I change jobs to Warrior, my "Blue Mage wardrobe" completely dissapears and doesn't load, isn't visible anywhere on my character (thus, not creating 22 more inventories worth of server load every zone, only one specific to my job, exactly like wardrobe is now.) There would only be one wardrobe-per-job-per-player.
To prevent items getting "lost", these wardrobes should only accept job-specific-equipment, so that you can't go, "Oh where did I put my Thaumas body?" But, you don't care about your Blue Mage AF's location when you're on Warrior, so it's fine to be completely unfindable, for example.