While this is a nice idea, it's just not going to work. You can't have an inventory that doesn't load when you zone, because that would allow you to smuggle Rare items. Even your mog house storage, which you need to be in a mog house or garden to access, is still loading when you change areas (and it loads before your sack, satchel, and case, which is silly - it would be nice to see loading order changed to put the mog house use only bags last.)

So, if the wardrobe is always going to be loaded anyway, what's the use in restricting it to one job? Maybe it could be made to load before the other wardrobes, but ten new wardrobes and a system to manually set the loading order would do that better.