I like your idea because it serves both as appearance customization and as a compromise between gear swaps and no gear swaps.

But.

I don't think it will work as gear swapping because of one major reason: there are tiered sets of gear available now, and there will likely continue to be tiered sets of gear. Someone who is using darklight gear now has no reason to go back and swap in gear from the beastman camps or older dungeons. Gear swapping has a requirement that getting new gear provides sidegrades and is used situationally, as opposed to straight up being an upgrade in every sense.

I also think that the major argument is not that there is gear swapping in general -- because that is something that you can do now to optimize your existing gear before a battle. The argument is allowing gear-swapping in the middle of combat, which I will argue up and down that it is a bad idea if only because it an obtuse mechanic that becomes the default gameplay style.