Once you've assigned retainer a job you cannot swap it without fully resetting their level back to 1 and leveling again. This can be done in the retainer menu, when they aren't on a venture and after you unequip all their gear.