I was hoping I was wrong, thank you for confirming.

What a weirdly unintuitive system. If retainers are already limited to what the character has done they shouldn't be further limited by being unable to swap from one job to another. It just doesn't feel like that's how it should work.