My biggest wish for retainers is that they would retain their job level even if you switch them to something else for awhile.

As is it now, they revert back to level 1, even if you had them at max previously.