Remember, you're retainer cannot level a class higher than you have leveled that class.

So, if you're still at 50, then they are stuck at 50, too.