What level did you get to? It doesn't really get slow until the last 15-20 levels or so.
And I'm pretty sure they inform you of the restriction when you choose the retainer's class.

The restriction is in place mostly so that people won't benefit from something like mining or battle retainers acquiring high level loot if they aren't high enough level to get said loot on their actual class. I'll agree that, since there is no distinction between battle retainers, the restriction is a little strange among them. However, most players use their main class's old gear as a hand-me-down to their retainer to keep the retainer's ilevel as high as possible, so it helps to have your battle retainer the same class as your main.