If your retainer is a DoW class, I have been getting rare dyes with ilvl110 and above on.
So at least level 50.

I have not gotten any rare items via my gatherer retainer, so I am unsure about that.