I once got Doma Castle while I was casually levelling my Scholar and I hadn't gotten any upgrades uh, at all since level 60.
Alliance roulette was keeping me levelled so fast I kinda didn't put my mind to it.

Well sure enough I was barely able to do the dungeon properly. I felt terrible, I was such a pain holding everyone back. We did it, the tank didn't want me to leave and was very forgiving, but I still felt bad the entire run lol.
On the flip side I do love a challenge so there was that. But yes, I do agree that some ilvl requirements would be helpful.