Maybe just don't massively spam invites?

What I do is, when doing low level dungeons in roulette, I'll ask the sprouts if they'd like to join, and if they say yes, I'll send them an invite, or have someone else do it. I wouldn't ever go around spamming invites in the overworld, that's just spam.