I don't always accept blind invites either. Sometimes I know why I'm getting the invite even though no one has asked me to join, I.E. Dungeon invite etc. depending on the time I have, and what I'm doing, depends on whether I'll accept the invite or not. If it's a blind invite I'll usually not say anything just decline, I don't see anything wrong with it, and if that's how you prefer to play, don't let anyone else dictate to you.