I've had this happen. Some complete stranger invites me to a party. I decline. He invites me again, and I decline. Seriously!? What compelled you to invite me the second time!? You thought maybe I'd changed my mind?

I think it might have been because I was taking part in a FATE at the time, so maybe he thought I wanted to join his stupid FATE-grinding party. I can do without that nonsense, but I understand that is other people's idea of a "good time." Even so, if I wanted to join a FATE-grinding party in the vicinity, I'd say so. And if you're looking for more people for your group, you can also say so. Don't just randomly invite every stranger around to your party. And when they decline, don't invite them again.