No, you're in the right there, sounds like they were a premade group of three abusing Vote Abandon.

Pretty terrible behavior really, and really, incredibly lazy of them to not make a PF Speed Run group.

If that happened to me and they were on my server I'd blacklist them instantly, it might not have any effect in the long term but at least you'll feel a bit better about it...