Usually if it is being abused, someone else will reject the vote. I doubt it is being abused. If it is being abused, I suggest using the report harassment in support desk.