If you're running into enough instances of this to where it is bothering you, you might be the problem causing your party members to want to grief you. I do my roulettes every day and only run into trolls once a week, if that.