It does indeed suck but it isn't, sadly, uncommon for some people to get the truly bad apples in pugs. There are 2 things you can do about this:

1. Report the player(s) for their abuse. If they are griefing you then you have every right to report them for it when you are not in the wrong.
2. Find a static group to play with that isn't so abusive. Your server's Novice Network, a Free Company, some friends, anyone you can find willing to go with you where you need to would work.