This happens all of the time, especially in places such as Aurum Vale where the dungeons are long and annoying. They're being punished for leaving if it makes you feel any better.

It's annoying but your attitude isn't going to help anything.