I don't know, seems to me like you're unlucky enough to be running Fates with a Fate party stealthing the area. I've had both hard and easy Fates in Heavensward areas and I run as Bard so my output isn't exactly top notch.