You're blaming SE. I don't feel you're gimping the parties you're leaving though, I think a light party could beat that with little help and empty spots get filled fast, but you're still abandoning something you agreed to potentially do. It's one dungeon out of quite a few, but you're still cherry picking.