Dungeon Problem - Am I missing something?

[ReiMakoto]

I've always been getting a penalty even if someone else has left first. What prompted me to write this was me literally experiencing this about half an hour ago.

Whats probably happened is if you waited about 30 seconds longer the replacement would have shown up, and because the replacement was about to load in, it didn't treat it as a partially empty party