You dont get the penalty if you access the dungeon via roulette, regardless of when you leave.

Tested it multiple times, even in the middle of the dungeon and never got any penalty.