I believe withdrawing when DF pop and Going into a dungeon to see if your in the right party and then pulling out is 2 different things.
If you withdraw 3 times when DF Pop, its a penalty
However if you que in progress, accepted it, see how you didnt get your friends party and pull out, it shouldnt give you a penalty.