It doesn't put you back at the start (though at times the DF does break), usually it sets your progress to just you alone but after a few seconds it will return you to the same group. However, it changes if the person withdraws is in a group and pulls everyone in that group out of the party, then it'll take more time to refill and sometimes move people around to fill a previous group up, etc etc which will take longer.