Well - but you can (and should) control who do you party with! I'm pretty sure this is working as intended (despite unclear description of the feature in - at least English - patch notes) otherwise the groups will "trade-off" withdraws within the party to still try to fish for a specific duty or one in progress. Imagine group of 7 players - each can withdraw twice before incurring any penalty so in total they can spam DF 14 times! Either way - if the "group" (although it's usually only one member) withdraws it affects other players who were paired with that group and such group should share penalty as whole, simple as that.