Easy fix for this..

Implement a stat tracking system behind the scenes that stores the number of times a person withdraws from a Duty.

Then use this information to put these people into the same pool for Duty Finder groups.