Duty finder will get people from all across the data center, not just your server. So even if you're on a low population server there shouldn't be an issue with duty finder. Your long queue could be explained by you queuing as a dps (maybe?) or by you trying during off time.