You're right. You can't participate in a Duty (story instance) while queue'd for the Duty Finder (dungeons you want to run). It actually makes sense. You can kill all the mobs, do FATEs or anything you want solo.. craft or gather.. fish... but if you join a group or go into a Duty then you get kicked from the Duty Finder.