The problem with that FATE is it moves very quickly through Old Gridania, and it's level synced, so if you're doing it on a high level class/Job if it happens to move the FATE boundary away from you, it deactivates the level sync and ends the FATE for you, thus the special duty action disappears. If I remember right, you also have to target the children with green nameplates (there are other children around too but they're just background npcs and are not part of the FATE).