I think the answer is "it depends." The circumstances of when you leave the duty, and why (if a GM can determine that), do matter. However, they do say you can land in hot water if you leave right after entering a duty (ex: if you get a dungeon you don't enjoy). So, yeah. Try not to do that.