It counts as a withdraw from the duty and you have to do it 3x to get a penality. It was implemented to prevent people from trying to cherry pick into duties and jamming up the queue. That was really fun when Crystal Tower came out.
It's annoying when it happens unintentionally, but it was the behavior of the player base that made SE implement the system. We only have ourselves to blame lol.