Pets can glitch and ignore retreat commands. They will retreat for one second and then resume hitting the target, even if the target is attacking the pet and not the summoner
I've seen this occur before, and I think I'd attribute it to the unique hate properties of Retreat, which according to Kaeko is -10 CE. I believe the avatar glitches depending on the current CE of the SMN at the time of use. Also, considering this occurs rather often on fresh mobs, my guess is if the SMNs existing CE is less than a certain level, then it will glitch.

Like, I don't really remember having Retreat glitch on me when I'm fighting mobs which I know I've pulled solid hate against, late into the battle (not auto-attack kind). Then again, its not something you use frequently.