I leave early on Fates all the time because folks tend to spread out and do multiple FATEs at a time so I try to hit as many as I can so I can get rewards faster.
I'm sorry if it bugs you. It bugs me that folks split up (intentionally or not), forcing me to break off from the pack so I can not miss out on the benefits...