I have this issue all the time. It has definitely gotten much worse since Heavensward came out and I do think latency plays a bit of a role in it (Bismark EX and Ravana EX are both very texture heavy so I lag in them horrifically), but I've also had it just not drop during a small trash pull despite having 50+ fps and a decent connection. I've also had a huge problem with my spells randomly canceling after I tried to turn off Cleric, in the end I'd still have Cleric on and whoever might have died because my Cure II never happened at all.