Turning off non-partied spell/skill effects has helped a little bit for me on PS3 with that issue... Until you start doing the fates in the heavily farmed zones... Like Costa Del Sol... Can't count the number of times I've had that giant crab fade from existence while I had it targeted only to die to it's AoE seconds later. Or another good one when the tank I'm partied with (and has aggro on said giant crab) suddenly disappears when I have him targeted, thus preventing me from even healing him.

It seems like it would be something as simple as giving party members / story/quest/fate mobs load priority over things being loaded.