Does it happen a lot in large combats? Like alliance raids? Beyond the driver situation I've found turning off spell affects for anyone outside your group massively improved performance.