It's definitely ping-dependent, and I'm sure that adds up over a fight. Unfortunately, this is universal to all burst-phases, and it will likely never change. The best choice is, like you said, to adjust your skill speed accordingly, though it's definitely painful to have to build for ping.