Is my issue latency or?

[XiXiQ]

Is this difference really so vast?

absolutely, the above post is correct. on 250 MS you can contact the server for updates 4 times a second max (though 3 is probably more realistic). With 50ms you can do it 20 times.
You will see aoes delayed/shortened, and will only get a single OGC in between skills without it clipping time. You end up losing about 10% dps for every 100ms or so.

I was on Odin, pings were about 270 at the time. When they moved it to EU it was impossible, more like 400. I moved to Tonberry, get about 120ms. The difference in game play is quite stark.