After some messing around, it seems it's not a ping issue (I usually average about 50ms overall) but my jitter seems to range between 0-20% on some tests... I'm not sure what to make of this, but I know it's not a good thing.

Judging by your last post (and thank you for the useful information btw) I think I might be able to check which hop might be causing the issue but I'm not sure.