Whose end is it?

[KraggyKor]

If TeamSpeak and iTunes Radio run just fine while I'm randomly lagging while playing, the problem shouldn't be on my side, right?

If others are playing on the same server as you without problems then clearly it's probably not SE's 'end' .. whether it's your 'end' or somewhere in between like your ISP throttling P2P is anyone's guess until you look into it.