It's a possibility an ISPs route to the Datacenter could be saturated. I believe SE use Akamai Edgesuite for their CDN (Content Delivery Network) Services to serve the downloads. But a CDN is only as good as the ISPs Peering to it. I'm based in the UK, so my connection is currently going to 195.101.0.233 which is a CDN cache in France where I'm getting 1.4MB/s but before the load balancer kicked in, I was connected to 195.10.11.59 which is registered to a Network Operations Center in Leeds, UK. This is much closer since I only have to go through 9 Routers to get there as opposed to the 17 Routers. But my speed started to get about 0.2MB/s which is probably why I got switched to a different route server once the current file was completed.

This is just the nature of CDNs. SE don't run them, they just send the files to them they want distributed. You're probably getting the best speed for that route that's possible but SE aren't the bottleneck here. It's the ISPs between your home connection and the CDN Cache you're connecting to.