I posted this in another thread, but it was requested that I post here as well:
It's hard to say where their instance servers are actually located. I've done some snooping around and I have found some information. Looking over the data I was able to acquire, it looks like Square rents its server space. There seems to be massive latency originating on the Seattle, Chicago, and Montreal (end point) hops owned by as6453.net, which are owned by Tata Communications. Their switches are showing quite of bit of slow down, (220ms+ latency) which in turn is causing more slow down and bottlenecking at the Montreal hops.
Tata Communications are a T1 telecomm in India, but they are at best a T3 in the US. I can't seem to find what T1 is hosting them in the US. If we can find who the backbone they are running through (AT&T, Century Link, Cogent, Level3, NTT, Savvis, SBC, Sprint, Verizon, XO) we can either field our own trouble tickets with them or we can contact our ISP's and request they avoid routing through their hubs.
If servers are being rented though Tata, then we can't reroute around them. We essentially have to deal with the latency until they redistribute their I/O paths or upgrade their switches.
Alternately, it could also be a problem with their MPLS circuits (hosted through Ormuco) reaching maximum bandwidth and causing latency.
The root issue more than likely lies between the two making it difficult to field any fix in a timely manner.
In short, it looks like Square doesn’t own the servers that are housing the North American and European worlds. This combined with the amount of traffic that their host is seeing is causing the latency issues.
3 184.99.64.201 (184.99.64.201) 18.925 ms 17.211 ms 17.251 ms
4 boid-agw1.inet.qwest.net (184.99.65.65) 23.503 ms 17.396 ms 36.997 ms
5 sea-brdr-02.inet.qwest.net (67.14.41.18) 44.756 ms 31.634 ms 37.997 ms
6 ix-1-0-0-0.tcore1.00s-seattle.as6453.net (64.86.123.77) 52.807 ms 304.980 ms 293.516 ms
7 if-0-0-0-2.core1.00s-seattle.as6453.net (64.86.123.2) 463.000 ms 294.448 ms 256.501 ms
8 if-9-3-3-0.tcore2.ct8-chicago.as6453.net (64.86.124.34) 323.445 ms 142.922 ms 96.168 ms
9 if-3-2.tcore1.w6c-montreal.as6453.net (66.198.96.45) 95.991 ms 96.259 ms 95.309 ms
10 66.198.96.50 (66.198.96.50) 99.636 ms 99.195 ms 99.700 ms
11 192.34.76.2 (192.34.76.2) 99.474 ms 98.574 ms 99.333 ms
whois 199.91.189.38
Ormuco ORMUCO-FIBER-NETWORK (NET-199-91-184-0-1) 199.91.184.0 - 199.91.191.255
Eidos ORMUC-SE (NET-199-91-189-0-1) 199.91.189.0 - 199.91.189.255
64 bytes from 199.91.189.38: icmp_seq=290 ttl=243 time=101.808 ms
64 bytes from 199.91.189.38: icmp_seq=291 ttl=243 time=101.363 ms
64 bytes from 199.91.189.38: icmp_seq=292 ttl=243 time=320.852 ms
64 bytes from 199.91.189.38: icmp_seq=293 ttl=243 time=422.690 ms
64 bytes from 199.91.189.38: icmp_seq=294 ttl=243 time=422.983 ms
64 bytes from 199.91.189.38: icmp_seq=295 ttl=243 time=484.630 ms
64 bytes from 199.91.189.38: icmp_seq=296 ttl=243 time=424.351 ms
64 bytes from 199.91.189.38: icmp_seq=297 ttl=243 time=445.440 ms
64 bytes from 199.91.189.38: icmp_seq=298 ttl=243 time=101.748 ms
64 bytes from 199.91.189.38: icmp_seq=299 ttl=243 time=101.246 ms
64 bytes from 199.91.189.38: icmp_seq=15 ttl=243 time=101.559 ms
64 bytes from 199.91.189.38: icmp_seq=16 ttl=243 time=100.622 ms
64 bytes from 199.91.189.38: icmp_seq=17 ttl=243 time=451.749 ms
64 bytes from 199.91.189.38: icmp_seq=18 ttl=243 time=104.527 ms
64 bytes from 199.91.189.38: icmp_seq=19 ttl=243 time=101.387 ms
64 bytes from 199.91.189.38: icmp_seq=25 ttl=243 time=101.613 ms
64 bytes from 199.91.189.38: icmp_seq=26 ttl=243 time=102.453 ms
64 bytes from 199.91.189.38: icmp_seq=27 ttl=243 time=402.926 ms
64 bytes from 199.91.189.38: icmp_seq=28 ttl=243 time=102.158 ms
64 bytes from 199.91.189.38: icmp_seq=29 ttl=243 time=101.519 ms
64 bytes from 199.91.189.38: icmp_seq=60 ttl=243 time=103.801 ms
64 bytes from 199.91.189.38: icmp_seq=61 ttl=243 time=100.424 ms
64 bytes from 199.91.189.38: icmp_seq=62 ttl=243 time=547.119 ms
64 bytes from 199.91.189.38: icmp_seq=63 ttl=243 time=467.761 ms
64 bytes from 199.91.189.38: icmp_seq=64 ttl=243 time=388.406 ms
64 bytes from 199.91.189.38: icmp_seq=65 ttl=243 time=557.122 ms
64 bytes from 199.91.189.38: icmp_seq=66 ttl=243 time=537.825 ms
64 bytes from 199.91.189.38: icmp_seq=67 ttl=243 time=102.962 ms
64 bytes from 199.91.189.38: icmp_seq=68 ttl=243 time=103.227 ms
http://www.as6453.net/
http://www.tatacommunications.com/
http://www.tatacommunications.com/ab...ontact_offices
http://en.wikipedia.org/wiki/Tata_Communications
http://www.ormuco.com/telecom-solutions/mpls/