Lag whilst in dungeons.. mainly Stone Vigil

[savarah]

Where are the instance servers located?

The lag within this dungeon is disgusting.... ='(
(friends from other countries also get it, so its not my issue)

[Marishi-Ten]

It's hard to say where their instance servers are actually located. I've done some snooping around and I have found some information. Looking over the data I was able to acquire, it looks like Square rents its server space. There seems to be massive latency originating on the Seattle, Chicago, and Montreal (end point) hops owned by as6453.net, which are owned by Tata Communications. Their switches are showing quite of bit of slow down, (220ms+ latency) which in turn is causing more slow down and bottlenecking at the Montreal hops.

Tata Communications are a T1 telecomm in India, but they are at best a T3 in the US. I can't seem to find what T1 is hosting them in the US. If we can find who the backbone they are running through (AT&T, Century Link, Cogent, Level3, NTT, Savvis, SBC, Sprint, Verizon, XO) we can either field our own trouble tickets with them or we can contact our ISP's and request they avoid routing through their hubs.

If servers are being rented though Tata, then we can't reroute around them. We essentially have to deal with the latency until they redistribute their I/O paths or upgrade their switches.

Alternately, it could also be a problem with their MPLS circuits (hosted through Ormuco) reaching maximum bandwidth and causing latency.

The root issue more than likely lies between the two making it difficult to field any fix in a timely manner.

In short, it looks like Square doesn’t own the servers that are housing the North American and European worlds. This combined with the amount of traffic that their host is seeing is causing the latency issues.

3 184.99.64.201 (184.99.64.201) 18.925 ms 17.211 ms 17.251 ms
4 boid-agw1.inet.qwest.net (184.99.65.65) 23.503 ms 17.396 ms 36.997 ms
5 sea-brdr-02.inet.qwest.net (67.14.41.18) 44.756 ms 31.634 ms 37.997 ms
6 ix-1-0-0-0.tcore1.00s-seattle.as6453.net (64.86.123.77) 52.807 ms 304.980 ms 293.516 ms
7 if-0-0-0-2.core1.00s-seattle.as6453.net (64.86.123.2) 463.000 ms 294.448 ms 256.501 ms
8 if-9-3-3-0.tcore2.ct8-chicago.as6453.net (64.86.124.34) 323.445 ms 142.922 ms 96.168 ms
9 if-3-2.tcore1.w6c-montreal.as6453.net (66.198.96.45) 95.991 ms 96.259 ms 95.309 ms
10 66.198.96.50 (66.198.96.50) 99.636 ms 99.195 ms 99.700 ms
11 192.34.76.2 (192.34.76.2) 99.474 ms 98.574 ms 99.333 ms

whois 199.91.189.38
Ormuco ORMUCO-FIBER-NETWORK (NET-199-91-184-0-1) 199.91.184.0 - 199.91.191.255
Eidos ORMUC-SE (NET-199-91-189-0-1) 199.91.189.0 - 199.91.189.255

64 bytes from 199.91.189.38: icmp_seq=290 ttl=243 time=101.808 ms
64 bytes from 199.91.189.38: icmp_seq=291 ttl=243 time=101.363 ms
64 bytes from 199.91.189.38: icmp_seq=292 ttl=243 time=320.852 ms
64 bytes from 199.91.189.38: icmp_seq=293 ttl=243 time=422.690 ms
64 bytes from 199.91.189.38: icmp_seq=294 ttl=243 time=422.983 ms
64 bytes from 199.91.189.38: icmp_seq=295 ttl=243 time=484.630 ms
64 bytes from 199.91.189.38: icmp_seq=296 ttl=243 time=424.351 ms
64 bytes from 199.91.189.38: icmp_seq=297 ttl=243 time=445.440 ms
64 bytes from 199.91.189.38: icmp_seq=298 ttl=243 time=101.748 ms
64 bytes from 199.91.189.38: icmp_seq=299 ttl=243 time=101.246 ms

64 bytes from 199.91.189.38: icmp_seq=15 ttl=243 time=101.559 ms
64 bytes from 199.91.189.38: icmp_seq=16 ttl=243 time=100.622 ms
64 bytes from 199.91.189.38: icmp_seq=17 ttl=243 time=451.749 ms
64 bytes from 199.91.189.38: icmp_seq=18 ttl=243 time=104.527 ms
64 bytes from 199.91.189.38: icmp_seq=19 ttl=243 time=101.387 ms

64 bytes from 199.91.189.38: icmp_seq=25 ttl=243 time=101.613 ms
64 bytes from 199.91.189.38: icmp_seq=26 ttl=243 time=102.453 ms
64 bytes from 199.91.189.38: icmp_seq=27 ttl=243 time=402.926 ms
64 bytes from 199.91.189.38: icmp_seq=28 ttl=243 time=102.158 ms
64 bytes from 199.91.189.38: icmp_seq=29 ttl=243 time=101.519 ms

64 bytes from 199.91.189.38: icmp_seq=60 ttl=243 time=103.801 ms
64 bytes from 199.91.189.38: icmp_seq=61 ttl=243 time=100.424 ms
64 bytes from 199.91.189.38: icmp_seq=62 ttl=243 time=547.119 ms
64 bytes from 199.91.189.38: icmp_seq=63 ttl=243 time=467.761 ms
64 bytes from 199.91.189.38: icmp_seq=64 ttl=243 time=388.406 ms
64 bytes from 199.91.189.38: icmp_seq=65 ttl=243 time=557.122 ms
64 bytes from 199.91.189.38: icmp_seq=66 ttl=243 time=537.825 ms
64 bytes from 199.91.189.38: icmp_seq=67 ttl=243 time=102.962 ms
64 bytes from 199.91.189.38: icmp_seq=68 ttl=243 time=103.227 ms

http://www.as6453.net/

http://www.tatacommunications.com/

http://www.tatacommunications.com/ab...ontact_offices

http://en.wikipedia.org/wiki/Tata_Communications

http://www.ormuco.com/telecom-solutions/mpls/

[LadyFreyja]

Sad. SE you're not that broke. Especially now. Please fix this latency with all the money you're making, thanks.

[Marishi-Ten]

It's not so much an issue with Square as it is with their providers. The base problem sounds like it's with Ormuco/Tata. Ormuco's MPLS circuits aren't robust enough to run the traffic they are getting reliably. The good news is, the fix times are extremely fast on this as it costs Ormuco money and more importantly, their reputation. Providers are quick to stop gap these issues because some ISP's can re route you around their data centers completely. The same goes for Tata. Their ports are seeing tons of slowdowns as well.

I would imagine the reason for the long maintenance on Wednesday is for Tata to upgrade their racks (physical additions or allocation and synchronization of existing partitions) and for Ormuco to upgrade their MPLS circuits.

I would imagine when the bandwidth issues were surfacing, the System Admins for Square were the first on the phone talking to them trying to alleviate the congestion. That they have to talk to two completely separate companies is by itself challenging. That coupled with the language barriers (Japanese, French, English) makes a complicated situation exponentially more complex and time consuming.

[KMarsara]

It's hard to say where their instance servers are actually located. I've done some snooping around and I have found some information. Looking over the data I was able to acquire, it looks like Square rents its server space. There seems to be massive latency originating on the Seattle, Chicago, and Montreal (end point) hops owned by as6453.net, which are owned by Tata Communications. Their switches are showing quite of bit of slow down, (220ms+ latency) which in turn is causing more slow down and bottlenecking at the Montreal hops.

Tata Communications are a T1 telecomm in India, but they are at best a T3 in the US. I can't seem to find what T1 is hosting them in the US. If we can find who the backbone they are running through (AT&T, Century Link, Cogent, Level3, NTT, Savvis, SBC, Sprint, Verizon, XO) we can either field our own trouble tickets with them or we can contact our ISP's and request they avoid routing through their hubs.

If servers are being rented though Tata, then we can't reroute around them. We essentially have to deal with the latency until they redistribute their I/O paths or upgrade their switches.

Alternately, it could also be a problem with their MPLS circuits (hosted through Ormuco) reaching maximum bandwidth and causing latency.

The root issue more than likely lies between the two making it difficult to field any fix in a timely manner.

In short, it looks like Square doesn’t own the servers that are housing the North American and European worlds. This combined with the amount of traffic that their host is seeing is causing the latency issues.

3 184.99.64.201 (184.99.64.201) 18.925 ms 17.211 ms 17.251 ms
4 boid-agw1.inet.qwest.net (184.99.65.65) 23.503 ms 17.396 ms 36.997 ms
5 sea-brdr-02.inet.qwest.net (67.14.41.18) 44.756 ms 31.634 ms 37.997 ms
6 ix-1-0-0-0.tcore1.00s-seattle.as6453.net (64.86.123.77) 52.807 ms 304.980 ms 293.516 ms
7 if-0-0-0-2.core1.00s-seattle.as6453.net (64.86.123.2) 463.000 ms 294.448 ms 256.501 ms
8 if-9-3-3-0.tcore2.ct8-chicago.as6453.net (64.86.124.34) 323.445 ms 142.922 ms 96.168 ms
9 if-3-2.tcore1.w6c-montreal.as6453.net (66.198.96.45) 95.991 ms 96.259 ms 95.309 ms
10 66.198.96.50 (66.198.96.50) 99.636 ms 99.195 ms 99.700 ms
11 192.34.76.2 (192.34.76.2) 99.474 ms 98.574 ms 99.333 ms

whois 199.91.189.38
Ormuco ORMUCO-FIBER-NETWORK (NET-199-91-184-0-1) 199.91.184.0 - 199.91.191.255
Eidos ORMUC-SE (NET-199-91-189-0-1) 199.91.189.0 - 199.91.189.255

64 bytes from 199.91.189.38: icmp_seq=290 ttl=243 time=101.808 ms
64 bytes from 199.91.189.38: icmp_seq=291 ttl=243 time=101.363 ms
64 bytes from 199.91.189.38: icmp_seq=292 ttl=243 time=320.852 ms
64 bytes from 199.91.189.38: icmp_seq=293 ttl=243 time=422.690 ms
64 bytes from 199.91.189.38: icmp_seq=294 ttl=243 time=422.983 ms
64 bytes from 199.91.189.38: icmp_seq=295 ttl=243 time=484.630 ms
64 bytes from 199.91.189.38: icmp_seq=296 ttl=243 time=424.351 ms
64 bytes from 199.91.189.38: icmp_seq=297 ttl=243 time=445.440 ms
64 bytes from 199.91.189.38: icmp_seq=298 ttl=243 time=101.748 ms
64 bytes from 199.91.189.38: icmp_seq=299 ttl=243 time=101.246 ms

64 bytes from 199.91.189.38: icmp_seq=15 ttl=243 time=101.559 ms
64 bytes from 199.91.189.38: icmp_seq=16 ttl=243 time=100.622 ms
64 bytes from 199.91.189.38: icmp_seq=17 ttl=243 time=451.749 ms
64 bytes from 199.91.189.38: icmp_seq=18 ttl=243 time=104.527 ms
64 bytes from 199.91.189.38: icmp_seq=19 ttl=243 time=101.387 ms

64 bytes from 199.91.189.38: icmp_seq=25 ttl=243 time=101.613 ms
64 bytes from 199.91.189.38: icmp_seq=26 ttl=243 time=102.453 ms
64 bytes from 199.91.189.38: icmp_seq=27 ttl=243 time=402.926 ms
64 bytes from 199.91.189.38: icmp_seq=28 ttl=243 time=102.158 ms
64 bytes from 199.91.189.38: icmp_seq=29 ttl=243 time=101.519 ms

64 bytes from 199.91.189.38: icmp_seq=60 ttl=243 time=103.801 ms
64 bytes from 199.91.189.38: icmp_seq=61 ttl=243 time=100.424 ms
64 bytes from 199.91.189.38: icmp_seq=62 ttl=243 time=547.119 ms
64 bytes from 199.91.189.38: icmp_seq=63 ttl=243 time=467.761 ms
64 bytes from 199.91.189.38: icmp_seq=64 ttl=243 time=388.406 ms
64 bytes from 199.91.189.38: icmp_seq=65 ttl=243 time=557.122 ms
64 bytes from 199.91.189.38: icmp_seq=66 ttl=243 time=537.825 ms
64 bytes from 199.91.189.38: icmp_seq=67 ttl=243 time=102.962 ms
64 bytes from 199.91.189.38: icmp_seq=68 ttl=243 time=103.227 ms

http://www.as6453.net/

http://www.tatacommunications.com/

http://www.tatacommunications.com/ab...ontact_offices

http://en.wikipedia.org/wiki/Tata_Communications

http://www.ormuco.com/telecom-solutions/mpls/

can you please put this in that ISP throttling thread

[savarah]

That is an awesome reply, thank you =)

Let's hope for a speedy resolution then! It'd not be so bad if i wasn't the healer :3

[Marishi-Ten]

I've posted my data in the stickied thread. I may end up using wireshark to see if I can get anymore visibility. I'll keep everyone posted if I come across anything new.

Thanks!