Request for information: ISPs and P2P throttling

[Ecksdee]

I get the feeling that there are two separate issues going on:

1.) ISP's may be throttling back some users as they think that the Final Fantasy XIV client is Peer To Peer (P2P) sharing. Some ISP's do this to stop gap large sharing clients (BitTorrent) causing latency in their networks.

2.) There are massive latency spikes on the server end causing "rubber banding" and delays when executing actions and movement.

If the ISP is throttling the actual client, one wouldn't see low ping rates followed by some extremely high latency followed by a drop to normal again. Throttling would be a low connection all the time with little to no jumping around (see my post here for reference http://forum.square-enix.com/ffxiv/t...ottling/page76 )

"Rubber Banding" is a symptom of packet loss and poor end point connection. When the server (Square) has to send the client (Player) packets there has to be an acknowledgement of the data on the client end and a confirmation of the data being received on the server end (Think MD5 checksum). If the packets are incomplete, the client can't confirm with the server it received all of the data so it tries again. In the meantime, all data stops parsing (the flow of data is interrupted). Once the packets are confirmed, the server and client then have to double down to send data and present them visually while syncing their clocks together (the speed up and all of the logs spilling in at once).

Hopefully Square can set up another sticky for the second issue (Easily duplicated. I've even set up a template for users if need be) so we can store aggregate values there and not clutter up an existing issue.

In the end, it may not be necessary as the "rubber banding" issue seems to stem from the MPLS (Ormuco) and rack (Tata) end. Until the MPLS circuits are upgraded to accommodate more bandwidth and the server ports are expanded and prioritized, latency and route paths will be impacted.

I am definitely experiencing the 2nd issue more than the 1st. Is there a viable fix and is it something on the end of the user or is it something that SE needs to fix?

[Conana]

ISP: Comcast XFinity
Country/Region: San Francisco, USA
Occur time: Throughout the day and night

No problem with the internet as I can browse other stuff without any problem. All routers/etc are up to date.

[deklyn]

ISP Shaw Cable
Country/Region: West Coast Canada
Time of occurrence: always since the last patch
Have not called ISP regarding issue as of yet

[Marishi-Ten]

I am definitely experiencing the 2nd issue more than the 1st. Is there a viable fix and is it something on the end of the user or is it something that SE needs to fix?

I can only speculate as to what the issue is as I don't have root access to Tata racks and Ormucos fiber.

I have no doubt that the Square is aware of the issue and that the servers going down for maintenance is, in part, to alleviate the latency spikes. Though the client is owned by Square, their servers and their fiber nodes are run by another company. The bonus to this is that Square doesn't have to field or staff techs to monitor and maintain servers. Square also doesn't have to buy servers so they save money. The drawback is when something goes wrong, they are helpless to actually correct the issue. All they can do is field master tickets on Tata and Ormuco CRM databases.

The maintenance scheduled for 9/11/13 is quite long. This actually tells us that there is going to be hardware changes and upgrades. 2-3 hour maintenance are more software (client) side. 8-12 hours signifies that hardware on the MPLS and servers are being modified.

If the fix today doesn't correct the issues users are seeing, there are a couple of things to try:

1.) If using wifi, attempt to hardline the connection. Does the congestion clear up?

2.) Be sure to open and forward the ports Square has detailed in a stickied thread in the tech support Forums.

3.) You may clear the issue using a third party program such as PingZapper or the like. I prefer to not use this method. Using a third party program leaves you at the mercy of the program and the company who services it. This is a band aid for the root problem. You would essentially be treating the symptom, not curing the disease.

4.) VPN's have been reported to help clear the issue. This is a gamble as VPN's work just like Tata does. Multiple users and the connection slows down and lags. The VPN may be able to re route you on a different route path essentially avoiding the high traffic data centers and ports.

Verizon DSL, Baltimore, Maryland

I had posted in here previously, reposting to add some ping and tracert info.

Lag started Friday, has continued since then. I can be all by myself with no one in visual range, I can be solo questing and fighting a mob, sitting in town, or in the Duty Finder and I will lag. At all times during the day. Got 90000'd a couple of times Friday in addition to the lag, but no disconnects since then, just strange inconsistent lagging. *VERY* small periods of normal gameplay, and then BAM.

Microsoft Windows [Version 6.2.9200]
(c) 2012 Microsoft Corporation. All rights reserved.

C:\Users\Jason>ping 184.107.107.176

Pinging 184.107.107.176 with 32 bytes of data:
Reply from 184.107.107.176: bytes=32 time=60ms TTL=53
Reply from 184.107.107.176: bytes=32 time=138ms TTL=53
Reply from 184.107.107.176: bytes=32 time=77ms TTL=53
Reply from 184.107.107.176: bytes=32 time=63ms TTL=53

Ping statistics for 184.107.107.176:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 60ms, Maximum = 138ms, Average = 84ms

C:\Users\Jason>ping 184.107.107.176

Pinging 184.107.107.176 with 32 bytes of data:
Reply from 184.107.107.176: bytes=32 time=60ms TTL=53
Reply from 184.107.107.176: bytes=32 time=61ms TTL=53
Reply from 184.107.107.176: bytes=32 time=61ms TTL=53
Reply from 184.107.107.176: bytes=32 time=61ms TTL=53

Ping statistics for 184.107.107.176:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 60ms, Maximum = 61ms, Average = 60ms

C:\Users\Jason>tracert 184.107.107.176

Tracing route to 184.107.107.176 over a maximum of 30 hops

1 <1 ms <1 ms <1 ms dslrouter.westell.com [192.168.1.1]
2 33 ms 34 ms 31 ms 10.4.42.1
3 32 ms 64 ms 34 ms G0-9-4-0.DLLSTX-LCR-22.verizon-gni.net [130.81.1
94.190]
4 33 ms 33 ms 33 ms so-1-1-0-0.RES-BB-RTR1.verizon-gni.net [130.81.1
99.2]
5 72 ms 34 ms 33 ms 0.ae1.BR3.IAD8.ALTER.NET [152.63.7.81]
6 50 ms 50 ms 50 ms ix-20-0.tcore1.AEQ-Ashburn.as6453.net [66.198.15
4.53]
7 60 ms * 67 ms if-2-2.tcore2.AEQ-Ashburn.as6453.net [216.6.87.1
]
8 66 ms 67 ms 61 ms if-11-2.tcore2.NJY-Newark.as6453.net [216.6.87.1
38]
9 59 ms 62 ms 60 ms if-4-4.tcore2.NYY-NewYork.as6453.net [66.198.111
.18]
10 60 ms 61 ms 67 ms if-2-2.tcore2.MTT-Montreal.as6453.net [64.86.226
.13]
11 62 ms 82 ms 62 ms if-0-2.tcore1.MTT-Montreal.as6453.net [216.6.115
.89]
12 63 ms 228 ms 153 ms 64.86.31.42
13 61 ms 62 ms 109 ms te8-3.dr10.mtl.iweb.com [67.205.127.238]
14 62 ms 60 ms 60 ms 72.55.128.44
15 64 ms 63 ms 62 ms 184.107.107.176

Trace complete.

EDIT: Saw mention of a program called Ping Zapper in a locked lag thread, tried it, and it does indeed work (at least for me). Downloaded, installed, and set up. 30 mins straight of lag-free playing. Logged off and closed Ping Zapper, and then logged back on. Immediate lag. Ran Ping Zapper again, launched XIV, lag-free. It's on a trial account at the moment, which limits you to 1 hour of game time before it stops working. I don't know if that's "1 hour, stops working, close program and reopen and it works fine", or "1 hour every 24 hours". I'd assume the latter, but I don't know. Don't have the time to test the trial account limit because I need to get some sleep before I go to work. But Hallelujah, it works flawlessly. If I lagged during my time using it, I could not tell. I'll see if tonight's maintenance fixes the problem, and if not, I'll be buying some prepaid Ping Zapper time tomorrow. My RAAAAAAAAAGE at having to possibly pay for a 3rd party tunneling program to make this game playable is mostly tempered by the fact that it's pretty damned cheap, and I'm still on my free time. Of course, tomorrow, if the lag persists outside of using it, I'll be dropping the four bucks for 30 days of Ping Zapper time; however, I can't say that I'll be completely happy doing it.

I had a few questions about your data. Are these polls before or after PingZapper? The reason I ask is that you have an extra hop through a data center that I haven't seen on any trace routes I've found on the Forums:

13 61 ms 62 ms 109 ms te8-3.dr10.mtl.iweb.com [67.205.127.238]

This data center is run by iweb: http://iweb.com/about/vision

I haven't seen this hop on either my own, or others trace routes. I've also noticed that your ping times are relatively low and are pretty close together (60-120ms). I wonder if this is because of the scrubber, or if you're routing through another data center that Square may have picked up.

Name of ISP: Verizon FiOS
Character name: Yuengling Light
Country of Residence: Fairfax County, Northern Virginia, United States

Same issue. Constant 5 second freezes.

I've isolated the issue down to one of these hops:

3 28 ms 14 ms 10 ms G0-9-2-6.WASHDC-LCR-21.verizon-gni.net [130.81.97.106]
4 9 ms 8 ms 8 ms ae1-0.RES-BB-RTR1.verizon-gni.net [130.81.209.206]
5 11 ms 12 ms 11 ms 0.ae5.XL1.IAD8.ALTER.NET [152.63.8.121]
6 9 ms 8 ms 8 ms 0.xe-10-0-1.GW9.IAD8.ALTER.NET [152.63.33.161]
7 * 14 ms 14 ms tinet-gw.customer.alter.net [152.179.50.30]
8 41 ms 39 ms 37 ms xe-1-0-0.mtl10.ip4.tinet.net [89.149.185.41]
9 41 ms 37 ms 40 ms ormuco-gw.ip4.tinet.net [216.221.156.110]
10 47 ms 41 ms * 192.34.76.2
11 47 ms 41 ms 42 ms 199.91.189.234
12 37 ms 39 ms 38 ms 199.91.189.74

Reply from 216.221.156.110: bytes=64 time=39ms TTL=250
Reply from 216.221.156.110: bytes=64 time=37ms TTL=250
Request timed out.
Request timed out.
Reply from 216.221.156.110: bytes=64 time=40ms TTL=250
Reply from 216.221.156.110: bytes=64 time=34ms TTL=250
Reply from 216.221.156.110: bytes=64 time=33ms TTL=250
Reply from 216.221.156.110: bytes=64 time=38ms TTL=250
Reply from 216.221.156.110: bytes=64 time=32ms TTL=250
Request timed out.
Reply from 216.221.156.110: bytes=64 time=38ms TTL=250
Request timed out.
Reply from 216.221.156.110: bytes=64 time=38ms TTL=250
Reply from 216.221.156.110: bytes=64 time=38ms TTL=250
Reply from 216.221.156.110: bytes=64 time=39ms TTL=250
Reply from 216.221.156.110: bytes=64 time=35ms TTL=250
Request timed out.
Reply from 216.221.156.110: bytes=64 time=38ms TTL=250
Reply from 216.221.156.110: bytes=64 time=42ms TTL=250


I tried getting Verizon to look into it, but they are blaming the "game provider". (0)

Verizon, though meaning well, is inaccurate in stating the issue is with the Game Provider. Looking over your parsing data, I see that you have low ping times. I don't see a whole lot of fluctuation. What I do see is the glaring time out requests. Your connection to the server port (Tata) is either congested or is dropping due to load. Yours is an excellent example of how the server you're connecting to behaves when congested or not specd to handle the amount of volume it's seeing.

Your issue unfortunately cannot be corrected by any via client side means. This is an issue with the actual server load and shows that the racks are congested and cannot negotiate with the amount of users it sees at this time. Until the server provider expands the servers (partition/addition) and redistributes the port assignments, the issue will persist.

Apologies for the long post!

[Yuengling]

Verizon, though meaning well, is inaccurate in stating the issue is with the Game Provider. Looking over your parsing data, I see that you have low ping times. I don't see a whole lot of fluctuation. What I do see is the glaring time out requests. Your connection to the server port (Tata) is either congested or is dropping due to load. Yours is an excellent example of how the server you're connecting to behaves when congested or not specd to handle the amount of volume it's seeing.

Your issue unfortunately cannot be corrected by any via client side means. This is an issue with the actual server load and shows that the racks are congested and cannot negotiate with the amount of users it sees at this time. Until the server provider expands the servers (partition/addition) and redistributes the port assignments, the issue will persist.

Apologies for the long post!

Verizon, though meaning well, is inaccurate in stating the issue is with the Game Provider. Looking over your parsing data, I see that you have low ping times. I don't see a whole lot of fluctuation. What I do see is the glaring time out requests. Your connection to the server port (Tata) is either congested or is dropping due to load. Yours is an excellent example of how the server you're connecting to behaves when congested or not specd to handle the amount of volume it's seeing.

Your issue unfortunately cannot be corrected by any via client side means. This is an issue with the actual server load and shows that the racks are congested and cannot negotiate with the amount of users it sees at this time. Until the server provider expands the servers (partition/addition) and redistributes the port assignments, the issue will persist.

Apologies for the long post!

Thanks for recognizing what I was actually trying to communicate. The issue isn't with Square-Enix at all, it's with an ISP someplace inbetween us and them. I dealt with this kind of crap in Iraq for a year, it's a bad cable or a overloaded device someplace no one really pays attention that has any interest in the game.

I haven't been able to isolate it, but from what I can tell, BOTH of these hops have issues. Both are TINET.net.

I've seen multiple other posts where people blame the ormuco-gw server IP, which is the same one I mention. They ARE NOT in Northern Virginia.

Logic would tell me that the tinet.net run systems are the problem.

I've had the same packet loss to both of these hops...

8 41 ms 39 ms 37 ms xe-1-0-0.mtl10.ip4.tinet.net [89.149.185.41]
9 41 ms 37 ms 40 ms ormuco-gw.ip4.tinet.net [216.221.156.110]

[KMarsara]

I can only speculate as to what the issue is as I don't have root access to Tata racks and Ormucos fiber.

I have no doubt that the Square is aware of the issue and that the servers going down for maintenance is, in part, to alleviate the latency spikes. Though the client is owned by Square, their servers and their fiber nodes are run by another company. The bonus to this is that Square doesn't have to field or staff techs to monitor and maintain servers. Square also doesn't have to buy servers so they save money. The drawback is when something goes wrong, they are helpless to actually correct the issue. All they can do is field master tickets on Tata and Ormuco CRM databases.

The maintenance scheduled for 9/11/13 is quite long. This actually tells us that there is going to be hardware changes and upgrades. 2-3 hour maintenance are more software (client) side. 8-12 hours signifies that hardware on the MPLS and servers are being modified.

If the fix today doesn't correct the issues users are seeing, there are a couple of things to try:

1.) If using wifi, attempt to hardline the connection. Does the congestion clear up?

2.) Be sure to open and forward the ports Square has detailed in a stickied thread in the tech support Forums.

3.) You may clear the issue using a third party program such as PingZapper or the like. I prefer to not use this method. Using a third party program leaves you at the mercy of the program and the company who services it. This is a band aid for the root problem. You would essentially be treating the symptom, not curing the disease.

4.) VPN's have been reported to help clear the issue. This is a gamble as VPN's work just like Tata does. Multiple users and the connection slows down and lags. The VPN may be able to re route you on a different route path essentially avoiding the high traffic data centers and ports.



I had a few questions about your data. Are these polls before or after PingZapper? The reason I ask is that you have an extra hop through a data center that I haven't seen on any trace routes I've found on the Forums:

13 61 ms 62 ms 109 ms te8-3.dr10.mtl.iweb.com [67.205.127.238]

This data center is run by iweb: http://iweb.com/about/vision

I haven't seen this hop on either my own, or others trace routes. I've also noticed that your ping times are relatively low and are pretty close together (60-120ms). I wonder if this is because of the scrubber, or if you're routing through another data center that Square may have picked up.



Verizon, though meaning well, is inaccurate in stating the issue is with the Game Provider. Looking over your parsing data, I see that you have low ping times. I don't see a whole lot of fluctuation. What I do see is the glaring time out requests. Your connection to the server port (Tata) is either congested or is dropping due to load. Yours is an excellent example of how the server you're connecting to behaves when congested or not specd to handle the amount of volume it's seeing.

Your issue unfortunately cannot be corrected by any via client side means. This is an issue with the actual server load and shows that the racks are congested and cannot negotiate with the amount of users it sees at this time. Until the server provider expands the servers (partition/addition) and redistributes the port assignments, the issue will persist.

Apologies for the long post!

you are just so awesome finally someone that can shed some light into this problem

[Jacin]

Marishi-Ten
"I had a few questions about your data. Are these polls before or after PingZapper? The reason I ask is that you have an extra hop through a data center that I haven't seen on any trace routes I've found on the Forums:

13 61 ms 62 ms 109 ms te8-3.dr10.mtl.iweb.com [67.205.127.238]

This data center is run by iweb: http://iweb.com/about/vision

I haven't seen this hop on either my own, or others trace routes. I've also noticed that your ping times are relatively low and are pretty close together (60-120ms). I wonder if this is because of the scrubber, or if you're routing through another data center that Square may have picked up."

Those ping times are all pre-Ping Zapper. I didn't bother running ping or tracert with Ping Zapper because the game was running without issue. If the problem persists after maintenance is over, and I suspect that it will due to there being no mention of a lag fix in any of the notes I could find, I'll run ping a tracert with Ping Zapper up and post those as well. I went ahead and bought 30 days of time for it, only $3.99, non-recurring billing, and it works.....so I figured what the hell.

[prymortal]

- Vodafone New Zealand (Priviously Telstra Clear)
- New Zealand
- Most times very frequent
- To hard to get ahold of.