Verizon Fios and Patch 2.3=Massive Lag

[Legendzerox]

I've been playing online games for a long time on Verizon FIOS along with FF14. I was completely fine playing ff14 and then 2.3 hits and I take a massive performance hit.

I talked to a guildy and she says she has Uverse and also took a big hit. If it was throttling, it wouldn't conveniently happen right after patch day.

[elpacolopez123]

Sure wish they would do something about that FCKING mog station and those ereor code BS

[Raist]

I've been playing online games for a long time on Verizon FIOS along with FF14. I was completely fine playing ff14 and then 2.3 hits and I take a massive performance hit.

I talked to a guildy and she says she has Uverse and also took a big hit. If it was throttling, it wouldn't conveniently happen right after patch day.

It isn't "throttling" in the sense that everyone typically interprets it. It is congestion control... or rather, a poor or virtually non-existent implementation of it. The north east corridors in the US are a MAJOR choking point for THIS game. More than half of all the traffic to Montreal is forced through there. There is also the little detour we take with the NSA check-in in Chicago, and the hops to get around the Great Lakes that have to be taken from the Midwest and people from the North West that get slung around British Columbia that can run into another bit of a bottleneck along the way. When ony one of the corridors in these areas get overloaded, everyone going through them suffers, regardless of who their local ISP.

It is a problem that exists between those local ISP's and the 4 top tier ISP's that SE's ISP peers with. Everyone uses these same people for transit/peering to connect each other's networks if they are not able to enter into more direct peering agreements with them. Comparing this game that is hosted in Montreal is not necessarily an apples to apples comparison. If you are in Texas and are playing a game hosted on a server in Chicago, you are in no way going to run into the same congestion issues you face coming through New York and New Jersey.

Because of the limited number of corridors amongst the big 3 routing partners that get the US into Montreal (it's about 20 combinations of provider and corridor), when one goes down it can have a sever impact on the players. If all things were distributed evenly, just one provider having issues in one corridor would affect 5% of the entire population at once. If it is the corridor in general that is affecting all 3 providers, then you are looking at 15%. If it is a more widespread regional thing and 5 paths are affected, then it could run anywhere from 25% (one provider) all the way up to 75% (5 paths, 3 providers each). Now, consider how the population and traffic flow is distributed in reality (more than half of the paths go through the north east) and the numbers ramp up considerably.

When the North East region is in trouble across the board (ie, major weather, massive traffic spike from seasonal events in retail, entertainment, politics, etc.), it has a major impact on quality of service between Montreal and the US. When 2.3 hit the wire, we also hit one of the biggest peaks in internet traffic for the entire YEAR in the US. There is a connection there... just look at the history in those links to Akamai I provided earlier.

A similar thing happens for the West Coast and the Pacific region. When Southern California is having a brain fart, a massive amount of people out in Australia, Hawaii, etc... all the way to Asia suffer when trying to get to services here in the US--because there are some major choking points for the undersea transit lines in Southern California. A similar phenomenon exists in the North East for the UK as well, but they have an escape route that comes around Nova Scotia that helps to mitigate it better.

The point is, there are patterns that you can track that explain a lot of this mess. The problem is a lot of local providers don't analyze their routing often enough (some only do it semi-anually or quarterly), so they don't compensate quickly enough. And to add insult to injury, when problems are brought to their attention, many don't sense the urgency of the problem or otherwise don't feel compelled to investigate further much less take appropriate action. Even when they are offered financial assistance to conduct upgrades (Level 3 offered to pay for and even install hardware upgrades for some of them), they still opt out of making improvements. This isn't speculation by the way, there have been articles linked to on these forums that detail this incident quite clearly.

We pay our ISP's for our service. We need to demand they provide an acceptable level of quality for that service. They've gotten away with running roughshod over us for nearly 2 decades now... time to hold their feet to the fire.

[Ballistic]

God, I thought I was going crazy, at least I know what's going on now. I thought it was just me. I lag like mad during those hours also. Same thing, FIOS and on Siren.

[LimitBreakOP]

I'm not in Siren, but I've been getting horrendous lag on Balmung. I'm able to play the game for a little while without any issues when I first start it up, but before long the lag starts to settle in. From there I just see players skipping along the map as they run and stop constantly from one destination to the next, NPC dialogue is delayed by a few seconds or so after I click on an npc, and in fights my character and the enemies around just stop moving every so often. Is yours the same as well? I'd like to clarify. My problem seems to persist all throughout the day rather than just 8PM to 10PM

[kiotsukete]

Same issue here, on FIOS but in So.Cal and also experiencing heavy lag over the past couple months. Even got 90k tonight during an alexandrite coffer ._.; Thought I was going crazy too but I'm glad to know I'm not the only one, even though it's still frustrating. I'll drop FFXIV way before FIOS though, since my Internet service has otherwise been great (Netflix, web surfing, Destiny, etc.)

[Angelzbl0od]

Microsoft Windows [Version 6.1.7601]
Copyright (c) 2009 Microsoft Corporation. All rights reserved.

C:\Users\Brock>tracert 199.91.189.74

Tracing route to 199.91.189.74 over a maximum of 30 hops

1 <1 ms <1 ms <1 ms Wireless_Broadband_Router.home [192.168.1.1]
2 4 ms 4 ms 4 ms L100.NYCMNY-VFTTP-125.verizon-gni.net [72.89.93.
1]
3 6 ms 10 ms 9 ms G0-11-1-6.NYCMNY-LCR-22.verizon-gni.net [130.81.
107.242]
4 * * * Request timed out.
5 6 ms 6 ms 7 ms 2.ae0.XT2.NYC4.ALTER.NET [140.222.227.27]
6 8 ms 11 ms 11 ms TenGigE0-7-0-4.GW8.NYC4.ALTER.NET [152.63.25.226
]
7 25 ms 25 ms 24 ms tinet-gw.customer.alter.net [152.179.72.122]
8 32 ms 34 ms * xe-1-3-0.mtl10.ip4.gtt.net [89.149.184.74]
9 33 ms * 35 ms ormuco-gw.ip4.gtt.net [216.221.156.110]
10 34 ms 36 ms * 192.34.76.2
11 36 ms 34 ms 34 ms 199.91.189.234
12 33 ms 34 ms 34 ms 199.91.189.74

Trace complete.

C:\Users\Brock>tracert 199.91.189.74

Tracing route to 199.91.189.74 over a maximum of 30 hops

1 <1 ms <1 ms <1 ms Wireless_Broadband_Router.home [192.168.1.1]
2 3 ms 4 ms 4 ms L100.NYCMNY-VFTTP-125.verizon-gni.net [72.89.93.
1]
3 7 ms 6 ms 6 ms G0-11-1-6.NYCMNY-LCR-22.verizon-gni.net [130.81.
107.242]
4 * * * Request timed out.
5 8 ms 6 ms 7 ms 2.ae0.XT2.NYC4.ALTER.NET [140.222.227.27]
6 11 ms 12 ms 9 ms TenGigE0-7-0-0.GW8.NYC4.ALTER.NET [152.63.22.45]

7 23 ms 25 ms * tinet-gw.customer.alter.net [152.179.72.122]
8 31 ms * 33 ms xe-1-3-0.mtl10.ip4.gtt.net [89.149.184.74]
9 * * * Request timed out.
10 35 ms 33 ms 34 ms 192.34.76.2
11 35 ms 34 ms 39 ms 199.91.189.234
12 35 ms * * 199.91.189.74
13 34 ms 33 ms * 199.91.189.74
14 * * 35 ms 199.91.189.74

Trace complete.

C:\Users\Brock>tracert 199.91.189.74

Tracing route to 199.91.189.74 over a maximum of 30 hops

1 <1 ms <1 ms <1 ms Wireless_Broadband_Router.home [192.168.1.1]
2 5 ms 4 ms 4 ms L100.NYCMNY-VFTTP-125.verizon-gni.net [72.89.93.
1]
3 9 ms 7 ms 8 ms G0-11-1-6.NYCMNY-LCR-22.verizon-gni.net [130.81.
107.242]
4 * * * Request timed out.
5 7 ms 7 ms 7 ms 2.ae0.XT2.NYC4.ALTER.NET [140.222.227.27]
6 16 ms 9 ms 12 ms TenGigE0-7-0-8.GW8.NYC4.ALTER.NET [152.63.16.125
]
7 * 24 ms * tinet-gw.customer.alter.net [152.179.72.122]
8 * 38 ms 41 ms xe-1-3-0.mtl10.ip4.gtt.net [89.149.184.74]
9 * 39 ms * ormuco-gw.ip4.gtt.net [216.221.156.110]
10 * * 34 ms 192.34.76.2
11 * 35 ms 34 ms 199.91.189.234
12 31 ms 34 ms 32 ms 199.91.189.74

Trace complete.

C:\Users\Brock>tracert 199.91.189.74

Tracing route to 199.91.189.74 over a maximum of 30 hops

1 <1 ms <1 ms <1 ms Wireless_Broadband_Router.home [192.168.1.1]
2 3 ms 4 ms 4 ms L100.NYCMNY-VFTTP-125.verizon-gni.net [72.89.93.
1]
3 10 ms 6 ms 9 ms G0-11-1-6.NYCMNY-LCR-22.verizon-gni.net [130.81.
107.242]
4 * * * Request timed out.
5 9 ms 7 ms 7 ms 2.ae0.XT2.NYC4.ALTER.NET [140.222.227.27]
6 13 ms 12 ms 9 ms TenGigE0-7-4-0.GW8.NYC4.ALTER.NET [152.63.21.133
]
7 24 ms 23 ms 26 ms tinet-gw.customer.alter.net [152.179.72.122]
8 32 ms * * xe-1-3-0.mtl10.ip4.gtt.net [89.149.184.74]
9 31 ms * 31 ms ormuco-gw.ip4.gtt.net [216.221.156.110]
10 * 34 ms 32 ms 192.34.76.2
11 31 ms * 34 ms 199.91.189.234
12 35 ms 35 ms 34 ms 199.91.189.74

Trace complete.

C:\Users\Brock>tracert 199.91.189.74

Tracing route to 199.91.189.74 over a maximum of 30 hops

1 <1 ms <1 ms <1 ms Wireless_Broadband_Router.home [192.168.1.1]
2 4 ms 3 ms 4 ms L100.NYCMNY-VFTTP-125.verizon-gni.net [72.89.93.
1]
3 9 ms 7 ms 7 ms G0-11-1-6.NYCMNY-LCR-22.verizon-gni.net [130.81.
107.242]
4 * * * Request timed out.
5 6 ms 6 ms 7 ms 2.ae0.XT2.NYC4.ALTER.NET [140.222.227.27]
6 18 ms 9 ms 12 ms TenGigE0-7-0-8.GW8.NYC4.ALTER.NET [152.63.16.125
]
7 * * 23 ms tinet-gw.customer.alter.net [152.179.72.122]
8 42 ms 28 ms 29 ms xe-7-3-0.mtl10.ip4.gtt.net [141.136.106.106]
9 32 ms 32 ms 32 ms ormuco-gw.ip4.gtt.net [216.221.156.110]
10 31 ms 36 ms 33 ms 192.34.76.2
11 * * 31 ms 199.91.189.234
12 32 ms * 32 ms 199.91.189.74

Trace complete.

C:\Users\Brock>tracert 199.91.189.74

Tracing route to 199.91.189.74 over a maximum of 30 hops

1 <1 ms <1 ms <1 ms Wireless_Broadband_Router.home [192.168.1.1]
2 4 ms 4 ms 4 ms L100.NYCMNY-VFTTP-125.verizon-gni.net [72.89.93.
1]
3 7 ms 7 ms 6 ms G0-11-1-6.NYCMNY-LCR-22.verizon-gni.net [130.81.
107.242]
4 * * * Request timed out.
5 17 ms 79 ms 79 ms 2.ae0.XT2.NYC4.ALTER.NET [140.222.227.27]
6 21 ms 6 ms 12 ms TenGigE0-7-0-6.GW8.NYC4.ALTER.NET [152.63.21.65]

7 24 ms 24 ms 27 ms tinet-gw.customer.alter.net [152.179.72.122]
8 31 ms 32 ms 32 ms xe-7-3-0.mtl10.ip4.gtt.net [141.136.106.106]
9 31 ms 32 ms * ormuco-gw.ip4.gtt.net [216.221.156.110]
10 31 ms 32 ms 32 ms 192.34.76.2
11 31 ms 32 ms 31 ms 199.91.189.234
12 * 31 ms 32 ms 199.91.189.74

Trace complete.

C:\Users\Brock>tracert 199.91.189.74

Tracing route to 199.91.189.74 over a maximum of 30 hops

1 <1 ms <1 ms <1 ms Wireless_Broadband_Router.home [192.168.1.1]
2 3 ms 17 ms 4 ms L100.NYCMNY-VFTTP-125.verizon-gni.net [72.89.93.
1]
3 9 ms 6 ms 7 ms G0-11-1-6.NYCMNY-LCR-22.verizon-gni.net [130.81.
107.242]
4 * * * Request timed out.
5 7 ms 7 ms 6 ms 2.ae0.XT2.NYC4.ALTER.NET [140.222.227.27]
6 12 ms 11 ms 12 ms TenGigE0-7-1-0.GW8.NYC4.ALTER.NET [152.63.21.125
]
7 23 ms 24 ms 24 ms tinet-gw.customer.alter.net [152.179.72.122]
8 * 33 ms 32 ms xe-7-3-0.mtl10.ip4.gtt.net [141.136.106.106]
9 31 ms 32 ms 32 ms ormuco-gw.ip4.gtt.net [216.221.156.110]
10 32 ms 32 ms 32 ms 192.34.76.2
11 31 ms * 34 ms 199.91.189.234
12 35 ms 34 ms * 199.91.189.74
13 * 33 ms 34 ms 199.91.189.74

Trace complete.

C:\Users\Brock>

[Raist]

Microsoft Windows [Version 6.1.7601]
Copyright (c) 2009 Microsoft Corporation. All rights reserved.

C:\Users\Brock>tracert 199.91.189.74

Tracing route to 199.91.189.74 over a maximum of 30 hops

1 <1 ms <1 ms <1 ms Wireless_Broadband_Router.home [192.168.1.1]
2 4 ms 4 ms 4 ms L100.NYCMNY-VFTTP-125.verizon-gni.net [72.89.93.1]
3 6 ms 10 ms 9 ms G0-11-1-6.NYCMNY-LCR-22.verizon-gni.net [130.81.107.242]
4 * * * Request timed out.
5 6 ms 6 ms 7 ms 2.ae0.XT2.NYC4.ALTER.NET [140.222.227.27]
6 8 ms 11 ms 11 ms TenGigE0-7-0-4.GW8.NYC4.ALTER.NET [152.63.25.226]
7 25 ms 25 ms 24 ms tinet-gw.customer.alter.net [152.179.72.122]
8 32 ms 34 ms * xe-1-3-0.mtl10.ip4.gtt.net [89.149.184.74]
9 33 ms * 35 ms ormuco-gw.ip4.gtt.net [216.221.156.110]
10 34 ms 36 ms * 192.34.76.2
11 36 ms 34 ms 34 ms 199.91.189.234
12 33 ms 34 ms 34 ms 199.91.189.74

Trace complete.

C:\Users\Brock>tracert 199.91.189.74

Tracing route to 199.91.189.74 over a maximum of 30 hops

1 <1 ms <1 ms <1 ms Wireless_Broadband_Router.home [192.168.1.1]
2 3 ms 4 ms 4 ms L100.NYCMNY-VFTTP-125.verizon-gni.net [72.89.93.1]
3 7 ms 6 ms 6 ms G0-11-1-6.NYCMNY-LCR-22.verizon-gni.net [130.81.107.242]
4 * * * Request timed out.
5 8 ms 6 ms 7 ms 2.ae0.XT2.NYC4.ALTER.NET [140.222.227.27]
6 11 ms 12 ms 9 ms TenGigE0-7-0-0.GW8.NYC4.ALTER.NET [152.63.22.45]
7 23 ms 25 ms * tinet-gw.customer.alter.net [152.179.72.122]
8 31 ms * 33 ms xe-1-3-0.mtl10.ip4.gtt.net [89.149.184.74]
9 * * * Request timed out.
10 35 ms 33 ms 34 ms 192.34.76.2
11 35 ms 34 ms 39 ms 199.91.189.234
12 35 ms * * 199.91.189.74
13 34 ms 33 ms * 199.91.189.74
14 * * 35 ms 199.91.189.74

Trace complete.

C:\Users\Brock>tracert 199.91.189.74

Tracing route to 199.91.189.74 over a maximum of 30 hops

1 <1 ms <1 ms <1 ms Wireless_Broadband_Router.home [192.168.1.1]
2 5 ms 4 ms 4 ms L100.NYCMNY-VFTTP-125.verizon-gni.net [72.89.93.1]
3 9 ms 7 ms 8 ms G0-11-1-6.NYCMNY-LCR-22.verizon-gni.net [130.81.107.242]
4 * * * Request timed out.
5 7 ms 7 ms 7 ms 2.ae0.XT2.NYC4.ALTER.NET [140.222.227.27]
6 16 ms 9 ms 12 ms TenGigE0-7-0-8.GW8.NYC4.ALTER.NET [152.63.16.125]
7 * 24 ms * tinet-gw.customer.alter.net [152.179.72.122]
8 * 38 ms 41 ms xe-1-3-0.mtl10.ip4.gtt.net [89.149.184.74]
9 * 39 ms * ormuco-gw.ip4.gtt.net [216.221.156.110]
10 * * 34 ms 192.34.76.2
11 * 35 ms 34 ms 199.91.189.234
12 31 ms 34 ms 32 ms 199.91.189.74

Trace complete.

C:\Users\Brock>tracert 199.91.189.74

Tracing route to 199.91.189.74 over a maximum of 30 hops

1 <1 ms <1 ms <1 ms Wireless_Broadband_Router.home [192.168.1.1]
2 3 ms 4 ms 4 ms L100.NYCMNY-VFTTP-125.verizon-gni.net [72.89.93.1]
3 10 ms 6 ms 9 ms G0-11-1-6.NYCMNY-LCR-22.verizon-gni.net [130.81.107.242]
4 * * * Request timed out.
5 9 ms 7 ms 7 ms 2.ae0.XT2.NYC4.ALTER.NET [140.222.227.27]
6 13 ms 12 ms 9 ms TenGigE0-7-4-0.GW8.NYC4.ALTER.NET [152.63.21.133]
7 24 ms 23 ms 26 ms tinet-gw.customer.alter.net [152.179.72.122]
8 32 ms * * xe-1-3-0.mtl10.ip4.gtt.net [89.149.184.74]
9 31 ms * 31 ms ormuco-gw.ip4.gtt.net [216.221.156.110]
10 * 34 ms 32 ms 192.34.76.2
11 31 ms * 34 ms 199.91.189.234
12 35 ms 35 ms 34 ms 199.91.189.74

Trace complete.

C:\Users\Brock>tracert 199.91.189.74

Tracing route to 199.91.189.74 over a maximum of 30 hops

1 <1 ms <1 ms <1 ms Wireless_Broadband_Router.home [192.168.1.1]
2 4 ms 3 ms 4 ms L100.NYCMNY-VFTTP-125.verizon-gni.net [72.89.93.1]
3 9 ms 7 ms 7 ms G0-11-1-6.NYCMNY-LCR-22.verizon-gni.net [130.81.107.242]
4 * * * Request timed out.
5 6 ms 6 ms 7 ms 2.ae0.XT2.NYC4.ALTER.NET [140.222.227.27]
6 18 ms 9 ms 12 ms TenGigE0-7-0-8.GW8.NYC4.ALTER.NET [152.63.16.125]
7 * * 23 ms tinet-gw.customer.alter.net [152.179.72.122]
8 42 ms 28 ms 29 ms xe-7-3-0.mtl10.ip4.gtt.net [141.136.106.106]
9 32 ms 32 ms 32 ms ormuco-gw.ip4.gtt.net [216.221.156.110]
10 31 ms 36 ms 33 ms 192.34.76.2
11 * * 31 ms 199.91.189.234
12 32 ms * 32 ms 199.91.189.74

Trace complete.

C:\Users\Brock>tracert 199.91.189.74

Tracing route to 199.91.189.74 over a maximum of 30 hops

1 <1 ms <1 ms <1 ms Wireless_Broadband_Router.home [192.168.1.1]
2 4 ms 4 ms 4 ms L100.NYCMNY-VFTTP-125.verizon-gni.net [72.89.93.1]
3 7 ms 7 ms 6 ms G0-11-1-6.NYCMNY-LCR-22.verizon-gni.net [130.81.107.242]
4 * * * Request timed out.
5 17 ms 79 ms 79 ms 2.ae0.XT2.NYC4.ALTER.NET [140.222.227.27]
6 21 ms 6 ms 12 ms TenGigE0-7-0-6.GW8.NYC4.ALTER.NET [152.63.21.65]
7 24 ms 24 ms 27 ms tinet-gw.customer.alter.net [152.179.72.122]
8 31 ms 32 ms 32 ms xe-7-3-0.mtl10.ip4.gtt.net [141.136.106.106]
9 31 ms 32 ms * ormuco-gw.ip4.gtt.net [216.221.156.110]
10 31 ms 32 ms 32 ms 192.34.76.2
11 31 ms 32 ms 31 ms 199.91.189.234
12 * 31 ms 32 ms 199.91.189.74

Trace complete.

C:\Users\Brock>tracert 199.91.189.74

Tracing route to 199.91.189.74 over a maximum of 30 hops

1 <1 ms <1 ms <1 ms Wireless_Broadband_Router.home [192.168.1.1]
2 3 ms 17 ms 4 ms L100.NYCMNY-VFTTP-125.verizon-gni.net [72.89.93.1]
3 9 ms 6 ms 7 ms G0-11-1-6.NYCMNY-LCR-22.verizon-gni.net [130.81.107.242]
4 * * * Request timed out.
5 7 ms 7 ms 6 ms 2.ae0.XT2.NYC4.ALTER.NET [140.222.227.27]
6 12 ms 11 ms 12 ms TenGigE0-7-1-0.GW8.NYC4.ALTER.NET [152.63.21.125]
7 23 ms 24 ms 24 ms tinet-gw.customer.alter.net [152.179.72.122]
8 * 33 ms 32 ms xe-7-3-0.mtl10.ip4.gtt.net [141.136.106.106]
9 31 ms 32 ms 32 ms ormuco-gw.ip4.gtt.net [216.221.156.110]
10 32 ms 32 ms 32 ms 192.34.76.2
11 31 ms * 34 ms 199.91.189.234
12 35 ms 34 ms * 199.91.189.74
13 * 33 ms 34 ms 199.91.189.74

Trace complete.

C:\Users\Brock>

Like most everyone else, you appear to be loosing packets (or at least suffering out-of-order delivery that is triggering retransmits) around/between the exchanges with your ISP's routing partner. Just in this case, it is TInet instead of the usual Level3/Cogent/TATA peers we see so often. That is Ormuco's fourth major peering partner, as I detailed in another thread back around Thanksgiving:

http://forum.square-enix.com/ffxiv/t...=1#post2594200

(http://bgp.he.net/AS20324)

Here's the breakdown of those ASN's:
AS6453 TATA COMMUNICATIONS (AMERICA) INC
AS3356 Level 3 Communications, Inc.
AS3257 Tinet SpA
AS174 Cogent Communications

Verizon peers with all of them, either directly or through their ALTER.NET segments (part of Verizon also). There is PLENTY they can do to remedy your problem... the trick is getting them to take action.

Edit:
Just to further detail the issue of limited corridors and how congestion can impact large groups of player's at once, here are some network maps for that TiNet/GTT group:

Source: http://www.gtt.net/our-network/network-maps/ (more maps posted there, with links to bigger ones)

[Angelzbl0od]

so they have been telling me that there is nothing wrong with my trace routes and refusing to send it to higher level techs. Is there anywhere I can go from here or some other information i can provide them with to get their attention?

[Arcueid]

This lag is the sole reason I quit FF14 a year ago and when i resubbed today, to my surprise the lag is still here. I also have verizon FIOS 50/25 in NYC area and on Sargantanas server.