myself and a few friends have started receiving massive lag after patch 2.3. We are all on Verizon Fios in the NY area and our other friends on Comcast have no problems. This usually happens between the hours of 8pm - 10pm est on the Siren server. I have spent over a month and a half troubleshooting this problem with Verizon. At this point all minor problems that have been found on my end and on verizons end have all been worked out. This leaves only one end left to check and that is the Square Enix side. Please dont tell me port forwarding or resetting my router. I will repeat just for those who do not like to read things first THERE ARE ZERO PROBLEMS WITH MY CONNECTION AND NONE FOUND ON VERIZONS SIDE. Connection is rated for 300Mbps download over 300Mbps upload confirmed and working 100%. No other games/applications have given me any trouble. It seems the only issue is with FFXIV. are any others on verizon having the same issue? Is anyone at Square Enix looking into this?
Verizon Fios and Patch 2.3=Massive Lag
Printable View
-
12-26-2014, 12:53 PMAngelzbl0odVerizon Fios and Patch 2.3=Massive Lag
-
12-26-2014, 01:49 PMSqua11_LeonhartTHEY ARE LYING TO YOU.Quote:
NONE FOUND ON VERIZONS SIDE
https://www.google.com.au/search?q=v...Fce9mgXkt4DYDw -
12-27-2014, 07:24 PMAngelzbl0od
no they are not lying. I have done the trace route and seen for myself where the connection is getting hung up and its always when it hits the hops near the monreal server. Also that link is about netflix and that is not a problem for me. I am capable of running netflix on 6 devices at the same time in HD quality and 2 more devices watching amazon prime in HD with no lag or buffering on any of the devices.
-
12-28-2014, 12:02 AMSqua11_Leonhart
Netflix is streamed across the same network links that your LYING THROUGH ITS TEETH 'Internet Service Withholder' use to get your game packets to the servers in montreal.
NetFlix is considered priority traffic as it is a multimedia packet where delay or packet loss can detrimentally affect the quality of the stream and experience.
As there are not enough links connected between the Verizon/Level 3 boundary (which btw, Level 3 has ready to connect any time Verizon chooses to actually provide adequate server), your game packets MAY get choked and end up either significantly delayed (high Latency) or Lost (high Packet Loss).
Please, develop an educated understanding of the issue -
12-28-2014, 03:57 AMAngelzbl0od
I am quite educated on the issue and I do not like or appreciate your condescending tone. I am glad that you do not like Verizon and I tend to agree with you, they piss me off. However I am nearly certain that the issue is on SE end. I think you know just enough about the topic to be confident but still incorrect about the issue. Unless you work for Square Enix or Verizon I do not think you have a clear picture of what is going on. I admit it could be on Verizons end but I do not think that it is. I have spent a month and a half troubleshooting this problem. I would be inclined to agree with you if I was having problems with any other service. And yes I have run other services at the same times as when my ffxiv normally lags out and have done heavy internet usage at these times with no lag or packet loss. No other trace routes cause a problem at any time during the day except for the ones going to ffxiv. And dont tell me that its where the network swaps to montreal because I have friends there who host other game servers from montreal and I host games servers that they connect to and never have any issues.
-
12-28-2014, 04:12 AMNashred
I am going to make this short because there is enough info on this already.
It is not because of SE it is Level 3 putting priority packets for netflix users and putting you in the slow lane.. It happens to allot of Verizon users and some ATT users and Time Warner users. It is spreading too and is costing SE users. SE is aware of the issue too. They have a thread to post your info if you have these problems.
SE choose one of the worst spot to put their servers because of the known issues with Level 3 though.
If you are on the pc you can get around the issue by using one of the vpn services. -
12-28-2014, 04:45 AMRaist
First off, Level3 is not the only one routing packets to SE's server. Ormuco peers with 3 of the majors for North America, and a 4th for overseas access. Likewise, your last mile ISP's will have agreements with some if not all of those same big 3 top tier ISP's for peering/transit across the country (and in some cases intercontinental traffic as well). Those three ISP's are Level3, Cogent, and TATA. Your ISP's policies are what is determining which of those three ISP's you are assigned to get across the border, into Canada, and over to Ormuco (SE's ISP). SE has NOTHING to do with that. It is your ISP that has put those agreements in place and their policies that are setting the metrics for directing you to them.
In the OP's case, we don't know who they are getting assigned for getting to Montreal... Squall was simply basing that on a long list of recent examples showing a known, and well documented pissing contest between Veriaon and Level3 that has repeatedly been reported elsewhere as well as here by players that have conducted a more proper investigation of their routes. For some of you, it may also be Level3... or it could be Cogent like I am using at the moment. If a proper investigation is done and it is found to be an issue with your current routing partner, your ISP actually has the means to switch you to an alternate partner---but you have to get in touch with your ISP's tier3 techs to get the proper escalation to make that happen. This can and does work. I've done it multiple times, and many others have done so as well--and it has resolved most if not all of their lag and stability issues.
Just curious Angelzbl0od... what level of support were you dealing with? And what address were you testing against? Way too often I see people trying to address this issue with the first level of support that answers the phone, or they are using an IP that is in no way tied into gameplay. Those helpdesk guys are usually only Tier1 support and at best put you in touch with Tier2 people by default. You need to specifically push past them and get to Tier3 support--some refer to them as Engineering. Also, if the address you were using did not start with 199, then you weren't even in the ballpark of where you need to be looking. I'm hoping you indeed have done the proper research and know either appropriate addresses to use or at least know how to find which one the game client is actually using when logged fully into the game. Just want to make sure you are in fact looking at the right things with the right people is all. More often than not when people are so vehemently ascerting they had everything checked and they passed with flying colors, they were misinformed or mislead at best... flat out lied to at worst. -
12-28-2014, 12:47 PMAngelzbl0od
i have been using ping and trace route on the following ip addresses. 199.91.189.74 for the NA data center and 199.91.189.29 for the siren server. Most of the time it is the data center that I have trace route problems. on both trace routes I have only 6-8 hops and requsts time out when it hits the last hop. Again this only happens on and off between the hours of 8-10pm est. I have been in touch with level 1 and 2 and have been emailing for a week with level 3 associates. They are willing to change my routing but we are trying to work out all problems while on the way to that solution. The surprising part is that my ping never goes above 40ms and that is at its absolute worst. my average ping to both data center and siren server are average 13ms
-
12-28-2014, 02:54 PMRaist
That doesn't sound like typical stats from US to Montreal... guessing you are along the Canada border or something? Even though you are stating low ping stats, you have a lot of jitter in there with a swing from 13 to 40---that's not normal, and could very well be the source of out-of-order packet delivery that triggers a lot of unnecessary retransmits.. Could be some ingress along the route if somewhere you are getting run on copper along the way that is causing some issues.
Are you by chance coming through Vancouver or Toronto on the way there? I know that sounds strange considering you are in NY... but I am in SC and at times have been sent on circuits assigned to Los Angeles to get to Montreal. There have been issues in those corridors off and on, might take something that tracks larger samples like pathping or a pingplotter to capture something more telling. You could even try some of the quality tests and such from VisualWare as well. You are likely hitting the main NY/NJ areas as well. Those are frequent hotbeds for issues lately. Might be able to tell more if we had a look at your actual route.
Also, might want to verify that your client is indeed using .29 if you haven't already. I know ARRStatus lists that as the IP for Siren, but it actually uses several IP's. You could actually be on a different one, and that might put you on a different VLAN channel or something on the way there that is having issues. We captured such an exception within Ormuco's section of my route a while back (300+ms spikes in the VLAN). We emailed their registered admin contact about it and the issue cleared up a few days later and it hasn't returned since.
Edit:
Just pulled up the Verizon page at downdetector, and there have been a lot of reports with internet problems for them lately. ATM, the northeast is not fairing to well in general---not uncommon for folks up there either.
https://downdetector.com/status/verizon-communications
https://cdn.serinus42.com/maps/20107/latest.jpg
(Note that image link is to a snapshot that gets archived periodically though out the day, so you can come back later and see if things have changed)
Just to further illustrate things, here is Time Warner's map--it's getting a lot of reports at the moment in the northeast as well:
https://downdetector.com/status/time-warner-cable
https://cdn.serinus42.com/maps/20096/latest.jpg
There's also a series of tools that Akamai puts out on data they collect about the state of the internet that demonstrates potential issues for the northeast as well--high utilization, high attack levels and such:
http://www.akamai.com/html/technology/dataviz1.html
http://www.akamai.com/html/technolog...b-monitor.html
http://www.akamai.com/html/technolog...try/index.html
The point is, there are a lot of contributing factors that could be at play here... many of which are completely out of SE's hands, but our ISP's can and should work to clean things up. -
12-29-2014, 02:23 AMAngelzbl0od
I will post some of my trace routes here next time I am getting lag spikes like that. The ping does not fluctuate a lot its pretty steady at 13 only hits 40 on rare moments. I am on Long Island so not really near the canada border. also not sure if im hitting toronto. I will download some of the trace apps you mention and see if i can get some more info out of them. I appreciate you taking time out of your day to throw out some suggestions that I may not have thought of earlier. Will post those traces either later tonight or tomm afternoon. I will get some good ones and then ones with lag so they can be compared
-
12-29-2014, 07:09 AMLegendzerox
I've been playing online games for a long time on Verizon FIOS along with FF14. I was completely fine playing ff14 and then 2.3 hits and I take a massive performance hit.
I talked to a guildy and she says she has Uverse and also took a big hit. If it was throttling, it wouldn't conveniently happen right after patch day. -
12-29-2014, 08:27 AMelpacolopez123
Sure wish they would do something about that FCKING mog station and those ereor code BS
-
12-29-2014, 11:57 AMRaistIt isn't "throttling" in the sense that everyone typically interprets it. It is congestion control... or rather, a poor or virtually non-existent implementation of it. The north east corridors in the US are a MAJOR choking point for THIS game. More than half of all the traffic to Montreal is forced through there. There is also the little detour we take with the NSA check-in in Chicago, and the hops to get around the Great Lakes that have to be taken from the Midwest and people from the North West that get slung around British Columbia that can run into another bit of a bottleneck along the way. When ony one of the corridors in these areas get overloaded, everyone going through them suffers, regardless of who their local ISP.Quote:
Originally Posted by Legendzerox
It is a problem that exists between those local ISP's and the 4 top tier ISP's that SE's ISP peers with. Everyone uses these same people for transit/peering to connect each other's networks if they are not able to enter into more direct peering agreements with them. Comparing this game that is hosted in Montreal is not necessarily an apples to apples comparison. If you are in Texas and are playing a game hosted on a server in Chicago, you are in no way going to run into the same congestion issues you face coming through New York and New Jersey.
Because of the limited number of corridors amongst the big 3 routing partners that get the US into Montreal (it's about 20 combinations of provider and corridor), when one goes down it can have a sever impact on the players. If all things were distributed evenly, just one provider having issues in one corridor would affect 5% of the entire population at once. If it is the corridor in general that is affecting all 3 providers, then you are looking at 15%. If it is a more widespread regional thing and 5 paths are affected, then it could run anywhere from 25% (one provider) all the way up to 75% (5 paths, 3 providers each). Now, consider how the population and traffic flow is distributed in reality (more than half of the paths go through the north east) and the numbers ramp up considerably.
When the North East region is in trouble across the board (ie, major weather, massive traffic spike from seasonal events in retail, entertainment, politics, etc.), it has a major impact on quality of service between Montreal and the US. When 2.3 hit the wire, we also hit one of the biggest peaks in internet traffic for the entire YEAR in the US. There is a connection there... just look at the history in those links to Akamai I provided earlier.
A similar thing happens for the West Coast and the Pacific region. When Southern California is having a brain fart, a massive amount of people out in Australia, Hawaii, etc... all the way to Asia suffer when trying to get to services here in the US--because there are some major choking points for the undersea transit lines in Southern California. A similar phenomenon exists in the North East for the UK as well, but they have an escape route that comes around Nova Scotia that helps to mitigate it better.
The point is, there are patterns that you can track that explain a lot of this mess. The problem is a lot of local providers don't analyze their routing often enough (some only do it semi-anually or quarterly), so they don't compensate quickly enough. And to add insult to injury, when problems are brought to their attention, many don't sense the urgency of the problem or otherwise don't feel compelled to investigate further much less take appropriate action. Even when they are offered financial assistance to conduct upgrades (Level 3 offered to pay for and even install hardware upgrades for some of them), they still opt out of making improvements. This isn't speculation by the way, there have been articles linked to on these forums that detail this incident quite clearly.
We pay our ISP's for our service. We need to demand they provide an acceptable level of quality for that service. They've gotten away with running roughshod over us for nearly 2 decades now... time to hold their feet to the fire. -
12-29-2014, 12:01 PMBallistic
God, I thought I was going crazy, at least I know what's going on now. I thought it was just me. I lag like mad during those hours also. Same thing, FIOS and on Siren.
-
12-29-2014, 12:23 PMLimitBreakOP
I'm not in Siren, but I've been getting horrendous lag on Balmung. I'm able to play the game for a little while without any issues when I first start it up, but before long the lag starts to settle in. From there I just see players skipping along the map as they run and stop constantly from one destination to the next, NPC dialogue is delayed by a few seconds or so after I click on an npc, and in fights my character and the enemies around just stop moving every so often. Is yours the same as well? I'd like to clarify. My problem seems to persist all throughout the day rather than just 8PM to 10PM
-
12-29-2014, 12:38 PMkiotsukete
Same issue here, on FIOS but in So.Cal and also experiencing heavy lag over the past couple months. Even got 90k tonight during an alexandrite coffer ._.; Thought I was going crazy too but I'm glad to know I'm not the only one, even though it's still frustrating. I'll drop FFXIV way before FIOS though, since my Internet service has otherwise been great (Netflix, web surfing, Destiny, etc.)
-
12-31-2014, 12:42 PMAngelzbl0od
Microsoft Windows [Version 6.1.7601]
Copyright (c) 2009 Microsoft Corporation. All rights reserved.
C:\Users\Brock>tracert 199.91.189.74
Tracing route to 199.91.189.74 over a maximum of 30 hops
1 <1 ms <1 ms <1 ms Wireless_Broadband_Router.home [192.168.1.1]
2 4 ms 4 ms 4 ms L100.NYCMNY-VFTTP-125.verizon-gni.net [72.89.93.
1]
3 6 ms 10 ms 9 ms G0-11-1-6.NYCMNY-LCR-22.verizon-gni.net [130.81.
107.242]
4 * * * Request timed out.
5 6 ms 6 ms 7 ms 2.ae0.XT2.NYC4.ALTER.NET [140.222.227.27]
6 8 ms 11 ms 11 ms TenGigE0-7-0-4.GW8.NYC4.ALTER.NET [152.63.25.226
]
7 25 ms 25 ms 24 ms tinet-gw.customer.alter.net [152.179.72.122]
8 32 ms 34 ms * xe-1-3-0.mtl10.ip4.gtt.net [89.149.184.74]
9 33 ms * 35 ms ormuco-gw.ip4.gtt.net [216.221.156.110]
10 34 ms 36 ms * 192.34.76.2
11 36 ms 34 ms 34 ms 199.91.189.234
12 33 ms 34 ms 34 ms 199.91.189.74
Trace complete.
C:\Users\Brock>tracert 199.91.189.74
Tracing route to 199.91.189.74 over a maximum of 30 hops
1 <1 ms <1 ms <1 ms Wireless_Broadband_Router.home [192.168.1.1]
2 3 ms 4 ms 4 ms L100.NYCMNY-VFTTP-125.verizon-gni.net [72.89.93.
1]
3 7 ms 6 ms 6 ms G0-11-1-6.NYCMNY-LCR-22.verizon-gni.net [130.81.
107.242]
4 * * * Request timed out.
5 8 ms 6 ms 7 ms 2.ae0.XT2.NYC4.ALTER.NET [140.222.227.27]
6 11 ms 12 ms 9 ms TenGigE0-7-0-0.GW8.NYC4.ALTER.NET [152.63.22.45]
7 23 ms 25 ms * tinet-gw.customer.alter.net [152.179.72.122]
8 31 ms * 33 ms xe-1-3-0.mtl10.ip4.gtt.net [89.149.184.74]
9 * * * Request timed out.
10 35 ms 33 ms 34 ms 192.34.76.2
11 35 ms 34 ms 39 ms 199.91.189.234
12 35 ms * * 199.91.189.74
13 34 ms 33 ms * 199.91.189.74
14 * * 35 ms 199.91.189.74
Trace complete.
C:\Users\Brock>tracert 199.91.189.74
Tracing route to 199.91.189.74 over a maximum of 30 hops
1 <1 ms <1 ms <1 ms Wireless_Broadband_Router.home [192.168.1.1]
2 5 ms 4 ms 4 ms L100.NYCMNY-VFTTP-125.verizon-gni.net [72.89.93.
1]
3 9 ms 7 ms 8 ms G0-11-1-6.NYCMNY-LCR-22.verizon-gni.net [130.81.
107.242]
4 * * * Request timed out.
5 7 ms 7 ms 7 ms 2.ae0.XT2.NYC4.ALTER.NET [140.222.227.27]
6 16 ms 9 ms 12 ms TenGigE0-7-0-8.GW8.NYC4.ALTER.NET [152.63.16.125
]
7 * 24 ms * tinet-gw.customer.alter.net [152.179.72.122]
8 * 38 ms 41 ms xe-1-3-0.mtl10.ip4.gtt.net [89.149.184.74]
9 * 39 ms * ormuco-gw.ip4.gtt.net [216.221.156.110]
10 * * 34 ms 192.34.76.2
11 * 35 ms 34 ms 199.91.189.234
12 31 ms 34 ms 32 ms 199.91.189.74
Trace complete.
C:\Users\Brock>tracert 199.91.189.74
Tracing route to 199.91.189.74 over a maximum of 30 hops
1 <1 ms <1 ms <1 ms Wireless_Broadband_Router.home [192.168.1.1]
2 3 ms 4 ms 4 ms L100.NYCMNY-VFTTP-125.verizon-gni.net [72.89.93.
1]
3 10 ms 6 ms 9 ms G0-11-1-6.NYCMNY-LCR-22.verizon-gni.net [130.81.
107.242]
4 * * * Request timed out.
5 9 ms 7 ms 7 ms 2.ae0.XT2.NYC4.ALTER.NET [140.222.227.27]
6 13 ms 12 ms 9 ms TenGigE0-7-4-0.GW8.NYC4.ALTER.NET [152.63.21.133
]
7 24 ms 23 ms 26 ms tinet-gw.customer.alter.net [152.179.72.122]
8 32 ms * * xe-1-3-0.mtl10.ip4.gtt.net [89.149.184.74]
9 31 ms * 31 ms ormuco-gw.ip4.gtt.net [216.221.156.110]
10 * 34 ms 32 ms 192.34.76.2
11 31 ms * 34 ms 199.91.189.234
12 35 ms 35 ms 34 ms 199.91.189.74
Trace complete.
C:\Users\Brock>tracert 199.91.189.74
Tracing route to 199.91.189.74 over a maximum of 30 hops
1 <1 ms <1 ms <1 ms Wireless_Broadband_Router.home [192.168.1.1]
2 4 ms 3 ms 4 ms L100.NYCMNY-VFTTP-125.verizon-gni.net [72.89.93.
1]
3 9 ms 7 ms 7 ms G0-11-1-6.NYCMNY-LCR-22.verizon-gni.net [130.81.
107.242]
4 * * * Request timed out.
5 6 ms 6 ms 7 ms 2.ae0.XT2.NYC4.ALTER.NET [140.222.227.27]
6 18 ms 9 ms 12 ms TenGigE0-7-0-8.GW8.NYC4.ALTER.NET [152.63.16.125
]
7 * * 23 ms tinet-gw.customer.alter.net [152.179.72.122]
8 42 ms 28 ms 29 ms xe-7-3-0.mtl10.ip4.gtt.net [141.136.106.106]
9 32 ms 32 ms 32 ms ormuco-gw.ip4.gtt.net [216.221.156.110]
10 31 ms 36 ms 33 ms 192.34.76.2
11 * * 31 ms 199.91.189.234
12 32 ms * 32 ms 199.91.189.74
Trace complete.
C:\Users\Brock>tracert 199.91.189.74
Tracing route to 199.91.189.74 over a maximum of 30 hops
1 <1 ms <1 ms <1 ms Wireless_Broadband_Router.home [192.168.1.1]
2 4 ms 4 ms 4 ms L100.NYCMNY-VFTTP-125.verizon-gni.net [72.89.93.
1]
3 7 ms 7 ms 6 ms G0-11-1-6.NYCMNY-LCR-22.verizon-gni.net [130.81.
107.242]
4 * * * Request timed out.
5 17 ms 79 ms 79 ms 2.ae0.XT2.NYC4.ALTER.NET [140.222.227.27]
6 21 ms 6 ms 12 ms TenGigE0-7-0-6.GW8.NYC4.ALTER.NET [152.63.21.65]
7 24 ms 24 ms 27 ms tinet-gw.customer.alter.net [152.179.72.122]
8 31 ms 32 ms 32 ms xe-7-3-0.mtl10.ip4.gtt.net [141.136.106.106]
9 31 ms 32 ms * ormuco-gw.ip4.gtt.net [216.221.156.110]
10 31 ms 32 ms 32 ms 192.34.76.2
11 31 ms 32 ms 31 ms 199.91.189.234
12 * 31 ms 32 ms 199.91.189.74
Trace complete.
C:\Users\Brock>tracert 199.91.189.74
Tracing route to 199.91.189.74 over a maximum of 30 hops
1 <1 ms <1 ms <1 ms Wireless_Broadband_Router.home [192.168.1.1]
2 3 ms 17 ms 4 ms L100.NYCMNY-VFTTP-125.verizon-gni.net [72.89.93.
1]
3 9 ms 6 ms 7 ms G0-11-1-6.NYCMNY-LCR-22.verizon-gni.net [130.81.
107.242]
4 * * * Request timed out.
5 7 ms 7 ms 6 ms 2.ae0.XT2.NYC4.ALTER.NET [140.222.227.27]
6 12 ms 11 ms 12 ms TenGigE0-7-1-0.GW8.NYC4.ALTER.NET [152.63.21.125
]
7 23 ms 24 ms 24 ms tinet-gw.customer.alter.net [152.179.72.122]
8 * 33 ms 32 ms xe-7-3-0.mtl10.ip4.gtt.net [141.136.106.106]
9 31 ms 32 ms 32 ms ormuco-gw.ip4.gtt.net [216.221.156.110]
10 32 ms 32 ms 32 ms 192.34.76.2
11 31 ms * 34 ms 199.91.189.234
12 35 ms 34 ms * 199.91.189.74
13 * 33 ms 34 ms 199.91.189.74
Trace complete.
C:\Users\Brock> -
12-31-2014, 02:15 PMRaistLike most everyone else, you appear to be loosing packets (or at least suffering out-of-order delivery that is triggering retransmits) around/between the exchanges with your ISP's routing partner. Just in this case, it is TInet instead of the usual Level3/Cogent/TATA peers we see so often. That is Ormuco's fourth major peering partner, as I detailed in another thread back around Thanksgiving:Quote:
Originally Posted by Angelzbl0od
http://forum.square-enix.com/ffxiv/t...=1#post2594200
Verizon peers with all of them, either directly or through their ALTER.NET segments (part of Verizon also). There is PLENTY they can do to remedy your problem... the trick is getting them to take action.Quote:
http://chart.googleapis.com/chart?ch...er&chco=346090
(http://bgp.he.net/AS20324)
Here's the breakdown of those ASN's:
AS6453 TATA COMMUNICATIONS (AMERICA) INC
AS3356 Level 3 Communications, Inc.
AS3257 Tinet SpA
AS174 Cogent Communications
Edit:
Just to further detail the issue of limited corridors and how congestion can impact large groups of player's at once, here are some network maps for that TiNet/GTT group:
Source: http://www.gtt.net/our-network/network-maps/ (more maps posted there, with links to bigger ones)
http://www.gtt.net/_includes/uploads...psmall0414.jpg
http://www.gtt.net/_includes/uploads...psmall0414.jpg -
01-06-2015, 01:59 AMAngelzbl0od
so they have been telling me that there is nothing wrong with my trace routes and refusing to send it to higher level techs. Is there anywhere I can go from here or some other information i can provide them with to get their attention?
-
01-06-2015, 11:51 AMArcueid
This lag is the sole reason I quit FF14 a year ago and when i resubbed today, to my surprise the lag is still here. I also have verizon FIOS 50/25 in NYC area and on Sargantanas server.