Connection issues.

[Hakken]

You aren't hearing from an SE employee about it because this is a user-to-user forum first and foremost (Welcome to "Technical Support" Forum! sticky). They don't post much in here beyond directing us to their established KB/FAQ answers or if there is breaking news on a hot topic--which the higher ups rarely give them the OK to share with us before it gets posted officially in something like the Lodestone.

It is also because in the majority of the cases, it isn't something wrong with the servers or their connection. Granted, there are cases where there has been corruption in character data, and early on we had issues with lobby queues and such... but that was more or less resolved with big infrastructure changes a good while back. Occasionally they still have issues with the servers, and they put notices up about it in the Lodestone--but they are fairly isolated and not the widescale fiasco we had back in the fall of last year. There are also issues that Sony needs to address with their PSN and PS3/4's themselves (BIG kerfuffle with the 2.00 firmware update, which they have acknowledged and are pushing to address some issues with 2.01 update--such notices go up on both the downdetector site and Sony's own twitter feeds).

The bulk of the problems seem to be coming from our problems in routing to SE's network. The routing partners and our ISP's have arrangements for getting us from our networks to Ormuco's (SE's ISP). They are simply straining under the load far more then many are aware of...despite all our efforts to get the word out, posting links to articles, graphs, monitoring/testing tools and guides--everyone still wants to blame everything on SE without even so much as testing their routes for signs of congestion/packet loss, which often proves to be an issue for them. Here is one such recent post I made the other day in another thread, and there are many others like it from multiple users on these forums:

http://forum.square-enix.com/ffxiv/t...=1#post2567884

Such things your ISP can and NEEDS to address--whether that is found within their networks or their partners, they have the means to resolve it on your behalf as their paying customer... be that working with their partners to improve things at the peering/transit exchanges, or switching you to another routing partner, or even breaking you off to a different local node (which, as a side effect may change your routing partner--it depends on how their metrics are set). Case in point, TWC has once again moved me this week from TATA over to Cogent, and aside from the bad jitter we still haven't remedied in our localized nodes the route is pretty tight and my gameplay on Midgard is just fine (and, barring some times SE actually had issues they had to address, it has been for over a year now, since my ISP has been working to address such issues for us in our area):

C:\Windows\System32>tracert neolobby02.ffxiv.com

Tracing route to neolobby02.ffxiv.com [199.91.189.74]
over a maximum of 30 hops:

  1     1 ms    <1 ms    <1 ms  LPTSRV [10.10.100.1]
  2    30 ms    21 ms    17 ms  cpe-075-176-160-001.sc.res.rr.com [75.176.160.1]
  3    18 ms    26 ms    25 ms  cpe-024-031-198-009.sc.res.rr.com [24.31.198.9]
  4    19 ms    20 ms    17 ms  24.31.196.212
  5    21 ms    21 ms    23 ms  be33.chrcnctr01r.southeast.rr.com [24.93.64.182]
  6    26 ms    29 ms    26 ms  bu-ether44.atlngamq46w-bcr00.tbone.rr.com [107.14.19.46]
  7    29 ms    26 ms    27 ms  107.14.19.11
  8    27 ms    26 ms    28 ms  te0-0-0-10.ccr21.atl02.atlas.cogentco.com [154.54.12.109]
  9    27 ms    26 ms    28 ms  be2050.ccr41.atl01.atlas.cogentco.com [154.54.0.165]
 10    37 ms    34 ms    35 ms  be2168.ccr21.dca01.atlas.cogentco.com [154.54.31.94]
 11    40 ms    40 ms    44 ms  be2148.ccr41.jfk02.atlas.cogentco.com [154.54.31.118]
 12    46 ms    45 ms    45 ms  be2106.ccr21.alb02.atlas.cogentco.com [154.54.3.50]
 13    51 ms    49 ms    51 ms  be2088.ccr21.ymq02.atlas.cogentco.com [154.54.43.17]
 14    63 ms    62 ms    64 ms  38.122.42.34
 15    49 ms    49 ms    49 ms  10.2.2.1
 16    47 ms    48 ms    52 ms  192.34.76.2
 17    50 ms    47 ms    50 ms  199.91.189.234
 18    49 ms    49 ms    48 ms  199.91.189.74

Trace complete.

C:\Windows\System32>tracert 199.91.189.30

Tracing route to 199.91.189.30 over a maximum of 30 hops

  1    <1 ms    <1 ms    <1 ms  LPTSRV [10.10.100.1]
  2    25 ms    28 ms    13 ms  cpe-075-176-160-001.sc.res.rr.com [75.176.160.1]
  3    78 ms    20 ms    27 ms  cpe-024-031-198-009.sc.res.rr.com [24.31.198.9]
  4    17 ms    20 ms    18 ms  24.31.196.212
  5    20 ms    22 ms    26 ms  be33.chrcnctr01r.southeast.rr.com [24.93.64.182]
  6    27 ms    30 ms    26 ms  bu-ether14.atlngamq46w-bcr00.tbone.rr.com [66.109.6.82]
  7    24 ms    23 ms    26 ms  107.14.19.99
  8    29 ms    27 ms    26 ms  te0-0-0-10.ccr21.atl02.atlas.cogentco.com [154.54.12.109]
  9    25 ms    25 ms    27 ms  be2050.ccr41.atl01.atlas.cogentco.com [154.54.0.165]
 10    36 ms    33 ms    34 ms  be2168.ccr21.dca01.atlas.cogentco.com [154.54.31.94]
 11    41 ms    39 ms    44 ms  be2148.ccr41.jfk02.atlas.cogentco.com [154.54.31.118]
 12    43 ms    45 ms    48 ms  be2106.ccr21.alb02.atlas.cogentco.com [154.54.3.50]
 13    50 ms    49 ms    49 ms  be2088.ccr21.ymq02.atlas.cogentco.com [154.54.43.17]
 14    62 ms    64 ms    63 ms  38.122.42.34
 15    47 ms    50 ms    48 ms  10.2.2.1
 16    50 ms    49 ms    48 ms  192.34.76.2
 17    49 ms    49 ms    51 ms  199.91.189.234
 18    48 ms    49 ms    48 ms  199.91.189.30

Trace complete.

Don't get me wrong, I understand you trying to explain to everyone whats happening, but bottom line is this. No one gets lag when playing other online games, only when playing FFXIV. Which means it's FFXIV and its completely unacceptable this hasn't been fixed after weeks and weeks of thread posts on here, we want a responce cause we want to know something is being done about it or at least they acknowledge that there's a issue. No responce from any mods or admins just makes everyone mad ontop of how mad they already are. You posting this stuff explaining stuff is just going to also make people more mad as well cause what your saying won't fix there problem.

[Raist]

Don't get me wrong, I understand you trying to explain to everyone whats happening, but bottom line is this. No one gets lag when playing other online games, only when playing FFXIV. Which means it's FFXIV and its completely unacceptable this hasn't been fixed after weeks and weeks of thread posts on here, we want a responce cause we want to know something is being done about it or at least they acknowledge that there's a issue. No responce from any mods or admins just makes everyone mad ontop of how mad they already are. You posting this stuff explaining stuff is just going to also make people more mad as well cause what your saying won't fix there problem.

Don't think you truly understand then... or you aren't reading the information that has been put out there for over a year on these forums, and nearly a decade on others. It isn't a game specific problem as much as it is a problem with the infrastructure of the hundreds upon hundreds of networks that interconnect to form what we simply call the internet. Oh, and SE has acknowledged there is a problem... they created a sticky about it over a year ago. They have in fact worked directly with some ISP's since then (there were threads about an issue in the UK that BT worked with them to identify the problem and BT--not SE--resolved it, and others have reported their ISP's resovling their problems as well and not just me).

It basically boils down to issues with when there is severe congestion along a backbone. It can affect all traffic going through there--doesn't matter if it's going to SE's servers or Blizzard's. You simply aren't seeing it happening with other games because you aren't getting crammed into the limited paths we have to take to get to Montreal. If you live on the west coast and play on a game server on the west coast (or pretty much anywhere else in the US, save maybe for the NY/NJ or Chicago areas)-- you have a slim to no chance of running into the congested corridors taking us specifically into places like Vancouver, Ottawa, Toronto, Montreal, or Nova Scotia. You may pick up something in Southern California, Seattle, Texas, Ohio, Atlanta, the Carolinas, Chicago, the north East---but not the handful of major paths going over the border and across Canada to Montreal.

As for the whole "it's not happening to other games/services" crowd, check out the forums for your ISP. You might find some threads in there linking to other games/services that are having issues. Here's the connectivity section for my ISP:

http://forums.timewarnercable.com/t5...p/connectivity

Just look at some of the recent thread titles posted there:

HUGE latency to League of Legends servers. (posted yesterday)

Frequent disconnects, packet loss,and lag spikes while gaming. Slow speed. (again, posted yesterday)

Speed keeps dropping (one of many threads that keeps getting back up to the top... this one is over a month old)

Constant night time slowdowns (this one started up last week)

Random lag issues (another older thread that started about a month ago that got brought back to the top this weekend)

And that's just the first page of that forum. It has tons of posts along these lines that continually come back up over and over again. You can see the same thing going on at other major ISP's as well. It isn't JUST FFXIV suffering from this problem. Just look at the Comcast and PSN maps that I put in that post I linked to earlier. Those aren't static images I put up either.. they are links to images that get updated periodically throughout the day. At times you will see them both lit up like a Xmas tree---lots of pretty red and yellow blotches all over the same areas on a recurring basis. Click the URL's above them for the source pages to each of them and read the comments the users have been posting. Click the "Companies" tab up top of the downdetector site and look at the pages and maps for other companies--you may even find one for your ISP or one of your favorite services. You will see a common them running through them---high congestion, packet loss, disconnects, unable to log in, stuttering streams... it is all over the place for all kinds of services, and seems to happen over and over in specific regions.

Why does this happen? Read the blog posts referenced in that post...don't even have to read them all really, that one post I quoted there gives a pretty good clue as to why it happens.

And once again.. for the record... I don't have these issues anymore. When I first start to see/feel the signs of it creeping up, my route gets changed and it clears up. Over and over again since last fall when I got things escalated to tier3 and things gradually got better and better to the point I have response times in the 70's most the time, or better and rarely experience the odd hiccup here and there (like a skip from a scratched DVD may be the best way to describe it). I'm in South Carolina too, nowhere near Canada, and sometimes go through the troublesome Raleigh/Durham and Charlotte corridors where we've seen players having problems. When that happens, I look at my route and if it looks like it's going south, I forward the data to them and it gets handled. Sometimes they even beat me to the punch and they bounce my modem before I can even look into it... a few minutes later, I'm up and running in the game again with a clean connection.

I had the same experiences with FFXI, Diablo I & II, Freespace Descent 1 & 2, Total Annihilation (used Microsoft's gaming platform), and a host of other things going all the way back into the 90's. And EVERY TIME it has been Road Runner/Time Warner Cable that went to bat for me and got it resolved...because it was an issue with where I was getting routed.

[Katsu78]

Jibber Jabber.

That's nice and all, but you notice that people around the world are having these problems, right? Also Someone pointed out that the lag spikes happen around 7pm EST - 12am EST, which matches what's been happening to me lately on the Leviathan server. It's been consistent for the last few days. Always around 7pm, and the lag stops at 12am. And it matches what other people have been saying, too. Other people's characters would stop for a few seconds, then move, stop, and so on. This is obviously something to do with FFXIV itself, if multiple people from different parts of the world are having the same problems.

I'd appreciate it if you not make it out to be some major conspiracy

[Raist]

That's nice and all, but you notice that people around the world are having these problems, right? Also Someone pointed out that the lag spikes happen around 7pm EST - 12am EST, which matches what's been happening to me lately on the Leviathan server. It's been consistent for the last few days. Always around 7pm, and the lag stops at 12am. And it matches what other people have been saying, too. Other people's characters would stop for a few seconds, then move, stop, and so on. This is obviously something to do with FFXIV itself, if multiple people from different parts of the world are having the same problems.

I'd appreciate it if you not make it out to be some major conspiracy

Please.. before you set out to try to make someone look like some sort of conspiracy theorist, at least take a few moments to look at the facts that have been presented. Especially when some of it has already been gathered and presented or linked in the prominent threads on the issue. It really makes you look bad when such information is even presented in the very posts/threads to which you are responding and it appears you haven't even bothered to read any of it (details that would explain your very argument were linked to not only in the post you replied to, but also another just a few posts up).

Take a look at the "world" via some of the maps here to see how this is impacting specific regions in tandem around the world:
https://downdetector.com/companies

Just in case no one even bothers to look at them, I will paste some of them here. These aren't static images... they will update periodically. See if you can spot anything they may have in common or any patterns forming with any particular service as you go through the week that might coincide with troubled times for you with any particular online service:
**it should be noted that even coming from Europe, you may get routed into the US before getting sent up to Montreal. It can and DOES happen.

https://downdetector.com/status/att


https://downdetector.com/status/comcast-xfinity


https://downdetector.com/status/time-warner-cable


https://downdetector.com/status/netflix


https://downdetector.com/status/playstation-network


https://downdetector.com/status/xbox-live


And just for good measure, here are some links to tools providing statistics--some of them allow you to zoom in on specific stats, industries, and regions:

http://www.akamai.com/html/technology/dataviz1.html
(Real Time overview of usage and calculates current usage compared to average usage. Can also toggle a view comparing average latency vs current latency stats)

http://www.akamai.com/html/technology/dataviz3.html
(Tracks HTTP hit and active streams over time, data (including peaks) goes back 6 months)

http://www.akamai.com/html/technolog...try/index.html
(tracks usage by industry type, can filter by region, shows current usage against average use, and has a graphs for last 24hrs and 365 days)

http://www.akamai.com/html/technology/dataviz2.html
(compare latency between two locations against across different routes (Akamia sells optimized routes for a premium). Interesting tool for the debate about altering routes)

http://www.internethealthreport.com/...spx?Period=RH4
(another interesting tool...can compare connectivity between some upper tier ISP's. When latency spikes are detected over 90ms, the boxes get color coded. Note this is ISP to ISP, not client to endpoint)

http://lg.he.net/
(a great tool...can select routers in specific regions and "probe" them to get ping, trace, and BGP Routing details)

***just an interesting quote from that article I linked to about the routing protocol:

Why do you need to understand BGP?

When BGP is configured incorrectly, it can cause massive availability and security problems, as Google discovered in 2008 when its YouTube service became unreachable to large portions of the Internet. What happened was that, in an effort to ban YouTube in its home country, Pakistan Telecom used BGP to route YouTube's address block into a black hole. But, in what is believed to have been an accident, this routing information somehow got transmitted to Pakistan Telecom's Hong Kong ISP and from there got propagated to the rest of the world. The end result was that most of YouTube's traffic ended up in a black hole in Pakistan.

More sinisterly, 2003 saw a number of BGP hijack attacks, where modified BGP route information allowed unknown attackers to redirect large blocks of traffic so that it travelled via routers in Belarus or Iceland before it was transmitted on to its intended destination.

In case you were unaware, earlier tonight the northeast corridors that more than 2/3 the traffic between the US and Montreal must go through were experiencing high lag and packet losses for several hours across multiple providers (it got REALLY bad around Newark for a while there). We're talking big names like AT&T, TWC, Comcast, amongst others. The backbones were overloaded again. As a result, it was negatively impacting multiple games and assorted services including but not limited to XBL, PSN, COD, Battle.net (as in Blizzard's services), FFXIV, ESPN, Hulu, NetFlix, and even 1Channel. It was happening in mass because a lot of people were getting crammed through those specific segments all at the same time trying to get to multiple places at once...all during primetime.

So, it's not a "conspiracy". People have been posting the statistics month after month demonstrating where all the congestion occurs. It's not even theoretical--it has been data collected and presented by the companies that manage these networks. These same statistics and concerns have recently been presented before the commission conducting oversight in the TWC/Comcast merger, as well as repeatedly before Congress over the Net Neutrality debate. People have also been posting that it also happens in the same regions to other services--not just FFXIV, but also LoL, WoW, GW2, Netflix, YouTube, the list goes on and on. And yet.. while there are people experiencing all these problems maintaining stable connections, there are people all over the place also not having ANY problems at all. People across the Atlantic Ocean even... not just people here in North America. It all depends on who gets routed through bad exchanges/segments and who doesn't.

We are hitting peak usage spikes that are double and at times close to triple of what they were 6 months ago. That's the internet as a whole... not just one game or one service. That is everyone hitting multiple services all at once during peak times for their region, and the exchange points are bottlenecking. Then congestion controls kick in to delay packets trying to stave off congestive failure. If line quality gets bad enough, the very TCP/IP protocol in use will flag you with a slow start (effectively throttling your receive window, slowing your throughput). Also, if conditions get bad enough out there along your route, shaping protocols will start to ignore lower priority traffic trying to recover bandwidth. If that gets far enough out of sorts, they start dropping packets.

It is up to the network admins to monitor, capture these exceptions, and adjust accordingly to resolve such issues. Period. SE has nothing to do with that process, nor do they have much of any way to influence it (unless it happens with their ISP or their own network). The problem is either such admins are not aware of how bad it is getting, or something is preventing them from moving forward with investigating/implementing countermeasures. That is why it is important for us to hold accountable the companies from whom we purchase our internet service--they've been getting away with it for decades, creating this grand mess. Collectively, we are ultimately their bosses--we are paying their bills and their payroll. We should expect and demand more for the money we have invested in their service.

[JayHiro]

well, im not sure if they check these posts or not. but that how people stop paying for services or make a switch. when things dont work the way they are intended. ):

[Oughackkin]

Greetings Vejjiegirl,

We understand that you are having Latency Issues, and we know how important a steady connection is to enjoying Final Fantasy XIV: A Realm Reborn.

Could you please provide your Internet Service Provider information? It will allow us to escalate this issue more thoroughly.

Thank You

-Oughackkin

[Windsar]

My problem is similar, I am on Midgardsormr. And when I run a trace route to the server address I get these results:

Tracing route to 199.91.189.28 over a maximum of 30 hops

1 8 ms 7 ms 7 ms d4-50-1-34.try.wideopenwest.com [50.4.34.1]
2 8 ms 9 ms 8 ms 76-73-166-97.knology.net [76.73.166.97]
3 9 ms 10 ms 8 ms 76-73-167-214.knology.net [76.73.167.214]
4 9 ms 8 ms 10 ms 76-73-167-89.knology.net [76.73.167.89]
5 365 ms 36 ms 9 ms 76-73-167-66.knology.net [76.73.167.66]
6 10 ms 10 ms 11 ms xe-5-1-0.bar1.Cincinnati1.Level3.net [4.59.40.17
]
7 38 ms 37 ms 38 ms ae-10-10.car2.Montreal2.Level3.net [4.69.153.86]

8 * 38 ms 38 ms ae-10-10.car2.Montreal2.Level3.net [4.69.153.86]

9 39 ms 39 ms 38 ms ORMUCO-COMM.car2.Montreal2.Level3.net [4.59.178.
74]
10 * * * Request timed out.
11 42 ms 42 ms 42 ms 192.34.76.2
12 43 ms 42 ms 42 ms 199.91.189.234
13 42 ms 43 ms 45 ms 199.91.189.28

Trace complete.


EDIT: This one was taken about 10 minutes later, at 7:10pm EST

Tracing route to 199.91.189.28 over a maximum of 30 hops

1 8 ms 8 ms 7 ms d4-50-1-34.try.wideopenwest.com [50.4.34.1]
2 8 ms 9 ms 8 ms 76-73-166-97.knology.net [76.73.166.97]
3 8 ms 8 ms 8 ms 76-73-167-214.knology.net [76.73.167.214]
4 8 ms 9 ms 9 ms 76-73-167-89.knology.net [76.73.167.89]
5 8 ms 9 ms 8 ms 76-73-167-66.knology.net [76.73.167.66]
6 11 ms 11 ms 11 ms xe-5-1-0.bar1.Cincinnati1.Level3.net [4.59.40.17]
7 * 38 ms 38 ms ae-10-10.car2.Montreal2.Level3.net [4.69.153.86]
8 * * * Request timed out.
9 39 ms 38 ms 39 ms ORMUCO-COMM.car2.Montreal2.Level3.net [4.59.178.74]
10 * * * Request timed out.
11 40 ms 41 ms 39 ms 192.34.76.2
12 41 ms 40 ms 41 ms 199.91.189.234
13 41 ms 42 ms 41 ms 199.91.189.28

Trace complete.

Seems to be right out of Level 3 is where the problems start. Not sure what's going on with hop 10.

The problems get worse as the night goes on. Between 7:00pm to 11:00pm EST are pretty bad.

I hope this helps a little bit. I'm sure other people are going to be having similar problems.

My ISP is Wide Open West

[Raist]

looks like you have issues before you get handed off to Level3... need to get your ISP's Tier3 techs to look into this further. Your ISP is putting you on the knology.net segments, where some typical signs of congestion are first showing up.

Oh, and that hop #10 (most likely 10.2.2.1) is probably going to show up with no responses for a lot of people because it is in a private address space. Depending on how your network and such is configured, it may always show triple stars because of the policies configured on that router. That one typically hovers around the 50-60ms mark for me here in SC, though at times it has been known to hit the 70's under high load conditions.

[EiraValkyrie]

I've been having this excessive lag as well on Ragnarok server during afternoon times (EU), making it impossible to do any crafting or gathering reliably. Here's my tracert (text in swedish, but you should get the gist).

Microsoft Windows [Version 6.1.7601]
Copyright (c) 2009 Microsoft Corporation. Med ensamrätt.

C:\Windows\system32>tracert 199.91.189.60

Spårar route till 199.91.189.60 över ett maximalt antal av 30 hopp

1 2 ms 2 ms 2 ms gw17.a212.priv.bahnhof.se [85.24.234.129]
2 2 ms 2 ms 2 ms vl109-zinken.tule.net.opennet.se [62.116.228.42]

3 2 ms 2 ms 2 ms h-81-170-190-65.na.cust.bahnhof.se [81.170.190.6
5]
4 6 ms 2 ms 2 ms sto-kn71.sto-mar-ar1.bahnhof.net [46.59.112.142]

5 2 ms 2 ms 3 ms sto-mar-ar1.sto-cr3.bahnhof.net [46.59.112.158]

6 3 ms 3 ms 2 ms sto-cr3.sto-cr1.bahnhof.net [46.59.112.162]
7 3 ms 9 ms 4 ms te0-7-0-9.ccr21.sto01.atlas.cogentco.com [149.6.
168.141]
8 5 ms 5 ms 4 ms be2396.ccr21.sto03.atlas.cogentco.com [130.117.5
0.121]
9 24 ms 24 ms 22 ms be2281.ccr41.ham01.atlas.cogentco.com [154.54.63
.1]
10 114 ms 117 ms 114 ms be2186.ccr41.ams03.atlas.cogentco.com [154.54.74
.121]
11 111 ms 111 ms 111 ms be2194.ccr41.lon13.atlas.cogentco.com [130.117.5
0.241]
12 114 ms 113 ms 114 ms be2391.ccr21.lpl01.atlas.cogentco.com [154.54.39
.150]
13 111 ms 110 ms 110 ms be2384.ccr21.ymq02.atlas.cogentco.com [154.54.44
.137]
14 115 ms 115 ms 115 ms 38.122.42.34
15 114 ms 114 ms 113 ms 192.34.76.10
16 116 ms 115 ms 115 ms 199.91.189.242
17 * * * Begäran gjorde timeout.
18 * * * Begäran gjorde timeout.
19 * * * Begäran gjorde timeout.
20 * * * Begäran gjorde timeout.
21 * * * Begäran gjorde timeout.
22 * * * Begäran gjorde timeout.
23 * * * Begäran gjorde timeout.
24 * * * Begäran gjorde timeout.
25 * * * Begäran gjorde timeout.
26 * * * Begäran gjorde timeout.
27 * * * Begäran gjorde timeout.
28 * * * Begäran gjorde timeout.
29 * * * Begäran gjorde timeout.
30 * * * Begäran gjorde timeout.

Spårning utförd.

C:\Windows\system32>

Just a string of timeouts in the end. That doesn't look good. I also tried pinging, but that just timeout'd as well.