Latency/Connection Issues Since the Launch of the EU Data Center

[Moonraker]

Following up from my previous post on this issue here, I have been monitoring the Data Center connection with PingPlotter and simple Ping/ Tracert to confirm the problems. There appears to be an issue each evening which is causing issues primarily with lag/ delays in-game caused by packet loss over the connection to the new EU server in Frankfurt.

The problem with lag & delays in-game are happening each evening during prime times and appear principally due to issues on the hosted data center network (with some issue in the Frankfurt network)

Previously there was some issues leaving France onto the international transfer to Germany. However this appears to have improved, at least in the last days. But the problem with hitching/ lag in-game especially when trying to craft or gather, with often seconds of delay between pressing spells and those spells going off making it a misery at these times.

It's why I have tried to get repeatable results and really try to narrow down the problem. This lag issue is really related directly to the network in Frankfurt but mostly the actual KDDI host for the European Data Center;

KDDI-DEUTSC.edge5.Frankfurt1.Level3.net

This can be seen clearly in this example from this evening over a period of three hours in this case, where the Packet Loss has reached over 12% which means frequent interruptions to data flow between the players client and the game server. Hence the lag and delays as the event entries get delayed and have to wait to synchronize again;



1. Ping is better

Note that the ping remains fairly normal from a 2MB connection in rural France and is similar to non primetime periods.

The ping now to the European DC is almost a third lower now from the previous DC, down from around 150-160ms to around 60-70ms. This is great.

However, whereas before even with higher ping all crafting & gathering, spells etc. were consistent and without any notable hitching or delay. Now, it happens each evening and more frequently over the weekend when more players are connected, even with the lower ping.

The issue mainly lies elsewhere as mentioned.

2. Lag & Hitching Spell/ Movement Delays are worse

What the image shows is the clear hike in Packet Loss (shown as PL% on the left side column) on entering the Frankfurt backbone at hop 7 and especially once entering the DC host service of KDDI at hop 9. From 1% to 5% then to 12% loss at the world server as the last hop 13.

The top graph in the bottom section is before entering; those below once in Frankfurt and then in the host network over three hours shows the problem over time.

12% Packet Loss really impacts play.

Compare the above during the evening to earlier this afternoon;



Some PL but noticeably less not in primetime.

Summary

This issue is repeatable and happens each evening/ primetime. it makes trying to craft or gather a real pain. Macros are impossible etc.

The problem is partly the Frankfurt network which has known issues, for instance in this post on another thread on these forums which also covers similar issues now.

But a big part appears to be directly impacted within the KDDI hosting network where the European Data Center is now.

Hopefully the host service can respond and find some way to deal with this because it really does seem to effect a sizeable number of players using this DC.

It may affect different players in various ways depending on how their net service is routed but in the end. Certainly for me it is impacting my enjoyment of the game I have recently joined and otherwise really love to play. It's why I have spent time to try and work out where the problem lies, including all the usual personal networking, computer diagnosis etc.

Given that the connection is essentially solid it appears that this is the problem, at least for my lag issue. Maybe others who can check in a similar way and via Support to try and get some resolution.

I have quite a lot more data and records I will gladly pass on if they can help and will continue to check things while this continues.

EDIT: Just to backup the PingPlotter data, I also run simple ping tests to the server at 195.82.50.55 (Lich, EU) which show a similar packet loss issue over a sample of 100 pings;

Friday 30/10/15 22:34

Ping statistics for 195.82.50.55:
Packets: Sent = 100, Received = 91, Lost = 9 (9% loss),
Approximate round trip times in milli-seconds:
Minimum = 62ms, Maximum = 69ms, Average = 64ms

Anyone can do this in Windows by opening the Command Prompt (right click Start menu bottom left in Windows 10) and entering;

ping 195.82.50.55 -n 100

This will ping the server 100 times to give a better sample and then give a result as above.

if you are on another world on Chaos then just check Resource Monitor for the active ip address of the ffxiv_dx11.exe process under TCP Connections (ffxiv_dx9.exe if using DX9 client)

You can ping the hops before to check for any issues before reaching the host DC network. Be aware that some hubs/ hops will not respond or give low priority to ICMP pings tests so can show ERR or no response. This is normal. However if it goes all the way to the final hop then there is a problem.

[Raist]

Thanks for sharing the data...it gives some nice examples of things to watch out for.

There are some issues with the data though, which may be causing you to miss some key points about what is going on with the connection. You are comparing a 2 minute sample against almost 2.5 hours (50 samples against 3450 samples)...which may not be painting a very accurate picture, and you seem to be glossing over the jitter factor, which can lead to retransmits as well.

In both samples you are experiencing the same incredibly high jitter earlier in your route that may be contributing to the issues you only caught with the longer sample (if the second sample ran just as long, you may have captured losses at hops 2 and 3 as well--and yes, you are showing losses within France Telecom). Take a closer look at the min/max values. In both samples, your France Telecom is spiking in a major way...at times up to almost 4x your average ping...and you should NOT be hitting 200ms+ response times at your localized gateways either.

Something is up much earlier in this route that your ISP needs to investigate.

[Moonraker]

Thanks for sharing the data...it gives some nice examples of things to watch out for.

There are some issues with the data though, which may be causing you to miss some key points about what is going on with the connection. You are comparing a 2 minute sample against almost 2.5 hours (50 samples against 3450 samples)...which may not be painting a very accurate picture, and you seem to be glossing over the jitter factor, which can lead to retransmits as well.

In both samples you are experiencing the same incredibly high jitter earlier in your route that may be contributing to the issues you only caught with the longer sample (if the second sample ran just as long, you may have captured losses at hops 2 and 3 as well--and yes, you are showing losses within France Telecom). Take a closer look at the min/max values. In both samples, your France Telecom is spiking in a major way...at times up to almost 4x your average ping...and you should NOT be hitting 200ms+ response times at your localized gateways either.

Something is up much earlier in this route that your ISP needs to investigate.

Thanks for your comments.

1. Samples.
Agreed, though the purpose was only to give a general snapshot to compare rather than a direct comparison.

2. Jitter.
I really should just switch the tag off in the testing mainly because it is not a true reflection of the connection. Any internet traffic load from my house will inflate the difference in min/ max latency and will skew the figures. Which, living in a home sharing the connection with my family will happen especially during primetime.

Considering that as mentioned, I live in a rural location and connect over a 2MB ADSL line, there will be some amount of jitter to reach Frankfurt, DE, though typically quality if the line doesn't change much over except in poor weather occasionally for me.

Given that I typically get decent Jitter test results, for example Pingtest (Java not installed so no PL) to Frankfurt ;



Using the wider test depth from DSLReport Ping/ Jitter Test gives a good idea that it is not a general issue for me;

31/10/2015 6:53:31PM calculating grade
31/10/2015 6:53:54PM grade A
31/10/2015 6:54:24PM grade A+
31/10/2015 6:54:54PM grade A

Here is a quick 10 minute sample to show that Jitter is not generally high yet still the packet loss rise from hop 7 at Frankfurt and even more clear at hop 9 into the KKDI host network during mid afternoon.



Regarding the underlying issue of bad lag/ hitching during primetime, it was not an issue until the data center move even with higher latency over a longer distance.

The data is showing a clear rise in packet loss hitting Frankfurt and then notably once in KKDI network.

It could be that it effects lower speed connections more than faster ones. Either way there is an issue for a lot of players and it lies somewhere which has some common root. This looks like a likely place to start.

[Laffek]

Name of Internet Service Provider (ISP)
Orange

Country of residence
Poland

Reference number from your ISP, if you have raised this matter with them
-

When does the problem occur (e.g. only during specific times such as in the evenings?)
Everytimes

Frequency of problem (e.g. every 2 minutes)
Every minute

Everywhere. Cities, dungeons, etc...

[Raist]

Edit: I must apologize if you feel I am singling you out or anything negative like that, it's just that I frequently see posts in these threads that miss important details, or otherwise may by applying something improperly to the specific problem--in this case, as detailed below, some of the test results cannot be used as a proper gauge of the QoS specifically to the demarcation point to the SE datacenter. I don't want to discourage the participation and detailed results from coming in...we just need to make sure that the data is relevant and potentially actionable information.

You are again loosing packets first at the exchange point between Opentransit and Level3---an election made by your ISP's routing policies.

Also still showing bad latency spikes in France---high enough to cause out of order packets. In the scope of this game, it can disrupt the quality of play (stuttering, pausing/speedup rubberbanding, etc.). Those ping test app ratings are not necessarily indicative of more critical dataflows---they are typically evaluating more in the scope of VOIP and video streaming, both of which are buffered and thus more tolerant of higher intermittent spikes and jitter.

Pingtest and such are also not necessarily testing along the same ISP's either. The Pingtest test is against Vodafone. More specifically, 145.253.248.206-- AS3209 is their ID for BGP. When I run a looking glass from Paris, I get slung through Telia to get to KDDI/SE---but Vodafone peers directly from that location (no Telia). When I trace to that Vodafone IP from home, I got bad spiking at hbg-bb1-link.telia.net [213.155.131.246]--an address not present in my pathing to KDDI, amongst other segments (after Ashburn's B2 hop, it can vary between B3 and B4, the B4 route it used for Vodafone routed differently to Germany then it did for SE last night--they appear to use different next-hop values in their tables).

Also when I trace from home to 95.172.92.166 (the IP for Germany in that DSL test you linked), I am using Cogent instead of Telia. From the same looking glass in Paris, it is using Telia but the next hop is 213.248.93.249--to get to SE the next hop is 213.248.93.81.

Not saying for certain it is the same for your ISP, but the possibility is there for the routing to be considerably different. So those ping test apps are not necessarily giving an indication of the specific path to SE's servers.

Also, here is the pingtest.net description of a B rating:

Very good! Your connection should work well for any Internet application. Some online games may not perform optimally.
MOS between 4.28 and 4.37 (Example - a ping around 90 ms with 0% packet loss)

**Their MOS scoring is an evaluation of the quality of a voice call. This game does not buffer in kind and is thus highly dependent on the timely, orderly delivery of the packets. Also note your score has an asterisk, indicating that one metric failed testing--it even tells you it failed the packet loss test in red text. Also note a very specific statement in there as well:

"Some online games may not perform optimally."

Should also note a statement at the DSL test as well:

If there is local congestion or ISP problems then you may see volatile latency numbers and the radar graph will show an inner green area and an outer green boundary. A stable connection will show stable numbers, a good grade, and the radar graph will be a solid color with very little movement and activity.

(in other words, localized issues may impact the results dramatically)

[Moonraker]

Edit: I must apologize if you feel I am singling you out or anything negative like that, it's just that I frequently see posts in these threads that miss important details, or otherwise may by applying something improperly to the specific problem--in this case, as detailed below, some of the test results cannot be used as a proper gauge of the QoS specifically to the demarcation point to the SE datacenter. I don't want to discourage the participation and detailed results from coming in...we just need to make sure that the data is relevant and potentially actionable information.

You are again loosing packets first at the exchange point between Opentransit and Level3---an election made by your ISP's routing policies.

Also still showing bad latency spikes in France---high enough to cause out of order packets. In the scope of this game, it can disrupt the quality of play (stuttering, pausing/speedup rubberbanding, etc.). Those ping test app ratings are not necessarily indicative of more critical dataflows---they are typically evaluating more in the scope of VOIP and video streaming, both of which are buffered and thus more tolerant of higher intermittent spikes and jitter.

Pingtest and such are also not necessarily testing along the same ISP's either. The Pingtest test is against Vodafone. More specifically, 145.253.248.206-- AS3209 is their ID for BGP. When I run a looking glass from Paris, I get slung through Telia to get to KDDI/SE---but Vodafone peers directly from that location (no Telia). When I trace to that Vodafone IP from home, I got bad spiking at hbg-bb1-link.telia.net [213.155.131.246]--an address not present in my pathing to KDDI, amongst other segments (after Ashburn's B2 hop, it can vary between B3 and B4, the B4 route it used for Vodafone routed differently to Germany then it did for SE last night--they appear to use different next-hop values in their tables).

Also when I trace from home to 95.172.92.166 (the IP for Germany in that DSL test you linked), I am using Cogent instead of Telia. From the same looking glass in Paris, it is using Telia but the next hop is 213.248.93.249--to get to SE the next hop is 213.248.93.81.

Not saying for certain it is the same for your ISP, but the possibility is there for the routing to be considerably different. So those ping test apps are not necessarily giving an indication of the specific path to SE's servers.

Also, here is the pingtest.net description of a B rating:

**Their MOS scoring is an evaluation of the quality of a voice call. This game does not buffer in kind and is thus highly dependent on the timely, orderly delivery of the packets. Also note your score has an asterisk, indicating that one metric failed testing--it even tells you it failed the packet loss test in red text. Also note a very specific statement in there as well:

"Some online games may not perform optimally."

Should also note a statement at the DSL test as well:

(in other words, localized issues may impact the results dramatically)

No problem with your desire to ensure that data posted is as useful as possible. It's great to have that interest and input.

I understand your reasoning and that general ISP/ network issue scan and will impact any test. However we can only provide data as an end user and hope that this gives some help/ indication for SE to move forward with some resolution to this issue which effects a number of their customers (hence this thread)

The links posted were really only to show that generally I do not suffer issues with poor jitter generally over my connection. it is what the test showed (as it was to a lot of sites around the globe and takes a aggregate result) This is not specific to the Data Center for the game.

All the points you made are totally valid and routing etc. will definitely impact results and potential issues. However, I must consider what I can which is the issues I have with lag/ hitching experienced each evening during primetime play ever since the data centre move. Even with lower latency this problem stems from that date.

Each evening I am looking at another PingPlotter analysis which shows the same thing; namely little packet loss until reaching the initial hop for KDDI which hosts the SE European Data Center and on through each hop thereafter to the final world server (PL at hop 6 is typical of hub not prioritising ICMP hence drops them but is not mirrored to PL after this hop. Generally it is expected)

This packet loss around 10%+ seems the most likely candidate for this issue from the data, like from this evening 10 minute sample;



If you have a better explanation for the jump in consistent packet loss from hop 9 into the KDDI network I would be interested to hear it.

Just to note that if you see spikes of latency in the samples it is mostly me just opening a web page etc. Given a 2MB ADSL connection even this will quickly take the available 264 Kb/sec ^^ This will also mess the Jitter calculation too.

All this is just trying to provide what data I can to help them find some root cause and fix it because it is continuing. If there is better way to do so I'm happy to do it.

[Kuinu]

ISP
Orange

Country of residence
Spain

Reference number from your ISP, if you have raised this matter with them
There's no one, even if I tried to talk to them loads of times. Useless bunch. The moment I get a better ISP offer, I'm getting off this train.

When does the problem occur (e.g. only during specific times such as in the evenings?)
Afternoons and evenings. Mornings were kind of bearable.

Frequency of problem (e.g. every 2 minutes)
10-20 seconds between each lagspike. Sometimes 5 seconds.

Game content in which the issues occur (e.g. dungeons, trials, certain areas)
Any map. Any instance. Any moment.

EDIT: Just in case personal posts to every user are sent with instructions; I'm not going to pay for another month until I'm sure that the problem has been issued and dealt with. I'll be glad to provide any information you need, but for instance, logging in or trying to log in is out of the question.

[Qba]

Name of Internet Service Provider (ISP)
Orange

Country of residence
Poland


When does the problem occur (e.g. only during specific times such as in the evenings?)
constant, worse during peak time

Frequency of problem (e.g. every 2 minutes)
Every 5sec : freeze / lag during peak time. More like every 15sec otherwise.

Game content in which the issues occur (e.g. dungeons, trials, certain areas)
Everywhere


It seems that wtfast helps, without it the game is unplayable.

[Tchyo]

Name of Internet Service Provider (ISP)
Orange

Country of residence
France

When does the problem occur (e.g. only during specific times such as in the evenings?)
more or less constant, but the symptoms are a lot more acute during peak time (evening)

Frequency of problem (e.g. every 2 minutes)
More or less random, but still regular enough (every five minutes at most).

Game content in which the issues occur (e.g. dungeons, trials, certain areas)
More or less everything, I feel it a lot more in instances but it could be just be doing things more sensible to latency there.

The exact symptoms are network packet loss and a lot of rubber banding, I can move my character around and cast skills, but nothing network-based works, the world is frozen, using skills doesn't trigger any proc or combo. I can only guess I appear frozen with the red "disconnecting" status to other people at this point. Most of the freezes only last a few seconds, but now and then it can go as far as 30s-1min.

Since I happen to rent a dedicated server at Online, I took some time to setup an OpenVPN-based tunnel on it and it completely solved the packet loss issue. The game is near unplayable for combat content without it.

[Zuri_54]

Name of Internet Service Provider (ISP)
Virgin Media (i think)

Country of residence
England

When does the problem occur (e.g. only during specific times such as in the evenings?)
Every time I have tried playing (I tested morning, afternoon and evening with similar results on each)

Frequency of problem (e.g. every 2 minutes)
Kind of random, but the lag hits between every 30 seconds to minute or so (if i manage to stay connected that long) and I am getting kicked with the 90002 error after a maximum of 5-7 minutes.

Game content in which the issues occur (e.g. dungeons, trials, certain areas)
Everywhere, although I have been unable to register for dungeons or trials because of getting diconnecting.