Latency/Connection Issues Since the Launch of the EU Data Center

[MkRadio]

Name of Internet Service Provider (ISP)
AT&T U-Verse

Country of residence
USA (Wisconsin)

Reference number from your ISP, if you have raised this matter with them
--No reference number given.

When does the problem occur (e.g. only during specific times such as in the evenings?)
Noticeable, yet manageable lag: All the time outside of "Significant lag period" below.
Significant lag: 8PM CST - Mid-morning (So Prime Time I assume)

Frequency of problem (e.g. every 2 minutes)
Constantly outside of raids along with every few seconds in raids. (A3S is somehow the most noticeable)

Game content in which the issues occur (e.g. dungeons, trials, certain areas)
Raids, idling, crafting, gathering, etc. (Once again, Arm of the Father (Savage) is the most noticeable)

IP address (optional)
--

[Phazonius]

Name of Internet Service Provider (ISP)
ONO

Country of residence
Spain

Reference number from your ISP, if you have raised this matter with them

When does the problem occur (e.g. only during specific times such as in the evenings?)
At all times.

Frequency of problem (e.g. every 2 minutes)
Between 1 and 5 minutes. Sometimes more often sometimes less.

Game content in which the issues occur (e.g. dungeons, trials, certain areas)
Everywhere. Specially, but not exclusively, while doing Alexander (Savage) or mining.

IP address (optional)

[MkRadio]

I'm not sure if it's actually WORSE in raids, but it's definitely more noticeable with heavy time-related mechanics being between those freeze moments... Especially when you lag during Digititis and have an moment of embarrassment when the static starts yelling at you.

[LZero]

Name of Internet Service Provider (ISP): Time Warner Cable
Country of residence: USA (Indiana)
Reference number from your ISP, if you have raised this matter with them: N/A
When does the problem occur (e.g. only during specific times such as in the evenings?): Started at the beginning of the week as a minor hiccup, only continued to get worse as the week has gone on. It now lags considerablly and kicks me off on occasion.
Frequency of problem (e.g. every 2 minutes): Every 20-30 secs
Game content in which the issues occur (e.g. dungeons, trials, certain areas): Everywhere, less so in battle and always when interacting with npcs or teleporting.
IP address (optional):

[Moonraker]

Following up from my previous post on this issue here, I have been monitoring the Data Center connection with PingPlotter and simple Ping/ Tracert to confirm the problems. There appears to be an issue each evening which is causing issues primarily with lag/ delays in-game caused by packet loss over the connection to the new EU server in Frankfurt.

The problem with lag & delays in-game are happening each evening during prime times and appear principally due to issues on the hosted data center network (with some issue in the Frankfurt network)

Previously there was some issues leaving France onto the international transfer to Germany. However this appears to have improved, at least in the last days. But the problem with hitching/ lag in-game especially when trying to craft or gather, with often seconds of delay between pressing spells and those spells going off making it a misery at these times.

It's why I have tried to get repeatable results and really try to narrow down the problem. This lag issue is really related directly to the network in Frankfurt but mostly the actual KDDI host for the European Data Center;

KDDI-DEUTSC.edge5.Frankfurt1.Level3.net

This can be seen clearly in this example from this evening over a period of three hours in this case, where the Packet Loss has reached over 12% which means frequent interruptions to data flow between the players client and the game server. Hence the lag and delays as the event entries get delayed and have to wait to synchronize again;



1. Ping is better

Note that the ping remains fairly normal from a 2MB connection in rural France and is similar to non primetime periods.

The ping now to the European DC is almost a third lower now from the previous DC, down from around 150-160ms to around 60-70ms. This is great.

However, whereas before even with higher ping all crafting & gathering, spells etc. were consistent and without any notable hitching or delay. Now, it happens each evening and more frequently over the weekend when more players are connected, even with the lower ping.

The issue mainly lies elsewhere as mentioned.

2. Lag & Hitching Spell/ Movement Delays are worse

What the image shows is the clear hike in Packet Loss (shown as PL% on the left side column) on entering the Frankfurt backbone at hop 7 and especially once entering the DC host service of KDDI at hop 9. From 1% to 5% then to 12% loss at the world server as the last hop 13.

The top graph in the bottom section is before entering; those below once in Frankfurt and then in the host network over three hours shows the problem over time.

12% Packet Loss really impacts play.

Compare the above during the evening to earlier this afternoon;



Some PL but noticeably less not in primetime.

Summary

This issue is repeatable and happens each evening/ primetime. it makes trying to craft or gather a real pain. Macros are impossible etc.

The problem is partly the Frankfurt network which has known issues, for instance in this post on another thread on these forums which also covers similar issues now.

But a big part appears to be directly impacted within the KDDI hosting network where the European Data Center is now.

Hopefully the host service can respond and find some way to deal with this because it really does seem to effect a sizeable number of players using this DC.

It may affect different players in various ways depending on how their net service is routed but in the end. Certainly for me it is impacting my enjoyment of the game I have recently joined and otherwise really love to play. It's why I have spent time to try and work out where the problem lies, including all the usual personal networking, computer diagnosis etc.

Given that the connection is essentially solid it appears that this is the problem, at least for my lag issue. Maybe others who can check in a similar way and via Support to try and get some resolution.

I have quite a lot more data and records I will gladly pass on if they can help and will continue to check things while this continues.

EDIT: Just to backup the PingPlotter data, I also run simple ping tests to the server at 195.82.50.55 (Lich, EU) which show a similar packet loss issue over a sample of 100 pings;

Friday 30/10/15 22:34

Ping statistics for 195.82.50.55:
Packets: Sent = 100, Received = 91, Lost = 9 (9% loss),
Approximate round trip times in milli-seconds:
Minimum = 62ms, Maximum = 69ms, Average = 64ms

Anyone can do this in Windows by opening the Command Prompt (right click Start menu bottom left in Windows 10) and entering;

ping 195.82.50.55 -n 100

This will ping the server 100 times to give a better sample and then give a result as above.

if you are on another world on Chaos then just check Resource Monitor for the active ip address of the ffxiv_dx11.exe process under TCP Connections (ffxiv_dx9.exe if using DX9 client)

You can ping the hops before to check for any issues before reaching the host DC network. Be aware that some hubs/ hops will not respond or give low priority to ICMP pings tests so can show ERR or no response. This is normal. However if it goes all the way to the final hop then there is a problem.

[Raist]

Thanks for sharing the data...it gives some nice examples of things to watch out for.

There are some issues with the data though, which may be causing you to miss some key points about what is going on with the connection. You are comparing a 2 minute sample against almost 2.5 hours (50 samples against 3450 samples)...which may not be painting a very accurate picture, and you seem to be glossing over the jitter factor, which can lead to retransmits as well.

In both samples you are experiencing the same incredibly high jitter earlier in your route that may be contributing to the issues you only caught with the longer sample (if the second sample ran just as long, you may have captured losses at hops 2 and 3 as well--and yes, you are showing losses within France Telecom). Take a closer look at the min/max values. In both samples, your France Telecom is spiking in a major way...at times up to almost 4x your average ping...and you should NOT be hitting 200ms+ response times at your localized gateways either.

Something is up much earlier in this route that your ISP needs to investigate.

[Laffek]

Name of Internet Service Provider (ISP)
Orange

Country of residence
Poland

Reference number from your ISP, if you have raised this matter with them
-

When does the problem occur (e.g. only during specific times such as in the evenings?)
Everytimes

Frequency of problem (e.g. every 2 minutes)
Every minute

Everywhere. Cities, dungeons, etc...

[Moonraker]

Thanks for sharing the data...it gives some nice examples of things to watch out for.

There are some issues with the data though, which may be causing you to miss some key points about what is going on with the connection. You are comparing a 2 minute sample against almost 2.5 hours (50 samples against 3450 samples)...which may not be painting a very accurate picture, and you seem to be glossing over the jitter factor, which can lead to retransmits as well.

In both samples you are experiencing the same incredibly high jitter earlier in your route that may be contributing to the issues you only caught with the longer sample (if the second sample ran just as long, you may have captured losses at hops 2 and 3 as well--and yes, you are showing losses within France Telecom). Take a closer look at the min/max values. In both samples, your France Telecom is spiking in a major way...at times up to almost 4x your average ping...and you should NOT be hitting 200ms+ response times at your localized gateways either.

Something is up much earlier in this route that your ISP needs to investigate.

Thanks for your comments.

1. Samples.
Agreed, though the purpose was only to give a general snapshot to compare rather than a direct comparison.

2. Jitter.
I really should just switch the tag off in the testing mainly because it is not a true reflection of the connection. Any internet traffic load from my house will inflate the difference in min/ max latency and will skew the figures. Which, living in a home sharing the connection with my family will happen especially during primetime.

Considering that as mentioned, I live in a rural location and connect over a 2MB ADSL line, there will be some amount of jitter to reach Frankfurt, DE, though typically quality if the line doesn't change much over except in poor weather occasionally for me.

Given that I typically get decent Jitter test results, for example Pingtest (Java not installed so no PL) to Frankfurt ;



Using the wider test depth from DSLReport Ping/ Jitter Test gives a good idea that it is not a general issue for me;

31/10/2015 6:53:31PM calculating grade
31/10/2015 6:53:54PM grade A
31/10/2015 6:54:24PM grade A+
31/10/2015 6:54:54PM grade A

Here is a quick 10 minute sample to show that Jitter is not generally high yet still the packet loss rise from hop 7 at Frankfurt and even more clear at hop 9 into the KKDI host network during mid afternoon.



Regarding the underlying issue of bad lag/ hitching during primetime, it was not an issue until the data center move even with higher latency over a longer distance.

The data is showing a clear rise in packet loss hitting Frankfurt and then notably once in KKDI network.

It could be that it effects lower speed connections more than faster ones. Either way there is an issue for a lot of players and it lies somewhere which has some common root. This looks like a likely place to start.

[Raist]

Edit: I must apologize if you feel I am singling you out or anything negative like that, it's just that I frequently see posts in these threads that miss important details, or otherwise may by applying something improperly to the specific problem--in this case, as detailed below, some of the test results cannot be used as a proper gauge of the QoS specifically to the demarcation point to the SE datacenter. I don't want to discourage the participation and detailed results from coming in...we just need to make sure that the data is relevant and potentially actionable information.

You are again loosing packets first at the exchange point between Opentransit and Level3---an election made by your ISP's routing policies.

Also still showing bad latency spikes in France---high enough to cause out of order packets. In the scope of this game, it can disrupt the quality of play (stuttering, pausing/speedup rubberbanding, etc.). Those ping test app ratings are not necessarily indicative of more critical dataflows---they are typically evaluating more in the scope of VOIP and video streaming, both of which are buffered and thus more tolerant of higher intermittent spikes and jitter.

Pingtest and such are also not necessarily testing along the same ISP's either. The Pingtest test is against Vodafone. More specifically, 145.253.248.206-- AS3209 is their ID for BGP. When I run a looking glass from Paris, I get slung through Telia to get to KDDI/SE---but Vodafone peers directly from that location (no Telia). When I trace to that Vodafone IP from home, I got bad spiking at hbg-bb1-link.telia.net [213.155.131.246]--an address not present in my pathing to KDDI, amongst other segments (after Ashburn's B2 hop, it can vary between B3 and B4, the B4 route it used for Vodafone routed differently to Germany then it did for SE last night--they appear to use different next-hop values in their tables).

Also when I trace from home to 95.172.92.166 (the IP for Germany in that DSL test you linked), I am using Cogent instead of Telia. From the same looking glass in Paris, it is using Telia but the next hop is 213.248.93.249--to get to SE the next hop is 213.248.93.81.

Not saying for certain it is the same for your ISP, but the possibility is there for the routing to be considerably different. So those ping test apps are not necessarily giving an indication of the specific path to SE's servers.

Also, here is the pingtest.net description of a B rating:

Very good! Your connection should work well for any Internet application. Some online games may not perform optimally.
MOS between 4.28 and 4.37 (Example - a ping around 90 ms with 0% packet loss)

**Their MOS scoring is an evaluation of the quality of a voice call. This game does not buffer in kind and is thus highly dependent on the timely, orderly delivery of the packets. Also note your score has an asterisk, indicating that one metric failed testing--it even tells you it failed the packet loss test in red text. Also note a very specific statement in there as well:

"Some online games may not perform optimally."

Should also note a statement at the DSL test as well:

If there is local congestion or ISP problems then you may see volatile latency numbers and the radar graph will show an inner green area and an outer green boundary. A stable connection will show stable numbers, a good grade, and the radar graph will be a solid color with very little movement and activity.

(in other words, localized issues may impact the results dramatically)

[AquaCasca]

#Name of Internet Service Provider (ISP)
-Meo

#Country of residence
-Portugal

#When does the problem occur (e.g. only during specific times such as in the evenings?)
-Notice continuous Lag spikes between 6pm till late night around 10pm. almost unplayed, worse then Montreal servers.

#Frequency of problem (e.g. every 2 minutes)
-everyday so far. normally since EU datacenter launch my ping was a steady 70ms in the first week(was really nice).
but now it jumps to 120-150ms (or more) around 6pm-10pm. servertime

#Game content in which the issues occur (e.g. dungeons, trials, certain areas)
-everywhere