Lack of SE Response to Lag

[Duuude007]

Obviously the moose threatened SE and they had no choice but to put them there. CLEAN HANDS.

Seeing that hundreds of thousands more people can play without lag caused by playing from japan datacenter, its a failure, yes?

[Allyra]

Yep - If it was purely SE, wouldn't XI's players also be suffering the same "unplayable lag"?

No one has said it's 100% SE's fault, stop exaggerating for one. Two, you can't compare the two because:

They are not using the same hardware
The netcode was not made by the same exact people
FFXI was specifically built for the ps2
FFXI is not as data intensive
FFXI doesn't even send the same kind of data
FFXI is 10 years old which means lag could have been fixed overtime
Rubberbanding does exist in FFXI (If you played it, I guarantee you would have seen several times people's toons running sporadically to catch up to you)
FFXI doesn't have "unplayable lag" because the game isn't dodge intensive, and you do not get locked out of content over lag.

If there is faulty "insert X" (netcode, hardware, data center placement, etc.) in one, it's not going to guarantee there is faulty "insert X" in the other.

Seeing that hundreds of thousands more people can play without lag caused by playing from japan datacenter, its a failure, yes?

I have no idea what you are asking, but the JP are having problems,too.

[Marishi-Ten]

No one has said it's 100% SE's fault, stop exaggerating for one. Two, you can't compare the two because:

They are not using the same hardware
The netcode was not made by the same exact people
FFXI was specifically built for the ps2
FFXI is not as data intensive
FFXI doesn't even send the same kind of data
FFXI is 10 years old which means lag could have been fixed overtime
Rubberbanding does exist in FFXI (If you played it, I guarantee you would have seen several times people's toons running sporadically to catch up to you)
FFXI doesn't have "unplayable lag" because the game isn't dodge intensive, and you do not get locked out of content over lag.

If there is faulty "insert X" (netcode, hardware, data center placement, etc.) in one, it's not going to guarantee there is faulty "insert X" in the other.

I have no idea what you are asking, but the JP are having problems,too.

1.) Agreed. Hardware differences can impact latency, but from every route parse and ICMP poll, the latency happens before the hosts internal network.
2.) Alot of people blame the netcode. Though I have seen in (rare) instances that the transition from server cluster to server cluster is messed up (3102, 90000, 10105) it seems much better.
3.) The PS2 can support and play XI, but traffic is traffic. It's all the same internet (save for an extra hop through the PS servers which the PS2 didn't have)
4.) I don't think any MMO in general is data intensive. It's not about the amount of data that is being transmitted, it's about the the stability of the connection. Stable/low latency is much more important that up/down ratios
5.) Data transmission is the same. I'm pretty sure both XI and XIV use ICMP packets (pretty sure XI doesn't use TCP/UDP I could be wrong though)
6.) I think the reason XI seemed more stable is because they didn't have multiple data centers employed over continents. Their servers were located in the HQ (or close to it) so they could jump on an issue quickly without having to wait for the mothership approval (like Montreal). It's easier to manage one data center that doesn't have to chain and relay.
7.) I remember a little bit of rubber banding, but it doesn't stand out in my mind (XIV STANDS OUT)
8.) Agreed
9.) They don't employ the same server clusters for XIV and XI. Would be too much of a hassle to find points of failure. Separate the clusters completely and you can completely rule out a possible failure mechanism.

Imagine you wanted to go from your house (client) to let's say Chicago (host). Imagine that every home had it's own airport and airplane (client traffic). Your airplane is little so you can't fly there to Chicago on your own. This means you have to fly to your nearest commercial airport and connect flights (hub) where other people are doing the same thing (traffic). The hub isn't all that big so you have to get to LAX to reach your destination (node) where a TON more people are trying to do the same thing. Imagine that it's Thanksgiving. You can imagine that flights would be booked to the point there are delays (latency) as there are only so many seats the planes have. Now, imagine Delta (Backbone/ISP) lost your luggage or overbooked the flight and say you gotta wait (packet loss) because there are so many people trying to get places and they don't care because hey, they got their money. They know the system is broken, but are doing nothing to fix it.

Imagine the map and lines as flight paths. Imagine this kind of thing happening every single hour of every single day and you would have the internet.

Japan is an island. They don't need as much raw material, maintenance, monitoring, and repairs as there isn't as much ground to cover and the hubs/nodes are substantially larger due to land mass. But when you try to force more data than the nodes can route (which can and WILL happen) problems arise. I think this is why we don't see as many reports of latency from Japan.

It seems the main problem isn't on Square's end, but on the backbones and the ISP (Ormuco/Level3/TATA) that services hubs in the areas. They just can't support and route the sheer amount of traffic that is being sent.

I'm confident of this because I make a living as an IT admin for a company that aggregates, stores, parses, and graphs data from monitoring gateways in the field that report said data into our servers so it can be viewed though websites (either ours or an API call out). Oh well, I'll probably just be flamed anyway for trying to offer an explanation.

[Billie21]

So you're saying it's a problem with the area where the datacenter is located?

[Marishi-Ten]

So you're saying it's a problem with the area where the datacenter is located?

More or less. You can test this yourself by pinging your servers IP address. Famfrit's IP is 199.91.189.39 - http://arrstatus.com/

Open your command prompt and type (without quotations) "tracert 199.91.189.39" and hit enter. Watch for asterisks and high latency values. You will have to run this multiple times as the bandwidth ebbs and flows. You should be able to tell if the issue is on Square (asterisks are after Montreal) or on the node/ISP/backbone (before the Montreal hop).

[Allyra]

FFXI stuff

I wasn't listing those to say that was the exact problems, just pointing out the differences to show why it's silly to say "if it doesn't happen in FFXI it can't be SE's fault for FFXIV."

As to the FFXI lag, I dunno how bad it was honestly, because the game didn't need you to have a perfect connection to play. I was just pointing out it definitely existed because of the running. But nothing in the game was going to kill you if you had rubberbanding. As well, it is a lot easier to notice in this game because of the red target circles letting us know exactly where things are going to hit, which FFXI doesn't have. So even if they were the same I don't think it would have stood out, or that anyone would even care.

I think this is why we don't see as many reports of latency from Japan.

We do see reports though, but I dunno how common they are. I have to rely on friends who can read the JP forums, but they do have it there.

It seems the main problem isn't on Square's end, but on the backbones and the ISP (Ormuco/Level3/TATA) that services hubs in the areas. They just can't support and route the sheer amount of traffic that is being sent.

But that would be on Square's end for putting all the data centers in one spot. Which is why I've been saying for awhile.

You can't honestly say that these hubs are able to handle WoW's traffic with millions of subscribers, but can't handle FFXI's traffic which is probably under a mil.

The difference would be Blizz has multiple datacenters so they aren't overloaded.

Again, not saying SE is 100% at fault, I never ever have. But they CAN do something about it. And some of us would like to know that they are actually doing something.

[Marishi-Ten]

snip

I see what you're saying, I do. But they way traffic routes and the way the entire structure is laid out wouldn't support the argument.

It's not Squares end. Their control starts and stops in Montreal (network wise). I'll show you (early hops omitted due to being behind my companies firewall):




Look at hop 6. I have an asterisk. This is command prompt saying that on the third poll, it could establish/verify the latency at that specific hop. Hop 6 happens to be in Seattle and is managed by Level3. Level3 is doing okay today, but has had issues in the past week: http://www.internetpulse.net/

That "data center" (it's actually a node) isn't managed by Square. Level3 is a Tier 1 backbone and are their own company that doesn't know that SE even exists. They are completely separate companies that do completely different things. It's unreasonable to expect Square to get on the phone to Level3 and say "hey, can you fix that node? Our players are having issues". Level3 would laugh them right off the phone and Square couldn't do anything about it.

Check hops 13-15. Now, these ARE Square's servers and internal route paths. If you see a spike in latency or asterisks here, then yeah, Square's bad.

This is just my route path. I'm one person out of MILLIONS that use the internet a day. Traffic ebbs and flows. There is literally nothing Square can do to fix a node that doesn't belong to them.

Find the break in the connection and run it back to the ISP that manages it. This is the only realistic way to fix the core problem.

WoW has better ROUTE paths. They probably have a contract with their ISP and full fiber connections to every switch. I'm sure Square asked for it. It's up to Ormuco (their ISP) to trench it.

[Allyra]

WoW has better ROUTE paths. They probably have a contract with their ISP and full fiber connections to every switch. I'm sure Square asked for it. It's up to Ormuco (their ISP) to trench it.

Do you understand how contradictory this statement is to the rest of what you are saying? You can't say "SE can't do anything" then say "Blizz did something."

Blizz is not a special snowflake. Blizz and SE have been in the MMO market for about as long. Nor is WoW the only other MMO on the market (just a lot easier to mention as all their information is out there.)

I understand SE doesn't own Level3, or Tata, etc. I never claimed they did. But they can do w/e it is Blizz and other MMO companies are doing in regards to their hardware placement. AND they can compensate for it in their game.

As I said earlier, if this type of lag existed in FFXI, no one would give a moments thought about it, because it wouldn't effect anything.

They built important fights in FFXIV that if you have 1 second of lag, that means you are dead. That's not the ISPs' problem, that's Square's.

[Drekthalon]

I may have missed it but something that also needs to be mentioned is that routing and latency is made worse by the game's design. To avoid hacking SE has designed the game to take place almost exclusively server side with the client not having the authority to tell the server "player was at position x not y and therefore was not actually in an aoe, standing still, etc."

Obviously this did not prevent hacking since everyone has seen teleporting bots on gathering nodes. Most other MMOs allow the client to do some of the work and calculations with timestamps to compensate for latency and packet loss.

This design decision certainly helps make the issues with routing and latency much more noticeable to us the end user.

edit: Blizzard's answer to hacking is to have a service run on the client computer scanning for any programs that are trying to modify traffic coming from the client to do things like teleport and to my knowledge that has worked out ok for them after the initial controversy about Blizzard monitoring your computer died down.

[Marishi-Ten]

Do you understand how contradictory this statement is to the rest of what you are saying? You can't say "SE can't do anything" then say "Blizz did something."

Blizz is not a special snowflake. Blizz and SE have been in the MMO market for about as long. Nor is WoW the only other MMO on the market (just a lot easier to mention as all their information is out there.)

I understand SE doesn't own Level3, or Tata, etc. I never claimed they did. But they can do w/e it is Blizz and other MMO companies are doing in regards to their hardware placement. AND they can compensate for it in their game.

As I said earlier, if this type of lag existed in FFXI, no one would give a moments thought about it, because it wouldn't effect anything.

They built important fights in FFXIV that if you have 1 second of lag, that means you are dead. That's not the ISPs' problem, that's Square's.

I worded my statement poorly. If I go to my ISP and say that the highest residential plan offered isn't enough, they will bump me up to a commercial plan (they won't because where I am, we don't have true fiber drops, but you get the gist) for a considerable amount of money. If the commercial plan isn't enough to meet my needs I can go to the ISP and REQUEST my own drop to the node. They may or may not do this and even if they did, it would take time (infrastructure, planning, prioritization, trenching, interconnect) and cost about 10 times more money (this is if they agree to it. They can say no to my request if they feel it's not feasible).

If Ormuco has the bandwidth, manpower, infastruture, money, engineering to drop more direct lines to Square and Square was willing to pay (it's a lot. Like an ASTRONOMICALLY high amount to do this by the way) and the ISP was willing to do it (these guys sign service agreements with one another) then yeah, I suppose it would be in Squares alley. That risk may be greater than the reward though (monetarily) and either Ormuco isn't willing to trench the miles of fiber to them or Square isn't going to pay the ransom to do it (IF the nodes having problems are local to Ormuco and Square).

Most of the problems I've seen are with Level3 (not surprising). Contact your ISP. Specifically ask them to route you away from the nodes and ask to have a trouble ticket opened at Level3 on your behalf. Contact Level3 every 48 hours with your ticket requesting a status update until the issue is resolved. If enough people do that, they will act.

You got me on the 1 second delay battle system. That was incredibly poor decision making on their end. I'm not trying to white knight Square or anyone (I have my problems with them) but if we want to force change, we have to go to the companies that have root responsibility and work up from there.