You gross 200 Million in subscriptions a month why are your servers junk?

[Valkyrie_Lenneth]

That was a lot of words to say "I'm not experienced in this kind of workload with these specific requirements"

Video streaming is a whole different thing from game servers.


And yes, there was a ddos attack at the time, people confirmed it from outside sources, providers were being hit as well, especially level3.

[Adrestia]

I could go into a much longer diatribe on interactive services at scale but that would take even _more_ words. If you’ve been in the MMO space for long you should know that EVE Online hosts 120,000+ concurrent users in a single-sharded service. Not “total subscribed accounts” or “active” over some broadly defined period of time. Concurrent. You can find some reader-friendly articles explaining the basics of the (10 year old) tech online.

“People” “confirmed” it by which you mean someone in the forum posted, without context, a map based on Arbor Networks data that showed pretty lines entering and leaving the US? Botnet activity is background noise. The botnets any random guy with a tor browser can hire for $10 are well-handled by commercial anti-DDoS solutions. If you want to lose sleep at night, think about what a DNS amplification attack against real critical services would do. But no one is investing that much in hitting MMOs. There are minor-scale DDoS attacks against games now and then when there’s a profit motive, like being able to dupe items in a P2W economy. But DDoSing games “for the lulz” is pretty much a dead art. Turns out when people from Lizard Squad and LulzSec started going to federal prison with decade-long sentences, people backed off of casually DDoSing games just because they could.

[Melichoir]

Assumnig that there are 650k world wide and the average sub is 14 USD ( I know it fluctuates based on region, but there is people who pay more or less, so for simple math lets say its 14USD), SE pulls down 9.1 Mil a month, or 109 Mil a year from subs alone for FFXIV.

That soudns like a lot of money, but then lets start looking at overhead costs. Se employs 4.3k people, with an average salary of 273k. Now that figure probably is including an overpaid management, so lets say the average is closer to 70k (average game dev salary). That means per year, 301 Million in salaries alone. Thats 3x the cost they get from Subs. Mind you, SE supposedly pulls down about 2.4 Billion a year. But subs only make 4% of their revenue, where as employee costs are about 13% of that revenue. Then we get into property costs, Tariff costs, server costs, equipment costs, third party costs (lawyers, janitorial staffs, etc), development on game projects (next xpac along with any other SE game) and you start quickly eating into how much money they make. This gets even more nutty when you find out they made less money in 2018 than they did in earlier years.

Point being, even with 650k subs, it's not so straight forward to have good servers. Not that they cant fix things and do better, but you need to be cautious about thinking that a high sub count means theyre swimming in cash. It's not that cut and dry.

[NanaWiloh]

OP claiming to work on Servers does not mean crap!! You are not the one handling SE servers. You dont know their server infrastructure your making blind anger filled guesses as to what the problem is, secondly any joe blow can claim to too work on servers so your wasting your breathe on that one.

[Adrestia]

OP claiming to work on Servers does not mean crap!! You are not the one handling SE servers. You dont know their server infrastructure your making blind anger filled guesses as to what the problem is, secondly any joe blow can claim to too work on servers so your wasting your breathe on that one.

Whoa. Calm down. Take a breath. I’m skeptical of some of the OP’s claims as well, but in your rush to post your outrage you’re one run-on or typo from a heart attack.

No matter what, remember that regardless of what SE’s current infrastructure is, there are industry standards. If they (and they certainly did) chose to ignore the collective wisdom of the DevOps community, that’s on them. Thankfully, there are larger businesses with older tech who are still managing to be dragged kicking and screaming into the 21st century. They can do this because the industry has put a massive amount of focus into providing layers of abstraction that can make the client, the server, or both completely blind to the fact that they’re being dynamically load-balanced and elastically scaled.

I can tell you from experience, the hardest part isn’t the technology. Once you get the right stuff approved for the job, it’s not too bad to implement. Not trivial, and there’s a reason DevOps professionals get paid well (Unlike most software developers. Holy crap do we get hosed by comparison). But seriously, the hardest part is convincing management that it’s time to catch up and do the right thing. Gaping security vulnerability? “They probably won’t hack us.” Critical performance instability? “We can just reboot every night until we decide to figure it out.” That’s what holds the dev and infrastructure teams back. Not that we’re being asked to solve unsolvable problems.

[NanaWiloh]

Whoa. Calm down. Take a breath. I’m skeptical of some of the OP’s claims as well, but in your rush to post your outrage you’re one run-on or typo from a heart attack.

No matter what, remember that regardless of what SE’s current infrastructure is, there are industry standards. If they (and they certainly did) chose to ignore the collective wisdom of the DevOps community, that’s on them. Thankfully, there are larger businesses with older tech who are still managing to be dragged kicking and screaming into the 21st century. They can do this because the industry has put a massive amount of focus into providing layers of abstraction that can make the client, the server, or both completely blind to the fact that they’re being dynamically load-balanced and elastically scaled.

I can tell you from experience, the hardest part isn’t the technology. Once you get the right stuff approved for the job, it’s not too bad to implement. Not trivial, and there’s a reason DevOps professionals get paid well (Unlike most software developers. Holy crap do we get hosed by comparison). But seriously, the hardest part is convincing management that it’s time to catch up and do the right thing. Gaping security vulnerability? “They probably won’t hack us.” Critical performance instability? “We can just reboot every night until we decide to figure it out.” That’s what holds the dev and infrastructure teams back. Not that we’re being asked to solve unsolvable problems.

I am well aware money holds the dev team back. They are at the mercy of the people upstairs, if Yoshi gets told no about a bigger budget then there is not much he can do now can he? I have no doubt he has asked and gotten told no as well as yes. But the server issue seems to be one he is not getting much of a yes on.

[Spirited-Away]

2hrs waiting and still not in, i’ve given up, they’re taking the **** now.

[Kirsten_Rev]

I could go into a much longer diatribe on interactive services at scale but that would take even _more_ words. If you’ve been in the MMO space for long you should know that EVE Online hosts 120,000+ concurrent users in a single-sharded service. Not “total subscribed accounts” or “active” over some broadly defined period of time. Concurrent. You can find some reader-friendly articles explaining the basics of the (10 year old) tech online.

In fairness, one cannot really compare EVE Online to most other MMOs. The technical side - as any of those reader-friendly articles explains - is an issue, yes, but there's an interrelated and critical content design factor as well. EVE Online was built with the viewpoint that players are the content. XIV, WoW, and most other MMOs are not - their content is presented to us by the development team in the form of stories, scripted fights, etc., all of which take place in a physical world that SE manually constructs for us. This physical world makes single shards unwieldy, because there's only so much space into which physical avatars can be placed. Could you imagine the visual clutter if every world-shard was combined for XIV? Just finding a Market Board to click in the Crystarium would be a nightmare. It's no different than why FPS titles limit the number of people in a given online match, rather than allowing for a global free-for-all on a single piece of land.

This also means the elasticity you're talking about really isn't something SE could achieve, because shards can't simply be spun up at will without consequences. Unlike a collection of movies on Netflix, worlds may start from the same stock template, but then they change in a way that has to remain persistent - people buy houses, goods are listed on the market board, etc. So one shard rapidly becomes distinct from all the others. That's not an architectural flaw; it's a feature. As a result, while there would be a couple of benefits to a more modern architecture - the removal of logical data center separation, for instance - it likely wouldn't really help with server congestion issues at the launch of an expansion.

[Adrestia]

This also means the elasticity you're talking about really isn't something SE could achieve, because shards can't simply be spun up at will without consequences. Unlike a collection of movies on Netflix, worlds may start from the same stock template, but then they change in a way that has to remain persistent - people buy houses, goods are listed on the market board, etc. So one shard rapidly becomes distinct from all the others. That's not an architectural flaw; it's a feature. As a result, while there would be a couple of benefits to a more modern architecture - the removal of logical data center separation, for instance - it likely wouldn't really help with server congestion issues at the launch of an expansion.

Good post.

It’s true there will always be a base capacity requirement to fit the needs of the game. The idea behind servers-on-demand is to minimize waste. You don’t often notice waste in housing because it’s basically all taken all the time, although you can easily see that because each large plot is “burdened” with several medium and small plots, the cost of adding a large plot now has the additional cost of adding a full district.

With split world servers, you see the waste even more dramatically. Why have low pop/ghost town worlds that are on hardware 6’ from hardware that’s at max capacity and bursting at the seams with new login attempts? A more modern approach would be to simply say those both of those servers are part of the same cluster. If one has a free vCPU and memory to spin up another zone server (say, a Crystarium instance to avoid market board area overcrowding), then it can do that. Under this model, if they don’t have the resources to spin up a new zone, it’s because they literally do not have them; not because they’re being under-utilized due to inefficient partitioning. Arguably at this point the answer is to then launch an instance on pay-per-use cloud computing, up to the point at which they have to say “hold on, a 4 million dollar AWS bill just isn’t in the cards this month.” But even if they wholly refuse to add capacity on demand, at least they’re no longer wasting the capacity they already own.

[Edax]

Angry You gross 200 Million in subscriptions a month why are your servers junk?

The population of the entirely of North America itself is 579 Million. I think you exaggerated the problems with the NA servers.