ELI5: Why the absolute hell did SE not use Azure or AWS as a temporary solution?

I'm just mentally frazzled after over 7 hours of trying to login now and I can't for the life of me understand why they would turn down and/or not use AWS or Azure as a temporary solution.

Is it pride? Is it not a possibility?

I just see a scenario where a company wouldn't use all available resources to assist their customers and maintain the positive image their brand and game had going into EW's launch.

Ask New World players how stable AWS is to actually play on.

1) Shareholders

2) See the above

Quote:

---

Originally Posted by SnowVix

Ask New World players how stable AWS is to actually play on.

---

They still got to play and didn't get kicked after six+ hours in queue. That's still an improvement over this shit show.

Quote:

---

Originally Posted by VisperCon

They still got to play and didn't get kicked after six+ hours in queue. That's still an improvement over this shit show.

---

getting booted and having multiple connection issues mid-play session is far worse, especially since you'd still be queueing to get back in, even if not for as long.

You just can't move a game server from dedicated hardware onto AWS

Think about when you play the game, every single action you and every other player and every monster needs to be calculated and the result sent back within 20 ms, 24/7, without even a single delay. Unlike Google or Amazon which can occassionally take a second or two longer to load a page and no one will notice, in a game if you lag even a little bit and even if its only once in an entire fight, it's really obvious. Even livestream sites like twitch don't need this kind of performance, because their video is buffered, it knows what to send seconds before it needs to. FFXIV needs to react to your actions almost instantly, even though the calculations are simple, the consistent responsiveness makes it pretty specialized.

And services like AWS and Azure just aren't designed for it because online games are smaller than the thousands of small and large tech companies that do use AWS, companies who care more about ease of development and scalability rather than microsecond performance. And AWS gets designed for those companies instead.

Quote:

---

Originally Posted by lisaa

You just can't move a game server from dedicated hardware onto AWS

Think about when you play the game, every single action you and every other player and every monster needs to be calculated and the result sent back within 20 ms, 24/7, without even a single delay. Unlike Google or Amazon which can occassionally take a second or two longer to load a page and no one will notice, in a game if you lag even a little bit and even if its only once in an entire fight, it's really obvious. Even livestream sites like twitch don't need this kind of performance, because their video is buffered, it knows what to send seconds before it needs to. FFXIV needs to react to your actions almost instantly, even though the calculations are simple, the consistent responsiveness makes it pretty specialized.

And services like AWS and Azure just aren't designed for it because online games are smaller than the thousands of small and large tech companies that do use AWS, companies who care more about ease of development and scalability rather than microsecond performance. And AWS gets designed for those companies instead.

---

What? Xbox Live runs on Azure....

Quote:

---

In a statement attributed to Azure Chief Technology Officer Mark Russinovich, Microsoft says:

"Most of Microsoft 365 services, including Teams, SharePoint Online and Office online, as well as Xbox Live services run primarily on Azure infrastructure today. Mailbox storage for Exchange Online and Outlook.com is also in the process of moving to standard Azure infrastructure."

---

Destiny 2 uses AWS
WoW friggin used AWS.

So much BS in your post.

Basically they wanted to save money. Bandwidth costs on the major cloud providers (especially AWS) can be very expensive.

Quote:

---

Originally Posted by VisperCon

I'm just mentally frazzled after over 7 hours of trying to login now and I can't for the life of me understand why they would turn down and/or not use AWS or Azure as a temporary solution.

Is it pride? Is it not a possibility?

I just see a scenario where a company wouldn't use all available resources to assist their customers and maintain the positive image their brand and game had going into EW's launch.

---

Your sub is 15 dollars a month because of that.
But if you and every other customer are okay with paying 50 dollars a month instead then I'm sure they can look into it.

Quote:

---

Originally Posted by PewPewPewPewPew

Your sub is 15 dollars a month because of that.
But if you and every other customer are okay with paying 50 dollars a month instead then I'm sure they can look into it.

---

Destiny 2 runs on AWS and costs $0/mo.
For Honor runs on AWS and costs $0/mo.
The Division 2 runs on AWS and costs $0/mo.

WoW used AWS to relieve player congestion.... Did you do any research before posting this?

Another game I play moved to Amazon's Cloud servers, and while it's not an MMO, it's absolutely HORRIBLE. Lag is atrocious, things don't want to render quickly at all. I shudder to even imagine how bad FFXIV would be. I've also played New World, performance of the game was ugh.

They said they have considered it and decided against it. Likely for good reasons.

Using AWS as a "temporary solution" is a suggestion born from ignorance - it is not a solution to the temporary problem we have now.

AWS doesn't magically just scale without effort. You need expertise to specifically design for it, otherwise you risk making things worse.

You can't simply copy-pasta your way into AWS. We're talking about complex infrastructure that was not built for cloud in the first place.
They would be looking at a migration process that takes months to years of planning, executing and testing.

Even if we entertain the fantasy that they could "just AWS 4head" in 2 days - migrating to a completely different environment for a temporary problem is a completely stupid business decision.

TLDR: AWS is not magic, stop pretending it is.

Its really not that simple either. It takes time and a likely amount of changes to code and functions. Even off loading something more logical like just the login service becomes a feat in itself and introduces several more points of failure. Even if you could find suitable hardware that allows your code to work without many changes having the extra tunnel or point to point connection can be a point of stress or failure itself.

I have seen a lot of devs turn to aws or azure and come out saying nope never again and going back to using a fully or self hosted non virtualized platform.

Quote:

---

Originally Posted by LariaKirin

They said they have considered it and decided against it. Likely for good reasons.

Using AWS as a "temporary solution" is a suggestion born from ignorance - it is not a solution to the temporary problem we have now.

AWS doesn't magically just scale without effort. You need expertise to specifically design for it, otherwise you risk making things worse.

You can't simply copy-pasta your way into AWS. We're talking about complex infrastructure that was not built for cloud in the first place.
They would be looking at a migration process that takes months to years of planning, executing and testing.

Even if we entertain the fantasy that they could "just AWS 4head" in 2 days - migrating to a completely different environment for a temporary problem is a completely stupid business decision.

TLDR: AWS is not magic, stop pretending it is.

---

Isn't failing to do anything for the last nine months to improve servers to prepare for this launch when they have the data to show their MoM growth and historical data to show the increase in returning players for an expansion even worse? It's not like the idea is "do it in 2 days" but they had MONTHS to prepare for this launch and have utterly failed in that regard.

They tried using AWS. It didn't work and they literally explained why.

If anyone thought of it then I assure you that they have already tried it.

Please at least Google search this before asking.

Quote:

---

Originally Posted by VisperCon

Isn't failing to do anything for the last nine months to improve servers to prepare for this launch when they have the data to show their MoM growth and historical data to show the increase in returning players for an expansion even worse? It's not like the idea is "do it in 2 days" but they had MONTHS to prepare for this launch and have utterly failed in that regard.

---

You work for them? Is that how you got that they aren't doing anything? Please.

Quote:

---

Originally Posted by Tanis_Ebonhart

They tried using AWS. It didn't work and they literally explained why.

If anyone thought of it then I assure you that they have already tried it.

Please at least Google search this before asking.

---

Source...that isn't from over 5 years ago? (Because obviously technology doesn't change in five years...) They use AWS for Dragon Quest X by the way so this isn't some new concept to SE.

Quote:

---

Originally Posted by VisperCon

Isn't failing to do anything for the last nine months to improve servers to prepare for this launch when they have the data to show their MoM growth and historical data to show the increase in returning players for an expansion even worse? It's not like the idea is "do it in 2 days" but they had MONTHS to prepare for this launch and have utterly failed in that regard.

---

1. They need a specific chip type to increase the amount of servers.
2. Said chip type is currently hard to find and we have a global shortage of, and have had, for over a year now.
3. All the WoW players came to FFXIV 6 months ago, increasing the active playerbase to a size NOBODY could have predicted.
4. Due to all of the above, they did everything within their power to take care of things while trying to avoid things getting worse.

Quote:

---

Originally Posted by RyuDragnier

1. They need a specific chip type to increase the amount of servers.
2. Said chip type is currently hard to find and we have a global shortage of, and have had, for over a year now.
3. All the WoW players came to FFXIV 6 months ago, increasing the active playerbase to a size NOBODY could have predicted.
4. Due to all of the above, they did everything within their power to take care of things while trying to avoid things getting worse.

---

These all sound like issues of a systemic failure that were years in the making compounded by either an unwillingness to invest or strategize. They had record profits in April of 2021, that was 8 months ago and the population continued to grow MoM. That's just basic trending and forecasting. Companies need to be able to pivot to deal with these types of things but SE's complacent approach to their servers have showed quite the failure. If they "did everything within their power to take care of things"...what exactly did they do?

Im also Mentally Frazzled how many people make new Threads of the Same Topic we already have.
And say the exact same things that were already said on another Thread.
Are people just Desperate for Upvotes?

Quote:

---

Originally Posted by VisperCon

These all sound like issues of a systemic failure that were years in the making compounded by either an unwillingness to invest or strategize. They had record profits in April of 2021, that was 8 months ago and the population continued to grow MoM. That's just basic trending and forecasting. Companies need to be able to pivot to deal with these types of things but SE's complacent approach to their servers have showed quite the failure. If they "did everything within their power to take care of things"...what exactly did they do?

---

8 months ago was in peak "there are no chips" season. They would have had to forecast this years ago, which means forecasting WoW falling to pieces and almost everyone and their mother jumping ship overnight. No one forecasts growth like that when its outside norms. Also COVID happened, which caused more people to have time at home, increasing counts of logged in players, and at that point, it was too late to fix things.

Quote:

---

Originally Posted by SnowVix

Ask New World players how stable AWS is to actually play on.

---

The servers were great when I played. They were about the only part of the New World experience that did work.

I am hoping that game can uh...fix its issues some day. It could be a very unique and interesting title if it had a coherent endgame and didn't try to explode all the time.

Based on the "experts" that start such threads they had to design for cloud support and infinite scalling up from beginning, ARR.
"Hey, we are trying to salvage this flop of an MMO by reworking it, lets design it so it can scale to be biggest MMO ever as well!" :)

Hey. :)

The problem here is, cloud instances are not really comparable to real servers. A typical cloud instance has a very low performance per CPU core. But you can rent thousands of them. If the FF14 server software is not prepared to run in those cloud instances then the results will be a real mess.

And yes, you can also rent real servers. But nobody knows whether the FF14 server software can run on standard PC server hardware or not. If they have something more special like Sun SPARC then it will be far more difficult to rent something like that.

Cheers

Quote:

---

Originally Posted by VisperCon

Isn't failing to do anything for the last nine months to improve servers to prepare for this launch when they have the data to show their MoM growth and historical data to show the increase in returning players for an expansion even worse? It's not like the idea is "do it in 2 days" but they had MONTHS to prepare for this launch and have utterly failed in that regard.

---

These posts are always the worst, what with being verifiably false. Hey look, they actually have made efforts to improve hardware and configuration not all that long ago.

Your problem is assuming "months" is a timescale they can work with. "Failing to do anything for the last nine months"? They've been planning to add an entire data center for over a year. Anything they would've or could've done would necessarily had to have been under consideration for after that went into operation, and OCE is already under delays. "They had historical data." Yeah, and the data say everything was going smoothly up until quite recently. They haven't had nearly enough time to adjust to the explosive growth they've experienced.

Quote:

---

Originally Posted by VisperCon

Isn't failing to do anything for the last nine months to improve servers to prepare for this launch when they have the data to show their MoM growth and historical data to show the increase in returning players for an expansion even worse? It's not like the idea is "do it in 2 days" but they had MONTHS to prepare for this launch and have utterly failed in that regard.

---

In the same vein, you had nine months to understand how bad it would be on week one. Also twenty years, give or take, to understand that every major MMO game launch in the history of time has been like this.

Quote:

---

Originally Posted by Floortank

In the same vein, you had nine months to understand how bad it would be on week one. Also twenty years, give or take, to understand that every major MMO game launch in the history of time has been like this.

---

This is not true. :) Rift: Planes of Telara had a very smooth start. And Trion was not very experienced because this was their first MMORPG. :)

Cheers

Quote:

---

Originally Posted by VisperCon

Isn't failing to do anything for the last nine months to improve servers to prepare for this launch when they have the data to show their MoM growth and historical data to show the increase in returning players for an expansion even worse? It's not like the idea is "do it in 2 days" but they had MONTHS to prepare for this launch and have utterly failed in that regard.

---

I don't get how people still don't understand, if CBUIII could acquire the hardware for new servers and be able to deploy them they would have done so by now. They were open to all options, were willing to spend however much money it could take to make sure launch was smooth, but (1) there was no supply due to the semiconductor shortage and (2) Square-Enix was not willing to pay top dollar for third-party servers to run one of their flagship games.

Like y'all are just thinking we're repeating this as if it were a meme that we use to insult players who are running into issues but it has been well documented that Square-Enix did not consider CBUIII for any more funding other than for content, and that they were not expecting such explosive growth these past few months. Increasing server capacity and login stability was likely in planning for years but they either never had the funds to do it or the hardware--and thanks to the pandemic they couldn't physically go to the datacenters to deploy any of the hardware they might have had.

AWS or Azure can be used as a last resort, if it takes to long for SE to get new servers. It maybe a little complicated to make it run stable, but with the right knowledge and coding it can

Quote:

---

Originally Posted by VisperCon

These all sound like issues of a systemic failure that were years in the making compounded by either an unwillingness to invest or strategize. They had record profits in April of 2021, that was 8 months ago and the population continued to grow MoM. That's just basic trending and forecasting. Companies need to be able to pivot to deal with these types of things but SE's complacent approach to their servers have showed quite the failure. If they "did everything within their power to take care of things"...what exactly did they do?

---

They ordered the OCE data center servers last spring (before they announced it during Fan Fest) and were warned they could not be delivered until 2022 which is why it is scheduled for patch 6.1.

Oh look.. another know it all who is going to tell the people running the server how they should be running their server. I assume you got some professional experience backing up your professional opinions?

Quote:

---

Originally Posted by VisperCon

Isn't failing to do anything for the last nine months to improve servers to prepare for this launch when they have the data to show their MoM growth and historical data to show the increase in returning players for an expansion even worse? It's not like the idea is "do it in 2 days" but they had MONTHS to prepare for this launch and have utterly failed in that regard.

---

You do realize that this chip shortage has been around since at least the summer of 2020 yeah? And that the big boom in the playerbase didn't happen for months afterward? Did you expect them to be psychic and predict this happening back before the shortage began?

Quote:

---

Originally Posted by VisperCon

They still got to play and didn't get kicked after six+ hours in queue. That's still an improvement over this shit show.

---

You clearly haven't played it.

Quote:

---

Originally Posted by VisperCon

Destiny 2 runs on AWS and costs $0/mo.
For Honor runs on AWS and costs $0/mo.
The Division 2 runs on AWS and costs $0/mo.

---

None of those are MMOs. They all use lobby-systems and instancing.

Quote:

---

Originally Posted by SnowVix

getting booted and having multiple connection issues mid-play session is far worse, especially since you'd still be queueing to get back in, even if not for as long.

---

don't try to argue with people that say that queues are worse than any game-breaking bug, lag, or connection issue while playing(I have PTSD from Neverwinter and archeage), they're clearly delusional.

Moving something from a datacenter to a cloud provider isn't something you "just" do. There are tons of variables involved, and shifting projects between these two types of platforms is a full time job for full teams of people. Compounding this, FFXIV wasn't designed to run on a public cloud provider. Their backend code isn't organized in a way that AWS or Azure expect things to work. Services don't communicate with cloud servers in mind, etc. It's like going to Chicago and expecting a map of New York to tell you where everything is.

I do this for a living; moving entire projects into the cloud takes months to years for *well* architected projects. I have no doubt they looked at what it'd take to move XIV's ancient spaghetti code onto AWS and concluded it would be easier to just make a new MMO designed on scaling cloud servers from scratch.

Quote:

---

Originally Posted by VisperCon

Destiny 2 runs on AWS and costs $0/mo.
For Honor runs on AWS and costs $0/mo.
The Division 2 runs on AWS and costs $0/mo.

WoW used AWS to relieve player congestion.... Did you do any research before posting this?

---

https://venturebeat.com/2018/03/19/a...ame-companies/

"to handle the ebbs and flows of demand on cloud-connected data centers"

It seems that the games do not run at 100% on AWS. Second, those games are not MMOs. And third, the server software must be developed in mind that it will propably run in cloud instances. If not then the server software will run very very VERY slow.

A cloud instance is more comparable to a Raspberry Pi than to a Xeon/Epyc-server. But you can rent thousands of those Raspberry Pi-like instances on demand and spawn them very quickly. And that is a problem. It is very unlikely that one cloud instance can handle more than a dozen players at once. And you do not want that only one dozen players are allowed per zone. Because it would not be an MMO anymore. :) And no, you cannot take 20 cloud instances and make one big instance of them because cloud instances are strictly separated from each other. You need to synchronize them permanently via network. And permanent synchronization creates a lot of network traffic.

Running on AWS may work for small group loot shooter but not for MMOs.

Cheers

Quote:

---

Originally Posted by VisperCon

They still got to play and didn't get kicked after six+ hours in queue. That's still an improvement over this shit show.

---

However its lag lag lag lol I guess you like failing and dying due to lag lol

Quote:

---

Originally Posted by SnowVix

Ask New World players how stable AWS is to actually play on.

---

Yeah it's not great.

Quote:

---

Originally Posted by Zanarkand-Ronso

Whats the point of your thread if you dont want people to click on them?

---

oh but he does want people to click on them. just not people that tell him he is to full of himself, only people that help him build the perfect circlejerk.

Quote:

---

Originally Posted by Larirawiel

https://venturebeat.com/2018/03/19/a...ame-companies/

"to handle the ebbs and flows of demand on cloud-connected data centers"

It seems that the games do not run at 100% on AWS. Second, those games are not MMOs. And third, the server software must be developed in mind that it will propably run in cloud instances. If not then the server software will run very very VERY slow.

A cloud instance is more comparable to a Raspberry Pi than to a Xeon/Epyc-server. But you can rent thousands of those Raspberry Pi-like instances on demand and spawn them very quickly. And that is a problem. It is very unlikely that one cloud instance can handle more than a dozen players at once. And you do not want that only one dozen players are allowed per zone. Because it would not be an MMO anymore. :) And no, you cannot take 20 cloud instances and make one big instance of them because cloud instances are strictly separated from each other. You need to synchronize them permanently via network. And permanent synchronization creates a lot of network traffic.

Running on AWS may work for small group loot shooter but not for MMOs.

Cheers

---

Apparently it does: https://aws.amazon.com/blogs/gametec...rpg-migration/

AWS is horrible, and migrating their services to AWS on such short notice is extremely unrealistic. The amount of resources they would have had to commit to offset a few days of player discomfort were probably assessed by the business division as not being worth it, and would have likely facilitated additional delays.

Source: I am AWS certified.