IT folks, realistically what can SE do to alleviate the current situation?

If this were you, in your work place what would you advise they do?

What are some constructive criticisms you have of the current situation?

Have players enter death arenas at the que screen for the right to play the rest of the game.

Save peoples spots in queue for longer then 2 seconds

In short: they should have bit the bullet and at the very least upgraded the login server. The queues, while bad, are expected from any expansion. But what is really disheartening is sitting in a queue for a few hours, getting 2002'd, and by the time you get back in you're at the back of the queue. That is the unacceptable part of all this, and the company needed to spend the money so paying customers wouldn't be in this mess.

Now it’s too late they should have limited their subscriber count to go with their servers capacity

Quote:

---

Originally Posted by Mialee

Now it’s too late they should have limited their subscriber count to go with their servers capacity

---

No, they have the capital to upgrade the servers and change how the login system works. They have more than enough resources to keep up with business expansion. Limiting sub count is counter productive to business. Expanding operations isn't.

Better question: Do people actually think there are steps that can be taken right now that they are not already taking? Folks really think devs are just sitting back laughing at all the frustration? Just ignoring the problem until it goes away?

Too many people asking "why aren't they doing [x]?" and not enough questioning "what makes you think they're doing nothing?"

Nothing, absolutely nothing. Delaying the release to wait for new hardware would have $@&&! Even more people off. Even with a bunch of new hardware once everything is “back to normal” there will just be lots of excess idle hardware because this type of situation only happens at a games release. They cant easily reprogam the game to work on other infrastructure that could just be rented. Would be more unstable if they did. We have the best solution. The game is out. Eventually people will finish and stop playing as heavily again and more people will get in. If there were queue problems this severe a month ago then we could determine other options, but even then they alreadg have plans to expand and release new data centers but they are stuck because of world evens and cost.

As my group at work calls it, this is a “wait longer” fix.

Quote:

---

Originally Posted by Hasrat

Better question: Do people actually think there are steps that can be taken right now that they are not already taking? Folks really think devs are just sitting back laughing at all the frustration? Just ignoring the problem until it goes away?

Too many people asking "why aren't they doing [x]?" and not enough questioning "what makes you think they're doing nothing?"

---

No, clearly they are sitting back munching on popcorn as they watch us plebs suffer. Why aren't they making additional servers appear from thin air? I want it now!

Quote:

---

Better question: Do people actually think there are steps that can be taken right now that they are not already taking? Folks really think devs are just sitting back laughing at all the frustration? Just ignoring the problem until it goes away?

---

Even if they're taking all the steps necessary it's still interesting to hear from those within the industry to help give the rest of us a better idea of what SE can do (or is doing) about it, if anything.

my below suggestion is taking into consideration that they are looking/working on implementing solutions...

at least make the queue have a grace period for the lobby errors. example.. instead of loging in and you're 4000 in queue and by the time you hit 3000 you get kicked out and then log back in to a 5000 queue

and/or

priorities PAYING SUBS.. i hate to say it but SQUENIX relies on us to pay their bills for the non paying members to enjoy a lot of content for free... both of these would go a long way... but ultimately the below will need to happen SOONER THAN LATER:


it should be like this... 4000 queue down to 3000 then get disconnected or lobby error. 5min grace to log back in and when you get back in your "grace period spot wasn't lost it , it still continued to go down the queue and you pick up where you left off... it would make this non sense more digestible …. and we all know this problem wont go anytime soon. it'll get worse tomorrow even and for atleast the next couple of months... they have to upgrade the infrastructure

Believe I saw or read that paying subs have priority in queue for logging into the game over free to play people.

Aside from that, I can't see a company like SquareEnix sitting back and watching us all with our heads exploding without some kind of plan in mind. Just doesn't make good business sense. As for everyone else that IS running around with their hair on fire--take it easy, I'm sure they will figure out a way to handle it.

Quote:

---

Originally Posted by ascensyon1

my below suggestion is taking into consideration that they are looking/working on implementing solutions...

at least make the queue have a grace period for the lobby errors. example.. instead of loging in and you're 4000 in queue and by the time you hit 3000 you get kicked out and then log back in to a 5000 queue

and/or

priorities PAYING SUBS.. i hate to say it but SQUENIX relies on us to pay their bills for the non paying members to enjoy a lot of content for free... both of these would go a long way... but ultimately the below will need to happen SOONER THAN LATER:


it should be like this... 4000 queue down to 3000 then get disconnected or lobby error. 5min grace to log back in and when you get back in your "grace period spot wasn't lost it , it still continued to go down the queue and you pick up where you left off... it would make this non sense more digestible …. and we all know this problem wont go anytime soon. it'll get worse tomorrow even and for atleast the next couple of months... they have to upgrade the infrastructure

---

I agree with this I don't mind waiting but it is frustrating getting pushed to the back of the line over and over again.

Quote:

---

Originally Posted by ascensyon1

priorities PAYING SUBS.. i hate to say it but SQUENIX relies on us to pay their bills for the non paying members to enjoy a lot of content for free... both of these would go a long way... but ultimately the below will need to happen SOONER THAN LATER:

---

I agree, free trial accounts need to be suspend until the issues are fixed. Telling your paying customers to get to the back of the line while other eat for free is maddening.

Let players who are willing to accept higher latency temporarily transfer to other regions for free. For example, some NA and EU players could temporarily play on JP realms where they can play during Japan's off-peak hours. Likewise, some Japanese players could temporarily play on NA or EU realms during our off-peak hours.

Quote:

---

Originally Posted by Blindheart

Nothing, absolutely nothing. Delaying the release to wait for new hardware would have $@&&! Even more people off. Even with a bunch of new hardware once everything is “back to normal” there will just be lots of excess idle hardware because this type of situation only happens at a games release. They cant easily reprogam the game to work on other infrastructure that could just be rented. Would be more unstable if they did. We have the best solution. The game is out. Eventually people will finish and stop playing as heavily again and more people will get in. If there were queue problems this severe a month ago then we could determine other options, but even then they alreadg have plans to expand and release new data centers but they are stuck because of world evens and cost.

As my group at work calls it, this is a “wait longer” fix.

---

This isn't like any other expansion for FFXIV. They had a massive increase in player base come over from other MMOs, more specifically WOW. Those people didn't come over for the End Walker expansion entirely. They switched because WOW lost its appeal. Chances are they will not be returning to WOW until Blizzard releases a new expansion which isn't expected anytime soon so the larger than typical player base is going to be here for some time. Maybe not quite as large as it is the past few days but still enough that they need to make changes to their infrastructure to grow.

I work in big data. We have tens of thousands of servers. A lot of our hardware runs 10 to 50k a pop for a complete server setup. We are just barely getting hardware that should have been delivered 9 months the ago. We have diversified our vendors and are putting up hundreds of thousands of dollars to different vendors in hopes we can get hardware from somewhere faster without sacrificing stability and service we provide while doing it. From a hardware perspective there is probably little they can do or even could have done if they started trying 6 months ago. There is also little way for a company that is not constantly ordering hardware to even have any idea it is this bad.

They have done an amazing job. They have provided compensation, apologized, they sent out a letter in advance setting the expectation that there would be queues like no one has ever seen and the reason for it. They sent out letting players know when peak times are. I have gotten on every day as has most of my FC and friends as we knew if we wanted to get in we do it when others are not.

As consumers of a product there is a number of people not doing things to help and that's where I see a big failing right now. Play when the lines are short and help balance the load. If you can't get on during low volume wait till another time. Don't stress yourself out and be a better person.

They could have upgraded their infrastructure in advance, or subbed it out to AWS or Azure. If they used AWS or Azure they literally could have deployed more servers / resources with a minimal amount of mouse clicks. The incompetence of the current situation is staggering when it could have and should have been preemptively mitigated when they knew full and well they had hordes of new players.

Fix the queue. I would be happy if I could actually get in line and actually log in instead of getting an 2002 halfway everytime. It is not that there is a line. For me is that I just can't log in

Quote:

---

Originally Posted by ascensyon1

my below suggestion is taking into consideration that they are looking/working on implementing solutions...

at least make the queue have a grace period for the lobby errors. example.. instead of loging in and you're 4000 in queue and by the time you hit 3000 you get kicked out and then log back in to a 5000 queue

and/or

priorities PAYING SUBS.. i hate to say it but SQUENIX relies on us to pay their bills for the non paying members to enjoy a lot of content for free... both of these would go a long way... but ultimately the below will need to happen SOONER THAN LATER:


it should be like this... 4000 queue down to 3000 then get disconnected or lobby error. 5min grace to log back in and when you get back in your "grace period spot wasn't lost it , it still continued to go down the queue and you pick up where you left off... it would make this non sense more digestible …. and we all know this problem wont go anytime soon. it'll get worse tomorrow even and for atleast the next couple of months... they have to upgrade the infrastructure

---

im sure they are actively monitoring the forums, their network analyticals and projecting when their network will be expanding and trying to find temp solutions. hopefully they will have more impact than the ones they've put in so far. i've only noticed the q getting worse and i see it get much worse starting tomorrow and for a very long time.

Quote:

---

Originally Posted by Razed

They could have upgraded their infrastructure in advance, or subbed it out to AWS or Azure. If they used AWS or Azure they literally could have deployed more servers / resources with a minimal amount of mouse clicks. The incompetence of the current situation is staggering when it could have and should have been preemptively mitigated when they knew full and well they had hordes of new players.

---

Someone will come along to tell you how its IMPOSSIBLE to utilize temporary cloud servers to lessen the load on their garbage infrastructure. Just wait 5 minutes.

Dropping players queue spot is definitely the issue.
From developer perspective, they have the feature to save players spot when the package drops and get kicked out with 2002 code.
But this feature also has time out.... once you're not able to log back in time. you have to re-queue.
Which make you have to set in front of the computer and make sure you have to log back on as soon as you get kick out with 2002 at random time. And queue time is HOURS.

There are definitely ways to scale up or out. but by the time devs finish their work/pass QA and ready to push out the changes..... everything should back to normal.... lol what's the point of doing it?

Quote:

---

Originally Posted by Feronar

Let players who are willing to accept higher latency temporarily transfer to other regions for free. For example, some NA and EU players could temporarily play on JP realms where they can play during Japan's off-peak hours. Likewise, some Japanese players could temporarily play on NA or EU realms during our off-peak hours.

---

That is only a marginal solution. I am a NA player and I already have a character on a JP server. I can't get far enough into the system to choose a datacenter to play on. I am 2002 long before that point over and over.

Quote:

---

Originally Posted by Psythe

They have done an amazing job. They have provided compensation, apologized, they sent out a letter in advance setting the expectation that there would be queues like no one has ever seen and the reason for it. They sent out letting players know when peak times are. I have gotten on every day as has most of my FC and friends as we knew if we wanted to get in we do it when others are not.

As consumers of a product there is a number of people not doing things to help and that's where I see a big failing right now. Play when the lines are short and help balance the load. If you can't get on during low volume wait till another time. Don't stress yourself out and be a better person.

---

I'm trying to cancel my Endwalker Digi Order until the issues are resolved but can't because I used the pre-order unlock code many many months ago. So no, they aren't doing everything.

Lots of people seem to be conflating "server capacity" issues with the login queue kicking y ou out. These are 2 completely different things.

Server queues are expected on launch. The queue randomly kicking you out is not.

I have a decade of experience in the industry and worked on very high traffic systems much higher than ffxiv so let me give my perspective:
- Their packet loss explanation makes zero sense. Unless they mean their server loses the packets in which case it's not really a packet loss problem.
- Possible fix #1: add retry logic to the client. If error 2002 or such, try a few times. Low cost and should solve a good chunk of the issues but doesn't get to the root cause. Would require a patch.
- Possible fix #2: tune and configure their login DB so it doesn't drop a random connection. I don't know enough about their infra, but dropping random connections is a common problem and can usually be mitigated by proper tuning. Might take a few days from a few weeks, not trivial, not sure if they have the expertise in house since it's peripheral to the game.
- Possible fix #3: Proper distributed queuing system implementation. Well-known problem with standard solutions. However building that and integrating in their infra is not trivial - we're talking weeks at best or months. Would likely solve all the issues.

In short I'd recommend fix #1 as a band-aid while they work on longer term solutions.

Quote:

---

Originally Posted by Razed

They could have upgraded their infrastructure in advance

---

Look, mate, I'm not even in the field, and even I understand that doing anything "in advance" for a company in this situation requires far more "advance" than a few months. Making proposals, getting approval, sorting out budget allocation, planning, acquisition, installation, and who knows what else.

And you know what? They did do all of that. It's why they've got an entire data center waiting for myriad delays to clear to get up and running. Anything else they could've done would've necessarily had to have happened after OCE was in operation, and, well, since that's stuck waiting, everything else they had planned is also being set back as well.

They can explain why we get kicked from the queue and why they aren't able to code a proper queue system that remembers your place in line. Oh and why does the damn game have to close when you get kicked out of the queue? Why does it not just go back to the title screen like any other proper online game does and all previous expansions did as far as I recall? Why do I have to go through all the hassle of putting in my authenticator every time I get 2002'd?

Fix the queue. I would be happy if I could actually get in line and actually log in instead of getting an 2002 halfway everytime. It is not that there is a line. For me is that I just can't log in

Quote:

---

Originally Posted by Razed

They could have upgraded their infrastructure in advance, or subbed it out to AWS or Azure. If they used AWS or Azure they literally could have deployed more servers / resources with a minimal amount of mouse clicks. The incompetence of the current situation is staggering when it could have and should have been preemptively mitigated when they knew full and well they had hordes of new players.

---

If only it was so simple. Remember character data is on their databases. So they would need to implement mirroring solution between public cloud and their datacenters which is a nightmare due to latency, bandwidth, security, VPNs etc.

Compare number of active players from year to year, add servers to help with the load accordingly. Solution to shortage is buy older hardware. May need to add network bandwidth as well.

They could implement ways to identify afk'rs using scripts not to be kicked - from other players being able to tag them to automated internal processes to help identify them. GM's could check tagged players and if they are found scripting to avoid the afk boot then suspend the account for 6 months - no appeal. The vast majority of the players doing this would not risk it and stop. Right now they are easily found and doing something obvious like jumping 3 times every 10 seconds - and this doesn't account for the people inside a house doing it out of sight. Right now by what I read we are only allowed 750 active players per world - and I'm sure a chunk of that is taken up by people afk scripting so they don't get booted because there is no downside if they are discovered other than being kicked by a GM.

Quote:

---

Originally Posted by Razed

They could have upgraded their infrastructure in advance, or subbed it out to AWS or Azure. If they used AWS or Azure they literally could have deployed more servers / resources with a minimal amount of mouse clicks. The incompetence of the current situation is staggering when it could have and should have been preemptively mitigated when they knew full and well they had hordes of new players.

---

Genuinely curious, no sarcasm, but which part of the SE team do you think is guilty of doing this? Business side, Yoshi-P, dev. team, or some other party I just don't know enough to mention?

Now, I know this game is paying SE's bills. It is vital to keep it running properly and keeping the user-base happy and paying. That alone makes me think this is just a combination of getting too popular at a time when the way things work violently shifted. No one (and everyone interested) is to blame... but it certainly stings when seeing advertisements for the game enticing even more people to sign up for the madness. There aren't enough resources for the current base they had, but they're trying to add more into the pool despite the water turning yellow.

Quote:

---

Originally Posted by Somnolence

If only it was so simple. Remember character data is on their databases. So they would need to implement mirroring solution between public cloud and their datacenters which is a nightmare due to latency, bandwidth, security, VPNs etc.

---

It should already be in the cloud, the entire industry (and government!) has been moving to the cloud as soon as it started to become viable in the early 2000s. Source: Me, I've worked for IBM, HP, big banks and government and have spent years of my life in datacenters (which I'm not bragging about, perpetual 67F temp gives you innate resistance to cold tho.)

Quote:

---

Originally Posted by TristanX

Compare number of active players from year to year, add servers to help with the load accordingly.

---

If only the game hadn't experienced an explosive population growth in the past year, faster than they would've been able to react to under ideal circumstances... Also, y'know, they were already in the process of adding servers, so.... kinda already doing all of this, except the time scales involved are a bit longer than we might expect/like.

Quote:

---

Originally Posted by Hasrat

If only the game hadn't experienced an explosive population growth in the past year, faster than they would've been able to react to under ideal circumstances... Also, y'know, they were already in the process of adding servers, so.... kinda already doing all of this, except the time scales involved are a bit longer than we might expect/like.

---

Yeah, shortage issues. Getting latest hardware takes time. Let's hope SE could get it sooner.

My workaround is queue earlier before the peak hours. Remote desktop just to queue is another way.

Quote:

---

Originally Posted by stu4355226

Dropping players queue spot is definitely the issue.
From developer perspective, they have the feature to save players spot when the package drops and get kicked out with 2002 code.
But this feature also has time out.... once you're not able to log back in time. you have to re-queue.
Which make you have to set in front of the computer and make sure you have to log back on as soon as you get kick out with 2002 at random time. And queue time is HOURS.

There are definitely ways to scale up or out. but by the time devs finish their work/pass QA and ready to push out the changes..... everything should back to normal.... lol what's the point of doing it?

---

In normal circumstances I'd agree, but this time things might not go back to normal because they have no option to increase capacity.

Improving the queue system is the last resort available to Square. The only other way right now for queues to improve is for huge swaths of players to give up on the game entirely, and that is not what Square wants.

Quote:

---

Originally Posted by Matrinka

Genuinely curious, no sarcasm, but which part of the SE team do you think is guilty of doing this? Business side, Yoshi-P, dev. team, or some other party I just don't know enough to mention?

---

I don't think it has anything to do at all with Yoshi-P or the dev team, they aren't the ones dealing with the engineering side of the business. To be honest I'm probably being way too harsh with saying it's incompetence, but they are behind the times. It works, until it doesn't in their case. It didn't have to get it where it doesn't work. They definitely should be moving everything to the cloud even if its their own personal datacenter fulfilling their cloud needs where they can spin up virtual assets but personally they probably aught to be farming everything network and server side to a company that specifically deals with offering cloud solutions like AWS (Amazon) or Azure (Microsoft), or somewhere else.

I have another idea that likely won't be popular, but I think would be the most fair solution: Ration everyone's playtime. Temporarily implement a daily playtime limit of X hours, and enforce it at the service account level. The limit could be increased, then eventually removed as the surge dies down. This would help deter AFKers, as going AFK would still count against your limited playtime. They could also incentivise playing during off-peak hours (I.e. early morning) by giving you an extra hour or two of playtime that can only be used during off-peak hours.

To start, perhaps they could give you 3 normal hours (Can be used any time of the day), plus 2 off-peak hours (Can only be used during off-peak hours), for 5 hours total per day if you are willing and able to play during off-peak hours.

Quote:

---

Originally Posted by Necrotica

Have players enter death arenas at the que screen for the right to play the rest of the game.

---

https://media.giphy.com/media/XWwIzh5GIWWf6/giphy.gif

Quote:

---

Originally Posted by Razed

They could have upgraded their infrastructure in advance, or subbed it out to AWS or Azure. If they used AWS or Azure they literally could have deployed more servers / resources with a minimal amount of mouse clicks. The incompetence of the current situation is staggering when it could have and should have been preemptively mitigated when they knew full and well they had hordes of new players.

---

I am constantly amazed that anyone still believes this game runs on Intel or AMD chip-based virtual systems, especially given the fact that they can't seem to get the systems they want. White-box Linux systems are readily available and have been so almost the year round. [Yeah, I work for a VAR who sells Sun/Oracle, Intel/AMD/SPARC/cloud, among other things].

I expect that a Japanese company laying out the architecture for an MMO might utilize RISC-based systems and select a Japanese computer company as their main vendor (Fujitsu and NEC come to mind).

We don't know, and it's pretty futile to even make a guess.

One thing I can guarantee right now. SE does not utilize any of the major Cloud companies for the game's infrastructure. What part of "Japanese/NA/EU Datacenter" don't people understand?