IT folks, realistically what can SE do to alleviate the current situation?

Quote:

---

Originally Posted by TristanX

Yeah, shortage issues. Getting latest hardware takes time. Let's hope SE could get it sooner.

---

Personally, I don't even see the "shortage issues" being the major factor here. I mean, obviously it is in getting OCE up. But that's kinda the point. OCE was already in the works. Even without shortages, new hardware would've necessarily been planned for after OCE, meaning after 6.0 launch. Even if shortages were never a problem, they never would've had new hardware ready to go immediately, and by the time they did, much of the chaos will have died down.

So, yeah, a classic "hands are tied" situation. They're doing the best with what they've got. Anyone that's kept up with their updates would know we're in a far better place than we were even a year or two ago, with the small internal improvements they've made. But, that only gets you so far when faced with a tsunami of new players.

Quote:

---

Originally Posted by Necrotica

Have players enter death arenas at the que screen for the right to play the rest of the game.

---

That would require them to set aside instance servers for that purpose, resulting in less server space to play the rest of the game.

Quote:

---

Originally Posted by Hasrat

Better question: Do people actually think there are steps that can be taken right now that they are not already taking? Folks really think devs are just sitting back laughing at all the frustration? Just ignoring the problem until it goes away?

Too many people asking "why aren't they doing [x]?" and not enough questioning "what makes you think they're doing nothing?"

---

Laughing? probably not, but their history proceeds them, for instance the housing dilemma, the game has had housing since 2.1 and they still haven't gotten it right. The simple fact is they got their money for the game and there are enough people that are willing to accept the status quo. Some even going so far as to suggest "working as intended".

Quote:

---

Originally Posted by Hasrat

Personally, I don't even see the "shortage issues" being the major factor here. I mean, obviously it is in getting OCE up. But that's kinda the point. OCE was already in the works. Even without shortages, new hardware would've necessarily been planned for after OCE, meaning after 6.0 launch. Even if shortages were never a problem, they never would've had new hardware ready to go immediately, and by the time they did, much of the chaos will have died down.

So, yeah, a classic "hands are tied" situation. They're doing the best with what they've got. Anyone that's kept up with their updates would know we're in a far better place than we were even a year or two ago, with the small internal improvements they've made. But, that only gets you so far when faced with a tsunami of new players.

---

Yup, massive increase of players. Whatever happening in their server/network team. They are finding solutions. Eventually these issues will be fixed.

Oh, i got 2002 issues on day 1. 3000 queue then 2002 multiple times. 7000 queue after that. I got no more 2002 issues. The wait time is hours though. On day 2, I just login early in the morning with 2 digit queue. My problem solved. Its good to work from home. On office weeks, I use remote desktop to queue 1 hour before peak hours.

Quote:

---

Originally Posted by Psythe

I work in big data. We have tens of thousands of servers. A lot of our hardware runs 10 to 50k a pop for a complete server setup. We are just barely getting hardware that should have been delivered 9 months the ago. We have diversified our vendors and are putting up hundreds of thousands of dollars to different vendors in hopes we can get hardware from somewhere faster without sacrificing stability and service we provide while doing it. From a hardware perspective there is probably little they can do or even could have done if they started trying 6 months ago. There is also little way for a company that is not constantly ordering hardware to even have any idea it is this bad.

They have done an amazing job. They have provided compensation, apologized, they sent out a letter in advance setting the expectation that there would be queues like no one has ever seen and the reason for it. They sent out letting players know when peak times are. I have gotten on every day as has most of my FC and friends as we knew if we wanted to get in we do it when others are not.

As consumers of a product there is a number of people not doing things to help and that's where I see a big failing right now. Play when the lines are short and help balance the load. If you can't get on during low volume wait till another time. Don't stress yourself out and be a better person.

---

I have been working in the industry as well for 25, and personally I see this as the "free and easy" way to address the issue of unhappy users.

It's really easy to say "Sorry" (or hard depending on the person) but it cost you nothing. I'm sorry, here's a fee coupon, is the basic of basics in any industry when you have issues and screw ups.

This is more of an issue of when the Accountants start the "Cost of buying new servers versus Return on existing equipment. That's a matter of how much depreciation is left on the existing life of a servers versus upgrading them for a launch.

They also don't like servers that cost them a lot to buy just sitting around having to be amortized because they were purchased for an initial rush, but now those users lose interest and those new servers are sitting idle, because they weren't purchased for growth, but for fan curiosity and they will be gone after a month or two.

And the point of playing during "Non-Peak" hours is great.

So I'll just tell my boss that I ether have to play during working hours, or I'll have to change my sleep schedule so that I can play after 2am. But that also means I may have to give up any family time I might have.

Yep, that is the better option the more I think of it. :rolleyes:

Sometimes I have a fantasy of enough people quitting that upper management has a heart attack over the accounting decisions they have been making to not upgrade any servers for this because they are waiting out the initial curiosity rush.

remove the queue limit?

Quote:

---

Originally Posted by Faelel

remove the queue limit?

---

I am not an IT person but I could see full server meltdown if the limit were removed. My non IT brain says the queue limit is there for a hardware reason. Could be wrong but that is how I imagine it.

almost seems like a perfect storm situation.
Shipping and silicon issues making getting parts to beef up servers or add new ones in any realistic time frame. For a company this size getting new servers isn't really a money constraint they would even consider. Spending money on the company tends to be a good thing for taxes anyways
Then like two months before endwalker we get this weird WoW implosion and giant exodus of players coming to FFXIV
Just from past experience it seems like the servers were improved to a point that could handle double but ended up getting like triple or quadruple

I would imagine the random 2002 isn't really as random as everyone thinks and it would definitely be nice if they could fix it. When it's not weekend/peak time and the queue is only around 2000 or so, it's not really happening at all.
I do feel bad for EU though, they're actually hitting the 12000 limit and getting 2002 for everyone at all times thing

If this were my software team, we would be:

* Investigating programmatic changes to the login queue to alleviate the 2002 issue. The one thing we can control is our own code, so that's the place to start.
* Contacting the IT vender we ordered the servers from (the OCE data center servers) and asking if we can jump the line for twice the price, paid to the other clients we'd be skipping. (This might not be legal.)


That's really all they can do.

They have terrible infrastructure. I don't know how they have things set up but they used the existing servers for so many excuses before this went down (Balmung being completely out of housing way before the pandemic but they can't add more instanced servers as an example), and I can't help but feel like they kept kicking the can down the road until it led to this.

I don't think you need to be an IT person to have some inclination. Just some common sense and familiarity with the modern world. There are even non computer technologies where you can reason by analogy that shutting everything down or stopping in response to a momentary loss of control is probably not the best solution. Right?

Realistically I doubt they can address the problem in short order. What seems to be going on is that they never designed to properly scale up to large player volumes. It would probably require some messy work that needed to be scheduled way in advance which is probably why they don't have a quick solution.

In either case the client doesn't need to shut down if there is packet loss. You know this too, you use the internet, you use apps. What happens when there are connection problems, does everything just shut down? Probably not.

How do other services handle large user volumes? Multiple entry points, letting the responsiveness lag, multiple retry connections on error, maybe they even shut down the connection and then reconnect at agreed upon intervals all automatically.

Either way it's unfathomable why a lobby needs a perfect and responsive connection to each client. Unless they just didn't design it with scaling to larger numbers in mind. I doubt they can easily redesign it, and I doubt new hardware can fix what is probably some kind of network bottleneck they need to software engineer around.

Realistically maybe they can make it so the game doesn't crash to desktop on 2002 and maybe make it so that there is a better memory for queue placement, or even that it tracks queue vs playtime and prioritizes people with higher queue proportions to the front of the queue. If that's too much they could add a special error sound for 2002, 3001 and so on errors, so people don't need to have the queue up the whole time.

In the long term they should invest in redesigning their network stuff and even to the point that we can see more people in an area at once and they don't pop in/out as much. Because it's possible the number of users just keeps on growing this decade.

Took some digging, but I was sure it was there, and here, back at the end of August, they had already mentioned their efforts to increase login caps and attempts to mitigate congestion.

All things considered, it gives me the impression that they may be at the limit of what they can improve without more physical hardware.

Quote:

---

Originally Posted by Faelel

remove the queue limit?

---

The limit.. and the 2002.. are there to ensure it doesn't go down entirely due to overload.

Quote:

---

Originally Posted by Hasrat

Took some digging, but I was sure it was there, and here, back at the end of August, they had already mentioned their efforts to increase login caps and attempts to mitigate congestion.

All things considered, it gives me the impression that they may be at the limit of what they can improve without more physical hardware.

---

I knew there would be issues, because of the hardware and current popularity. SE knew it. We knew it. Some people buried their heads in the sand. It was unavoidable.

There are major issues with hardware worldwide right now, and an MMO is the least of the first world problems over this shortage. SE is just as powerless as we are. We're on the same ship.

Quote:

---

Originally Posted by Catwho

If this were my software team, we would be:

* Investigating programmatic changes to the login queue to alleviate the 2002 issue. The one thing we can control is our own code, so that's the place to start.
* Contacting the IT vender we ordered the servers from (the OCE data center servers) and asking if we can jump the line for twice the price, paid to the other clients we'd be skipping. (This might not be legal.)

That's really all they can do.

---

Yea, I have no problem staying in a queue, it's getting kicked and hour in then starting over.

I guess SE is going to need to make some hard decision. No new server is coming and everyone in the queue is supposedly a paying customer, the only option they have is people quitting to free up room. The only way to do that is offer free refunds to any who want it. People who are vested most likely won't quit but people who just bought the game most likely will.

Another distasteful option is unannounced rolling world restarts to remove all the botting characters from the worlds; not just normal players but we know RMT is working it 24/7.

Hot take: play time caps. Hard limit of twelve hours a day or something. Stop letting people queue for duties at ten so you can ensure people aren't in content when it happens.

Quote:

---

Originally Posted by Feronar

I have another idea that likely won't be popular, but I think would be the most fair solution: Ration everyone's playtime. Temporarily implement a daily playtime limit of X hours, and enforce it at the service account level. The limit could be increased, then eventually removed as the surge dies down. This would help deter AFKers, as going AFK would still count against your limited playtime. They could also incentivise playing during off-peak hours (I.e. early morning) by giving you an extra hour or two of playtime that can only be used during off-peak hours.

To start, perhaps they could give you 3 normal hours (Can be used any time of the day), plus 2 off-peak hours (Can only be used during off-peak hours), for 5 hours total per day if you are willing and able to play during off-peak hours.

---

so, instead of just ticking off people in the queue, now add in anyone that can get into the game? not to mention the amount of coding needed to be done to even allow this?

I know, you think "its easy peasy, just have a timer start when you get into the game"

sure... why not. but then, what if you are in the middle of a dungeon? a cutscene? obviously have to code in things to not log you out in those cases. and then code in how much time after those events, and code the events to be taken into consideration.

you are playing, game is working great, you get to a new area... and your time expires.. but DONT WORRY, you can run through the queue later during off hours to continue your journey for 2 hours... or you get in at night... and after 2 hours off peak.. booted..

yeah, that dog dont hunt as they say. all that would do is annoy the rest of the player base.

Quote:

---

Originally Posted by Kes13a

so, instead of just ticking off people in the queue, now add in anyone that can get into the game? not to mention the amount of coding needed to be done to even allow this?

I know, you think "its easy peasy, just have a timer start when you get into the game"

sure... why not. but then, what if you are in the middle of a dungeon? a cutscene? obviously have to code in things to not log you out in those cases. and then code in how much time after those events, and code the events to be taken into consideration.

you are playing, game is working great, you get to a new area... and your time expires.. but DONT WORRY, you can run through the queue later during off hours to continue your journey for 2 hours... or you get in at night... and after 2 hours off peak.. booted..

yeah, that dog dont hunt as they say. all that would do is annoy the rest of the player base.

---

It's fun how quickly people turn to "Fuck you got mine" when the solution to a problem involves being considerate of others who don't have any of what you do. People will always be delightfully selfish and self centered creatures.

Quote:

---

Originally Posted by Dewslam

It's fun how quickly people turn to "Fuck you got mine" when the solution to a problem involves being considerate of others who don't have any of what you do. People will always be delightfully selfish and self centered creatures.

---

This isn't a problem for the players to solve or fix. The consumer shouldn't have to control how the service is delivered to other consumers.
Both players in this theoretical situation are paying for a service. Neither are responsible for the other one to get access to the service they paid for. This is up to Square to fix. Being selfish or not has nothing to do with it. Square is the party to be held responsible in this situation.

Bear with me on this idea, their play servers have a cap, and once over you get a queue. This leads me to wonder if they can apply the same to the log server. The log server by their own information is full at 17000 players in queue, this leads to 2002 errors and possible instability causing the packet loss errors.

Might be possible to add a second Q, click start, get a small Q to the logging server, once that lets you in you click for your server and its Q.

The second Q would be annoying but may make for a more stable experience on the play server Q, just a thought

Nothing.

The complainers will never be satisfied. They have this ridiculous need to act like they're slighted victims who demand sympathy and validation.

If I'm in line waiting for 30m, I don't want to suddenly get kicked out of line for reasons beyond my control and lose my spot. I'm OK with waiting as long as I know I'll eventually get in given time.

1. Allow up to 5 to 10m for clients to reconnect and get back in queue
2. In regards to the post they made, make the queue more tolerate of network issues, I can stay in game for hours at a time, but can't stay in queue for 30m without a 2002 error
3. Redesign the queue system so that its not reaching hardware limits possibly (throw the queue in a file, have clients ping the server at fixed intervals rather than maintain connection or whatever it is that actually happening, etc.)

Quote:

---

Originally Posted by Dewslam

It's fun how quickly people turn to "Fuck you got mine" when the solution to a problem involves being considerate of others who don't have any of what you do. People will always be delightfully selfish and self centered creatures.

---

thats not what I am saying at all. but that solution requires pre planning and announcement. implementing it after the fact would raise the ire of the people who do get in now. annoying the entire player base is not a workable solution.

in case you have been sleeping in a cave the last few years, being considerate of others is a dying trait in humanity. basic human respect and decency is in short supply.

if it was announced months in advance, there might be a fair bit of grumbling, but people would work with it, but still likely be irked. the person who posted this idea was saying it should be implemented now. and I am saying, its a non solution. at that point no one would be happy. period. little to do with "got mine" mentality. the profanity is uncalled for.

I would advise people to read the media letters that have been posted the couple of months. Especially the one released prior to early access. It says whay measures they have currently taken.

Quote:

---

Originally Posted by NanaWiloh

I would advise people to read the media letters that have been posted the couple of months. Especially the one released prior to early access. It says whay measures they have currently taken.

---

why read when indignation works so well and "blow off steam"

in this day and age facts and reading something months old is just being a "librarian"

outrage over something everyone saw coming is apparently so much more instantly gratifying

I've said this in another thread but I'll say it again. The chip/covid narrative is a meme because I started October of 2018 and my server was already bear cap. I had queue times everyday logging in to SB and SHB. The servers should have been addressed long before EW announcement because of the incline in players since ARR. Trends and patterns are something humans are supposed to be noticing even before the vid and the wow exodus.

Quote:

---

Originally Posted by Kes13a

why read when indignation works so well and "blow off steam"

in this day and age facts and reading something months old is just being a "librarian"

outrage over something everyone saw coming is apparently so much more instantly gratifying

---

Lol Its just easier to say reading is hard.

Quote:

---

Originally Posted by Ibage

This isn't a problem for the players to solve or fix. The consumer shouldn't have to control how the service is delivered to other consumers.
Both players in this theoretical situation are paying for a service. Neither are responsible for the other one to get access to the service they paid for. This is up to Square to fix. Being selfish or not has nothing to do with it. Square is the party to be held responsible in this situation.

---

And yet nothing in the world is infinite. Even in a perfect world, what do you expect square to do to ensure every person ever is happy? Own an entire country's worth of servers to ensure none ever go down, ever have issues and always have enough space for every member of an ever expanding userbase? The basic problem, as seen in a massive magnitude right now, is that there's always going to be limited space. Both on worlds and on the log in queue. Too many people logging in at once will fuck it up. If too many people are logged on the world will be full and you can't get in even if you're next in line. Most people getting outraged over this only care in so far as it matters that they get in and they get to play; which makes it even more galling when those same people start talking about ways to avoid logging out or getting caught by the anti-afk measures.

So yes, it has everything to do with being selfish. Should square have fixed this shit years ago? Certainly. But they didn't. Yelling and screaming about it isn't going to magic up more servers. Instead people should think more about their actions and how it impacts other people beyond "Well I have what I want I don't care what happens to anyone else" or "I don't have what I want, everyone else should suffer until I do". Just look at how many threads there have been about people *demanding* other players get kicked from the game because they, personally, can't log in and want to. It's entirely self centered bullshit, my dude. We all pay for the service. Or do you think priority should be given to players who've paid the most on mogstation?

Quote:

---

Originally Posted by RinaB

I've said this in another thread but I'll say it again. The chip/covid narrative is a meme because I started October of 2018 and my server was already bear cap. I had queue times everyday logging in to SB and SHB. The servers should have been addressed long before EW announcement because of the incline in players since ARR. Trends and patterns are something humans are supposed to be noticing even before the vid and the wow exodus.

---

ok, since this is all about personal opinions now, I'll say that in my experience, you're wrong. and I have more experience than you because I say so.

Quote:

---

Originally Posted by Hoosier

This isn't like any other expansion for FFXIV. They had a massive increase in player base come over from other MMOs, more specifically WOW. Those people didn't come over for the End Walker expansion entirely. They switched because WOW lost its appeal. Chances are they will not be returning to WOW until Blizzard releases a new expansion which isn't expected anytime soon so the larger than typical player base is going to be here for some time. Maybe not quite as large as it is the past few days but still enough that they need to make changes to their infrastructure to grow.

---

this might have been mentioned but I just ran across this and I would like to point out the following: Those that came from other MMO's as you said will be here for a while till those mmo's appeal to them again (or expansion). thing is they were here and though our queues were high it wasn't extreme like it is fore this, no real crashing or issues till the early access. Now some will have gotten the game and expansion but only started on the 3rd sure but that's the minority. the problem is the people that had done everything and were tired of everything that had stopped playing a while back came back because of the new content! just like you said those from the other mmo's will go away when they offer new content. You are correct though, they NEED to continue getting the oceanic Data Center going, upgrading the EU Data Centers and expanding it, and even a little bit more work on the NA Worlds or new Worlds for the NA. yes they do have to keep on that but to be honest the extreme numbers will dampen down without the new people that moved over long before the drop of the early access. This is the thing people are saying is they shouldn't have sold more licenses than they could support, but the real problem is not brand new players really it is the older players that are reactivating for a time to enjoy the new content.

And just think, tomorrow is the official launch day... More people =D

Quote:

---

Originally Posted by Jeslynn

And just think, tomorrow is the official launch day... More people =D

---

My plan for tomorrow is the same as today. Get in a little time early morning and call it done for the day. Not worth fighting the queue war at this point for me.

Quote:

---

Originally Posted by Myrany

My plan for tomorrow is the same as today. Get in a little time early morning and call it done for the day. Not worth fighting the queue war at this point for me.

---

Wish I could to. A slight downside to a new M-F 7-5 job lol.

Quote:

---

Originally Posted by Jeslynn

Wish I could to. A slight downside to a new M-F 7-5 job lol.

---

You and me both. I will be getting up early to play. LOL

Quote:

---

Originally Posted by Razed

They could have upgraded their infrastructure in advance, or subbed it out to AWS or Azure. If they used AWS or Azure they literally could have deployed more servers / resources with a minimal amount of mouse clicks. The incompetence of the current situation is staggering when it could have and should have been preemptively mitigated when they knew full and well they had hordes of new players.

---

Sure, but look at the launch of New World, which went through pretty much the same issue, despite the game being done by the same company that is making AWS. If even a juggernaut like Amazon -which owns the cloud service itself- cannot do this, why would we expect SE to be able to?

if i was me? rent a crappy server somewhere close to the original one, put that server in the same cluster that primal or aether and then open 4 or 5 ghost servers with character creation disabled so people can use them as a proxy with the world visit feature to log and play while you upgrade the current datacenter

Regarding capacity it is what it is, and from the few times I got in things were working well. If you try to push beyond the comfort level you'll get other issues.

But for login/disconnect issue maybe something could change. Have a flag for if the player voluntarily logged out. If raised then so long, good bye, and if they try to get back in its like new. However if they disconnect involuntarily then hold position for a grace period of a reasonable time, if the time expires then it's considered a full logout. The downside to this though is it would increase que times.

To be fair I don't think they were expecting this level of demand. It's not just that the "other MMO" is losing players, this seems to be "the game" of 2021.

The problem is, we "It folks" can not give reliable tipps because we know literaly nothing about their infrastructure. We don't know which CPU architecture the servers need, which programming languages are used, which databases are used, where the bottlneck are etc.

SQEX is not very transparent here.

Cheers

Quote:

---

Originally Posted by Larirawiel

The problem is, we "It folks" can not give reliable tipps because we know literaly nothing about their infrastructure. We don't know which CPU architecture the servers need, which programming languages are used, which databases are used, where the bottlneck are etc.

SQEX is not very transparent here.

Cheers

---

This is a core problem for anyone who thinks they can "solve" the problem. We don't know anything but the tiny info nuggets they've given us about the 2002 error being a supposed overflow valve to prevent larger crashes. We don't know why they can't just make the system deny logins until it can actually handle them full term or why certain people seem to get right back into place even if logging back in 3-5 minutes after the error while others claim it's less than 30 seconds (though I am of the mind they're likely exaggerating).


What we can surmise though is that SE understands this is a bad customer experience and they're likely having emergency meetings to try and solve it themselves since it DOES mean lost profits when people don't continue their subs. So any actually implementable "quick" fix has likely been considered and either is in the works or been deemed unfeasible. The actual programmers and engineers working for them are NOT slouches. They've wrung some impressive performance and features out of what was a huge outsourced kludge of a game.

The idea that the community can solve the problem for them is pretty laughable..... BUT if anyone can prove me wrong with a response that isn't "well they should just *cloud server/ do what other game does/have fixed this years ago*" and can speak to the actual process and lines of code or interaction that needs to be fixed.. please have at and we can all work to get that to SE's attention with a thread and upvotes. There are many armchair network engineers here apparently but none of them have gotten out of their danged chairs or proven any credentials or knowledge of the actual process.

Quote:

---

Originally Posted by Jeslynn

And just think, tomorrow is the official launch day... More people =D

---

Good thing I have to work, so I can do something Productive with my time instead of wasting it trying to log in.
I do dread how bad its gonna be on Official Release @_@