Data center travel suspended

[Stormpeaks]

Did you or anyone actually really expect this to end up any other way...?

I did expect a little bit more preparation than this yeah, maybe dont release features that are going to be massively used and that will break in the first hour. Internal testing maybe ?

[GeminiReed]

I did expect a little bit more preparation than this yeah, maybe dont release features that are going to be massively used and that will break in the first hour. Internal testing maybe ?

Perhaps they should have some better rate limiters but the volume problem is going to correct itself once the novelty wears off. They release these things in an off-patch both as a volume test and to let that novelty period soak a bit. The remaining unknown will be if the DCTS servers can handle the steady-state traffic.

Depending on how the feature was designed they might even be able to temporarily add hardware to deal with the surge.

[Kaurhz]

I did expect a little bit more preparation than this yeah, maybe dont release features that are going to be massively used and that will break in the first hour. Internal testing maybe ?

Regardless of how much testing they would do, the first thing people are always going to do is dogpile a new feature, and this will always cause congestion to happen, especially when practically everyone and their grandmother dogpile a new feature. Managing that congestion is not very easy to do, especially when the number of requests potentially far exceeds the capacity that the servers can handle. This is basically akin to the community doing a DDoS attack.

[TaleraRistain]

Regardless of how much testing they would do, the first thing people are always going to do is dogpile a new feature, and this will always cause congestion to happen, especially when practically everyone and their grandmother dogpile a new feature. Managing that congestion is not very easy to do, especially when the number of requests potentially far exceeds the capacity that the servers can handle. This is basically akin to the community doing a DDoS attack.

It's also entirely possible to do heavy stress testing in a development environment but then have something unexpected manifest when it's released to the live environment. You try to plan for us much as you can and have a plan for rolling back if necessary if something unexpected happens.

[Kaurhz]

It's also entirely possible to do heavy stress testing in a development environment but then have something unexpected manifest when it's released to the live environment. You try to plan for us much as you can and have a plan for rolling back if necessary if something unexpected happens.

Indeed, the thing that people seem to forget is that simulations are often done in accordance with abstract models and these often don't align very well with the outcomes you find when doing it in a real, live environment, so whilst you may be able to simulate the load, you won't necessarily catch the issues that you might otherwise find when doing it in a live environment.

I sympathize with people being unable to log in or being caught on a DC as a visitor when they may otherwise be set for raiding or have plans with friends. But honestly let's face it, there was a fairly good chance this feature was going to crash in some way or another.

To those that were complaining about the Monday release schedule, this is why.

[TaleraRistain]

Indeed, the thing that people seem to forget is that simulations are often done in accordance with abstract models and these often don't align very well with the outcomes you find when doing it in a real, live environment, so whilst you may be able to simulate the load, you won't necessarily catch the issues that you might otherwise find when doing it in a live environment.

Not to mention the system you're working on may be fine, but it may or may not be connected to the great hive of systems that could affect things. Like what happened with the lottery server and the housing server. Sometimes there isn't time or resources to completely mock up exactly the same environment. I know I certainly have times I say "I have tested X, Y, Z, A, B" and then acknowledge "C" could still happen and develop a plan for that.

I definitely sympathize with anyone unable to log in and play or are stuck. But I also sympathize with the developer side because I've been in those shoes having to roll back something that didn't do quite what was expected. That they were able to suspend it so fast says to me that they did have a plan and were watching for excessive activity outside their predictions.