There was civil war raging prior to the Calamity that destroyed the Allagan Empire.

Otherwise, it was a lot of factors:

The earthquake sunk the Crystal Tower into the earth and CT at the time was a source of unlimited energy for the Allagans, without it most of their technology would not work. Coupled with the initial mass death of the earth quake, they were left with no (competent) leader since Xande was sealed in the CT.

I'd also imagine that the remaining pure Allagan were hunted and killed. Azys Lla most likely was evacuated since it was sent way up into the sky and the key was left with the Vanu Vanu to guard. Not sure how this could have played out, whether it was the WoL/ rebels that did it to prevent the Empire from using it or it was the Empire itself to do it to safeguard against the rebels.