Rome fell to neglected tribes from the north. It certainly happens.

I wouldn't be surprised if the Giants had a hand in it, presuming they were poorly treated slaves of the Allag.