The Gigas are descendants of the upper class Allagans who made themselves giants. As the centuries rolled by, memories of Allag civilisation faded and got corrupted. The crystal tower, which their anscestors remembered as a symbol of the might of the Allagan empire became a holy place only heard of in stories, which is why the Gigas are drawn to it. The giants and hecatoncheir are descendants of a slave race created by the Allagans to act as builders and bodyguards for their masters, along the same lines as what happened to Phlegthorn. According to the Gigas guildhest there is a hierarchy with giants serving the Gigas, so this backs up my theory.

Also I can't be the only one that sees the similarities between Xande's model and the giants....