It should be noted that the areas these towers were appearing are very close to where primals were defeated as pointed out in the 5.4 spoiler thread.

Floating City of Nym -> O'gomoro Mines -> Titan
Ala Mhigan Quarter -> Royal Promenade -> Shinryu (Also relatively close to where Lakshmi was defeated. Just a zone over.)
Sangolii Desert -> Zahar'ak -> Ifrit
Lost City of Amdapor -> Enkidu (This is where it gets interesting since it was a side story summon. Also while not a summon, Diabolos was trapped here for a long time. If he is to be believed, he was rather powerful prior to his imprisonment.)
Dravanian Forelands -> Loth ast Gnath -> Ravana
Yanxia -> Castrum Fluminis -> Tsukuyomi

As said poster also mentioned, it is possible they are drawing on the aether from these primals in order to help fuel their tempering properties. It can also be assumed that they are also in other areas as we do know of other primals that have been defeated in recent memory that the WoL has never faced. (Sorrow of Werlyt)