The issue with the Castrums in Eorzea is that, aside from the ones in Gyr Abania, they are all in Eorzean-held territory. These Castrums were, largely, built during the 5-year time skip between the ending of 1.x and 2.0. This was the timeframe during which Gauis' XIVth Legion went rogue... and all deemed traitors punishable by death, which is why Nero didn't return to Garlemald after Gaius' death.
It's not that Garlemald abandoned its loyal soldiers, it's that Garlemald's loyal soldiers abandoned it. Kinda. They were more loyal to Gaius than Garlemald. I suspect things would be quite different had Gaius succeeded in subjugating Eorzea... but I digress.
That said, I suspect the soldiers in the Keeper of the Lake may have been sent by Garlemald proper. It's unclear. We do know that the Empire is able to carry out operations in Eorzea if it so wishes though - see Grynewaht attacking us while we're attempting to access Omega's control room in Mor Dhona, the heart of Eorzea.