Its worth pointing out that Ishgard was actually attacked. They mention the attack later on. I think his comment suggested that it indicated that someone was feeding the Dravanians information for them to know to attack when he wasn't there, ie. Dravanian spies in Ishgard.