Ramuh had been summoned at some point in the past, which was what tempered the sylphs, but we have every indication that he's no longer there. We were sent to look into the sylphs to ensure they weren't about to bring him back, and once that was determined, that was why they were left alone. With Ramuh running loose in the Shroud, I don't think anyone would have been so comfortable turning a blind eye (that, and we can go full into the heart of the Sylphlands; I'm pretty sure we would have seen some indication of him there, as a "guardian deity" to the sylphs he probably wouldn't be so comfortable letting us walk in and start killing them).