That really doesn't matter, because even if you had so much more spell speed that the Fire IV cast time was absurdly fast like 1.5s, it would still be longer than the GCD.
Idolon is right in attributing it to how every single offensive action a BLM can take is a GCD action. 100% of BLM damage is on GCD, so if cast times drop by, say, 5%, BLM damage goes up to 105.26% (1.0 ÷ 0.95). Compare to another DD job that might do 70-80% of damage on the GCD and 20-30% off it, a 5% cast time reduction only results in 103.68-104.21% damage (0.7 ÷ 0.95 + 0.3) ~ (0.8 ÷ 0.95 + 0.2).