meh. not entirely correct but not exactly wrong. only light spirit comes with different modes, for which its buffing mode is indeed 1/2 its regular casting time. light spirit also has its unique healing mode triggered by low hp. and you can direct who you want light spirit to buff (in buff mode) by facing it at any alliance members.
using assault/retreat simply resets the timer, and this is something you can actually use to your advantage when they start casting AM because you can prematurely reset it before the spell actually casts.
My theory is actually impossible to prove wrong. But there are other theories that work just as well, they are just not as logical.

If you summon a fresh spirit and catch aggro, they'll cast in exactly half time. This means that either you summon spirits in offensive mode at half initial recast, or they are summoned in buff mode with max recast. There is no reason SE would set "create new spirit with half timer" as code, opposed to the likelihood they put it as "create new spirit with max timer". Considering elemental behavior is buff mode or attack mode, all evidence point at similar for spirits.

You can also retreat a non-light spirit, then claim a monster to trigger aggro protection, and see if it takes half or full time to attack. If it resets their only attack recast, it will be full time. If they are in fact in buff mode as I predict, they'd cast in half time. (I believe I tested this, but I forgot so can't claim this as true)

Another option is to wait 10 seconds after your light spirit buffs you, and claim a monster. Light spirit should cast a nuke within the same time span as he would if you had let him stay and buff you. It's timer is the buff timer until it has cast one spell or is told to assault.

Everything about this behavior fits with my theory. But I will admit that you can make up excuses that could also be true. Like pets being summoned with half their casting time (except light spirit which is summoned in buff mode with full? seems like random programming in that case, instead of a logical one)