I've never had this issue with Scourge or Banish II. It always drops to "9 sec recast" after I cast the spell.