I don't have swiftcast, but I understand the delay issue. If I cast Ice and then switch between umbral ice to astral fire too quickly, the game won't recognize that I even had umbral ice to begin with. I'd imagine it is the same for swiftcast.