SP is independent of actions taken or conditions (color of orb) during the process.
All you need to do is Hasty Hand something twice and you'll get 2 different SP values.