This system could work, but can be seriously abused.

Those that wish to abuse the system can just as many said, keep the monster alive while spamming abilities till SP is capped. There is no incentive in killing the monster fast due to the limited SP gained for faster kills, would also limit certain classes that don't have many helpful abilities to use.

Would turn the system on itself, people will go the easiest route which would gain the most SP possible.

Now if there was a way to balance the usage of abilities to set a standard SP gain, I for one would like to see how this system would work. could be a new direction in the game.

Post the math/explanation and hopefully we can all come up with a method to increase the structure as well find flaws with the system, only true way is to work the "idea" and find/tweak/remove any flaws that may be in place.