I think the best way to put it is the speed drop bothers me, a lot. I like going fast. I don't want to lose that. Riddle of Fire /almost/ feels like you're losing stacks.
... But, the damage buff is pretty big. So I guess I don't like how the ability functions, but I like the damage?