I feel like it's much harder to balance around this. This is one reason the timewalking dungeons in WoW are so horribly balanced. They are either to hard to to easy.

I'm perfectly fine with how it is.