The problem with the heavy debuff is that it's annoying enough to be uncomfortable, but It isn't annoying enough to make world enemies relevant

You can still ignore them, you simply will go slower