Once Heavensward is out, there would be no harm in lowering the difficulty for the sake of the story content, or even loot.