One person turning it off will make no difference though. If you're a DPS then you'll still make tight DPS checks you may not have because the 3 other DPS are buffed. If you're a Tank even if you screw up rotating your CDs, as long as you're not one shotted, buffed Healers will have no problem keeping you up. And the reverse for Healers, less intense healing periods on buffed Tanks, plus your other Healer can cover for you.
Unless you're able to talk over half the party into switching it off then it will still effect your run.