The encounters are balanced with there being a full party buff. Simply removing the buff does not mean more difficult battles. Future encounters would be balanced with there not being a full party buff. The result is a zero net change in difficulty.

The ONLY thing that determines difficulty, is the developers arbitrary choice of mechanics and gameplay elements incorporated into a specific encounter.