The population of the game mode is irrelevant in regards to SE's obligation to fix it.
The savage raiding community is tiny in comparison to the whole of the player base, yet every single issue that is raised within those raids is subsequently addressed and resolved in short order. The same could be said for things that are seemingly insignificant, such as Chocobo racing, Lords of Verminion, Triple Triad, or even minion collectors. We could sit here all day and divide the game up into it's many different parts and quantify the number of players in each, but the bottom line is that all of them, no matter how niche, are paying customers. We all pay the same subscription. SE is very firmly in the wrong if they think it is okay to discriminate against a portion of their subscription base, no matter how small that portion is. That is especially the case considering the alleged toxicity is by no means exclusive to any single part of this game.