Well, I believe part of the reasoning here comes from emotional attachment to your own character, and the voice plays a part in that, so hearing something being forced onto your own character that doesn't sound right would be upsetting. Rather than continue dealing with this, some might find it easier to switch to a different voice because it would be one they're not already attached to or enjoyed. Maybe a good comparison would be something like, so I never played AST during stormblood, however i did play SCH and DRK, and when shadowbringers rolled around, I was very upset by the changes they had made to two of my favorite jobs. On the other hand, I never played AST, so I dont feel quite as much about the loss that others do about it, though I very much understand why it's a widely not well-received change. I kinda like AST how it is right now but I'd be willing to bet if I had played it previously I would probably be more bothered by it as I am with the jobs I did play.
And back to the main subject, I might be speaking for a small (or maybe larger) group of people here without going into too much detail, but a lot of my friends including myself consist of people that kinda have some mental stability issues at times, and I think changing something like this that we care a lot about is more likely to affect us emotionally than it may others. I know it probably sounds ridiculous but I have a friend who shares the exact same voice as me and when I told them about what had gotten changed, it was just about breakdown-inducing. While I wasn't impacted as strongly by this, it still was significant enough that I was put off by it enough to stop playing and stay pretty upset about it for a number of days before getting over the initial shock.
I'm not exactly saying that's healthy or reasonable probably, but that's just the way things are for some of us and with all due respect I have tried to post here in this thread about my complaints/concerns with these changes without letting my own emotions/state of mind affect how I talk about it. (but I know you can probably see some of that influence in my previous posts anyway.) Coming here and trying to use all of my own emotions/experiences as a justification for change wouldn't be helpful or contributional and would largely come across as "I don't like this, change it." I'm just glad that I'm not the only person here that shares the same sentiments and that there are fair arguments as to why this change wasn't implemented very well or fairly. I think if we had always had more voices than just two for combat, this never would have been an issue, because the voice actor would be right there.
Sorry if I'm not making any sense or am being off topic right now, I'm kind of scatterbrained and have a hard time putting words together