Many spoilers are referenced here, please skip if needed.
How about this then? If this is a genuine fear, don't play the game. By doing so you knowingly decide that the pacifist path is not for you to take. In order to gain the upper echelons of power one must struggle, fight, and be willing to die. The argument of who do we answer to has already been broached and not just once but multiple times. you need only actually look and read the text of the story you are playing.
Long before Dark Knight we are faced with the mortality of those that follow the path of adventurer's, you need only look at story of your fellow adventure's during the first three dungeons. Then again we shown the price of failure and what the happens when we sit idly by with the tempering process Ifrit subjects people to. Few games ever show that, people ask why this is happening to them, why they must face this, at one point you are given the chance to run away... But you didn't you? You went back and killed that primal because you had to stop it's machinations, and the more time you put it off the more he would temper.This was all at level twenty five...
The work continues through the msq, we learn more of the machinations of not only the beastmen, the Empire and their super weapon, the plans to subjugate the entire on Erozea through military force with the help of Ascians no less, bringers of chaos and those who have for millennia untold have corrupted civilizations into primal summoning. The very act slowly killing the planet.
We are spurned on finding that many of our allies are murdered by the Empire. After which we personally bury the bodies of our fallen compatriots, individually, one by one we carry their bodies to proper burial. All the while we grid ourselves to free our friends from capture by the Empire, even over hear what the soldiers at the castrum have in store for female companions... only after rescuing them do we find out who it was who betrayed us. One of our own is possessed and forced to betray his friends and allies through other worldly means (a burden he carries to eve this day). Still we fight on, through insurmountable odds all the while trying to save as many as possible even allowing ourselves being the last ones out of a destroyed hellfire of a building on the back of a breaking magitek reaper while saving our mentally tortured friend no less.
Oh yeah as a 1.0 player it is revealed that w did this all while under a memory fog that keeps previous allies and friends from remember us. We protected the world from Galemald and attempted to stop the Primal Bahumut, failed and the were wiped from the memory of nearly every person we saved. And then we did it again, with no expectation of reward for past accomplishments. Only afterward the mother crystal lifting the veil from their memory to recall the individual who with all their power struggled to keep all of Erozea safe twice over content to be forgotten. That concludes A Realm Reborn msq.
Many things were left out of that. Many people just buy the story line skip potion and never see it. We have under gone many, many other trials since then. There is a reason why i and so many challenge this train of thought. It falls flat, makes no sense and is a false, unneeded edge to our character. You are playing the game, full knowing what you as the character is, and what it is you are doing and why you do it. Whats being asked for, the question of what if people saw you in a negative light has already been addressed. If you skipped it that's no fault of the content writers.