The other thing to keep in mind is that our primary goal in travelling to the past was to witness what did happen there so we could fix it in the present. If we had changed it to be something else (and if we had the knowledge of what to change in the first place) then we would no longer be seeing the events that led to our present, and the knowledge would not be useful to us.

Alternately, if we did go back in the time with the mentality that "Hermes will cause all our present-day problems, so we should kill Hermes in the past to fix it", it would probably have to turn out that Hermes wasn't the problem after all and actually when Elidibus sent us to talk to him, he was unaware of the fact that shortly before the Final Days, Hermes was murdered by a mysterious assailant and someone else was elected to his position, and that person ended up being the one we were dealing with in the present day.