Nothing in the lore shows this to be true with how time travel is handled. The lore shows two types of time travel. Branching alternate time lines, based on large decisions that change the future, and small self contained loops that do not change the future significantly.
Shadowbringers was the former and EW and Alex the latter. There is NO evidence that events in an alternate branch timeline affect the other branches without some kind of time travel device (Alexander, The Tycoon.)
Gra'ha changes his future by stopping the 8UE, so he can no longer return to that timeline even though it still exists. Elidibus' warning is that if we try to change the timeline by stopping the End of Days in the past, because we and everything we're fighting for exists because of the Sundering, will not exist when we return, thus creating another branch. This is the whole reason for the closed loop paradox and the convergence with Venat/Hydaelyn.
We are currently on a branch timeline, but the game, by factor of we're the main character, is shown from OUR POV.
Timeline 1
Ancient world -> Meteia -> Zodiark -> Sundering -> ARR -> Crystal Tower Raids -> HW -> SB -> B -> We Die -> 8UC -> Tycoon Created -> Time Travel -> A ->Midgarsormr Sidestory
Graha leaves at A ends up at B.
Timeline 2
Ancient world -> Meteia ->B -> Zodiark -> Sundering -> ARR -> Crystal Tower Raids -> HW -> SB -> Graha ends up on First -> SB -> EW -> A -> Current Patch
We leave at A end up at B, but return to A.
In Timeline 1, we do not EXIST to be able to go back in time. Without Graha, going back in time we can NEVER go back to Elpis. Our actions in Elpis are irrelevant to the timeline from the Ancient World up to EW, because things happening in Elpis would have happened anyway. The Metia, the summoning of Zodiark, and the Sundering, all happen whether we show up there or not. Elidibus all but straight up says this in his line about not meddling.
So what changes with us going back? The knowledge that Venat has, but refuses to act upon, and the knowledge we gain and DO act upon.