I don't see any reason to assume that's what caused the timelines to diverge - otherwise our time travels in Alexander split the timeline twice, and that couldn't have happened because both time-travel instances rely on time acting as a loop. Particularly the second time during A12 where we save our past self - there simply cannot be a version of events where we fail to do so, because that would be a paradox.
It seems logical to me that the point where the timelines diverge is the point where the original future is no longer possible because new events are incompatible with what G'raha knows to be true. Specifically, preventing the calamity that is a key event in creating the world as it exists in that timeline.
It can be hard to explain but a change in known circumstances is not the same thing as a change. The "three years ago" incident in Alexander is the easiest example - the timewarp and the "interfering" goblins were always part of the single version of events that played out there, but Mide doesn't know about them until she witnesses it a second time from a different perspective.
By the same logic, when the Crystal Tower first appears in Lakeland, that wouldn't be a change to some previous "untouched" version of events, but part of the single version of that world's history.
Whatever exactly triggers the split, my personal line of thinking is to place it as late as possible in the chain of events. That well could be the first time the Exarch makes contact and Thancred is taken - though (dependent on some specific hypotheticals) suppose the record of this event doesn't survive through to the far future to be relayed to G'raha. Suppose all the Scions are "taken" but the records that survive the calamity are unclear about this, and they are misunderstood to be victims of Black Rose. This would mean that even after the Exarch starts affecting events in the Source, the timelines have not yet diverged until we do something that more conclusively changes things.
Perhaps it's whether or not the Exarch successfully "interferes" with our fight with Zenos. Perhaps it's whether we choose to listen to him or not. Perhaps it doesn't even diverge until we succeed or fail in defeating Hades? But again this is very speculative.
I have speculated on this idea before - specifically that our defeat of (what the Garlean army believes to be) Zenos is the tipping point that eventually drives them to employ Black Rose. However, we have no reason to consider it a fact, unless perhaps we're intended to interpret the trailer as an alternate "how it went in the other timeline" version.
Suppose (for some arbitrary reason) I decide to throw a ball at a window.
"The glass might break," I say. "You should stand back so you don't get hit by it."
I throw the ball. It bounces harmlessly off the window.
Your argument regarding the tower is equivalent to insisting that because the glass is not broken, I must not have thrown the ball yet.
The Exarch planned to change the course of time. He warned people to stay away from the thing that might change in a harmful way as a consequence of his plan. His prediction of that consequence turned out to be inaccurate.
Time has been changed, and there was no great fanfare to announce that it had happened. But you can observe that the situation has changed in other ways, and those changes mean that the events G'raha knows as "the history of how the Eighth Calamity occurred" cannot come to pass. The Light is gone from the First and cannot supercharge Black Rose in the diminished chance that it is still used in the Source. Therefore, unless those specific circumstances return, we cannot still be on the same timeline that leads to the original future he came from.
As for "no going back to the Seventh Astral Era" for the other timeline? No it won't, because time moves forward. They can't return to the Seventh Astral Era but they can work towards an Eighth.
This is not the cause and effect you are looking for.
Making the tower vanish will not avoid the calamity.
The tower might have vanished as a side effect of the other actions we undertook to prevent the calamity. It did not. It does not have to.
To return to my analogy: throwing the ball = acting to change time; breaking the window = making the tower vanish.
The breaking glass does not cause the ball to be thrown.
As others have already covered at this point, there are variables. We don't know the timing of when Black Rose would have been used - perhaps it was before Zenos made his move to assassinate Varis. Perhaps he missed that one bomb that they ultimately set off.
We know that Emet-Selch was scheming to make use of Black Rose to trigger the calamity. We know it has aether-halting properties. And we have no reason to doubt G'raha's recounting of the history of his timeline - or Omega's more immediate records in A World Forsaken.
It's less of a narrative stretch to assume Zenos didn't stop it in the other timeline than to assume that the calamity (the exact one as known to G'raha) is still on track to happen despite all evidence indicating that it has been averted.