There is actually some missable dialogue from G’raha right after he joins the Scions in which he states that without the tower, his aether usage is harder to perform, but he still feels more useful with those skills than returning to being an archer. Also, apparently the most annoying side effect of his unlinking from the tower is how hungry he is all the time.