There's another problem, too: Azem was a lot taller than us!

We've seen that people from the other reflections aren't any smaller than us. Further, if people from the Source got bigger with each rejoining, I feel like that would have come up? People would be hitting their heads on their doorframes after the Seventh Umbral Calamity. That implies that no matter how many times we're rejoined, we won't ever be as tall as the Ancients.