I think the idea that they're tempered is flawed from a number of angles.
First and pretty obviously, the Echo. It seemingly (Hydaelyn theories notwithstanding) allows all those who have it to effectively be immune to tempering, we even see Echo-users capable of outright shielding others from the aether of Primals. So, if all the Ancients had the Echo, how were they tempered?

Attendant to this is the issue of what Zenos says regarding the Echo and tempering, that the Echo in fact makes your will superior to that of a Primal in some way, and thus their will cannot override your own. However, he demonstrates this by forcibly mantling Shinryu and controlling him. This creates a secondary issue regarding Elidibus as Zodiark's heart - If Zodiark was summoned through Elidibus, and seemingly Elidibus' will controlled him, why would he temper anybody?

Next are the issues presented by Emet-Selch and Mitron. As we know, the tempered are driven to carry out the will of their Primal - Yet, Emet actively worked against Zodiark in leaving the Azem stone to WoL and then even saving WoL in the battle with Elidibus. And Mitron, his plans had nothing to do with Zodiark and arguably worked against Zodiark as well, he wished to compress time simply to be with Gaia for all eternity.

Related to Mitron and Gaia is the fact that, so far as we know, tempering is undone upon someone's death. As all the sundered Ascians died, none of them should be tempered. It being a matter of memory doesn't make much sense, as it's always been shown as an aetherial issue with the Primal's aether imprinting into someone, and a trip through the Lifestream really should wash away any such aetherial influence upon the body and soul.

And Fandaniel, he's an even worse can of worms. The Final Days was literally what Zodiark was summoned to stop, if he was tempered why would he go about trying to restart it? Moreover he suggests that he was shackled to the wills of the Unsundered somehow, and now that they're gone he's completely free to live however he wishes. That in itself really doesn't read as, "I'm free - To follow Zodiark the way I want to", it would be pretty weird for it to be like that considering his dialog and actions.