Once you max out your zodiac weapon you should be able to challenge Gilgamesh in a 1 vs 1 ex battle and if you lose he keeps your relic and you have to start over.