What would be the point in having summoners then? Since I'm hoping it becomes an available class/job in the future.

My ideal of a summoner class is one which requires assistance from his/her fellow adventurers, (Like Yuna and her crew), to defeat primals in an exciting and difficult battle in order for the summoner to obtain them. Perhaps you can make it difficult to obtain the summoner class itself as well. A level 50+ battle or some sort. This would indeed be a "special" class compared to the rest.

Perhaps there can be a limit on how long each summoner can hold on to a particular primal? To which they would have to defeat it again to access its power. Although, having to defeat the same primal again might become tedious but it could possibly solve the issue of seeing too many summoners out there walking around with their Ifrits.

I would say very long cool downs but that would punish people who change to the class itself when signing on or frequently switching jobs.