I don't think the retainers can get them, you need to have any +1 equipped while you do the fates (for example i'm a main scholar but I used my bard +1 to collect the atmas because physical damage is faster than cast time). It is a hard grind I was in some areas for 6 hours with no drop. also 2 of my friends finished collecting way before me and I started before they did. I did get them all though and have just about finished my first book, im just waiting around in the area doing nothing now because they make you do 3 more fates in the book, and the respawn time on them is really long.

also I found that in the early mornings or in less populated maps or if you go do a dungeon or just take a break from it and come back that when you come back you get it within the first 5 fates