It is annoying, but stouter is right. You could get the fate items on times when your not doing the quests themselves. but worry not because SE has already talked about reducing the times for these in a live letter a bit ago I believe....again..