I think the FATE times are fine really, I don't mean to sound harsh but it was your own decision to stay in your DF queue and not join the party.

I like to do the FATE's whenever they are up so I have the item already before even accepting the quest. I recommend doing this ^^