Let me elaborate a bit as to why I stopped fertilizing.
I used to diligently fertilize every time I stopped by the house, with the thought that I was speeding up the harvest time, however at some point I realized that this was becoming a moot point.
It was getting to the point where the crops were coming up ready to harvest when no one was around to do the harvesting, somewhat wasting any speed gains.
What's the point of pushing the harvest time to, say for example, 4am, when I was certainly not going to be online?
Also, unless your planting times are very controlled (only planting at say 4pm in your local timezone) it was too much of a bother to schedule "accelerated" harvest times that would actually line up to when someone could harvest.
Once I came to this realization, I simply plant and harvest when I have the time.
I control the garden, the garden does not control me