If it reliably happens to you in the evening, every evening at the same time, the problem is more than likely with your ISP.
You may be getting throttled for being over a certain download limit, or others in the area also on your ISP begin heavy use which affects everyone.
Lastly, you may have people in your home downloading heavily in that time, or even people jacking wireless from you, so check your security. Perhaps you yourself have a torrent program running, or even your AV program doing it.

There are quite a few things you can check before going to the forums and screaming murder at SE demanding they should fix it for you.
Because its happening at a certain time, with regularity, you cant blame SE alone for it, otherwise everyone on your server would report it at the same time.