This happens all the time or only on peak hours ?
Also, can you post a tracert to your server IP ? (you can hide first 3 jumps)