I am taking this that you are all on pc version. Another question is if you guys are using vista-windows 8. If you are a simple fix that fixed this problem for me is Leatrix Latency Fix. Here is the site directly so people don't accidentally download the wrong thing. http://www.leatrix.com/leatrix-latency-fix

For some reason microsoft got this ingenious idea to make your tcp/ip protocol send and receive files at 2-4 at at time. Under heavy traffic times this can cause you to experience delays resulting in you performing an action and nothing happening for seconds at a time. Leatrix fixes this by rewriting the tcp/ip protocols to send and receive 1 file at a time so even under heavy traffic times you don't get that 3-5 second delay.