Can still be issues with your ISP, if they have packet shaping and things enabled.
Which ISP are you using?