Latency issues starting from 11/5

Starting yesterday latency has been extremely high from 4pm PST through 11pm PST or so. Tonight spikes were as high as 380ms. Here is the relevant pathping:

Code:

Computing statistics for 475 seconds... Source to Here This Node/Link Hop RTT Lost/Sent = Pct Lost/Sent = Pct Address 0 xxxxxx.socal.rr.com [192.168.1.115] 0/ 100 = 0% | 1 0ms 0/ 100 = 0% 0/ 100 = 0% 192.168.1.1 0/ 100 = 0% | 2 11ms 0/ 100 = 0% 0/ 100 = 0% cpe-104-33-160-1.socal.res.rr.com [104.33.160.1] 0/ 100 = 0% | 3 12ms 0/ 100 = 0% 0/ 100 = 0% 24.30.170.13 0/ 100 = 0% | 4 17ms 0/ 100 = 0% 0/ 100 = 0% agg33.tustcaft01r.socal.rr.com [72.129.27.66] 0/ 100 = 0% | 5 17ms 0/ 100 = 0% 0/ 100 = 0% 107.14.19.30 0/ 100 = 0% | 6 18ms 0/ 100 = 0% 0/ 100 = 0% 107.14.19.121 0/ 100 = 0% | 7 17ms 0/ 100 = 0% 0/ 100 = 0% te0-0-0-29.ccr23.lax05.atlas.cogentco.com [154.54.10.249] 0/ 100 = 0% | 8 17ms 0/ 100 = 0% 0/ 100 = 0% be2180.ccr21.lax01.atlas.cogentco.com [154.54.41.57] 0/ 100 = 0% | 9 --- 100/ 100 =100% 100/ 100 =100% be2065.ccr21.iah01.atlas.cogentco.com [154.54.5.65] 0/ 100 = 0% | 10 67ms 0/ 100 = 0% 0/ 100 = 0% be2172.ccr41.atl01.atlas.cogentco.com [154.54.29.17] 0/ 100 = 0% | 11 78ms 0/ 100 = 0% 0/ 100 = 0% be2170.mpd21.dca01.atlas.cogentco.com [154.54.31.106] 0/ 100 = 0% | 12 85ms 0/ 100 = 0% 0/ 100 = 0% be2518.ccr41.jfk02.atlas.cogentco.com [154.54.80.161] 0/ 100 = 0% | 13 --- 100/ 100 =100% 100/ 100 =100% be2106.ccr21.alb02.atlas.cogentco.com [154.54.3.50] 0/ 100 = 0% | 14 93ms 0/ 100 = 0% 0/ 100 = 0% be2088.ccr21.ymq02.atlas.cogentco.com [154.54.43.17] 0/ 100 = 0% | 15 88ms 0/ 100 = 0% 0/ 100 = 0% 38.122.42.34 0/ 100 = 0% | 16 --- 100/ 100 =100% 100/ 100 =100% 10.2.2.1 0/ 100 = 0% | 17 110ms 1/ 100 = 1% 1/ 100 = 1% 192.34.76.2 0/ 100 = 0% | 18 108ms 0/ 100 = 0% 0/ 100 = 0% 199.91.189.234 0/ 100 = 0% | 19 200ms 0/ 100 = 0% 0/ 100 = 0% 199.91.189.42

Something odd going on there with Cogent's lines trying to get out of Southern California (not uncommon either--that area has been a hotbed for congestion for a long time now). Looks like there maybe an issue with the load balancing at your hop 9 (154.54.5.65). Testing against it, after about 7 failed pings, it responds from the alternate IP (154.54.12.109). It's almost a repeating pattern too... every 5th to 7th one it responds back on the alternate. It also doesn't resolve the DNS name either.

May want to get in touch with TWC's Tier3 support about this... get some snapshots of both good and bad time frames, and also gather samples of your modem's signal and error logs for submission as well. You can go to their forums and/or twitter for assistance--I've found you wind up getting more progress that way since you can actually get the data to them when contacting them online versus the phone. Ultimately, they should provide you either an email to submit logs and traces and such, or they may direct you to the DSL Reports forum they set up for their reporting. Run some standard traces to your server's IP, as well as some other popular services. See if you can catch samples with those hops responding some times and not at others--pretty good sign they are struggling with congestion through that path. Push them for escalation if you can provide such examples. They will likely make you jump through the hoops of having a tech check your lines and such--they have to work their way through the process so they can push for each escalation. Stick with it and they should eventually get it forwarded to their "engineering" tier, where it will hopefully move forward with routing you around the congested corridor(s) somehow.

It can be done... I've gone through the process here in SC and they've been doing a pretty good job of staying ahead of it so far this year. It took a couple months in 2013 to get the localized nodes split up first (which resolved a lot of other issues), but that left us with the problems still at the peering exchanges. Fortunately, the changes they've been making in the routing to remedy it only takes a few days to put in place at the worst (sometimes it happens the same night). Each time I go back to a previous partner (I regularly get switched amongst TATA, Level3, and Cogent--the PSiNet you are using is affiliated with Cogent), it seems to get a little better. I have particularly noticed I am staying with TATA the longest of the three before it catches up to us. Sign of progress I guess...hopefully it's an indication they are actually finally working to upgrade their peering agreements.

Firstly, I just wanted to thank you for your detailed response. I also agree that hop 9 is an issue and I have actually tried to get something done about it via TWC's direct forums on DSL reports. However, the main issue over the past few days has been on hop's 16-19. Here is a pathping that shows what my connection normally looks like:

Code:

Computing statistics for 475 seconds... Source to Here This Node/Link Hop RTT Lost/Sent = Pct Lost/Sent = Pct Address 0 xxxxxx.socal.rr.com [192.168.1.115] 0/ 100 = 0% | 1 0ms 0/ 100 = 0% 0/ 100 = 0% 192.168.1.1 0/ 100 = 0% | 2 11ms 0/ 100 = 0% 0/ 100 = 0% cpe-104-33-160-1.socal.res.rr.com[104.33.160.1] 0/ 100 = 0% | 3 11ms 0/ 100 = 0% 0/ 100 = 0% 24.30.170.13 0/ 100 = 0% | 4 17ms 0/ 100 = 0% 0/ 100 = 0% agg33.tustcaft01r.socal.rr.com [72.129.27.66] 0/ 100 = 0% | 5 17ms 0/ 100 = 0% 0/ 100 = 0% 107.14.19.30 0/ 100 = 0% | 6 19ms 0/ 100 = 0% 0/ 100 = 0% 107.14.19.121 0/ 100 = 0% | 7 17ms 0/ 100 = 0% 0/ 100 = 0% te0-0-0-29.ccr23.lax05.atlas.cogentco.com [154.54.10.249] 0/ 100 = 0% | 8 17ms 0/ 100 = 0% 0/ 100 = 0% be2180.ccr21.lax01.atlas.cogentco.com [154.54.41.57] 0/ 100 = 0% | 9 --- 100/ 100 =100% 100/ 100 =100% be2065.ccr21.iah01.atlas.cogentco.com [154.54.5.65] 0/ 100 = 0% | 10 67ms 0/ 100 = 0% 0/ 100 = 0% be2172.ccr41.atl01.atlas.cogentco.com [154.54.29.17] 0/ 100 = 0% | 11 78ms 0/ 100 = 0% 0/ 100 = 0% be2170.mpd21.dca01.atlas.cogentco.com [154.54.31.106] 0/ 100 = 0% | 12 84ms 0/ 100 = 0% 0/ 100 = 0% be2518.ccr41.jfk02.atlas.cogentco.com [154.54.80.161] 0/ 100 = 0% | 13 --- 100/ 100 =100% 100/ 100 =100% be2106.ccr21.alb02.atlas.cogentco.com [154.54.3.50] 0/ 100 = 0% | 14 92ms 0/ 100 = 0% 0/ 100 = 0% be2088.ccr21.ymq02.atlas.cogentco.com [154.54.43.17] 0/ 100 = 0% | 15 86ms 0/ 100 = 0% 0/ 100 = 0% 38.122.42.34 0/ 100 = 0% | 16 --- 100/ 100 =100% 100/ 100 =100% 10.2.2.1 0/ 100 = 0% | 17 96ms 0/ 100 = 0% 0/ 100 = 0% 192.34.76.2 0/ 100 = 0% | 18 94ms 0/ 100 = 0% 0/ 100 = 0% 199.91.189.234 0/ 100 = 0% | 19 94ms 0/ 100 = 0% 0/ 100 = 0% 199.91.189.42 Trace complete.

As you can see, everything before hop 16 is about the same as the pathping I posted last night. The main difference is that hop 16+ is around 95ms, which is what I usually play at. When it was at its worst last night, hop 17 was at 367ms and hop 19 was 633ms.

may want to run a series of normal traces and not pathping. You're getting the average latency over 100 pings for each hop in those reports and not seeing the potential jitter in play. You may be averaging 19ms coming through LA, but could be getting congestion that causes a 180+ lag spike to show every 12th time or something odd like that. This can compound with additional jitter as you move further down the route, so your issue may not necessarily be at the VLAN coming through Ormuco but actually earlier in the route but it's just not showing up in a pathping report. When I run traces, those last hops (including 10.2.2.1) are typically on par or slightly lower than where I cross the IPX from TATA/Cogent/Level3 and into Ormuco.

Code:

C:\Windows\System32>tracert 199.91.189.30 Tracing route to 199.91.189.30 over a maximum of 30 hops 1 2 ms <1 ms <1 ms LPTSRV [10.10.100.1] 2 25 ms 26 ms 46 ms cpe-075-176-160-001.sc.res.rr.com [75.176.160.1] 3 26 ms 16 ms 25 ms cpe-024-031-198-009.sc.res.rr.com [24.31.198.9] 4 19 ms 18 ms 19 ms 24.31.196.212 5 21 ms 23 ms 26 ms be33.chrcnctr01r.southeast.rr.com [24.93.64.182] 6 26 ms 25 ms 27 ms bu-ether14.atlngamq46w-bcr00.tbone.rr.com [66.109.6.82] 7 26 ms 24 ms 24 ms 107.14.19.99 8 26 ms 26 ms 27 ms te0-0-0-10.ccr21.atl02.atlas.cogentco.com [154.54.12.109] 9 27 ms 28 ms 26 ms be2050.ccr41.atl01.atlas.cogentco.com [154.54.0.165] 10 36 ms 34 ms 34 ms be2168.ccr21.dca01.atlas.cogentco.com [154.54.31.94] 11 39 ms 39 ms 42 ms be2148.ccr41.jfk02.atlas.cogentco.com [154.54.31.118] 12 44 ms 44 ms 45 ms be2106.ccr21.alb02.atlas.cogentco.com [154.54.3.50] 13 48 ms 48 ms 50 ms be2088.ccr21.ymq02.atlas.cogentco.com [154.54.43.17] 14 63 ms 65 ms 65 ms 38.122.42.34 15 49 ms 48 ms 50 ms 10.2.2.1 16 50 ms 51 ms 49 ms 192.34.76.2 17 47 ms 49 ms 51 ms 199.91.189.234 18 49 ms 49 ms 51 ms 199.91.189.30 Trace complete.

That's taken on Wifi from the back of the condo, so there's some odd jitter that creeps in at the start, but notice the variances in my nodes here in the Carolinas. A good bit of jitter likes to creep in there during high volume times, and when it gets really bad I start to feel it in game. Otherwise, things run great now that they're staying on top of things with their peering partners. They typically catch it once it starts to spike above the 230-280 range and change it up without me contacting them now. Early on, I had to generate the reports and forward them and it would take about 3 days to get things changed. But I haven't had to contact them about things on that end for almost 2 months now. We are back to trying to address the congestion in our local area again though. To that end, I am on a new gateway now (75.176.160.1 which is assigned to Conway... I used to be assigned to Hilton Head, which started with 66).

Code:

C:\Windows\System32>ping -n 25 75.176.160.1 Pinging 75.176.160.1 with 32 bytes of data: Reply from 75.176.160.1: bytes=32 time=23ms TTL=254 Reply from 75.176.160.1: bytes=32 time=11ms TTL=254 Reply from 75.176.160.1: bytes=32 time=11ms TTL=254 Reply from 75.176.160.1: bytes=32 time=10ms TTL=254 Reply from 75.176.160.1: bytes=32 time=12ms TTL=254 Reply from 75.176.160.1: bytes=32 time=11ms TTL=254 Reply from 75.176.160.1: bytes=32 time=12ms TTL=254 Reply from 75.176.160.1: bytes=32 time=10ms TTL=254 Reply from 75.176.160.1: bytes=32 time=23ms TTL=254 Reply from 75.176.160.1: bytes=32 time=10ms TTL=254 Reply from 75.176.160.1: bytes=32 time=11ms TTL=254 Reply from 75.176.160.1: bytes=32 time=12ms TTL=254 Reply from 75.176.160.1: bytes=32 time=11ms TTL=254 Reply from 75.176.160.1: bytes=32 time=11ms TTL=254 Reply from 75.176.160.1: bytes=32 time=11ms TTL=254 Reply from 75.176.160.1: bytes=32 time=9ms TTL=254 Reply from 75.176.160.1: bytes=32 time=9ms TTL=254 Reply from 75.176.160.1: bytes=32 time=12ms TTL=254 Reply from 75.176.160.1: bytes=32 time=10ms TTL=254 Reply from 75.176.160.1: bytes=32 time=10ms TTL=254 Reply from 75.176.160.1: bytes=32 time=11ms TTL=254 Reply from 75.176.160.1: bytes=32 time=11ms TTL=254 Reply from 75.176.160.1: bytes=32 time=10ms TTL=254 Reply from 75.176.160.1: bytes=32 time=22ms TTL=254 Reply from 75.176.160.1: bytes=32 time=11ms TTL=254 Ping statistics for 75.176.160.1: Packets: Sent = 25, Received = 25, Lost = 0 (0% loss), Approximate round trip times in milli-seconds: Minimum = 9ms, Maximum = 23ms, Average = 12ms

Code:

C:\Windows\System32>pathping 75.176.160.1 Tracing route to cpe-075-176-160-001.sc.res.rr.com [75.176.160.1] over a maximum of 30 hops: 0 G75VXLAP [10.10.100.10] 1 LPTSRV [10.10.100.1] 2 cpe-075-176-160-001.sc.res.rr.com [75.176.160.1] Computing statistics for 50 seconds... Source to Here This Node/Link Hop RTT Lost/Sent = Pct Lost/Sent = Pct Address 0 G75VXLAP [10.10.100.10] 0/ 100 = 0% | 1 1ms 0/ 100 = 0% 0/ 100 = 0% LPTSRV [10.10.100.1] 0/ 100 = 0% | 2 11ms 0/ 100 = 0% 0/ 100 = 0% cpe-075-176-160-001.sc.res.rr.com [75.176.160.1] Trace complete.

Notice the difference in the stats amongst those reports. Pathping is showing an average latency of 11ms, but in the tracert it was showing a swing of 25-46, and when I pinged it directly 25 times it swung from 9-23, with right at 1/8 of them clocking in at 2x the average reported by the pathping report. From that small ping sample, the line is showing jitter of just over 3ms against what pathping was showing as an average of 11ms. That comes out to ~30% of the pathping's reported latency... which is not a good result.

Even looking at the short results of the one tracert, it is ~10ms jitter against an average of ~32ms latency which still comes out to around 30% again, which is simply not were you want to be... especially if it is happening at multiple points in your route. If you consistently see those 10, 20, 30ms or higher spikes show up, it can mean your route is in for a bumpy ride. But you won't be seeing that in a pathping report... but you may capture it with some tracerts.

I decided to start up PingPlotter and have been running it for the past few hours. I set the interval to 1 second and set graphs for each individual hop (I prefer a visual representation of the data at a glance). I definitely see what you were talking about in terms of jitter, but am a bit surprised by how bad things have started getting right at 4pm:

http://i.imgur.com/LLMctob.png

I'll keep an eye on this as the night progresses, hopefully collecting enough data to inform TWC of what's going on. I just want to thank you again for all of the help you've provided.

At least for the short term, WTFast has allowed me to play the game tonight.

http://i.imgur.com/iTFazBD.png