Bad hop in Canada Horrible packet loss Cali user

[Sunako_Nakahara]

The hop ae-10-10.car2.Montreal2.Level3.net over the last few days has been getting worse and worse. Ran the scan over about 2 hours from 4:00 PM (PST) till around 6:00 PM (PST)

|------------------------------------------------------------------------------------------|
| WinMTR statistics |
| Host - % | Sent | Recv | Best | Avrg | Wrst | Last |
|------------------------------------------------|------|------|------|------|------|------|
| ROCKY4662 - 12 | 3466 | 3054 | 0 | 1 | 23 | 1 |
| 10.72.210.1 - 1 | 4994 | 4964 | 7 | 12 | 280 | 22 |
| ip68-4-11-146.oc.oc.cox.net - 1 | 4995 | 4965 | 8 | 12 | 150 | 10 |
| ip68-4-11-93.oc.oc.cox.net - 1 | 4999 | 4970 | 8 | 13 | 225 | 14 |
| ip68-4-11-92.oc.oc.cox.net - 1 | 4943 | 4900 | 7 | 13 | 250 | 11 |
| langbprj01-ae1.rd.la.cox.net - 1 | 4943 | 4900 | 10 | 16 | 253 | 12 |
| 4.53.230.93 - 2 | 4838 | 4769 | 9 | 17 | 616 | 13 |
| vlan70.csw2.LosAngeles1.Level3.net - 9 | 3767 | 3430 | 81 | 87 | 132 | 87 |
| ae-72-72.ebr2.LosAngeles1.Level3.net - 11 | 3633 | 3262 | 81 | 86 | 272 | 86 |
| ae-3-3.ebr3.Dallas1.Level3.net - 9 | 3805 | 3477 | 81 | 87 | 276 | 85 |
| ae-83-83.csw3.Dallas1.Level3.net - 9 | 3857 | 3542 | 81 | 87 | 293 | 88 |
| ae-81-81.ebr1.Dallas1.Level3.net - 9 | 3809 | 3482 | 81 | 86 | 234 | 87 |
| ae-14-14.ebr2.Chicago2.Level3.net - 10 | 3705 | 3352 | 81 | 87 | 133 | 87 |
| ae-1-100.ebr1.Chicago2.Level3.net - 10 | 3709 | 3357 | 81 | 87 | 277 | 85 |
| ae-6-6.ebr1.Chicago1.Level3.net - 9 | 3809 | 3482 | 81 | 87 | 274 | 88 |
| ae-10-10.car2.Montreal2.Level3.net - 67 | 1403 | 475 | 0 | 107 | 292 | 277 |
| ORMUCO-COMM.car2.Montreal2.Level3.net - 10 | 3701 | 3347 | 82 | 87 | 134 | 87 |
| 192.34.76.10 - 11 | 3607 | 3230 | 84 | 90 | 307 | 86 |
| 199.91.189.242 - 15 | 3231 | 2760 | 0 | 89 | 177 | 86 |
| 199.91.189.30 - 8 | 3897 | 3592 | 84 | 89 | 128 | 90 |
|________________________________________________|______|______|______|______|______|______|

[Raist]

looks like you may have some issues in general across most of your route. Would be interesting to see if pathping flags anything in it's forwarding statitistics.

[Lurkios]

A ton of folks have posted about this since the data center change in 2.1 (I never bothered to do a tracert then, but I'd venture a guess that server was not in-route until the change). Unfortunately, other then change ISPs or move the datacenter out of Canada (Why the hell is it in Canada anyway? What happened to Chicago?), I'm not sure what SE could really do much it. Level3 has to fix their router, which they don't seem overly interested in doing.

[Wobbler]

Sunako yeah quite a few people suffer from the level 3 problems in Canada, cox, verizon, twc customers alike.

Don't listen to the uninformed that say that you have many problems, the packetloss is the main issue here.

Yes square can do something about it, they need to give level 3 a call and apply pressure from a "We are loosing money because people can't play our game" perspective as well as your ISP needs to either a) swap your specific peering port to use a different peering network (may not be available for you) or b) give level 3 a nice high priority trouble ticket to investigate car11.

Alternatively you can purchase a server and route your own traffic thru it if its not going via level 3 and you can avoid the poor server.

[Raist]

Don't listen to the uninformed that say that you have many problems, the packetloss is the main issue here.

Guessing this is pointed at my post, at least in part?

Go back and look at the report again. It's not JUST that one level3 hop in Canada causing problems for that route. The OP is flirting with 10% losses before even leaving the LA area, and that follows through Dallas, into Chicago (where it actually hit 10% loss), and up into Canada where it compounds the problem into 67%. Afterwards, almost all the hops after that are floating around the 10% mark again (10, 11, 15, then 8)--and let's not forget latency seems to be all over the map throughout the entire route

This may well indicate a highly consistent problem closer to home that is having a full-time impact on every hop along the way. Why I mentioned another tool that might shed some more light on the situation.

I have a level3 hop in Canada that exhibits similar behavior, just not as bad. ae-11-11.car2.Montreal2.Level3.net [4.69.141.1] routinely likes to kick off around 30-40% losses for me (it just returned 37% on a pathping), but there are problems on the hop just before that (ae-5-5.car1.Montreal2.Level3.net [4.69.141.5]). Latency will spike from around 80 to 250+ on that hop, and then ae-11-11 will start dropping packets--it also shows losses on the forwarding statistics between these hops as well. When ae-5-5 goes back to it's normal 80-ish latency, the two hops behave just fine.

Yet, in spite of all that, I am still able to play the game for the most part without much incident. My latency across almost the entire route is below 100 except for one in NY that runs 100-130, and those two hops in Montreal that want to act up. The reason is because the rest of my route is pretty clean and is able to recover more effectively. For the OP though, things look rough all over so it has a much bigger impact--and even when that one hop runs clean, there may still be a consistent problem with lag/skip that may be problematic. I'd hazard a guess that ~10% packet loss with periodic 1/4 second or higher lag spiking will surely be felt in game.

Cutting in a new modem tonight (finally getting to upgrade to DOCSIS 3 in our area, looking forward to better streams once this thing gets going). Who knows... maybe they'll get me rerouted again this weekend and I can get off that bad hop again myself <crosses fingers>.

[Raist]

HOLY CRAP! Did NOT expect the new modem to have much of an impact, but... WOW, this thing is smooth. Speedtest graphs are nearly flat now--nearly everyone I run is either nearly perfectly flat on the download, or has a steady gentle slope downward after the "turbo" boost crap that TWC does (like to Canada, it ramps up to 12M+, then slopes slowly to just under 9M and holds there). Uploads are steadier too.. it quickly and smoothly runs up to my 1M cap, and then holds steady at just a hair over 1M throughout the test. Before on the Time Warner supplied Ubee box, both would spike and dip... was like watching a heart monitor of someone strung out on crack or something.

IDK if it's just coincidence or what... could have been a small change in the path I'm missing, but this pathping I just took right after 11:30 Eastern on a Friday night from South Carolina on TWC-SC is the cleanest I've seen in a long time:

Tracing route to 199.91.189.25 over a maximum of 30 hops

<initial hops removed for security>

  4  24.93.64.134  (first hop off my modem, goes straight to Charlotte, NC)
  5  bu-ether24.atlngamq46w-bcr00.tbone.rr.com [107.14.19.18]
  6  107.14.19.11
  7  twc-telus.pr0.chi10.tbone.rr.com [66.109.9.70]
  8  vlan51.ebr1.Atlanta2.Level3.net [4.69.150.62]
  9  ae-6-6.ebr1.Washington12.Level3.net [4.69.148.106]
 10  ae-1-100.ebr2.Washington12.Level3.net [4.69.143.214]
 11  4.69.201.65
 12  ae-71-71.csw2.NewYork1.Level3.net [4.69.134.70]
 13  ae-72-72.ebr2.NewYork1.Level3.net [4.69.148.37]
 14  ae-5-5.car1.Montreal2.Level3.net [4.69.141.5]
 15  ae-11-11.car2.Montreal2.Level3.net [4.69.141.1]
 16  ORMUCO-COMM.car2.Montreal2.Level3.net [4.59.178.74]
 17  192.34.76.10
 18  199.91.189.242
 19  199.91.189.25

Computing statistics for 475 seconds...

            Source to Here   This Node/Link
Hop  RTT    Lost/Sent = Pct  Lost/Sent = Pct  Address

                                0/ 100 =  0%   |
  4   28ms     0/ 100 =  0%     0/ 100 =  0%  24.93.64.134 (Charlotte, NC)
                                0/ 100 =  0%   |
  5   33ms     0/ 100 =  0%     0/ 100 =  0%  bu-ether24.atlngamq46w-bcr00.tbone.rr.com [107.14.19.18]
                                0/ 100 =  0%   |
  6   32ms     0/ 100 =  0%     0/ 100 =  0%  107.14.19.11
                                0/ 100 =  0%   |
  7   36ms     1/ 100 =  1%     1/ 100 =  1%  twc-telus.pr0.chi10.tbone.rr.com [66.109.9.70]
                                0/ 100 =  0%   |
  8   33ms     0/ 100 =  0%     0/ 100 =  0%  vlan51.ebr1.Atlanta2.Level3.net [4.69.150.62]
                                0/ 100 =  0%   |
  9   49ms     1/ 100 =  1%     1/ 100 =  1%  ae-6-6.ebr1.Washington12.Level3.net [4.69.148.106]
                                0/ 100 =  0%   |
 10   50ms     0/ 100 =  0%     0/ 100 =  0%  ae-1-100.ebr2.Washington12.Level3.net [4.69.143.214]
                                0/ 100 =  0%   |
 11  102ms     0/ 100 =  0%     0/ 100 =  0%  4.69.201.65
                                0/ 100 =  0%   |
 12   85ms     0/ 100 =  0%     0/ 100 =  0%  ae-71-71.csw2.NewYork1.Level3.net [4.69.134.70]
                                0/ 100 =  0%   |
 13   68ms     0/ 100 =  0%     0/ 100 =  0%  ae-72-72.ebr2.NewYork1.Level3.net [4.69.148.37]
                                0/ 100 =  0%   |
 14   75ms     0/ 100 =  0%     0/ 100 =  0%  ae-5-5.car1.Montreal2.Level3.net [4.69.141.5]
                                0/ 100 =  0%   |
 15   93ms     0/ 100 =  0%     0/ 100 =  0%  ae-11-11.car2.Montreal2.Level3.net [4.69.141.1]
                                0/ 100 =  0%   |
 16   64ms     1/ 100 =  1%     1/ 100 =  1%  ORMUCO-COMM.car2.Montreal2.Level3.net [4.59.178.74]
                                0/ 100 =  0%   |
 17   67ms     1/ 100 =  1%     1/ 100 =  1%  192.34.76.10
                                0/ 100 =  0%   |
 18   67ms     1/ 100 =  1%     1/ 100 =  1%  199.91.189.242
                                0/ 100 =  0%   |
 19   66ms     0/ 100 =  0%     0/ 100 =  0%  199.91.189.25

Trace complete.

And here's a standard tracert, just to show some extra ping results for each hop... just took it at midnight. Still a little bit of weird Jitter I'd like to see tweaked, but I really shouldn't complain with results like this after what we've had the last 3 1/2 months:

Tracing route to 199.91.189.25 over a maximum of 30 hops

  4    27 ms    26 ms    26 ms  24.93.64.134
  5    39 ms    30 ms    30 ms  bu-ether24.atlngamq46w-bcr00.tbone.rr.com [107.14.19.18]
  6    54 ms    27 ms    26 ms  107.14.19.11
  7    28 ms    37 ms    28 ms  twc-telus.pr0.chi10.tbone.rr.com [66.109.9.70]
  8    58 ms    58 ms    60 ms  vlan51.ebr1.Atlanta2.Level3.net [4.69.150.62]
  9    48 ms    49 ms    49 ms  ae-6-6.ebr1.Washington12.Level3.net [4.69.148.106]
 10    50 ms    49 ms    50 ms  ae-1-100.ebr2.Washington12.Level3.net [4.69.143.214]
 11    52 ms    70 ms    52 ms  4.69.201.65
 12    51 ms    49 ms    50 ms  ae-71-71.csw2.NewYork1.Level3.net [4.69.134.70]
 13    50 ms    49 ms    49 ms  ae-72-72.ebr2.NewYork1.Level3.net [4.69.148.37]
 14    49 ms    49 ms    50 ms  ae-5-5.car1.Montreal2.Level3.net [4.69.141.5]
 15    58 ms    50 ms    50 ms  ae-11-11.car2.Montreal2.Level3.net [4.69.141.1]
 16    61 ms    58 ms    59 ms  ORMUCO-COMM.car2.Montreal2.Level3.net [4.59.178.74]
 17    64 ms    65 ms    66 ms  192.34.76.10
 18    64 ms    63 ms    65 ms  199.91.189.242
 19    63 ms    84 ms    63 ms  199.91.189.25

Trace complete.

I cut over to a new Zoom 5341J modem tonight so I can take TWC's modem back in the morning and stop that stupid leasing fee. If your ISP offers you the same option and you've been waffling over doing it... might be worth it, and not just for the $8 or so savings each month. Picked this thing up for $75 (with tax) at my local Best Buy tonight. Can't wait to see how the Roku's behave tomorrow night.

[warace]

i am having an issue here also im not a pro at networking but i am witnessing horrible "rubber band" and the trace route is:
1 <1 ms <1 ms <1 ms xxxxxxxxxxx
2 * * * Request timed out
3 11 ms 14 ms 10 ms 24.144.1.89
4 10 ms 11 ms 13 ms 66.194.86.233
5 27 ms 26 ms 25 ms 64.129.238.118
6 55 ms 54 ms 60 ms 4.53.234.17
7 53 ms 53 ms 63 ms 4.69.150.62
8 55 ms 53 ms 51 ms 4.69.148.106
9 68 ms 56 ms 58 ms 4.69.143.214
10 53 ms 53 ms 58 ms 4.69.201.61
11 52 ms 57 ms 52 ms 4.69.134.66
12 69 ms 52 ms 51 ms 4.69.148.33
13 51 ms 53 ms 55 ms 4.69.141.5
14 68 ms 73 ms * 4.69.141.1
15 69 ms 75 ms 65 ms 4.59.178.74
16 77 ms 66 ms 67 ms 192.34.76.10
17 68 ms 69 ms 70 ms 199.91.189.242
18 67 ms 67 ms 68 ms 199.91.189.57

Finished

it is the step 14 every time is this causing my issue?

[Raist]

That is Level3 service getting across Canada, so could very well be. They've been known offenders for years. There is also a possibility of problems at your gateway as well, depending on what you see happening frequently at hop #2. I am currently trying to get an issue addressed with my TWC residential gateway at the moment. I've found evidence of it happening on their business class gateways as well for several clients nearby, so it is looking like it may be a systemic issues in our area with TWC's lines in general. OR it could just be their worn out CMTS equipment that dates back to when Clinton was still in office--something they are supposedly in the process of updating to give everyone a speed increase this year. Won't know for sure until maintenance gets the bucket truck(s) into my neighborhood. Saw them out in force today a few miles away... keeping my fingers crossed.

C:\Windows\System32>tracert 199.91.189.25

Tracing route to 199.91.189.25 over a maximum of 30 hops

  1     2 ms    <1 ms    <1 ms  LPTSRV [10.10.100.1]
  2  1697 ms  1412 ms    72 ms  cpe-098-025-064-001.sc.res.rr.com [98.25.64.1]
  3    13 ms    14 ms    11 ms  cpe-024-031-198-005.sc.res.rr.com [24.31.198.5]
  4    26 ms    21 ms    26 ms  xe-7-0-0.rlghncpop-rtr1.southeast.rr.com [24.93.64.40]
  5    30 ms    29 ms    32 ms  107.14.19.44
  6    30 ms    32 ms    31 ms  ae-2-0.pr0.dca10.tbone.rr.com [66.109.6.169]
  7    35 ms    33 ms    36 ms  te0-16-0-23.ccr41.iad02.atlas.cogentco.com [154.54.10.209]
  8    33 ms    33 ms    35 ms  be2113.mpd21.dca01.atlas.cogentco.com [154.54.6.170]
  9    42 ms    40 ms    41 ms  be2150.mpd21.jfk02.atlas.cogentco.com [154.54.31.130]
 10    43 ms    42 ms    45 ms  be2106.ccr21.alb02.atlas.cogentco.com [154.54.3.50]
 11    66 ms    67 ms    65 ms  be2088.ccr21.ymq02.atlas.cogentco.com [154.54.43.17]
 12    56 ms    53 ms    54 ms  38.122.42.34
 13    44 ms    45 ms    45 ms  192.34.76.10
 14    46 ms    47 ms    46 ms  199.91.189.242
 15    46 ms    44 ms    46 ms  199.91.189.25

Trace complete.

C:\Windows\System32>tracert www.google.com

Tracing route to www.google.com [74.125.228.18]
over a maximum of 30 hops:

  1     2 ms     1 ms     1 ms  LPTSRV [10.10.100.1]
  2  2354 ms   618 ms    64 ms  cpe-098-025-064-001.sc.res.rr.com [98.25.64.1]
  3    13 ms    11 ms    11 ms  cpe-024-031-198-005.sc.res.rr.com [24.31.198.5]
  4    25 ms    22 ms    26 ms  24.93.64.128
  5    32 ms    31 ms    30 ms  107.14.19.44
  6    28 ms    31 ms    30 ms  107.14.19.133
  7    93 ms     *      111 ms  66.109.9.66
  8     *       30 ms     *     209.85.252.46
  9    42 ms    40 ms    42 ms  72.14.238.173
 10    36 ms    36 ms    36 ms  iad23s05-in-f18.1e100.net [74.125.228.18]

Trace complete.

See what happened there? When my gateway is all out of whack like that, I get timeouts. When it is behaving as it should (less than 30ms, often hovering in the teens or better), the game and video streaming is near flawless. But when it's spiking all over the map like this, it's rough going at best, unusable at worst.

There has also been some issues reported in the Arkansas area recently as well, which is where that third hop is registered (Conway Corporation). So there could be some wonkiness close to home that is kicking in and compounding an issue that might otherwise be recoverable on Level3's segment. Without ruling out an issue amongst your first few hops, hard to say that it definitely is Level3.

Either way though, there seems to be something going on along your route that your ISP could look into further for you--it's either in their gateway/CMTS (if cable), or on their partner's segments. May want to try some other sites as well like Google/YouTube or Yahoo to see if you see similar behavior. If you can demonstrate a similar pattern to another location, then there is definitely something your ISP needs to look into.

[warace]

caught it lagging hard again

1 <1 ms <1 ms <1 ms xxxxxxxxx
2 * * * Request timed out
3 21 ms 10 ms 11 ms 24.144.1.89
4 11 ms 10 ms 20 ms 66.194.86.233
5 29 ms 26 ms 28 ms 66.192.249.222
6 127 ms 57 ms 55 ms 4.53.234.17
7 95 ms 98 ms 86 ms 4.69.150.62
8 52 ms 61 ms 52 ms 4.69.148.106
9 90 ms 92 ms 98 ms 4.69.143.214
10 93 ms 92 ms 92 ms 4.69.201.61
11 64 ms 55 ms 53 ms 4.69.134.66
12 60 ms 52 ms 71 ms 4.69.148.33
13 52 ms 53 ms 51 ms 4.69.141.5
14 74 ms * * 4.69.141.1
15 81 ms 80 ms 82 ms 4.59.178.74
16 * 70 ms 70 ms 192.34.76.10
17 * 78 ms 72 ms 199.91.189.242
18 79 ms * * 199.91.189.57
19 * * 98 ms 199.91.189.57

Finished

ran a few extras i know for sure its not my router/modem which is "masking?" the report at #2 had a few time outs at 6 then at 14 + . is there someone i should send this info too? SE or ... idk where to even start

[Raist]

Yeah.. that's the rub. It really falls under your ISP's responsibility to address. They made the agreements that are putting you on that level3 segment... or rather, they've signed up to services whose analytics are determining that route. So, you have to pass the info to them so they can pass it on up the chain. Won't lie... it can get frustrating with some companies. When you call in, you get tier1 people that will look at your modem logs and see everything fine locally, and more often then not, that's pretty much it. They may have you reboot devices, call a tech in to check the quality of your lines...but all they are doing is what they have the power to do, check your local connectivity out.

When you contact them, let them know you've spotted signs of congestion or packet loss and you have some trace logs you want to forward to tier3 for review. You may get lucky and they give you an email address or a different number to call.

That is ultimately who you will need to investigate issues beyond your local wiring: tier3 support. Unfortunately, sometimes you have to go through those first 2 levels to prove to them it isn't something locally causing issues before they'll escalate it to tier3. It's good to go through the motions sometimes though--may get new cables or splitters, maybe even a new modem if you are renting. I've found I get better results with Time Warner now through their online options. Some have forums, some have a ticket generating system, or they may simply have an email or twitter account. Twitter is fun sometimes... they don't like to see bad publicity out there.

Not sure if www.conwaycorp.com is your ISP (24.144.1.89 at hop 3 is theirs), or someone they are using as an exchange, but there are some named contacts in the whois/ARIN data you might be able try if that is your ISP and you aren't getting anywhere with the normal channels. Here's some links to those digs if you want the addresses and phone numbers:

http://myip.ms/info/whois/24.144.1.89
http://whois.arin.net/rest/net/NET-24-144-0-0-1/pft