Severe rubberbanding,Fios user and I included traceroutes

[Darthenron]

Been having some severe rubber-banding and went to trace-route to see what was going on. Hoping this info could possible help solve whatever has happened to the servers since the 2.1 update introdced rubberbanding to fios players.
For those who don't know how to read it the numbers 1-16 show the hops or how the data is flown and the numbers are 3 attempts to get how long it took. It seems all the level3 stuff is taking forever.

1 63 ms 1 ms 3 ms Wireless_Broadband_Router.home [192.168.1.1]
2 58 ms 6 ms 4 ms L100.DLLSTX-VFTTP-53.verizon-gni.net [173.71.42.1]
3 8 ms 8 ms 11 ms G0-3-1-1.DLLSTX-LCR-22.verizon-gni.net [130.81.107.118]
4 34 ms 59 ms 58 ms so-4-1-0-0.DFW9-BB-RTR2.verizon-gni.net [130.81.51.244]
5 8 ms 8 ms 31 ms 0.xe-3-0-3.BR2.DFW13.ALTER.NET [152.63.100.69]
6 16 ms 8 ms 8 ms ae8.edge10.dallas3.level3.com [4.68.62.33]
7 48 ms 58 ms 49 ms vlan70.csw2.Dallas1.Level3.net [4.69.145.126]
8 50 ms 69 ms 51 ms ae-71-71.ebr1.Dallas1.Level3.net [4.69.151.138]
9 49 ms 64 ms 52 ms ae-14-14.ebr2.Chicago2.Level3.net [4.69.151.117]
10 50 ms 50 ms 48 ms ae-5-5.ebr2.Chicago1.Level3.net [4.69.140.193]
11 49 ms 49 ms 49 ms ae-1-100.ebr1.Chicago1.Level3.net [4.69.151.177]
12 51 ms 51 ms 49 ms ae-10-10.car2.Montreal2.Level3.net [4.69.153.86]
13 49 ms 49 ms 50 ms ORMUCO-COMM.car2.Montreal2.Level3.net [4.59.178.74]
14 56 ms 52 ms 52 ms 192.34.76.10
15 52 ms 51 ms 51 ms 199.91.189.242
16 56 ms 57 ms 64 ms 199.91.189.57

1 5 ms 3 ms 3 ms Wireless_Broadband_Router.home [192.168.1.1]
2 5 ms 6 ms 21 ms L100.DLLSTX-VFTTP-53.verizon-gni.net [173.71.42.1]
3 7 ms 8 ms 13 ms G0-3-1-1.DLLSTX-LCR-22.verizon-gni.net [130.81.107.118]
4 8 ms 8 ms 10 ms so-4-1-0-0.DFW9-BB-RTR2.verizon-gni.net [130.81.151.244]
5 8 ms 7 ms 7 ms 0.xe-3-0-3.BR2.DFW13.ALTER.NET [152.63.100.69]
6 8 ms 11 ms 9 ms ae8.edge10.dallas3.level3.com [4.68.62.33]
7 60 ms 50 ms 48 ms vlan70.csw2.Dallas1.Level3.net [4.69.145.126]
8 54 ms 50 ms 50 ms ae-71-71.ebr1.Dallas1.Level3.net [4.69.151.138]
9 49 ms 50 ms 50 ms ae-14-14.ebr2.Chicago2.Level3.net [4.69.151.117]
10 50 ms 56 ms 49 ms ae-5-5.ebr2.Chicago1.Level3.net [4.69.140.193]
11 51 ms 49 ms 49 ms ae-1-100.ebr1.Chicago1.Level3.net [4.69.151.177]
12 49 ms 48 ms 69 ms ae-10-10.car2.Montreal2.Level3.net [4.69.153.86]
13 50 ms 49 ms 48 ms ORMUCO-COMM.car2.Montreal2.Level3.net [4.59.178.74]
14 69 ms 52 ms 51 ms 192.34.76.10
15 54 ms 51 ms 54 ms 199.91.189.242
16 57 ms 57 ms 57 ms 199.91.189.57

1 2 ms 1 ms 1 ms Wireless_Broadband_Router.home [192.168.1.1]
2 5 ms 8 ms 5 ms L100.DLLSTX-VFTTP-53.verizon-gni.net [173.71.42.1]
3 11 ms 7 ms 8 ms G0-3-1-1.DLLSTX-LCR-22.verizon-gni.net [130.81.107.118]
4 10 ms 10 ms 7 ms so-4-1-0-0.DFW9-BB-RTR2.verizon-gni.net [130.81.151.244]
5 8 ms 8 ms 9 ms 0.xe-3-0-3.BR2.DFW13.ALTER.NET [152.63.100.69]
6 11 ms 11 ms 10 ms ae8.edge10.dallas3.level3.com [4.68.62.33]
7 49 ms 50 ms 49 ms vlan70.csw2.Dallas1.Level3.net [4.69.145.126]
8 50 ms 51 ms 51 ms ae-71-71.ebr1.Dallas1.Level3.net [4.69.151.138]
9 49 ms 50 ms 49 ms ae-14-14.ebr2.Chicago2.Level3.net [4.69.151.117]
10 49 ms 49 ms 50 ms ae-5-5.ebr2.Chicago1.Level3.net [4.69.140.193]
11 50 ms 51 ms 50 ms ae-1-100.ebr1.Chicago1.Level3.net [4.69.151.177]
12 227 ms 208 ms 211 ms ae-10-10.car2.Montreal2.Level3.net [4.69.153.86]
13 50 ms 49 ms 49 ms ORMUCO-COMM.car2.Montreal2.Level3.net [4.59.178.74]
14 52 ms 52 ms 51 ms 192.34.76.10
15 107 ms 62 ms 52 ms 199.91.189.242
16 57 ms 57 ms 56 ms 199.91.189.57

[Jamal]

Before 2.1 my trace was pretty straightforward from london to new york to montreal and I didn't go through a single level3 router. Didn't have rubberbanding and latency was tad high but constant.

2.1
1 <1 ms <1 ms <1 ms KOTIBOKSI [192.168.100.1]
2 20 ms 14 ms 14 ms priova1.fi.elisa.net [88.112.160.1]
3 15 ms 14 ms 15 ms 6-2-1.raurau1.fi.elisa.net [139.97.19.189]
4 15 ms 16 ms 15 ms ge1-1-0.rairlu-p1.fi.elisa.net [139.97.19.185]
5 17 ms 17 ms 17 ms ae0.helpa-p1.fi.elisa.net [139.97.0.201]
6 * 17 ms 17 ms ae2.helpa-gw1.fi.elisa.net [139.97.6.250]
7 17 ms * 17 ms ae1-10.bbr1.hel2.fi.eunetip.net [213.192.191.53]
8 23 ms 24 ms 24 ms so7-0-0-0.bbr1.sto1.se.eunetip.net [213.192.191.210]
9 24 ms 24 ms 24 ms xe-10-2-2.bar1.Stockholm1.Level3.net [213.242.110.169]
10 141 ms 141 ms 141 ms ae-212-3602.ebr1.Berlin1.Level3.net [4.69.159.6] dafug is happening here?
11 142 ms 142 ms 142 ms ae-1-100.ebr2.Berlin1.Level3.net [4.69.148.206]
12 141 ms 142 ms 141 ms ae-26-26.ebr2.Dusseldorf1.Level3.net [4.69.200.165]
13 142 ms 141 ms 141 ms ae-45-45.ebr1.Amsterdam1.Level3.net [4.69.143.197]
14 142 ms 141 ms 141 ms ae-21-21.ebr2.Amsterdam1.Level3.net [4.69.143.54]
15 144 ms 141 ms 142 ms ae-48-48.ebr2.London1.Level3.net [4.69.143.82]
16 142 ms 141 ms 142 ms ae-41-41.ebr1.NewYork1.Level3.net [4.69.137.66]
17 142 ms 142 ms 141 ms ae-81-81.csw3.NewYork1.Level3.net [4.69.134.74]
18 141 ms 142 ms 141 ms ae-82-82.ebr2.NewYork1.Level3.net [4.69.148.41]
19 142 ms 142 ms 142 ms ae-5-5.car1.Montreal2.Level3.net [4.69.141.5]
20 320 ms 243 ms 232 ms ae-11-11.car2.Montreal2.Level3.net [4.69.141.1] Pinging this address shows that it has no consistency what-so-ever.
21 142 ms 142 ms 142 ms ORMUCO-COMM.car2.Montreal2.Level3.net [4.59.178.74]
22 138 ms 137 ms 137 ms 192.34.76.10
23 137 ms 137 ms 137 ms 199.91.189.242
24 137 ms 137 ms 137 ms 199.91.189.22

Why does my connection play ping-pong in europe before going over to the other side? Did my ISP change something overnight? I dont think so.

[Rivienne]

I too have been having problems, also fios. And like the person on the other thread, I too am mostly seeing this evening only, specifically after 5PM est. (It may start earlier, but after 5pm is when I play most) Mornings I rarely have problems.

Here is the trace on my side and as you noted, I also see the problem is in level3 somewhere:

11 ae-72-72.ebr2.newyork1.level3.net (4.69.148.37) 23.896 ms
ae-82-82.ebr2.newyork1.level3.net (4.69.148.41) 25.194 ms
ae-92-92.ebr2.newyork1.level3.net (4.69.148.45) 24.406 ms
12 ae-5-5.car1.montreal2.level3.net (4.69.141.5) 385.623 ms 582.139 ms 408.990 ms
13 ae-11-11.car2.montreal2.level3.net (4.69.141.1) 32.451 ms 24.717 ms 24.453 ms
14 ormuco-comm.car2.montreal2.level3.net (4.59.178.74) 24.675 ms 21.997 ms 25.229 ms

If you recall, right after 2.1 there was a NA only maintenance for the datacenter ISP. I think this problem isn't on SE's end, but the ISP provider for the NA data center but hopefully SE can get the right people to solve this. I have been unable to try any of the new content because of this problem. I can't do anything past Ifrit HM anymore unless I do it first thing in the morning when I rarely have enough time.

[Lurkios]

I'm getting some pretty severe packet loss on the ae-11-11.car2.Montreal2.Level3.net jump as well, coming from East Coast USA via Verizon FiOS. Didn't have the problem since 2.1, and haven't been able to do much since. Disappearing Garudas, teleporting Titans, etc. etc.

[BlueBanana]

SE never replied to the sticky thread with all the FiOS complaints.. that 2.1 network hotfix started the rubberbanding issue, forced me to play on VPN


Tracing route to traceip.ffxiv.com [202.67.52.81]
over a maximum of 50 hops:

1 <1 ms * <1 ms DD-WRT [192.168.1.1]
2 1 ms * 5 ms L100.NYCMNY-VFTTP-173.verizon-gni.net [71.183.95
.1]
3 5 ms * 4 ms G0-13-2-6.NYCMNY-LCR-21.verizon-gni.net [130.81.
139.76]
4 3 ms * 40 ms ae3-0.NY325-BB-RTR1.verizon-gni.net [130.81.199.
162]
5 69 ms 70 ms 70 ms 0.xe-7-1-0.XL3.LAX15.ALTER.NET [152.63.113.90]
6 79 ms 71 ms 71 ms TenGigE0-6-0-2.GW5.LAX15.ALTER.NET [152.63.30.10
1]
7 69 ms 69 ms 70 ms kddia-gw.customer.alter.net [152.179.21.114]
8 69 ms 127 ms 69 ms lajbb002.int-gw.kddi.ne.jp [59.128.2.21]
9 178 ms 178 ms 178 ms otejbb206.int-gw.kddi.ne.jp [203.181.100.21]
10 171 ms 171 ms 172 ms jc-ote301.int-gw.kddi.ne.jp [118.155.197.178]
11 185 ms 200 ms 187 ms 106.187.28.198
12 220 ms 187 ms 187 ms 61.195.56.133
13 171 ms 171 ms 172 ms 219.117.144.74
14 164 ms 164 ms 163 ms 219.117.144.45
15 171 ms 172 ms 171 ms 219.117.146.129
16 162 ms 163 ms 163 ms 219.117.146.46
17 168 ms 171 ms 171 ms 202.67.52.81

Trace complete.

[Raist]

It's looking more and more like everyone else's traces.... showing it's an issue with who your ISP has partnered with for carrying your data off their local network over to Ormuco (SE's ISP). In the majority of cases it's been TATA, Cogent, or level3. We had i-web and a few others in the mix at one time as well, but for the most part what's been reported are those first three.

Need to forward your findings to your ISP and request tier3 support or higher to look into it further on your behalf. This is an issue with someone your ISP has contracted with as an extension of your service contract (in a round-a-bout way), and thus have a responsibility to work on resolving the problem on your behalf.

[Wobbler]

I played on excalibur (or should I say I am trying to play on excalibur.

I went to FFIV:ARR Server Status and grabbed the IP of my server (http://www.arrstatus.com)

I then used a tool that I used in the past when playing another online game when I experienced issues with their Texas based data center. The tool is called WinMTR.

I did a set of 100 tests and this was the result;

|------------------------------------------------------------------------------------------|
| WinMTR statistics |
| Host - % | Sent | Recv | Best | Avrg | Wrst | Last |
|------------------------------------------------|------|------|------|------|------|------|
| Wireless_Broadband_Router.home - 0 | 100 | 100 | 0 | 3 | 90 | 1 |
| L101.VFTTP-21.NYCMNY.verizon-gni.net - 0 | 100 | 100 | 6 | 12 | 194 | 9 |
| G0-6-5-6.NYCMNY-LCR-21.verizon-gni.net - 0 | 100 | 100 | 7 | 13 | 99 | 10 |
| ae3-0.NY325-BB-RTR2.verizon-gni.net - 0 | 100 | 100 | 6 | 16 | 192 | 22 |
| 0.xe-3-2-0.BR2.NYC4.ALTER.NET - 0 | 100 | 100 | 30 | 42 | 102 | 40 |
| 204.255.173.54 - 0 | 100 | 100 | 26 | 36 | 195 | 40 |
| vlan52.ebr2.NewYork2.Level3.net - 0 | 100 | 100 | 35 | 42 | 192 | 40 |
| ae-45-45.ebr2.NewYork1.Level3.net - 0 | 100 | 100 | 35 | 41 | 103 | 42 |
| ae-5-5.car1.Montreal2.Level3.net - 0 | 100 | 100 | 33 | 66 | 341 | 38 |
| ae-11-11.car2.Montreal2.Level3.net - 23 | 53 | 41 | 0 | 47 | 386 | 18 |
| ORMUCO-COMM.car2.Montreal2.Level3.net - 0 | 100 | 100 | 37 | 44 | 198 | 40 |
| 192.34.76.10 - 8 | 77 | 71 | 33 | 39 | 76 | 38 |
| 199.91.189.242 - 4 | 88 | 85 | 33 | 47 | 302 | 44 |
| 199.91.189.38 - 0 | 100 | 100 | 32 | 37 | 187 | 36 |
|________________________________________________|______|______|______|______|______|______|
WinMTR v0.92 GPL V2 by Appnor MSP - Fully Managed Hosting & Cloud Provider

The bolded lines are kinda interesting. So it appears that for me (FIOS in NYC) the issue is both Level3 and inside squares network.

Does anyone know if square is looking into this or just assuming that it is some P2P shaping.

[Raist]

I played on excalibur (or should I say I am trying to play on excalibur.

I went to FFIV:ARR Server Status and grabbed the IP of my server (http://www.arrstatus.com)

I then used a tool that I used in the past when playing another online game when I experienced issues with their Texas based data center. The tool is called WinMTR.

I did a set of 100 tests and this was the result;

|------------------------------------------------------------------------------------------|
| WinMTR statistics |
| Host - % | Sent | Recv | Best | Avrg | Wrst | Last |
|------------------------------------------------|------|------|------|------|------|------|
| Wireless_Broadband_Router.home - 0 | 100 | 100 | 0 | 3 | 90 | 1 |
| L101.VFTTP-21.NYCMNY.verizon-gni.net - 0 | 100 | 100 | 6 | 12 | 194 | 9 |
| G0-6-5-6.NYCMNY-LCR-21.verizon-gni.net - 0 | 100 | 100 | 7 | 13 | 99 | 10 |
| ae3-0.NY325-BB-RTR2.verizon-gni.net - 0 | 100 | 100 | 6 | 16 | 192 | 22 |
| 0.xe-3-2-0.BR2.NYC4.ALTER.NET - 0 | 100 | 100 | 30 | 42 | 102 | 40 |
| 204.255.173.54 - 0 | 100 | 100 | 26 | 36 | 195 | 40 |
| vlan52.ebr2.NewYork2.Level3.net - 0 | 100 | 100 | 35 | 42 | 192 | 40 |
| ae-45-45.ebr2.NewYork1.Level3.net - 0 | 100 | 100 | 35 | 41 | 103 | 42 |
| ae-5-5.car1.Montreal2.Level3.net - 0 | 100 | 100 | 33 | 66 | 341 | 38 |
| ae-11-11.car2.Montreal2.Level3.net - 23 | 53 | 41 | 0 | 47 | 386 | 18 |
| ORMUCO-COMM.car2.Montreal2.Level3.net - 0 | 100 | 100 | 37 | 44 | 198 | 40 |
| 192.34.76.10 - 8 | 77 | 71 | 33 | 39 | 76 | 38 |
| 199.91.189.242 - 4 | 88 | 85 | 33 | 47 | 302 | 44 |
| 199.91.189.38 - 0 | 100 | 100 | 32 | 37 | 187 | 36 |
|________________________________________________|______|______|______|______|______|______|
WinMTR v0.92 GPL V2 by Appnor MSP - Fully Managed Hosting & Cloud Provider

The bolded lines are kinda interesting. So it appears that for me (FIOS in NYC) the issue is both Level3 and inside squares network.

Does anyone know if square is looking into this or just assuming that it is some P2P shaping.

Looks like you may have some stability issues earlier in your route that may be compounding along the way, ultimately affecting things at the endpoint as well. Remember, you pass through all previous hops getting to the next one down the line (test hop 1, then goes through 1 to test 2, then through 1 and 2 to test 3, etc.). It's kind of hard to read that report as laid out, may be able to line it up better if you use the code tags. See if I can clean it up a bit so we can see it better:

 WinMTR statistics |
| Host - 					% | Sent|Recv |Best|Avr|Wrs|Last|
|------------------------------------------------|------|-----|----|---|---|----|
| Wireless_Broadband_Router.home - 		0 | 100 | 100 | 0 | 3 | 90 | 1 |
| L101.VFTTP-21.NYCMNY.verizon-gni.net -	0 | 100 | 100 | 6 | 12 | 194 | 9 |
| G0-6-5-6.NYCMNY-LCR-21.verizon-gni.net - 	0 | 100 | 100 | 7 | 13 | 99 | 10 |
| ae3-0.NY325-BB-RTR2.verizon-gni.net - 	0 | 100 | 100 | 6 | 16 | 192 | 22 |
| 0.xe-3-2-0.BR2.NYC4.ALTER.NET - 		0 | 100 | 100 | 30 | 42 | 102 | 40 |
| 204.255.173.54 - 				0 | 100 | 100 | 26 | 36 | 195 | 40 |
| vlan52.ebr2.NewYork2.Level3.net - 		0 | 100 | 100 | 35 | 42 | 192 | 40 |
| ae-45-45.ebr2.NewYork1.Level3.net - 		0 | 100 | 100 | 35 | 41 | 103 | 42 |
| ae-5-5.car1.Montreal2.Level3.net - 		0 | 100 | 100 | 33 | 66 | 341 | 38 |
| ae-11-11.car2.Montreal2.Level3.net - 		23 | 53 | 41 | 0 | 47 | 386 | 18 |
| ORMUCO-COMM.car2.Montreal2.Level3.net - 	0 | 100 | 100 | 37 | 44 | 198 | 40 |
| 192.34.76.10 - 				8 | 77 | 71 | 33 | 39 | 76 | 38 |
| 199.91.189.242 - 				4 | 88 | 85 | 33 | 47 | 302 | 44 |
| 199.91.189.38 - 				0 | 100 | 100 | 32 | 37 | 187 | 36 |

Pay close attention to the wide variances along your route. Best/Worst, versus your average at each hop. You've got a lot of jumping around with your latency well before you get to SE...your worst is frequently 3 or more times your best (as well as your average) at many hops. Something strange going on in general there that needs to be looked at more closely by higher end support with your ISP.

[Wobbler]

Raist as much as I appreciate the formatting of the thread, I think that packetloss is much more important that poor worst results. If the average was closer to worst than best I could see that as being an issue however the majority of the team the worst is far from the average.

23% packetloss is not something that is acceptable, even 8% packetloss (inside square by the looks of things) is very borderline.

Bottom line is this, if square want people to continue with their game especially those who are only doing endgame PvP / PvE they need to start the conversations with the ISP / Peering networks now rather than putting this back on the end users.

[ZohnoReecho]

it's also possible that you lost those packets inside square because e-11-11.car2.Montreal2.Level3.net gave issues.