It's the nodes on the way to the server to California. Honestly the only way SE will ever fix this issue is to move the servers to a more central location in NA since they have no control over the nodes along the way to the state.

I get constant issues during prime time as well from Comcast.

Yes my ports are forwarded.

Yes the game is the only program running.

No it's not my ISP's fault.

It's the server location.