It's worth noting that games are more sensitive to network trouble than most other internet applications. Video and music will buffer data ahead of time in your computer and keep playing when the connection has trouble. You typically won't notice a web page taking 300ms rather than 90ms to respond.

Your impairment appears to be intermittent also so you might not notice it at all with other internet applications. Games just make these sorts of things more obvious. You could always try a traceroute to google rather than SE when the issue starts. You'll likely see high pings to them also.