Ok, so the unifying thing seems to be our ISP, two of them seem to be the culprits...

BrightHouse and Comcast