It's possible that it's neither your ISP or SE, but instead a bad node you have to pass through in between that they don't have direct control over. Have you ever done a traceroute to see where the connection is being lost?