UPDATE for everyone: I tried connecting from my tethered phone...and voila! It always works!

I had my suspicions it had to do with the ISP. At least for me that seems to be the problem...