I'm on Chaos DC in EU, I've done some digging.
Latency seems to increase over the course of the day, so it seems like a congestion issue.
Traceroute shows the exact pont where its happening, I even contacted the provider, but as I'm not the service owner, they cannot investigate.
VPN helps out.