I have a steady 72-74ms to my server, it hardly ever spikes, the server itself has a 0.1s (100ms) so it means the only delay should be 172-174ms, which is less than 0.5s, yet this keeps happening.

At this point I just slow down a bit to not fail it but that already means a DPS loss.