Major Latency Problems

I'm having huge latency problems on the Midgardsormr server. According to the in-game connection status, latency is up to 500 sending and 4000 receiving, even in lightly populated areas. I press a button and get a response seconds later. NPC's take 30 seconds to a minute to appear. I checked my internet connection and it seems to be fine. I'm not having problems on the other online games I play.

Is there anything I can do? Is anyone else having this problem?

Thanks!