This is a shining example of horrible game design. Doing a small casual event like Skirmish should NOT be this annoying.
You only need to travel to the site once. Then you can teleport there. I don't think a little trouble on a one time trip is a huge deal.