For story reasons, it'd be a really, really bad idea to plop an aetheryte in the middle of Vesper Bay. It's like asking your enemies to come walking through the front door.

In terms of gameplay, I'm going with what everyone else says. Chocobo porter from Horizon, go do something for a minute, then you'll be there, or take the ferry route.