In the Main Scenario, you fight a Siren on the beaches of the Isles of Umbra that later becomes the dungeon boss for Pharos Sirius. The reason the locals are so freaked is that allegedly the sirens are supposed to all be dead - the pirate king Mistbeard killed them all in a great culling when he ruled the high seas. The half-cracked Lalafellin smithy Mimidoa claims to have been a part of this culling; he's been claiming this since 1.0, and it always seemed like one of his many tall tales. Suddenly, here you realize that it's obviously, comically true: his specialized earplugs help you fight the siren without being charmed.
Now, a different legend about Mistbeard claims that he once raided the wedding of Nanamo Ul Namo's parents, making off with a ton of gold and allegedly the (willing) virginity of the bride. That last part is probably an embellishment added as the story spread through ships and pubs, of course, but take a look at the map next time you're in Sastasha. If you look closely, the final room (the dock where ye ... weight your anchor) is called The Sultana's Lap. Once upon a time, it was one of Mistbeard's many treasure coves.
Eventually (Leviathan / Hullbreaker Island), you learn Mistbeard's identity, which adds a whole new layer onto his legend and connects to the legends of admiral Merlwyb and her father, as their pirate crew was a bitter rival of Mistbeard's. When you understand the relationship that the two now share, you then understand in a new way how Limsa Lominsa has evolved in becoming a legitimate nation state to ally against the Garlean Empire. And on and on it goes.
It spirals downwards and interconnects, from obvious ties to subtle hints. When you have all of that buried in a main narrative that itself is pretty good, people are willing to fawn over it a bit, lol. Not everyone, but enough to explain why you've read what you've read, even if you don't see what they saw.