For small shards, you will know when you have touched them all if you have access to teleporting directly outside the gates (There'll be a notification you can, and additional areas added to the list).

As for which is which? The majority are right next to the guilds and say so straight up (eg "Conjurer's Guild" or "Marauder's Guild" etc); the remaining few are locations you will very quickly learn (there's only about two per town - for instance, "Gridania Aetheryte Plaza" is the Old Gridania northeast, and used to get to most places on that map, while "Mih Khetto's Amplitheatre" is up in New Gridania and roughly centered there).

It's something you'll learn pretty quickly honestly.