Banditry varies from region to region. In La Nosca, there isn't a lot of banditry based FATES (most FATEs are wild beast related), and the few that are specify that the bandits are pirates that have lost their pirate ships. I'd assume most La Noscian banditry happens on the water where we can't participate in FATEs. In Gridania, a lot of the bandits are people who don't want to follow the Elemental's laws in addition to those who are just normal bandits. Ul'dah seems to feature the most typical bandits of the lot.