I think that it can be assumed that there are other NPC adventurers around at every stage of the game, doing odd jobs and exploring dungeons (likely not the same dungeons we do) and whatnot. Our paths don't usually intersect with theirs, and they serve only to prop up "Adventuring" as a well-known career option - a career at which our character happens to excel. We're the best of the best, so it's not unexpected that NPCs may have run into other adventurers that don't measure up to our awesomeness, and prejudge us based on those bad experiences.

There are, of course, places where adventurers are not welcome; Ishgard, for example is isolationist and kept all foreigners out (including adventurers). We were a special exception. I don't believe that any of the NPCs in Ishgard complained about such-and-such adventurers they met.