I don't think the fear (for lack of a better term) of losing XP had anything to do what people I chose to play with. It's a no brainer that if you have the opportunity to play with people that you had a good experience with you'll do so again, but leveling up was such an integral part of FFXI that you basically took what you got in terms of parties so you could progress through the game.