People use the term "friend" way too lightly. If they refuse to go with you, for whatever reason that may be, I'd say "associate" is more fitting.