Hypothetical situation: I remove someone who I've either forgotten or don't recognize from my friend list. They don't remove me from theirs. At some point I'm standing near them, but I don't realize it because they're not highlighted. However, I'm still on their friend list, so I'm highlighted for them. I'm not interacting with them. They feel ignored or snubbed. They contact me. I treat them like a stranger. They feel snubbed. Everyone is having a bad time. End of hypothetical situation.

So back in the real world, I never remove anyone from my friend list because that would mean their names wouldn't be highlighted when they're nearby. And if their names aren't highlighted when they're nearby, and I can't see them first and run into an entirely different area to mitigate the potential for any interaction ever again.

I've played several games that allow you to write a short note about each contact in your friend list. I would find such a feature helpful to remind me who these people I presumably once shared a connection with are. ^^

Also, removing someone from your friend list should remove you from theirs. That would address the stalking problem while still allowing OP to better keep track of their online contacts.