Not everyone is avoided, usually because they have a persona that isn't like some sort of tedious old man. Desire is equivalent to occurrence percentage. If people don't bother you its because they'd rather avoid you like the plague. A simple enough concept; one that aligns with the same issues real world celebrities 'enjoy' these days.
Only thing that will save you is developing a thick skin and refusing to play their game. Troll them until they stop, the fire with fire strategy. If that's too tedious, surround yourself with those who have a forceful persona and develop a bubble. Safety lies in co-operative numbers. You won't find that in the complacent incompetence of SE.