I've found with controller that there's the r1 button filtering you can setup, and in that there's an enemies only setting that doesn't get people's pets. I don't normally see people with their summons out near the retainer bell. Does the Other/NPC option work to only get the bell in that situation? When I'm in town, I put it on a filter that is only the other/npc option with all others off.