Haha, so I'm not the only one!

Both, actually. If it's someone I know, I'm pretty obvious about it. Even with strangers, if I know them to be socially very interactive.

If I don't want the other one to notice I am spying on them I stand as far away from them as possible so they don't notice me, or if it's close to an NPC or a retainer bell I just click on a few other people too in rapid succession, to make it seem as if I was just going through tab targets xD