The only place where AFK/idle for a while bothers me is out in the field. (seeing a miner just idle out near their last node or people just generally standing idle out in the open because I'm not sure if they're benefiting from anything)

In towns with Aetherites I see a reason...the idle bonus for exp (which is what I do), I make sure I'm in a sanctuary before idling. I rarely idle outside of towns (if I do I'm usually doing DoL stuff).