They actually indirectly touched on this in the whole Nanamo thing with the transition from ARR to HW.. Not as direct as you're talking about... I do agree it would make for an interesting plot point - people getting to the point of fearing us.

I will admit I am getting a little tired of everyone basically kissing our feet whenever we move a pebble for them. I wouldn't mind the odd mistake, such as killing the wrong person and finding out they were innocent or something :P