When I had my where are the rest of my orcs problem in jugner they were going in a circle or something on wide scan until I approached them. Then they very slowly made their way to the fortification without agroing me. It's like they went to the bar and were so hung over they forgot they needed to go fight the next day. I think something similar is happening with the yags at the fort.