Hrm... I had the mind to add one thing here, mentioning that when I updated my drivers (using an AMD card) the issue here vanished (the particular 'graphics lag when panning' issue). Howevre, Ranebow, what you just said made me think...
That updated improved my FPS as a whole, and now that you mention occlusion culling I think I might know WHY the panning suddenly drops your FPS and it's quite... Logical.

Occlusion culling doesn't renders things which are offscreen, that saves you on processing power and memory but it ALSO requires you to suddenly load and render everything outside your view range when you pan your screen.
Which means that when that is on you are causing a sudden spike in processing/memory requirements by asking a lot of new things to be rendered suddenly, causing a huge drop in FPS temporarily. Notice how your issue is worse the more things are there around you? (try that in Revenant's Toll, then try that in, say... Urth's Gift? Or, hum... Nym? Somewhere that is kinda barren playerwise)