Google'd and found someone has the same problem, he got advised to get more ram.
But my PC has enough ram. I don't have problem with Diablo 3 as well.