Alright, so I managed to resolve the issue by reducing my mouse's polling rate from 1000 to 125 Hz. Now the game is unaffected FPS-wise by mouse movement.