I have had that problem when I was playing on DX9. Since I switched to DX11, Afterburner has been recording and screenshotting perfectly in windowed mode. (It also recently updated, don't know if that's related.)