This often gets tripped up because of invalid (or null) registry keys. If that's it, the simple fix for it is to go into the FFXI config and set/save valid resolution settings (might need to change them, then flip them back so it saves on exit if it already shows the settings you wanted).

Otherwise, there may be an issue with your hardware/driver/DirectX/system settings causing an issue.