sounds like your config file got knackered somehow? Can try manually editing the resolution values. It's nested under your documents folder, under My Games. Default path is like this:

C:\Users\<UserName>\Documents\My Games\FINAL FANTASY XIV - A Realm Reborn

It's a plain text file called FFXIV.cfg and can be opened in any standard text editor. Note that depending on your file browser settings, you may not see the full name...in which case you'd be looking for FFXIV with no extension.

There will be a section with two sets of screen height/width dimensions, one will be for use when windowed and the other for FullScreen. Make sure they are saved with valid values for your screen.