Do players have permission to make the ffxiv.exe process large address aware?

[KoolAidPitcher]

Hello Square-Enix, I recently submitted a bug report about the ffxiv.exe process not being large address aware and crashing when it attempts to utilize >2GiB of virtual memory; however, this bug report was moved to the "not a bug report" forum. You can see the original report here: http://forum.square-enix.com/ffxiv/t...-address-aware

The problem is reproducible when one is playing the game on a 4K display (3840x2160 resolution), as a larger display requires a larger raster and more memory. As computer displays continue to become higher resolution, this will be a larger problem.

I can hypothetically fix this problem myself by modifying the ffxiv.exe file using the Microsoft COFF/PE Editor to make the process large address aware. Large address awareness uses an unsigned 32-bit integer to address virtual memory instead of a signed 32-bit integer. This allows the process to address 4GiB of memory (0x00000000 - 0xFFFFFFFF) as opposed to 2GiB (0x00000000 - 0x7FFFFFFF). There is more technical information included in my original forum post.

My question is whether I can make this change to prevent the game from crashing on my computer. According to the terms of service:

2.6 Hacking and Circumvention. You may not hack, disassemble, decompile, or otherwise modify the Game or server computer code, whether the Game code is located on a DVD, Blu-rayTM disc, your computer/console or on Square Enix's servers, except as expressly permitted by Square Enix or applicable law.

Can players be granted permission to make the process large address aware? For example, Blizzard has granted World of Warcraft players the ability to modify their executable. (See: http://us.battle.net/wow/en/forum/to...8706617?page=6)

It would also be awesome if the game's executable was large address aware by default, so we wouldn't have to patch it ourselves. The patch has no side-effects except for being able to address an additional 2GiB of virtual memory, and has been a part of World of Warcraft since 4.1+.

Note that I am not actually using the patched files while playing the game, as that could be potentially considered against the terms of service since I modified the binary. I am simply requesting to use an interim solution while we are waiting for a 64-bit version of FFXIV to be released.

[Raist]

Issues bound to legacy 32-bit OS technologies. More specifically DX9 from circa Windows XP. With any luck much of our oddball issues like this may get addressed with he coming DX11 client.

[KoolAidPitcher]

Issues bound to legacy 32-bit OS technologies. More specifically DX9 from circa Windows XP. With any luck much of our oddball issues like this may get addressed with he coming DX11 client.

Hi Raist. The specific issue I am seeing could indeed be resolved by using DirectX 10 or higher. Unfortunately, Final Fantasy XIV is a DirectX 9 application. Microsoft has a knowledge base article explaining how Direct3D applications allocate virtual address space which pretty much summarizes the problem I am having. Also, as Microsoft notes at the bottom of the knowledge base article, applications will eventually reach the 2GB virtual address space limit on their own, for other reasons.

It would be awesome if I could get a response as to whether I can self-patch the executable to be large address aware, or whether Square Enix will acknowledge there is a bug instead of moving it to the "not a bug report" forum. Either one will help toward fixing the problem...

According to Microsoft's excellent documentation (https://support.microsoft.com/en-us/kb/940105):

On a modern operating system such as Windows Vista, applications run within their own private virtual address space. Typically, the size of the virtual address space is fixed at 2 gigabytes (GB) for 32-bit applications. How much virtual address space is available is not related to how much physical memory there is on the computer.

Every memory allocation, file mapping, or library that is loaded by an application consumes space in this virtual address space. When the application consumes all its virtual address space, any additional such operations fail. Although all applications should be coded to handle memory allocation failures, many applications do not recover correctly from such failures. Therefore, the programs may become unstable or stop responding after they recover from such failures.

Existing games and other graphics applications frequently allocate virtual memory for a copy of the video memory resources that the application uses. The application uses this copy to restore the display quickly if the contents of video memory are lost. For example, the application uses this copy if the user presses ALT+TAB or if the user puts the computer in standby. Typically, the DirectX run time manages the copy on behalf of the application when the application creates a managed resource. However, an application can also manage the copy itself. The virtual memory that the copy uses is directly proportional to the video memory resources that the application allocates.

A modern graphics processing unit (GPU) can have 512 MB or more of video memory. Applications that try to take advantage of such large amounts of video memory can use a large proportion of their virtual address space for an in-memory copy of their video resources. On 32-bit systems, such applications may consume all the available virtual address space.

With the introduction of DirectX 10 and Windows Display Driver Model (WDDM) in Windows Vista, it is no longer necessary for an application to maintain a copy of its resources in system memory. Instead, the video memory manager makes sure that the content of every video memory allocation is maintained across display transitions. For compatibility reasons, Windows Vista emulates "device lost" for DirectX versions that are earlier than DirectX 10 to make sure that no application-visible API behavior changes.

To virtualize video memory, the video memory manager in Windows Vista assigns a virtual address range to every video memory resource. This range is conceptually similar to the copy that an application might create. However, the video memory manager manages the process more efficiently than the application might. The video memory manager uses the virtual address range to handle transitions or over-commitment of video memory. However, the virtual address range is typically unused on a system that has lots of video memory. As long as this virtual address range remains unused, no physical memory is allocated for it. In contrast, the system memory copy that is maintained in the older driver model is guaranteed to be fully populated with physical memory.

If an application creates its own in-memory copy of its video resources, or the application uses DirectX 9 or an earlier version, the virtual address space contains the WDDM video memory manager's virtualized range and the application's copy. Applications that use graphics APIs that are earlier than DirectX 10 and that target GPUs that have large amounts of video memory can easily exhaust their virtual address space.

To address this problem, Microsoft is changing the way that the video memory manager maintains the content of video memory resources. This change is being made so that a permanent virtual address range does not have to be used for each virtualized allocation. With the new approach, only allocations that are created as "lockable" consume space in the virtual address space of the application. Allocations that are not created as "lockable" do not consume space. This approach significantly reduces the virtual address space that is used. Therefore, the application can run on large video memory configurations without reaching the limits.

Although this approach reduces virtual address consumption, it does not eliminate the 2 GB virtual address space limit that many applications are quickly approaching on their own. Eventually, applications will reach the limit for other reasons.

[VoltaAsura]

Why not wait 2 more months to see what the DX11 client has to offer and worry about it then?

[KoolAidPitcher]

Why not wait 2 more months to see what the DX11 client has to offer and worry about it then?

I look forward to the DX11 client. In the meantime, I am curious if Square-Enix will allow me to fix the problem myself by using the Microsoft COFF/PE Editor to make the ffxiv.exe process large address aware (see: https://msdn.microsoft.com/en-us/library/wz223b1z.aspx). By using an unsigned integer instead of a signed integer to reference virtual memory addresses, this will provide the ffxiv.exe process an additional 2GiB of virtual address space and will resolve my crashes.

[Raist]

The problem though may be in the DX9 emulation layer, possibly even Winblows own x86 thunking in general. Not sure if those runtimes will work well if you exceed the expected address space limits of the older x86 environments. Not entirely up to speed on whether the x86 implementation OS limits are still enforced somehow even within the x64 OS either. There could be some wonkiness in there where you still run up against the 2GB (or tweaked 3GB) user space cap like we did in 32-bit.

Microsoft has been f-cking with programmers for as long as I can remember. So glad I ran away from networking when Vista launched. Such a shame *nix and OGL never got the momentum it deserved.

[KoolAidPitcher]

The problem though may be in the DX9 emulation layer, possibly even Winblows own x86 thunking in general. Not sure if those runtimes will work well if you exceed the expected address space limits of the older x86 environments. Not entirely up to speed on whether the x86 implementation OS limits are still enforced somehow even within the x64 OS either. There could be some wonkiness in there where you still run up against the 2GB (or tweaked 3GB) user space cap like we did in 32-bit.

Microsoft has been f-cking with programmers for as long as I can remember. So glad I ran away from networking when Vista launched. Such a shame *nix and OGL never got the momentum it deserved.

Hi Raist. I am pretty sure that thunking is only used when emulating 16-bit processes designed for DOS and Windows 3.x, while 32-bit processes are handled natively without an emulation layer. (Actually, this is only partially true. Thunking is still used in some cases: https://msdn.microsoft.com/en-us/lib...=vs.85%29.aspx). As for large-address awareness in an actual 32-bit environment, according to Microsoft's documentation, the /LARGEADDRESSAWARE flag will be ignored in an actual 32-bit environment unless an additional /3G operating system flag is also being used.

I'm a software engineer, and while I prefer to work in unix-like systems, I have also developed plenty of software for Windows. As a matter of fact, most of my C++ code works in all operating systems with little effort. In college I designed a CPU which processed a limited MIPS instruction set, wrote a toy operating system kernel, wrote a toy compiler, etc. etc.

Anyway, nice to meet you.

[Raist]

My concern though is whether unforeseen issues may come into play though with the DLL's and such that get called. Whether that is in the scope of the DX9 emulation layer, or something somehow buried elsewhere within the OS layers or not. That's one thing I was always curious about x64...since it has the designated x86 programs folder, does it also call specific 32-bit support files from the OS, or is it all x64. You never know with Microsoft.

It would be an interesting experiment though...provided you don't loose your sanity getting it to work.

Edit:
Uggh.. out of curiosity I poked around in the executable files. References to old Xinput/Xaudio, 32-bit GDI, kernel, etc...even DirectInput8? IDK... might be setting yourself up for some headaches.

[Etrius]

TL;DR
"Do players have permission to make the ffxiv.exe process LAA?"
You can try to... tell me if you get past this: