Well I had gave up on SE support on this, so sorry for not replying Einmimiria. There goes my planned update:

I ended up changing the whole modem/router configuration in my house just so I can bring the Ethernet cable to my living room and connect to my gaming pc. It's a far from ideal solution and I would really appreciate any ideas so I can keep cable clutter away from my living room but it works.

So yes I can connect to the game using the Ethernet cable. I also tested that I can connect to the game using the WiFi in my primary computer (which is now running on the WiFi). This really throw away my WiFi network as the root of the problem theory, since my primary computer don't even have any port forwarding configuration set for it.

This post and Einmimiria reply made me think that maybe there is some incompatibility between my wifi card and FFXIV? I never thought something like that would be possible but my pc has an integrated wifi card, I do have a usb adapter available I will try using that...