Sudden lag spikes

Timeline of what I've been doing

May - August 2021: Started experiencing severe lag and packet loss at certain timing of the day, 5pm - 1am (AEDT). Consulted Optus (previous ISP), troubleshooted, and figured that it's most likely due to my old modem. Optus then sent me a new modem which resolved the issue for 3 days, then the horrible lag spikes returned again (which lasted for the entire day) and my ISP were unable to identify the root of the problem. I got frustrated and decided to swap to a different ISP.

September - November 2021: New ISP (Tangerine) was doing great until about early November of which, the same severe lag started happening again but this time it starts at approximately 1am (AEDT). Consulted Tangerine and they re-configured my modem and resetted my IP address, but the issue still persist.

This issue seems to be happening primarily for FFXIV as I don't seem to have that issue for other games/casual net browsing/Netflix/etc.

I'm playing FFXIV on PS4, and net is LAN-connected (connection is much worse on WiFi). However, net is stable when using hotspot (tether from my mobile phone, although speed is not the greatest, but no lag spikes, at least).

Symptoms I've noticed
- Always happening around the same time
- Skills are rubberbanding
- Nothing moves during raid
- When casting, I can move and cast and it will not interrupt
- Cast doesn't consume dualcast buff (for RDM)

Is there anybody experiencing the same issue? If yes, is there a solution to this problem as I don't think continuously swapping ISPs will remedy the problem.