Can you check if the memory usage of the game also increases over time as the fps goes down?
It sounds like a memory leak which might be caused by an issue with the drivers.

I dont have invidia or intel components but my SO does have a 4060 and she has some frame drops in areas where my AMD card runs flawlessly