Xbox series X framerate drops

I can't really find a rhyme or reason for it. I know it's not my internet as I have great rate and download speeds. During my Xbox network tests. I can look in direction a. Get great frames turn to direction B, and get bad framerates. Then turn around again and go back to direction B and the framerates are fine. It's very odd. Is anyone else seeing this issue?

Are you in a highly populated area at the time, as in part of a city where there are just dozens upon dozens of crafters sitting on top of each other? If so, that's unfortunately normal and there's not much you can do about it. The PC version tanks in those instances too, though not as noticeable on stronger hardware with VRR.