Sounds like your Graphics card may be overheating just enough to become unstable, or something with your power regulation has some components that have worn out just enough to become unstable as they warm up a bit.
Some of this can get a bit complicated to investigate, as it can be capacitors that have dried out over time and such... you may need to check voltages of specific components and such to verify the power regulation side of it on the motherboard and such. But, temperatures and some core generic voltages are usually pretty easy to monitor. Your graphics card or your motherboard may have some monitoring tools built into their software package that can give you such information (usually something used for overclocking and such). There are some free tools that can also give you some voltage/temp ratings that might give you some clues as well.
www.cpuid.com has a nice HWMonitor tool that can give you a lot of info on your system (depending on what sensor data it can find). You can also go to Tech-PowerUp for their GPU-Z for information on your graphics card if you can't get it's temps in the HWMonitor.
Edit: The v1.24 link for HWMonitor over on the right side of the page is for the free version--not the paid Pro version. Looks a little misleading at first
Voltages will fluctuate depending on load, but basically you want your 12V lines to be staying at least 12V, your 5V to be 5V, etc., without them going real far above their expected ranges (ie: seeing 13v on a 12V line or 6V on a 5V line wouldn't be cool). You also don't want your temps ramping up really high. The limits will vary for each component--you can post what you are getting for your CPU and GPU along with information on what they are for some feedback on what you are seeing (Intel/AMD CPU and model number, nVidia or AMD and it's model number, etc.--this info should show up in these utilities).