AMD Comments on GPU Stuttering, Offers Driver Roadmap & Perspective on Benchmarking

Final Words

Bringing things to a close, when AMD first came to us about this, we had mixed feelings about what they were proposing. Typically, manufacturers only take issue with benchmarking methodologies when they have some deficiency on their part. Surprisingly enough, AMD's stance was equal parts recognition of where they had issues and offering their input on the right way to do things.

Meanwhile, we’ve wanted to do an article like this for some time now; specifically a dissection of FRAPS so that we could better explain why we have not been using FRAPS for investigating stuttering. To our surprise, we found AMD repeating many of the positions we held, and more importantly they were offering up further facts and additional data that we didn’t have access to that would help support our own position. So despite whatever AMD’s intentions were, this worked out well for us.

Ultimately AMD’s message has been one of information, explanation, and admission of oversight. AMD has been clear with us from the start that the primary reason they even ended up in this situation is because they weren’t doing sufficient competitive analysis, and that they have revised their driver development process so that they now do this analysis to prevent future problems. The fact that NVIDIA seemed to have figured all of this out much earlier was a point of frustration for AMD. The company likely left non-negligable amounts of performance on the table over the years, which could've definitely helped in close races.

At the same time they’ve been hard at work on fixing existing stuttering problems, with many of those fixes being delivered while fixes for more DX10+ games are right around the corner. 

At the same time however AMD’s message is not just one about stuttering, but also one about benchmark and analysis methods with FRAPS. FRAPS, despite its limitations, has clearly exposed problems with AMD’s drivers that resulted in stuttering that AMD needed to fix. Meanwhile measuring frame intervals with FRAPS has become an increasingly common technique in reviews, only to really become popular at the same time as when AMD has finally fixed many of these issues.

AMD’s concern – and one that NVIDIA has shared with them in the past – is that measuring the rendering pipeline at the beginning of the pipeline like FRAPS goes about it does not accurately represent what the end user is seeing, due to the various buffers in the Pipeline and how the Present mechanism works. While FRAPS was good enough to pick up on the major stuttering issues in AMD’s drivers, as these issues get resolved it’s far too coarse a tool to pick up on finer issues, and in fact what FRAPS is now seeing is decoupled from what the user is seeing due to the presence of the context queue and other buffers. All of these, for the record, are points we agreed with AMD on, even before our meeting.

The end result of all of this is that change is in the air. Just as how quickly as Scott Wasson and others changed the nature of GPU reviewing by using FRAPS to measure frametimes, things must change again for GPU reviewers. If FRAPS is no longer an adequate tool to measure stuttering and frame intervals – as both AMD and NVIDIA insist – then new methods and new tools must be created to measure those factors at the end of the rendering pipeline, where the results would match what the end user is seeing. Though on the subject of tools, AMD for their part is favoring double-blind trials as the ultimate method of detecting stuttering. They’re fundamentally right since the perceptibility of stuttering depends on the person, but admittedly this is also the least objective/qualitative way of evaluating stuttering.

In any case, just as how change is in the air for GPU reviews, AMD has had to engage in their own changes too. They have changed how they develop their drivers, how they do competitive analysis, and how they look at stuttering and frame intervals in games. And they’re not done. They’re already working on changing how they do frame pacing for multi-GPU setups, and come July we’re going to have the chance to see the results of AMD’s latest efforts there.

For us this is what we hope to be the start of our own changes. There are tools in development that meet our criteria for better measuring frame intervals, and hopefully in the not too distant future we’ll be able to discuss those tools to a much greater degree, and to use those tools to go about measuring frame intervals in the manner we’ve always wanted to. But that is a story for another day, so until then you’ll have to stay tuned to find out.