We Have A Benchmarking Problem - And Here's How We Are Fixing It (Patreon)

This has been DF's biggest tools investment for some time - arguably since 2008 when we built the first HDMI console performance analysis tools. We're looking to solve the PC benchmarking problem once and for all by - and this is putting it in very simplistic terms - getting games to benchmark themselves.

We've been voicing dissatisfaction with our hardware reviews for some time, so it's time to do something about it. Key to the problem is the fact that we get about a week to review any given piece of hardware - benchmarking, validating benchmarks, writing, scripting, product shots, filming, editing. And as we work remotely, one person does the job and it's becoming less viable with each release.

By far the biggest time sink is benchmarking. Way too much of that week is spent running numbers - and there's never enough of them. In the process, we end up talking about percentages and not much else. At best, reviews are group tests when perhaps they could be more.

So for some months now, we've been working with the modder Illusion on our automated benchmarking solution - a way to liberate staff from the most tedious process in tech journalism.

You've perhaps seen the 'MarkBench' system LTT has created - well, we think we've gone a couple of steps beyond that. We have game mods to handle all types of benchmarking scenarios, even down to custom camera mods to simulate the manual runs you need on some games - and arguably the majority of games on benchmarks designed to stress test CPUs.

In the process, we're aiming to massively increase the amount of data we get, to the point of building a database of performance metrics that will enable vastly more PC hardware coverage. The process is ongoing but with the release yesterday of Will Judd's Ryzen 9600X/9700X reviews, we saw the first use of this system for actual coverage... and it's working beautifully.

The video above is for all supporters - and supporters only - and shows our progress so far. Once we reach new milestones, we'll keep you updated. We'll go public with the new tools once they're fully completed. However, for me (Rich), it's important that Supporters are fully informed of what we're up to and where we are making key investments. Please enjoy the #content!

Downloads; https://www.digitalfoundry.net/we-have-a-benchmark-problem-and-heres-how-we-are-fixing-it