Rearchitecting Testing for 2H17 (Patreon)

Wanted to run a Patreon-only update!

We tend to build a new test plan once per year for each set of devices. The test plan covers data organization (a big, unseen element of what goes on behind the scenes), custom software development, test automation where possible, and determining which software is used for which products.

We have been on our current CPU testing suite for nearly a year now, so it's getting to about the point where we want to update. An update will allow us to take a year's worth of learning, of finding shortcomings or inconsistencies, and massively improve our testing for the next year. This is an iterative process, and is why our tests often slowly change with each new product launch: it is iterative and incremental -- those new tests don't happen overnight.

Sadly, this also means we'll be throwing away a year of data, as it'll likely be no longer applicable to our tests. We can keep some synthetics (like Cinebench), but will probably update Blender (thus losing Blender data), update Premiere, and change or add games.

Patrick and I have learned a lot about how we want to improve our CPU testing going forward. The first major step is to clean up the spreadsheets and data sorting and organization, then build-in some semi-automated chart generation. That should save me around 2 hours of rather tedious work per test. No need to manually generate that many charts for each review. We also want to add a few games, but we're not sure what to use yet. Possibly Civ6, maybe some others. Metro might be retired at some point soon, though we're not sure yet.

Anyway, the GPU test suite is still good for now. We're about to add at-rails power draw for RX Vega, which will be new as it's our first GPU review since getting the clamp, but will otherwise stick with the current list of games. Might add one or two slowly, and we're looking forward to Destiny 2 being a potential bench title.

The CPU test suite revisions will begin once Threadripper is fully out of our content pipeline, so that'll be a while yet. Once that's out, we'll start fresh with no data, sadly, and begin building a data-set as CPUs launch.

Cheers, everyone!

(PS - next round of shirts shipping this week)