If you’ve looked at some of our reviews in the last couple of days, you may have noticed a few different benchmarks and some new charts that we weren’t using before. We don’t usually do this, but behind the scenes we’ve just given our benchmark suite a comprehensive update for the first time since 2013. In the interest of keeping you all informed and letting you know what we’re thinking here on the Ars Orbiting HQ, this is a good opportunity to run through the tests we do, why we do them, and why we care about benchmarks in the first place.
First, you’ll notice that we’re using some new colors in our charts. As much as we liked the bright colors in our previous charts—colors chosen to match the Ars color palette, incidentally—we got semi-regular feedback from colorblind folks that they were hard to read. Ars Creative Director Aurich Lawson chose our new chart colors to be easily legible by people with all common forms of color blindness.
What we’re using: CPU and GPU compute benchmarks
We may use benchmarks other than these when we’re doing in-depth component reviews of the latest flagship processors or graphics cards, but generally speaking in phone and laptop reviews these are the standard benchmarks we’ll be running on everything.
Our primary CPU test is Geekbench 4 from the fine people at Primate Labs. It replaces the previous version, Geekbench 3, and scores from Geekbench 3 are not comparable to those from Geekbench 4.
Geekbench measures many different aspects of single- and multi-core CPU performance, cryptography performance (increasingly important as enabled-by-default encryption becomes more prevalent), and memory performance. Primate Labs has provided extensive detail on Geekbench 4’s CPU workloads here.
If you’re interested in knowing more about what has changed in Geekbench 4, Primate Labs’ John Poole gave a great interview to XDA last year talking about how the test had changed. At the high level: huge performance increases in mobile chips enabled better parity between the desktop versions of Geekbench and the mobile versions, and the tests have been tweaked to make it easier to make performance comparisons across platforms and across architectures. The way the test runs was also changed to minimize throttling in heat-constrained systems.
Geekbench 4 also adds a new GPU compute benchmark that works with OpenCL, Nvidia’s CUDA, and Android’s RenderScript APIs. Lots of apps these days will use GPUs to accelerate specific tasks and take some of the load off of the CPU, and this number will be of particular interest to people who use GPUs to run apps like Photoshop or Premiere or to crunch numbers rather than run games.
On our desktop platforms, we’ll also continue to use Maxon’s Cinebench R15, a combined CPU and GPU benchmark. It spits out one big number for both single-core and multi-core CPU performance, and since it takes a while to run, it can be a decent indicator of heat-related throttling issues.
Historically, browser benchmarks have been used mostly to compare different browsers running on the same system and how the performance of different versions of the same browser evolved over time. In the early days before decent smartphone benchmarking apps became a thing, they were also useful as a rudimentary way to compare performance between different devices. Today’s benchmarking tools have gotten good enough that we don’t need to rely on them as much, so we don’t.
We’re still using three browser benchmarks that we consider to be reasonably modern: Google Octane, Mozilla’s Kraken, and Browserbench.org’s JetStream. Just be aware of their limitations; they’re not super helpful when comparing different platforms, and they’re also primarily an indicator of single-core CPU performance and not multi-core CPU performance.
Our primary cross-platform GPU benchmark is Kishonti’s GFXBench. The test offers a variety of high- and low-level benchmarks, and we stick to the high-level ones: we’re still using the T-Rex and Manhattan tests, and now we’ve added the more punishing Manhattan 3.1 and Car Chase tests. On Windows and Android, these are all OpenGL graphics benchmarks. In iOS and macOS, we run the Metal versions, which don’t include the Car Chase test yet (the OpenGL versions aren’t being updated on Apple’s platforms, presumably because Apple itself appears to have given up on keeping its OpenGL implementation even remotely up-to-date).
GFXBench offers two different types of tests: “onscreen” tests and “offscreen” tests. There has been some confusion about the two in the past, but the difference should be fairly easy to understand. Onscreen tests run at the native resolution of the device’s display panel, which tells us how good a given GPU is at driving graphics on a particular display. If you have one laptop with a 1080p screen and one with a 4K screen and both are using the same model of GPU, the 4K system is going to score significantly lower in the onscreen tests because that GPU is pushing more pixels. Offscreen tests render the scenes at 1080p on every device regardless of the screen’s resolution, which puts all the GPUs on even footing so that we can definitively say “all else being equal, GPU X is better than GPU Y.”
On Windows PCs, we also run four tests from the most recent version of 3DMark: Cloud Gate, Sky Diver, Fire Strike, and Time Spy. Where GFXBench helps us measure OpenGL performance, 3DMark covers DirectX performance, and those four tests cover several versions of DirectX up to and including version 12. And the Cinebench R15 GPU test rounds things out with another OpenGL-based test on both Windows and Macs.
This part of our suite is unchanged. In Windows and Android, we measure sequential read and write speeds (behavior you’d see when downloading or copying a single large file to disk) and random read and write speeds (what you’d see if multiple programs were making a bunch of small writes to the disk, as happens often if you’re multitasking). AndroBench is the utility we use to make these measurements in Android, and in Windows we use CrystalDiskMark.
Things are more complicated on Apple’s side of the fence because the storage benchmarks are either not as robust or nonexistent. The few storage benchmarks that exist in iOS are badly out of date and offer no customization options, so rather than provide potentially bad numbers, at this point in time we err on not providing iOS storage speed numbers. For macOS we use QuickBench, which isn’t as good as CrystalDiskMark but does at least let us measure peak sequential read and write numbers.
Let’s begin by stating the obvious: benchmarks don’t always tell you much about what it will be like to run actual apps. You can’t point to a specific Geekbench score and declare that it meets the minimum requirements for running Photoshop well. Same for 3DMark and any given game at a given resolution or detail level; they can give us some idea, but it’s also not really the point.
Benchmarking is primarily about comparing the relative performance of two or more systems using a consistent, repeatable set of tasks. We can use these numbers both to track generational improvements as new hardware replaces old hardware and to track how the same chips perform in different devices. The first part doesn’t always feel important today, since the performance of mainstream laptop and desktop chips has largely plateaued, and smartphone chips are beginning to show signs of doing the same thing, but it’s still useful for people replacing something that’s more than a year or two old.
That second part has become especially important in the age of smartphones, tablets, and fanless laptops, where the design of any given device’s heatsink can have a big impact on performance. Under ideal circumstances, a Qualcomm Snapdragon 821 or an Intel Core m3-7Y30 is going to run about the same in just about any system. In the real world, this is heavily dependent on how good each individual device is at dissipating heat and letting the chips run at their maximum speeds.
Because benchmarks usually put a respectable amount of strain on the components inside these gadgets, then, they’re also a good way to suss out certain kinds of design flaws. Look at the benchmarking charts for something like Huawei’s MateBook tablet from last year or a Snapdragon 810 phone, for instance, and it becomes clear that heat is a problem for these systems. This means that not only will heat impact the day-to-day speed of your device, but that it may not last as long since excessive heat shortens the life of batteries and other components.
Listing image by Andrew Cunningham