Tech savvy readers know that benchmarks have little significance in real life. They are a relatively good way to compare devices in relation with each other, but other than that, benchmarks tell us quite little about the real life performance of a smartphone or tablet.
Here's another reason we should collectively give up our obsession with benchmarks and leaderboards – they can be gamed.
Gaming benchmarks has been a time-honored tradition in the PC GPU industry, but so far, we haven't seen the practice adopted in the mobile world. Until now, because it looks that Samsung has just been caught red-handed.
Here's the story – a user on the Beyond3D forum noticed that the GPU of the Exynos 5 Octa processor in the Galaxy S4 runs at a higher frequency when running benchmarks than it does when it runs games or other regular apps.
Screenshot of AnTuTu benchmark with the GPU frequency in the background (Credit: AnandTech)
The folks over at AnandTech investigated the claim and their conclusion is that Samsung does indeed set the GPU of the Exynos 5 Octa version of the Galaxy S4 to run at 533MHz during benchmarking, as opposed to 480MHz during normal operation. The benchmarking apps that trigger the behavior are GLBenchmark 2.5.1, AnTuTu, and Quadrant.
AnandTech spotted a similar behavior when it comes to the CPU – when running GLBenchmark 2.5.1, AnTuTu, Linpack, and Quadrant, the cluster of Cortex A15 cores @ 1.2GHz kicks in. When running GFXBench 2.7 (a benchmark that wasn't whitelisted in the firmware), the lower-powered A7 cluster clocked at 500MHz is used. Moreover, it appears that the CPU of the Snapdragon 600 version of the Galaxy S4 is also set to maximum speeds when benchmarks are ran.
In addition to these empirical observations, AnandTech also analyzed the firmware of the Exynos version of the Galaxy S4 and found the hard code that instructs the GPU and CPU to kick into high gear when specific benchmarks are running.
While Anand Lal Shimpi and Brian Klug of AnandTech elegantly call the behavior "benchmark optimization", it's clear that we're looking at Samsung's underhanded attempt to make the Galaxy S4 look better in benchmarks.
The moral of the story? Don't ever make buying decisions based on benchmark scores.
No comments:
Post a Comment