Benchmarks – if it didn’t have so many letters, we would call it a four-letter-word. Some people live and die by benchmarks and the vast majority could care less as long as their device runs fast and smooth. The thirst for benchmarks may be driven by techie websites that, much like scientists, demand proof – finite numbers – they will not settle for “fast and smooth,” they must have numerical confirmation. Whenever a manufacturer knows that their potential customers may be looking at these numbers, there could be a tendency for them to “fudge” their numbers somewhat – and that is exactly what Samsung was accused of doing with their Galaxy Note 3 benchmarks.
We did an article the first of October, where Samsung defended their testing procedures and it is not like they overclocked the processor – they simply ran the tests with the processor running at their advertised specifications. So did they fudge the truth or were the well-respected test sites, like Ars Technica or Anandtech lying? We may never know the entire truth, but GameBench ran some of their own tests between the Samsung Galaxy S4 and the HTC One to try and get to the truth, and Engadget shared them with us.
From what our friends at Engadget are saying, “there is growing evidence” that not only Samsung, but also other Android phone manufacturers may also bend the truth. In most cases, it involves programming a device into ramping up its performance if it detects that a benchmark test has been launched. GameBench claims that they have developed an “uncheatable” performance tests, and though in beta, they performed some tests on the Galaxy S4 and HTC One, and it seems as though Samsung really did not have anything to worry about.
Rather than simply running predictable benchmarks, GameBench took the approach of actually running a genuine game and recording results as their app runs in the background. By doing this, BenchMark is able to look at and measure the median frame rate, frames-per-second (FPS), and the battery drain, in real world operations. Here they tested the Galaxy S4 and HTC One using four games, and claim you would never get these kind of results using a standard GFXBench benchmark tests.
Above you can see that the Galaxy S4 ran the games “much smoother,” but at a higher battery drain, which sometimes is a necessary trade-off. What good does it do to save your battery from draining, if the game you are playing is choppy?
The moral of this article is not to rely strictly on standard benchmark tests that cannot show how the device will work when actually playing a game. The GameBench app should be ready in early 2014 and may help take some of the mystery out of benchmarks. Let us know in the comments of on Google+ what you think about all of these tests – do you follow benchmarks closely, or go by how your device operates for you.