“The relevance and accuracy of mobile benchmarking has come under fire over the past several months.” This is the line Qualcomm chose to start the press release issued last Friday when they announced that they’re backing BDTI to create a new benchmark standard. That first line is in obvious reference to the fact that Samsung has recently been caught cheating on the benchmarks for their flagship device, the Galaxy S4.
BDTI, which stands for Berkeley Design Technology Inc., have been sponsored by Qualcomm to independently create a new user experience rating for mobile devices. In theory, this means that the new rating system should be completely neutral and not favor any kind of device. The fact that Qualcomm is the one sponsoring this doesn’t exactly scream neutrality but that doesn’t mean that it will favor Qualcomm either so we can keep hoping for a true, real-world benchmark system.
The real-world aspect of the benchmark is another important topic that was addressed in the press release. As Jeff Bier, President of BDTI stated:
“Many existing mobile benchmarks do not measure attributes relevant to consumers and can be easily manipulated. There is a striking lack of mobile benchmarks that measure actual user experience. Our user-experience-focused tests will include real-world measurements of application performance, battery life, and data transfer speed and delay.”
Qualcomm repeated several times the difference between everyday activity and pure math solutions. Nobody cares how fast a processor do a complex equation if they can’t run a simple app properly so if this new standard is able to provide real answers that relate to a standard use of a smartphone they might have a chance of making benchmarks relevant again. There’s been a trend lately where people do benchmarks just for the sake of doing them. We’ve learned that it’s just a number and it doesn’t represent how a device performs. Hopefully BDTI can provide an answer that would allow us to compare devices better than we can today.
BDTI said that they would have an initial public release in late 2013 and general availability of the benchmark in early 2014. I’m sure one of the first tests performed on this new system will be the Nexus 5 vs the iPhone 5s, both of which are probably going to be announced in the Fall.