DS Forums

 
 

Samsung caught doping the benchmarks… again (Galaxy Note 3)


Reply
Thread Tools Search this Thread
Old 02-10-2013, 12:54
jonner101
Forum Member
 
Join Date: Apr 2008
Posts: 3,153
I would suspect all companies that ship a device to the people that do the benchmarks, ship a benchmark optimised device. If you performed the exact same task on each phone, then I would say yes. Maybe not by 20% though.



The benchmarks are not there to mislead, they are a guide. Its a dick waving contest between manufacturers, and they are all at it. Samsung arent the only one that do it. As I said earlier, only a naive person would think a benchmark reflects real life useage - thus there is no attempt to mislead.

I agree that they should also provide a benchmark for daily use...i.e. what the user will actually experience. Provide both figures if you want, nothing wrong with that.
The benchmarks are hard coded into the OS. Sorry but how devious and underhand can you get
jonner101 is offline   Reply With Quote
Please sign in or register to remove this advertisement.
Old 02-10-2013, 13:06
Stiggles
Forum Member
 
Join Date: Jan 2011
Location: Dundee, Scotland
Posts: 9,292
The benchmarks are hard coded into the OS. Sorry but how devious and underhand can you get
Eh?

No they aren't! The code simply allows the device to perform at it's absolute maximum while the test is being run. It's not adding anything to the benchmark!
Stiggles is offline   Reply With Quote
Old 02-10-2013, 13:12
jonner101
Forum Member
 
Join Date: Apr 2008
Posts: 3,153
Eh?

No they aren't! The code simply allows the device to perform at it's absolute maximum while the test is being run. It's not adding anything to the benchmark!
The relevant info is here they are hard coded when the app was rebuilt with a different package name the performance dropped by 20% for the exact same app.


The file we ended up with is called "DVFSHelper.java," and it contains a hard-coded list of every package that is affected by the special CPU boosting mode. According to this file, the function is used exclusively for benchmarks, and it seems to cover all the popular ones. There's Geekbench, Quadrant, Antutu, Linpack, GFXBench, and even some of Samsung's own benchmarks. The two functions applied to this list seem to be "PACKAGES_FOR_BOOST_ALL_ADJUSTMENT," which is no doubt the CPU booster, and "PACKAGES_FOR_LCD_FRAME_RATE_ADJUSTMENT," which makes it sound like the phone is also altering the display frame rate.

The inclusion of GFXBench is surprising given that it shows no unusual idling behavior in System Monitor. Between the inclusion of that and the suspicious "frame rate adjustment" string, it's clear that Samsung is doing something to the GPU as well, though those clock speeds are more difficult to access than the CPU speeds (a method used by AnandTech on the international S 4 no longer works on the Note 3).

The "DVFS" in "DVFSHelper" stands for "Dynamic frequency scaling," also known as CPU throttling, which has many legitimate uses to manage both heat and power draw. This file contains a few special settings for the camera, Gallery, and some other packed-in apps, but nothing like what is in the above section. Benchmarking apps are the only type of app that is systematically called out and boosted.

To see how some other benchmarks are affected, we made "stealth" versions of those, too—the exact same app, just with a different package name. These results back up the Geekbench findings: we're seeing artificial benchmark increases across the board of about 20 percent; Linpack showed a boosted variance of about 50 percent.
jonner101 is offline   Reply With Quote
Old 02-10-2013, 15:09
IvanIV
Forum Member
 
Join Date: May 2006
Posts: 25,199
Samsung switch off processor's power save mode when a programme with a certain package name is running. By "coincidence" they are all benchmark programmes. Testers repackaged a benchmark programme using a name not on the Samsung list and they got different (worse) results, because power save mode kicked in this time.
IvanIV is offline   Reply With Quote
Old 02-10-2013, 15:20
Philip Wales
Forum Member
 
Join Date: Feb 2013
Location: South Wales
Posts: 5,866
In my world it's called "not playing by the book", while nothing wrong with showing the capabilities of a phone, they should make it perfectly clear that these figures are not attainable in "real world settings" and then publish the actual "real world" figures.
Philip Wales is offline   Reply With Quote
Old 02-10-2013, 15:21
flagpole
Inactive Member
 
Join Date: Jan 2003
Posts: 43,524
This is very devious of Samsung if they have actually hard coded their OS build to look for these benchmarks and basically allow the processor to be over-clocked when running them.

It would be funny if it were actually possible to fry the chip if you kept running one of these benchmarks. Someone should try it and sue their ass off.
seems like a lot of trouble to go to for a replacement of an already working phone.
flagpole is offline   Reply With Quote
Old 02-10-2013, 15:23
Dark 1
Forum Member
 
Join Date: Nov 2002
Location: Nottingham
Posts: 3,975
The thread title is incredibly misleading!!

All this does, as has been explained already, is allow the absolute maximum performance of the device to be used for a few mins at a time to show what in theory it's capable of.

Nothing wrong is being done here. I dare say many other companies do this as well.
So it's just coincidental that the only apps that can tap into this "maximum performance" are known benchmarking apps which serve no purpose other than to provide numbers which can be compared to competitors products?

And it's just coincidental that only Samsung has been caught doing this? Twice!

No, I think the title is appropriate. It's a cynical tactic to mislead consumers into believing their phones are significantly more powerful than those of their direct competitors.

"They are without honour" - Worf
Dark 1 is offline   Reply With Quote
Old 02-10-2013, 15:28
Philip Wales
Forum Member
 
Join Date: Feb 2013
Location: South Wales
Posts: 5,866
^^^ agree it's bloody dodgy and unfair, I suppose to use the car thing again would be for BMW to claim there 3 Series can top 180mph, then actually say well the M3 can.
Philip Wales is offline   Reply With Quote
Old 02-10-2013, 15:28
Dark 1
Forum Member
 
Join Date: Nov 2002
Location: Nottingham
Posts: 3,975
In my world it's called "not playing by the book", while nothing wrong with showing the capabilities of a phone, they should make it perfectly clear that these figures are not attainable in "real world settings" and then publish the actual "real world" figures.
But Samsung aren't generally providing the benchmarks. That's left to reviewers who won't know that their results have been 'doped'... until now.
Dark 1 is offline   Reply With Quote
Old 02-10-2013, 15:31
Philip Wales
Forum Member
 
Join Date: Feb 2013
Location: South Wales
Posts: 5,866
^^true, but Samsung must run these benchmarks too, what's to stop them handing over a phone with a totally mad processor, get the benchmarks, then change the processor for the full production model, which in a way is what they are doing.
Philip Wales is offline   Reply With Quote
Old 02-10-2013, 15:35
Dark 1
Forum Member
 
Join Date: Nov 2002
Location: Nottingham
Posts: 3,975
seems like a lot of trouble to go to for a replacement of an already working phone.
More likely for someone (an American probably) to buy one just so they can have a punt at suing Samsung, claiming they only did so purely based on the benchmarking scores.
Dark 1 is offline   Reply With Quote
Old 02-10-2013, 15:36
flagpole
Inactive Member
 
Join Date: Jan 2003
Posts: 43,524
More likely for someone (an American probably) to buy one just so they can have a punt at suing Samsung, claiming they only did so purely based on the benchmarking scores.
they still wouldn't be able to 'sue their ass off'
flagpole is offline   Reply With Quote
Old 02-10-2013, 15:39
Philip Wales
Forum Member
 
Join Date: Feb 2013
Location: South Wales
Posts: 5,866
^^^ no but the Americans do love a "class action" as seen with Microsoft and the Xbox 360
Philip Wales is offline   Reply With Quote
Old 02-10-2013, 15:40
ACU
Forum Member
 
Join Date: Aug 2009
Posts: 7,916
So it's just coincidental that the only apps that can tap into this "maximum performance" are known benchmarking apps which serve no purpose other than to provide numbers which can be compared to competitors products?

And it's just coincidental that only Samsung has been caught doing this? Twice!

No, I think the title is appropriate. It's a cynical tactic to mislead consumers into believing their phones are significantly more powerful than those of their direct competitors.

"They are without honour" - Worf
Whats the point in running at full pelt when you are reading tweets? If Samsung made their phones that ran a full pelt all the time, the battery would last around 5 hours and be dead. What good is that to anyone? It seems what Samsung have done, is make the choice at what apps run at full pelt, which is a sensible way of doing things.

Not sure why you think its a Samsung issue, like I said earlier all companies do it, including Apple. Its nothing new. Funny seeing people grab at straws to have a pop at Samsung. The words glass, houses and stones come to mind.

Good luck finding a company with honour. Care to let me know of one? I cant think of one, that makes smartphones.
ACU is offline   Reply With Quote
Old 02-10-2013, 15:42
Philip Wales
Forum Member
 
Join Date: Feb 2013
Location: South Wales
Posts: 5,866
^^ Do Apple fudge benchmarks, I've not seen a 70 page thread on that yet, there must be one if you say they do, every other Apple Thread gets the treatment.
Philip Wales is offline   Reply With Quote
Old 02-10-2013, 15:48
alanwarwic
Forum Member
 
Join Date: Oct 2003
Location: the wild world web
Posts: 28,132
You know what the outcome of this seems to be?

It is that the hyped up Snapdragons are more than a bit crap.
We already had Samsung limiting Exynos to allow the Snapdragon and Exynos S4s to work at the same speed.

This amplifies everything even further. Or behold, is it a case of the exact same?
alanwarwic is offline   Reply With Quote
Old 02-10-2013, 15:53
jabbamk1
Forum Member
 
Join Date: Jan 2012
Location: London, UK
Posts: 8,759
You know what the outcome of this seems to be?

It is that the hyped up Snapdragons are more than a bit crap.
We already had Samsung limiting Exynos to allow the Snapdragon and Exynos S4s to work at the same speed.

This amplifies everything even further. Or behold, is it a case of the exact same?
What...?

Qualcomm are not crap.
jabbamk1 is offline   Reply With Quote
Old 02-10-2013, 16:16
IvanIV
Forum Member
 
Join Date: May 2006
Posts: 25,199
Samsung want to have their processor and eat it. First with big fanfares they announce a mad 8-core processor, then it turns out that a four is a new eight. Then they say they use special multicore processor that can switch to a power save mode and save battery. There's only a small problem, benchmarks are worse, because waking up the processor from power save mode is not for free. No problem, turn it off for benchmarking. This is not so harmless as some state here. There's enough people who take benchmarks into consideration when choosing their phones. Phones nowadays are mostly very good and one has to decide based on something, why not the benchmarks, especially if Samsung phones are way ahead of the competition. IMO it's quite dodgy.
IvanIV is offline   Reply With Quote
Old 02-10-2013, 16:18
alanwarwic
Forum Member
 
Join Date: Oct 2003
Location: the wild world web
Posts: 28,132
....I've not seen a 70 page thread on that yet, there must be one if you say they do, every other Apple Thread gets the treatment.
I found one on "5S show it wipes floor with S4 and HTC One" but simply cannot find the Safari one.

I was thinking of that time when Safari benchmarks were recording an early finish time, thus increasing reported speed.
alanwarwic is offline   Reply With Quote
Old 02-10-2013, 17:24
paulbrock
Forum Member
 
Join Date: Apr 2002
Location: Wapping, London
Posts: 16,222
if mobile benchmark tests actually did model real world user usage and then samsung came along and spoiled the test then it'd be right to call foul.

But they don't, they never have. they tend to throw excessive amounts of calculations/polygons at the processors, far more than would ever be encountered in normal daily use, and count how long it takes. Given its not real world cases, there's nothing wrong with manufacturers *optimising* their devices for the benchmarks, because that's really all that's happening. There's no secret overclocking going on, or "set the engnes to 110% captain" or "if benchmark then unlock secret superfast mode".
paulbrock is offline Follow this poster on Twitter   Reply With Quote
Old 02-10-2013, 18:28
alanwarwic
Forum Member
 
Join Date: Oct 2003
Location: the wild world web
Posts: 28,132
What...?Qualcomm are not crap.
But wasn't Samsung shuffling benchmarks to make the two S4's look identical speed.

I'm sort of baffled as to why this is not some of the same.
We have the cheating benchmark mantra and have moved away from reasoning.
alanwarwic is offline   Reply With Quote
Old 02-10-2013, 19:09
jonner101
Forum Member
 
Join Date: Apr 2008
Posts: 3,153
Can't believe some people think it's ok to hack your OS so it over clocks when all the well known benchmark apps are run.

Has this been done by other companies, well probably but that doesn't make it OK for Samsung to do it.

Clear fraud on Samsung's behalf.
jonner101 is offline   Reply With Quote
Old 02-10-2013, 19:21
alanwarwic
Forum Member
 
Join Date: Oct 2003
Location: the wild world web
Posts: 28,132
I thought it was actual default underlocking which was prevented.

What is the overlock?
alanwarwic is offline   Reply With Quote
Old 02-10-2013, 19:23
sancheeez
Forum Member
 
Join Date: Feb 2005
Posts: 6,762
Don't care. I'll be having one regardless.
sancheeez is offline   Reply With Quote
Old 02-10-2013, 19:26
alanwarwic
Forum Member
 
Join Date: Oct 2003
Location: the wild world web
Posts: 28,132
http://en.m.wikipedia.org/wiki/Note_3
Oh look, just like the S4 there is both the Exynos and the Snapdragon chipsets.

Like I said, they are making the Snapdragon look less crap.
Yes, I overplay the word 'crap' but that counteracts the hype of those last generation ARM CPUs.
alanwarwic is offline   Reply With Quote
 
Reply




 
Forum Jump


All times are GMT. The time now is 13:10.