PDA

View Full Version : Re: Stonewalling: (Re: Ooops... Nvidia was and is cheating on 3dmark03 after all)


Derek Wildstar
June 22nd 03, 04:02 PM
"Roger Squires" > wrote in message
.com...

> Don't kid yourself. Nvidia silently and surreptitiously cheated on
> 3dmark in multiple, shockingly blatant ways, with the deliberate intention
> to deceive consumers and reviewers about the performance of an entire
> product line, and they are continuing to do so.


Prove it.

Ben Pope
June 22nd 03, 04:29 PM
Derek Wildstar wrote:
> "Ben Pope" > wrote in message
> ...
>> Derek Wildstar wrote:
>>> "Roger Squires" > wrote in message
>>> .com...
>>>
>>>> Don't kid yourself. Nvidia silently and surreptitiously
>>>> cheated on 3dmark in multiple, shockingly blatant ways, with the
>>>> deliberate intention to deceive consumers and reviewers about the
>>>> performance of an entire product line, and they are continuing to
>>>> do so.
>>>
>>>
>>> Prove it.
>>
>> I think cheating in 8 (or so) different ways, including replacing
>> code, setting clipping planes for particular frames which can and do
>> only work for the most respected and utilised consumer graphics
>> benchmark program is evidence enough.
>>
>> 8 seperate mistakes that all just so happen to improve scores in the
>> benchmark do not and cannot happen by mistake - the intention to
>> mislead customers is obvious.
>>
>> Ben
>> --
>> I'm not just a number. To many, I'm known as a string...
>
> That's not proof Ben.

Are we trying to prove that they cheated or that they intended to mislead
customers? I think one follows from the other.

Any court would go on reasonable doubt and nVidia would lose.

Tests that prove that nVidia was cheating*:

http://www.futuremark.com/companyinfo/3dmark03_audit_report.pdf

*I'll define cheating as: "Optimisations or modifications that improve
performance."

To prove that nVidia intentionally wanted to mislead customers is a little
harder, but why would they go to any effort at all to increase scores?
Since scores are a fundamental part of many of not all comparisons, and that
customers rely on this to make purchasing decisions, then I would say that
beyond reasonable doubt, nVidia wanted to mislead customers over the
performance of their graphics chipset. Bear in mind that these
optimisations DO NOT affect any other game or application, ONLY the
benchmark in question.

Ben
--
I'm not just a number. To many, I'm known as a string...

Merkutio
June 22nd 03, 05:24 PM
It doesn't detect the benchmark program by name, but by the way the program
tries to access features. Benchmarking is much different from gaming and
rendering.

"Mark B" > wrote in message
...
> To prove it, 3dmark03 on an Nvidia card then rename 3dmark03.exe to
> something else and run the test again.
>
>
> Mark
>
>

Lithurge
June 22nd 03, 05:38 PM
"Merkutio" > wrote in
:

> It doesn't detect the benchmark program by name, but by the
> way the program tries to access features. Benchmarking is
> much different from gaming and rendering.
>

Link to Futuremarks audit:

http://tinyurl.com/citj

Some will argue this is still not proof Nvidia deliberatley
tried to cheat. However taking into account the fact merely
changing certain call names 'fixed' it to work as intended is
strong circumstantial evidence.

John Russell
June 22nd 03, 07:50 PM
"Pete" > wrote in message
...
> Derek Wildstar wrote:
>
> > That's not proof Ben.
>
> The 3dmark affair was pretty conclusive proof of intent to deceive, as far
> as I'm concerned. It takes a serious stretch of the imagination to call
the
> file-renaming incident a 'bug'. I'm trying to evaluate the top 2 cards for
> a prospective purchase (as opposed to justifying a purchase in
retrospect -
> that always clouds your judgement) and this don't make nVidia any easier
to
> trust. If they're going to divert time & money into cheating and away from
> genuine driver improvements, then they deserve to get caught with their
> pants down. Let's hope they learn a lesson from it.

So why should you believe a test program? Nvidia's arguement is that the
coding of this test does not reflect the advice given by Nvidia on how to
code for their cards. With only 2 major GPU makers it's rediculous to argue
that real games wouldn't use Nvidia's advice. I want games to run as fast as
possible by using both Nvidia and ATI's advice. I'm not interested in how a
few Finns decide to code a benchmark So many people feel they can't trust
Nvidia. Believing that does not make 3dmark a decent test of how cards play
games!
If you don't want time and effort wasted on 3dmark, then campaign for real
games to be used for tests. It's the fact people place so much importance on
3dmark results which is creating this situation.

John Russell
June 22nd 03, 07:55 PM
"Lithurge" > wrote in message
...
> "Merkutio" > wrote in
> :
>
> > It doesn't detect the benchmark program by name, but by the
> > way the program tries to access features. Benchmarking is
> > much different from gaming and rendering.
> >
>
> Link to Futuremarks audit:
>
> http://tinyurl.com/citj
>
> Some will argue this is still not proof Nvidia deliberatley
> tried to cheat. However taking into account the fact merely
> changing certain call names 'fixed' it to work as intended is
> strong circumstantial evidence.

Nvidia where trying to make the test work they way they advise coders to
program for their cards. I cannot believe with only 2 major GPU makers that
real game coders would ignore the limitations of these makers products. As a
Nvidia owner I want game coders to get the best from my card, so should ATI
owners.

Lithurge
June 22nd 03, 08:26 PM
"John Russell" > wrote in
:
Both ATI and Nvidia users will expect
> games to run well on their cards and complain if they
> don't.
>
>
>

If they have any sense then they will view 3dmark as one of a
large range of tools to test hardware, whether by others or
themselves. If 3dmark is showing it as running slower, then
people should be pleased when game x runs faster because of
these 'optimisations'.

John Russell
June 22nd 03, 08:29 PM
"Lithurge" > wrote in message
...
> "John Russell" > wrote in
> :
>
> >
>
> > Nvidia where trying to make the test work they way they
> > advise coders to program for their cards. I cannot believe
> > with only 2 major GPU makers that real game coders would
> > ignore the limitations of these makers products. As a
> > Nvidia owner I want game coders to get the best from my
> > card, so should ATI owners.
> >
> >
> >
>
> Did you look at the screenshots in that PDF?
>
> I wouldn't want my games to look like that.
>
> You seem to be missing the point this is not a chip maker
> advising a coder on how to get the best out of their chipset,
> it's about fiddling with the way a driver works to produce the
> fastest results, in one set of circumstances that works to their
> advantage in reports on the speed of their card.
>
> Your argument might hold water if NVidia had publicly stated
> they were doing this before (or even admitted to after being
> found out), but they did not.
>
> And if they were really that altruistic they would pay the money
> to be in the 3dmark beta program. (Not that I necessarily agree
> with the way futuremark are running things)
>
> Lets wait for them to specifically code drivers to play games
> the way they want to shall we? ;-)

If there coded as bad as 3dmark I hope so!

bp
June 22nd 03, 09:20 PM
On Sun, 22 Jun 2003 20:13:15 +0100, "John Russell"
> wrote:

>> "with the deliberate intention to deceive consumers and reviewers about
>> the performance of an entire product line"
>
>The opposite was true. It is futuremark who are decieving everyone as to how
>good cards are at running games.

Then they should have made their cheating drivers look for game
benchmarks and cheated on them instead of 3d marks. ;) or they could
,and should, have been up front about their feelings toward 3dmarks.
They didn't and the line from them now, which you now spew, is simply
back peddling to cover their ass.

> Nvidia intent was to make the test reflect
>how nvidia advise coders to code games, and hence how fast their cards would
>be running games.

So are you planing on running your games with features turned off
just to get a better FPS rate than an ATi card ? I thought not


>Clearly the degree to which this can be done is crude when done at the
>driver level. Never the less,

Never the less it shouldn't have been done at all.

<snip Nvidia's company, we got caught now what, line>

>

John Russell
June 22nd 03, 09:48 PM
"bp" > wrote in message
...
> On Sun, 22 Jun 2003 20:13:15 +0100, "John Russell"
> > wrote:
>
> >> "with the deliberate intention to deceive consumers and reviewers about
> >> the performance of an entire product line"
> >
> >The opposite was true. It is futuremark who are decieving everyone as to
how
> >good cards are at running games.
>
> Then they should have made their cheating drivers look for game
> benchmarks and cheated on them instead of 3d marks. ;) or they could
> ,and should, have been up front about their feelings toward 3dmarks.

They did when they left the 3dmark support group. No one listened. Their
rather crude tactic has created a debate about whole issue of graphic card
testing which wasn't there before. Now you can go on with the mistaken
believe that a handful of Finns have a benchmark that reflects how thousands
of game coders world wide will code games if you want too.
If we all give up on 3dmark, because we either believe it's coded badly, or
we can't trust the drivers, we will all be better off. The focus will then
be on how fast games run, and preferable the games each of like to run, not
somebody elses.

Pete
June 22nd 03, 10:33 PM
John Russell wrote:

>
> So why should you believe a test program? Nvidia's arguement is that the
> coding of this test does not reflect the advice given by Nvidia on how to
> code for their cards. With only 2 major GPU makers it's rediculous to
> argue that real games wouldn't use Nvidia's advice. I want games to run as
> fast as possible by using both Nvidia and ATI's advice. I'm not interested
> in how a
> few Finns decide to code a benchmark So many people feel they can't trust
> Nvidia. Believing that does not make 3dmark a decent test of how cards
> play games!
> If you don't want time and effort wasted on 3dmark, then campaign for real
> games to be used for tests. It's the fact people place so much importance
> on 3dmark results which is creating this situation.

I agree with 100% of that. It's not that I even care about 3dmark scores, I
always skip over them in the reviews. They have to be taken with a strong
pinch of salt at the best of times.
If the Detonator drivers had refused to run 3dmark, saying instead:

"3dmark detected. nVidia do not believe this to be a representative test for
benchmarking and comparison, please run something else."

- well, that would have been OK by me. It's awkward, but it's honest. If
they'd run with their "optimisations" but popped up a window explaining
what they'd done - that would have been OK too.
Whereas instead they cheated. They took the exam, but they sneakily altered
the questions to make them easier.
So I do agree with you that nVidia have a legitimate grievance, but they
still acted dishonestly and deceitfully under pressure.

Roger Squires
June 23rd 03, 12:33 AM
> > "with the deliberate intention to deceive consumers and reviewers about
> > the performance of an entire product line"
>
> The opposite was true. It is futuremark who are decieving everyone as to
how
> good cards are at running games. Nvidia intent was to make the test
reflect
> how nvidia advise coders to code games

John, this statement makes it clear that you are not aware of what
Nvidia was doing with these cheats. One example: one major cheat was to
introduce clipping planes that cut out everything in the scene not covered
by the camera. This only works when the camera path is fixed and known
beforehand, in other words, it will not work in actual gameplay! It's only
possible purpose is to deliberately cheat on the benchmark.

>Nvidia intent was to make the test reflect
> how nvidia advise coders to code games, and hence how fast their cards
would
> be running games.

There is now some evidence that Nvidia is cheating on the builtin
timedemos that games contain. This is not optimizing real gameplay, but
cheating for artificially high benchmark scores. I hope to see articles
soon on hardware websites investigating this non-3dmark cheating further.

rms

Derek Wildstar
June 23rd 03, 07:33 AM
"ZOD" > wrote in message
thlink.net...
> > Prove it.
>
> Prove they didn't....hehe
>

Hot damn, you got me. :) Actually, I'm glad I don't. I don't want to defend
their behavior, but the sky isn't falling on nvidia, not over this issue at
least.

Ben Pope
June 23rd 03, 08:34 AM
Derek Wildstar wrote:
> "ZOD" > wrote in message
> thlink.net...
>>> Prove it.
>>
>> Prove they didn't....hehe
>>
>
> Hot damn, you got me. :) Actually, I'm glad I don't. I don't want to
> defend their behavior, but the sky isn't falling on nvidia, not over
> this issue at least.

Thats 'cos the sky has been clipped away!!

:-P

Ben
--
I'm not just a number. To many, I'm known as a string...

Courseyauto
June 23rd 03, 12:16 PM
<<
"ZOD" > wrote in message
thlink.net...
> > Prove it.
>
> Prove they didn't....hehe
>

Hot damn, you got me. :) Actually, I'm glad I don't. I don't want to defend
their behavior, but the sky isn't falling on nvidia, not over this issue at
least.>>

really...the Fx 5900 are selling as fast as they get in stock.

bp
June 23rd 03, 02:43 PM
On Sun, 22 Jun 2003 21:48:05 +0100, "John Russell"
> wrote:

>They did when they left the 3dmark support group. No one listened.
You can't be up front after you're caught. No one will listen.

Folk
June 23rd 03, 05:31 PM
On Mon, 23 Jun 2003 06:29:53 GMT, "Derek Wildstar" >
wrote:

>More effort should be put into innovation and optimization, the leading
>players are too busy dicking around with 3dmark, and they are both guilty of
>disappointing their customers.

Problem is, in the enthusiast press arena, those companies live and
die by benchmark results. So what are they to do? Both ATI and
Nvidia would be dumb to simply ignore 3DMark, so they spend time
optimizing for it. If review sites would ignore the synthetic
benchmarks, and gauge performance based on actual games, then we would
all benefit.

ZOD
June 23rd 03, 06:10 PM
> Thats 'cos the sky has been clipped away!!
> :-P

OH man, Quake3 bad quality sky flashback...hehe

ZOD
June 23rd 03, 06:11 PM
> Hot damn, you got me. :) Actually, I'm glad I don't. I don't want to
defend
> their behavior, but the sky isn't falling on nvidia, not over this issue
at
> least.

Hey...it never hurt ATI in the long run...hehe

ZOD
June 23rd 03, 06:14 PM
> really...the Fx 5900 are selling as fast as they get in stock.

All 7 of them?
Hehe....

bp
June 23rd 03, 07:28 PM
On Sun, 22 Jun 2003 20:29:15 +0100, "John Russell"
> wrote:

>>
>> Lets wait for them to specifically code drivers to play games
>> the way they want to shall we? ;-)
>
>If there coded as bad as 3dmark I hope so!
So John what exactly do you know about the way 3dmarks coded it
other than what Nvidia told you ? Who btw way are simply trying to
cover their ass after being found out.

If you look at what they did then I would say Futuremarks coded it
much better. You have looked haven't you ?

Derek Wildstar
June 23rd 03, 07:35 PM
"Ben Pope" > wrote in message
...

>
> The retraction was basically to say "NVIDIA's driver design is an
> application specific optimization and not a cheat ".
>
> In my humble opinion I would say that when the application is a benchmark,
> the optimisation is cheating.


And that sentiment is more damaging than any inflated number. Because both
you and I, with differing degrees of intensity over the claim of 'cheat'
both feel 'cheated' by nvidia. This is what needs to be addressed, not the
final score in 3DM.

Maybe this is simply growing pains in the industry, a maturing of how we are
soon to quanitfy performance. Benchmarks are now suspect due to the ability
of GPU's to adapt to the running application, so anything that might be
previosuly-known-as-cheating is now, application specific optimization.

Surely, you wouldn't call it cheating if there was a table, in the driver
suite, that modified certain aspects of the rendering engine based on what
application was running, if the goal was to get a 'better' end result? But
as you say, when the end result is treated as a yardstick to the general
capacity of the card, you delve into murky ethical waters.

Derek Wildstar
June 23rd 03, 07:37 PM
"Folk" > wrote in message
...
> On Mon, 23 Jun 2003 06:29:53 GMT, "Derek Wildstar" >
> wrote:
>
> >More effort should be put into innovation and optimization, the leading
> >players are too busy dicking around with 3dmark, and they are both guilty
of
> >disappointing their customers.
>
> Problem is, in the enthusiast press arena, those companies live and
> die by benchmark results. So what are they to do? Both ATI and
> Nvidia would be dumb to simply ignore 3DMark, so they spend time
> optimizing for it. If review sites would ignore the synthetic
> benchmarks, and gauge performance based on actual games, then we would
> all benefit.


I posit it's now *not* dumb to ignore 3DMark, you with me?

I'd rather throw my support behind nvidia *and* ATI, and toss out
Futuremark. They have done nothing to bring gaming forward!

Derek Wildstar
June 23rd 03, 07:42 PM
"Ben Pope" > wrote in message
...

> >
> > Hot damn, you got me. :) Actually, I'm glad I don't. I don't want to
> > defend their behavior, but the sky isn't falling on nvidia, not over
> > this issue at least.
>
> Thats 'cos the sky has been clipped away!!
>

Only the part I wasn't looking at. :)

Lithurge
June 23rd 03, 07:48 PM
"John Russell" > wrote in
:


>> >
>>
>> If they have any sense then they will view 3dmark as one
>> of a large range of tools to test hardware, whether by
>> others or themselves. If 3dmark is showing it as running
>> slower, then people should be pleased when game x runs
>> faster because of these 'optimisations'.
>
> Anyone with any sense will NOT be using 3dmark to make
> judgements as to how well cards run GAMES! Anyone who does
> advise people to do so is being as deceitful as they would
> make Nvidia out to be.
>
>
>

Isn't that what I just said?

Please don't be an nvidia fanboy. All card makes have their pros
& cons as do various means of benchmarking them.

Personally I've gone with the best card available at the time I
have the money/desire to upgrade after checking a wide variety
of sources. This time nvidia have dropped the ball, as voodoo
did way back when. However they have the money & the skill to
pick it back up, and may well be my card of choice at the next
upgrade.


What I won't do is listen to pointless posts like yours that
ignore facts from unbiased sources & simply quote those of a
single biased source as you can't see beyond your own fandom.


If 3dmark is an 'unfair' benchmark it is unfair for all
manafacturers, or can you provide evidence otherwise? Rather
than fanboy raving?

Derek Wildstar
June 23rd 03, 07:48 PM
"Roger Squires" > wrote in message
.com...

> Websites now realize that benchmarking is a problem:
> http://www.hardocp.com/article.html?art=NDkx
>
> rms

This benchmarking issue might be brand spanking new news to Kyle, but it's
not to those who have treated benchmarks with healthy suspicion ever since
we learned to count to three.

Notice how the focus of the talk is on Futuremark, and their methodologies,
not some scandalous trickery by nVidia. It's absurdly ironic that the
on-line press, supposedly the most up to the minute in breaking news and
trends, lag so far behind when it comes to issues such as this.

hOCP is like the dingy saloon in the basement of the frathouse. Always
entertaining when frequented (I check them everyday), but utterly devoid of
meaningful commentary.

The fact that they finally acknowledge that *they* are part of the problem,
gives me great hope that they might graduate with honors!

Roger, you have toned down your condemnation of nvidia, have you changed
your opinion?

Courseyauto
June 23rd 03, 08:17 PM
> really...the Fx 5900 are selling as fast as they get in stock.

All 7 of them?
Hehe....

>>>

new egg got in the 5900 and 5900 ultra and sold them all in
less than an hour...HE....HE...HE...HE

Toby Groves
June 23rd 03, 08:18 PM
In article >, Ben Pope
> writes
>Toby Groves wrote:
>>
>> Now let's look at Ben's claim:
>
>Careful when quoting please - I did not say that.

My apologies, you're quite correct, my mistake.
--
Toby

ZOD
June 23rd 03, 11:51 PM
> Only the part I wasn't looking at. :)

...or the part you didn't know was there...hehe

ZOD
June 23rd 03, 11:55 PM
> My apologies, you're quite correct, my mistake.

Now there's a shocker...hehe

Derek Wildstar
June 24th 03, 02:28 AM
"ZOD" > wrote in message
hlink.net...
> > Only the part I wasn't looking at. :)
>
> ..or the part you didn't know was there...hehe
>
>

Is that a bad thing?

We really must stop meeting like this.

ZOD
June 24th 03, 05:44 AM
> Is that a bad thing?

Not if you don't know what you're missing....hehe

> We really must stop meeting like this.

Hehe....

Ben Pope
June 24th 03, 07:57 AM
Toby Groves wrote:
> In article >, Ben Pope
> > writes
>> Toby Groves wrote:
>>>
>>> Now let's look at Ben's claim:
>>
>> Careful when quoting please - I did not say that.
>
> My apologies, you're quite correct, my mistake.

No problems.

Ben
--
I'm not just a number. To many, I'm known as a string...

Folk
June 24th 03, 05:12 PM
On Mon, 23 Jun 2003 18:37:03 GMT, "Derek Wildstar" >
wrote:

>
>"Folk" > wrote in message
...
>> On Mon, 23 Jun 2003 06:29:53 GMT, "Derek Wildstar" >
>> wrote:
>>
>> >More effort should be put into innovation and optimization, the leading
>> >players are too busy dicking around with 3dmark, and they are both guilty
>of
>> >disappointing their customers.
>>
>> Problem is, in the enthusiast press arena, those companies live and
>> die by benchmark results. So what are they to do? Both ATI and
>> Nvidia would be dumb to simply ignore 3DMark, so they spend time
>> optimizing for it. If review sites would ignore the synthetic
>> benchmarks, and gauge performance based on actual games, then we would
>> all benefit.
>
>
>I posit it's now *not* dumb to ignore 3DMark, you with me?
>
>I'd rather throw my support behind nvidia *and* ATI, and toss out
>Futuremark. They have done nothing to bring gaming forward!
>

HardOCP posted an editorial yesterday afternoon that follows along
these same lines. I don't see eye to eye with Kyle Bennett on a lot
of things, but I do agree with him here. It's easy for review sites
to simply run a few synthetic benchmarks, through up some graphs and
call it a review. It will be much harder to come up with how graphic
card 'A' provides a better gaming experience than graphic card 'B' in
any given game.

One of the areas where HardOCP still doesn't get it is with the focus
of a graphics card. Yes, they are geared toward 3D processing power,
but even the most hardcore gamers like to play videos and do other 2D
tasks, but you can search high and low and no review sites give any
lip service to 2D quality or DVD/video features. As pc's become more
integrated with home entertainment systems, the ability of a card to
handle those types of tasks will become more important.

Derek Wildstar
June 25th 03, 02:23 AM
"ZOD" > wrote in message
thlink.net...


>
> Not if you don't know what you're missing....hehe

I wouldn't if you didn't tell me.