Welcome to Viper's Lair - Benchmarking

With the boom in the use of computers over the past few years we have also seen a extreme rise in the number of users on the internet. And with many people wanting to know more about their computers and how they can upgrade it most effectively. Such sites as Tomshardware, Anandtech and HardOCP have sprung up as a direct result of this fact.

As a reader of other people's reviews, consumer and also as a hardware reviewer I try to look at reviews (my own included) and try to see what good things are in them and try to see what can be improved on. The following is more or less what I've inferred from what I've read others say, and from my own reading. This article is meant as a minor rant, with some logic (hopefully) behind it.

One of the major tools used by any site to convey performance is benchmarks (this site included). But do most sites convey all the information that they can? Are they as impartial as they can be? Is there anything that sites, such as Viperlair, can do to provided a more complete picture of products we look at? This article will look at a few ways that we, as reviewers can give more information in greater detail in a few areas.

Video Card Benchmarks

When you buy a 3D video card, you buy it knowing that you want to play games with it, and not programs that just spit out a number at you. The benchmarks used in video card reviews haven't really changed in the past while, which is saddening. Many people still rely on programs such as Quake III and 3D Mark 2001SE to test video cards. But with Quake III we can see that the video card is no longer really a limiter, even at 1600*1200 (taken from Huberts review of the PNY Ti4600):

Many people consider anything over 60fps to be playable, and we can see with even the current/previous generation average graphics cards, they have no trouble reaching frame rates over 100fps for a card that was only about $100 (US). We can see even the more mainstream video cards today can easily play at 1600*1200, thus making Quake III a benchmark that really has no meaning with newer video cards. This is because most people would not be hindered by buying one of these video cards instead of a faster/more expensive one. Another thing to note is that the frame rate range used is from 100fps to 160fps, which exaggerates the differences between the cards tested here. What about 3D Mark 2001?

With 3D Mark 2001 we get a number that is supposed to represent how your computer will perform in games but is not always accurate. Overall it is a decent program to look at, but it has many flaws. Some people obviously try to optimize their systems for this program, while not showing any performance gain in normal games. Also the number that you are given is just that, a number, it doesn't necessarily reflect the true performance of the card. As an example of this let us look at how the Kyro II performs in 3D Mark 2001 compared to the GeForce 2 Pro, which in most games performs similarly.

As you can see from a sampling of the top five results in the Mad Onion database (for all XP's between 1481-1547MHz) the Kyro II is about 2500 'Marks' less than the GF2 Pro's, yet in most games it performs as well as or better than a GF2 Pro. We can see that this helps show the ineffectiveness of 3D Mark 2001 to show the real life performance of a video card, and how it tries to reduce the performance of games to a simple number but in some cases fails. This is more a problem of synthetic benchmarks, including , , , and to a lesser extent and due to the fact that they only simulate real life games or in some cases don't even do that. Another point is that many of the tests can be run using Max Payne based on the same engine as parts of 3D Mark 2001. We seem to have 'bashed' some of the most used benchmarks so let us change focus and look at what can be done to improve the benchmarking procedure.