Benchmarking - How and Why?: Benchmarks exist to determine how a particular piece of hardware performs in relation to itself, and to others. Question is, are readers really getting the information they really need?

    With the boom in the use of computers over the past few years we have also seen a extreme rise in the number of users on the internet.  And with many people wanting to know more about their computers and how they can upgrade it most effectively.  Such sites as Tomshardware, Anandtech and HardOCP have sprung up as a direct result of this fact.  

    As a reader of other people's reviews, consumer and also as a hardware reviewer I try to look at reviews (my own included) and try to see what good things are in them and try to see what can be improved on.  The following is more or less what I've inferred from what I've read others say, and from my own reading.   This article is meant as a minor rant, with some logic (hopefully) behind it.

    One of the major tools used by any site to convey performance is benchmarks (this site included).  But do most sites convey all the information that they can?  Are they as impartial as they can be?  Is there anything that sites, such as Viperlair, can do to provided a more complete picture of products we look at?   This article will look at a few ways that we, as reviewers can give more information in greater detail in a few areas.

Video Card Benchmarks

    When you buy a 3D video card, you buy it knowing that you want to play games with it, and not programs that just spit out a number at you.  The benchmarks used in video card reviews haven't really changed in the past while, which is saddening.  Many people still rely on programs such as Quake III and 3D Mark 2001SE to test video cards.  But with Quake III we can see that the video card is no longer really a limiter, even at 1600*1200 (taken from Huberts review of the PNY Ti4600):

    Many people consider anything over 60fps to be playable, and we can see with even the current/previous generation average graphics cards, they have no trouble reaching frame rates over 100fps for a card that was only about $100 (US). We can see even the more mainstream video cards today can easily play at 1600*1200, thus making Quake III a benchmark that really has no meaning with newer video cards. This is because most people would not be hindered by buying one of these video cards instead of a faster/more expensive one. Another thing to note is that the frame rate range used is from 100fps to 160fps, which exaggerates the differences between the cards tested here. What about 3D Mark 2001?

    With 3D Mark 2001 we get a number that is supposed to represent how your computer will perform in games but is not always accurate.  Overall it is a decent program to look at, but it has many flaws.  Some people obviously try to optimize their systems for this program, while not showing any performance gain in normal games.  Also the number that you are given is just that, a number, it doesn't necessarily reflect the true performance of the card.  As an example of this let us look at how the Kyro II performs in 3D Mark 2001 compared to the GeForce 2 Pro, which in most games performs similarly.

    As you can see from a sampling of the top five results in the Mad Onion database (for all XP's between 1481-1547MHz) the Kyro II is about 2500 'Marks' less than the GF2 Pro's, yet in most games it performs as well as or better than a GF2 Pro.   We can see that this helps show the ineffectiveness of 3D Mark 2001 to show the real life performance of a video card, and how it tries to reduce the performance of games to a simple number but in some cases fails.  This is more a problem of synthetic benchmarks, including , , , and to a lesser extent and due to the fact that they only simulate real life games or in some cases don't even do that.  Another point is that many of the tests can be run using Max Payne based on the same engine as parts of 3D Mark 2001.  We seem to have 'bashed' some of the most used benchmarks so let us change focus and look at what can be done to improve the benchmarking procedure.

    As I stated before, we buy video cards to play games, so therefore we should focus on this, instead of the synthetic results that don't really mean much in the way of real life performance.  We would also want to be flexible to a certain extent, while holding on to some benchmarks for comparison's sake.  Quake III might be a choice for comparisons, but there are many other programs that came out after that, that are just as good.  A couple of examples of this are Jedi Knight II (same engine as QIII), Serious Sam SE, or even Max Payne.  We as reviewers want to be flexible because the gaming world around is changing at a fast pace, with new games being released with new features, there is a need to show how video cards perform in these environments which can't be done with current games (think Doom3).

    This brings us to the next point.  Until the release of UT2003, most Direct X benchmarks used in reviews consisted of 3D Mark 2001.  There are other options for DirectX benchmarks, one just has to look around.  The guys at devised a way to benchmark Max Payne which turned out to be a very good benchmark.  An interesting benchmark I found out about at was for a DirectX 8 game called Ballistics, which is a very nice 'racing' game that can be used to test video cards in a non fps game.  The last one I'll mention is that of AquaMark, it is harder to come by, but is an interesting real/synthetic benchmark.

    Another thing I have noticed sorely lacking from many reviews (even some of mine and others who write for this site) is a lack of quality tests, and more specifically 2D quality tests.  Since you are looking at a 2D display for the entire time you use your computer (unless you have some new 3D display), it therefore deserves, rather requires, testing to see its quality in everyday use.  The best way I've seen is to have a blind test with multiple people (preferred, to represent regular computer users), using one video card as a base for comparison to the others.  Good tests are very importent as well, including text and color images, to test the saturation and quality of the images produced by the video card (not the monitor as it is a control in this case).  If more people did this, it would help many people in their purchases (mine included) as I don't want to buy a video card with pathetic 2D quality (and yes many are sold even today).  

    One last thing we will look at with video card benchmarking is detail.  We need to provide you the reader with as much unbiased information as possible to counter any bias we might have.  How can we do this?  An interesting way of doing this is by providing a graph of the second by second frame rate, while also providing the minimum/average/maximum frame rates.  This has been made possible by the program Fraps, which doesn't use up much in the way of resources (~1% of the frame rate) and allows for the frame rate to be recorded every second.  Lets look at an example of this in action using the results from Serious Sam SE at 1600*1200 from my Parhelia review.  First we'll look at just a bar graph, and then look at the graph found in that review.

    Nothing unusual here, it looks as if the Parhelia is just a little bit faster than the Kyro II, apart from that we can't infer much else.   What about if we chart the results?