Methodology for testing video cards: 2007. Use of FRAPS
Author:
Date: 06.12.2006
|
|
Tests in NFS Most Wanted using FRAPS
The test setup and the video card are the same. We chose the circular track "Ironwood", drove two rounds, with three rivals and the road traffic set to the maximum. That is, the game conditions are quite real and absolutely not predictable in terms of repeatability. It rains, gets dark or light, and the rivals behave like real hooligans. Now let's take the results file "… frametimes.csv" and build first a graph of instant FPS values, and then a distribution diagram.
As you see, the FPS values are quite uniformly distributed around the 60 FPS point. To be more precise, the average value is 59.2 FPS.
The shape of the distribution diagram resembles a "bell" even more than in the F.E.A.R. test, even without discarding "redundant harmonics". The vertical red line, as before, marks the average FPS value and is almost in the very middle of the imaginary "bell". Therefore, we can state that the resultant average FPS value does reflect the real performance of the video card in this game.
Let's turn to the FPS graph and its division into four parts. We did that to clarify the following point – how long should we ride along the track to produce the normal "bell"? In other words – How does the form of the diagram and the average FPS value deduced with FRAPS depend on the time of tests.
In our conditions of tests, we drove two rounds along the circular track. That is, each quarter from the division of the FPS graphs is equivalent to approximately half the round. Let's build a diagram of FPS distribution for the first quarter (or first half of the first round).
First quarter. The time of test is 42 s, with the average value being 56.5 FPS. The distribution diagram does not look like a "bell" at all.
Two quarters (first round). The time of test is 84 s, with the average value being 59.9 FPS. On the diagram, we gradually get the sought for "bell".
Three quarters. The time of test is 126 s, with the average value being 58.8 FPS. There are no essential differences from the previous diagram. The "bell" becomes more oblong, but its shape almost doesn't change.
Four quarters (two full rounds). The time of test is 169 s, with the average value being 59.2 FPS.
It follows from the above that the criterion of minimum time required for tests – a duration of tests - should be such that further increase in the time of tests not result in the change of the shape of the "main bell" of the distribution diagram.
The resultant series of average FPS values for each sector looks like this – 56.5, 59.9, 58.8, 59.2. As you can see, only the value for the first quarter makes an essential difference from that of the other three. If we discard it (as per the formulated criterion) and find the average of them for each sector, we get 59.3 FPS. At the same time, the deviation amounts to merely 0.6 FPS both sides, that is, no more than ± 1% !
In our case, when testing in NFS Most Wanted it suffices to "drive" a round (84 sec of the testing time) or one and a half rounds (126 sec of testing time). Knowing the time of tests and the average FPS value, we can approximately estimate the overall counts using the following formula – "multiply the testing time by the average FPS value". Therefore, the required number of counts when the statistical method starts working is 5000-7000. We intentionally draw your attention to this fact. If we take a rather weak video card, the average FPS will be much lower, therefore, the overall number of counts will go down if the testing time remains unchanged. Therefore, when testing "weak" video cards it is advisable that the time of tests be longer.
Final Words
FRAPS is a powerful and precise tool for measuring performance of video cards. To produce true data,
- choose a demo scene most adequate to the typical gameplay
- run the test for quite a sufficient time
- use the FRAPS results file of the type "… frametimes.csv"
- upon transformation of source data of the file " … frametimes.csv", calculate the sought-for average FPS values and, if needed, the minimum and maximum FPS.
Conclusion
The presented method of tests using FRAPS and further data processing, it is possible to produce precise and adequate enough values of video cards performance in games that have no integrated benchmark. Even more than that. The specified method allows to analyze the "quality" of demo scenes in benchmarks integrated into games as well as get the idea of the nature of the game engine.
Unfortunately, in this review we had to omit one rather vital point - what should be regarded the minimum FPS value? As you saw on the example of test integrated into F.E.A.R., the minimum FPS values shown by the integrated benchmark may substantially differ from reality. Believe me, that is merely a tip of the iceberg. This issue requires an in-depth investigation. And we'll tell about that next time.
Discuss the review in a forum
 |
Content: |
 |
|
|
Top Stories: |
|
 |
|
MoBo:
|
|
|
|
VGA Card:
|
|
|
|
CPU & Memory:
|
|
|
