Exploring “Frame time” measurement – Part 1 – Is Fraps a good tool?
Faster. Smoother. Richer. These three words define Nvidia’s Kepler marketing campaign which attempts to convey what a gamer wants to experience – (1) Faster; more raw frame rates as measured as frames per second (fps). (2) Smoother; the gamer should not experience hiccups or stuttering which we are now able to easily measure and chart. And (3) Richer; the overall gaming experience should be excellent and fully-featured.
One of the tools most often used to convey the performance differences between video cards is Fraps. Most commonly, it is used to measure fps (frames per second), or frame rates which convey how evenly the frames are delivered. Fraps is quite popular among enthusiasts and there is a free version that can also measure frame times as well as frame rates. It is a versatile tool as seen from the Fraps website:
It has been noted for nearly a decade that just measuring fps does not convey the full experience if the frames are delivered unevenly – if there is stutter, micro stutter, or jitter. Recently, Scott Wasson of the Tech Report began to measure “frame times” which are also measured by Fraps. He is not the first, as Lost Circuits experimented with this in 2008 but returned to fps charts for simplicity sake.
It is pretty well-established that Fraps is a very good measure of frame rates (fps), but what about its ability to measure frame times? Fraps also outputs these results that can be charted in Excel or a similar spreadsheet program if “frametimes” is checked (along with “FPS” and “MinMaxAvg” which are usually measured, under the “FPS” tab).
We are going to look at two in-game benchmarks that also measure frame times and we will compare them using Fraps frame time measurements. We are using the STALKER, Call of Pripyat stand-alone benchmark and Metro 2033’s official benchmark, and we shall also look at Far Cry 2.
We are using the stock-clocked GTX 680 and if our testing shows Fraps to be a useful tool, later benching will pit its frame time smoothness of delivery versus the HD 7970 at GHz edition clocks. Raw frame rates simply do not convey the complete gaming experience and this can be easily illustrated with multi-GPU – where otherwise satisfactory frame rates may feel much slower than then they should due to uneven delivery of frames. 
One thing to note is that we overclocked our Core i7-3770K CPU to 4.5GHz to make sure that there is no CPU bottleneck. We are testing at 1920×1080. Although we are only testing three games in this Part 1, generally we use 2x/4xAA or 8xAA plus 16xAF and with maximum (ultra) DX11/10.1/10/9c details whenever it is available. We are benching with GeForce 310.70 WHQL drivers.
Please continue on to the next page for the complete hardware and software setup of our testing platform.
Test Configuration
Test Configuration – Hardware
- Intel Core i7 3770K (overclocked to 4.5GHz); Turbo is on.
- EVGA Z77 FTW motherboard (Intel Z77 chipset, latest beta BIOS, PCIe 3.0 specification; CrossFire/SLI 16x+16x using Plex chip.)
- 8GB Kingston DDR3 PC1866 Kingston RAM (4×2 GB, dual-channel at 1866MHz; supplied by Kingston)
- Power Color Radeon HD 7970 (3GB, overclocked to GHz Edition speeds 1050/6000MHz)
- Nvidia GTX 680 (2GB, 1006/6008MHz, reference clocks), supplied by Nvidia
- Onboard Realtek Audio
- Two identical 500 GB Seagate Barracuda 7200.10 hard drives configured and set up identically from drive image; one partition for Nvidia GeForce drivers and one for ATI Catalyst drivers
- Kingston 240GB HyperX SSD used for AMD partition, supplied by Kingston.
- Kingston 240GB HyperX 3K SSD used for Nvidia partition, supplied by Kingston
- OWC 240GB Mercury EXTREME Pro 4G used for both AMD and Nvidia, on loan from OWC.
- Thermaltake ToughPower 775 W power supply unit supplied by Thermaltake
- Thermaltake Overseer RX-I full tower case, supplied by Thermaltake
- Sapphire Vapor-X CPU cooler on loan from Sapphire
- Philips DVD SATA writer
- ASUS VG278 27″ Light Boost enabled 3D Vision 2 ready 120Hz display
Test Configuration – Software
- Fraps 3.5.9 full version
- ATi Catalyst 12-11 Beta11 drivers; highest quality mip-mapping set in the driver; use application settings; surface performance optimizations are off. Latest CAPs used.
- Catalyst Control Center used to set power draw to maximum for the HD 7970; clocks, voltage and fan profiles are stock
- EVGA PrecisionX used to set power draw to maximum for Nvidia cards; clocks voltage and fan profile are stock
- NVIDIA GeForce WHQL 310.70 High Quality
- Windows 7 64-bit; very latest updates
- Latest DirectX
- All games are patched to their latest versions.
- vsync is forced off in the control panels.
- Varying AA enabled as noted in games; all in-game settings are specified with 16xAF always applied; 16xAF forced in control panel for Crysis.
- All results show average frame rates and frame times
- Highest quality sound (stereo) used in all games.
- Windows 7 64, all DX9 titles were run under DX9 render paths, DX10 titles were run under DX10 render paths and DX11 titles under DX11 render paths.
The Benchmarks
-
DX11
- STALKER, Call of Pripyat
- Metro 2033
- Far Cry 2
Let’s head to the charts and see the frame time comparison between Fraps results and the Call of Pripyat and Metro 2033 engine results.
Metro 2033
Let’s first look at Metro 2033. Everything is maxed out at 1920×1080 except for DOF which provides very little in the way of visuals but takes a large performance hit. We are going to examine one representative run and compare the frame time chart as reported by the engine and made into a chart as compared to a Fraps run of the same run and compared with the frame time chart as reported by Fraps.
First up is the fps chart that the game engine reports:
Now here is an Excel fps chart of the very same run as charted by Fraps. Thanks to Grstanford of ABT forum for making the calculations for all of the following charts!
We note that in both cases, the results are very similar. What about Frame Times? Here is a comparison of how the Fraps and the Metro 2033 game engine displays frame times in Excel:
On the left are the frame times as reported by the game engine opened in an Excel spreadsheet. In the center is the same run as reported by Fraps; and on the right is the Metro 2033 game engine’s individual frame times after being recalculated.
Here are the Excel charts that were produced – first the Metro 2033 frame times as reported by the game engine:
Now the same run as reported by Fraps:
As you can see, Fraps reports very similarly to Metro 2033 in both frame rate and frame times. The Fraps run would be a second or two different from the official benchmark due to starting Fraps benching immediately before or after the bench actually begins and stopping it slightly before or after it ends. When watching this benchmark, one can actually see the stutters. It appears to have been optimized for performance and higher fps, not smoothness.
Let’s now look at STALKER, Call of Pripyat
STALKER, Call of Pripyat
We use maxed out settings but only used 2xAA as we want to average over 60 fps.
Here are our fps averages as reported by the SunShafts benchmark:
As with Metro 2033, we are going to examine one run and compare the frame time chart as reported by the engine and made into a chart as compared to a Fraps run of the same run and the frame time chart as reported by Fraps. First up is the chart that the game engine reports:
Now the same run as reported by Fraps:
And again, the runs are so similar as to convey approximately the same information. It fits very well in with the experience of watching this benchmark – a small chug at the beginning, then it settles down until the scene changes and lighting comes into play – smoothness alternates with slight jitter until we reach the last scene that settles down nicely. There is a better balance of good performance with overall smoothness compared with Metro 2033 which jitters more noticeably.
These frame time charts are all well and good in supporting frame rates, but who is going to want to spend the time to make the calculations necessary to create these individual frame time charts? Fortunately there is an easy solution.
Enter Frafs Bench Viewer
Lindsay Bigelow, an established member (raffriff) of FrapsForum.com, developed an uncomplicated tool for simply dragging and dropping Fraps frame time csv files into a program, named Frafs Bench Viewer, that quickly makes a chart. We would like to support his efforts by also using his tool and encouraging our readers to also make these simple charts and perhaps share them on ABT forum.
Frafs Bench Viewer can be downloaded at SourceForge.com or from Softpedia.com as freeware. All you do is drag and drop a Fraps frametimes.csv file into the program and it automatically creates customizable graphs!
Here is a graph produced by the Frafs Bench Viewer of the same Call of Pripyat benchmark as the last graph above:
It may look different from the frame time Excel graph that we are repeating again here . . .
. . . until we compress it:
The Frafs Benchmark Viewer is a very useful and easy-to-use tool to display frame times and we encourage our readers to submit their own graphs and experiences on ABT forum.
Let’s look next at Far Cry 2.
Far Cry 2
Far Cry 2 benchmark also reports the frame times but it rounds them off quite differently so that the spikes are not as apparant as they are with Fraps. Here is the game engine frame times (in brown) superimposed over the Fraps frame times (in blue):
In all three cases, Fraps frame times follows the three game engine frame times closely enough to be useful in conveying the experience of “smoothness” and some of the spikes in the chart can be noticed by a careful observer – or as we shall show in this continuing “smoothness” series – by a high-speed digital camera.
Let’s head to our conclusion about Fraps frame time measurements.
Conclusion
This has been quite an enjoyable hands-on experience comparing Fraps frame time runs with three game engines actual reporting of frame times. We feel reasonably confident as we proceed that Fraps is as useful a tool in reporting frame times as it is is for frame rates. It is not perfect, but it conveys a reasonable assessment of what is actually being experienced in-game by the user regarding smoothness or its lack.
We also like Frafs Benchmark Viewer as a simple and easy tool to create nearly instant graphs out of Fraps frame time runs. We plan to continue to use this tool as we explore “smoothness” by measuring frame times, relating them to frame rates and perhaps posting Youtube videos of our own GTX 680 versus Radeon HD 7970 side-by-side runs at 60 fields per second, 120 fps, and 240 fps.
If you would like to weigh in on our future testing methods or to specifically discuss micro stutter and frame time testing methods, feel free to comment below, ask questions, make requests, or have a detailed discussion at ABT forum.
Stay tuned, there is a lot coming from us at ABT including Part Two of “Exploring Frame Time Measurement”. Don’t miss it this week as we compare the GTX 680 versus the HD 7970 as to smoothness. Stay tuned to ABT for the latest regular news and Top Apps, as well as a Genius PC 2.0 and 4.0 speaker evaluation as well as a Newer Technology Voyager USB 2.0/3.0 Drive Dock test drive.
Happy gaming and Happy New Year!
Please join us in our Forums
Become a Fan on Facebook
Follow us on Twitter
For the latest updates from ABT, please join our RSS News Feed
Join our Distributed Computing teams
- Folding@Home – Team AlienBabelTech – 164304
- SETI@Home – Team AlienBabelTech – 138705
- World Community Grid – Team AlienBabelTech
wow. These results are extremely similar. More so than i ever thought. Fraps result do differ by a tiny amount but this is because the game engine and fraps grab their data completely different. Fraps monitors DX/openCL calls after the fact and it being an external program throws in its own latencies, however small they may be. But through all this the trend graphs show near identical trends for fraps vs the internal game engines. Its amazingly accurate and the results are like mirror images.
Some people may not know the point in these testings so to be clear i might add it. Most games do not have any way to gather frame times from the engine at all. Fraps can do this rather easily anytime while playing or benchmarking. The question was whether fraps was accurate enough to rely on at all. Whether the spikes meant what they were being represented as. Having these test with the game engines producing near identical trend graphs gives high credence to fraps being an acceptable tool to capture frame times. We also have the reviewers experience which seemed to align with the frame time data very nicely. He seen the small hitching or studdering that the trend charts represent.
It’s remarkably easy, actually. Fraps and in game timing work by time stamping the ‘swap buffer’ command which indicates end of frame. As the timestamp is an internal CPU counter the only difference would be either side of procedure call so both would differ by at most a few nanoseconds. Fraps would be at most a few kilobytes of code mostly being a stub for forwarding calls to the ‘real’ driver.
The question “whether fraps was accurate enough to rely on at all” was not answered “at all”, as Fraps and games used the same measure of “frame time”: time between calls to “swap buffer”, as Pancake mentioned here.
But the real question: does it measure correctly all the stuttering we see on the screen? And the answer is almost certainly: “No”. As after call to Present() driver can do a quite a lot of work before finally presenting the frame to the display device.
And although it doesn’t matter in the terms of average frame counts it does matter when you want to measure “jitter”.
Sadly, the only way to correctly measure the frame time is to use profiling tools from the vendor (Nvidia or AMD), and the game needs to support this “profiling” mode (none of them obviously ships with it “on”) or some clever way to circumvent it by making game to run with “hacked” D3D library.