When it comes to benchmarking software, Futuremark’s 3DMark Fire Strike is the gold standard among PC enthusiasts. But there are limits to what Fire Strike can measure; as a DirectX 11 implementation, the tool can’t quantify how graphics cards perform with Microsoft’s cutting-edge DirectX 12 graphics technology. That’s a problem as more and more DX12 games hit the streets.
Or rather, it was a problem.
Today, Futuremark released its new Time Spy benchmark for the 3DMark suite, which is loaded with DirectX 12 features to put graphics cards to the next-gen test.
Time Spy isn’t Futuremark’s first DirectX 12 benchmark; that honor goes to the API Overhead Feature Test released in March 2015. But the Overhead Feature Test merely measured draw call performance in DirectX 11 versus DirectX 12 (or the now-defunct Mantle). Time Spy is a full-blown benchmark composed of several subtests that factor into an overall combined score. It’s the DX12 equivalent of Fire Strike, basically.
And it lets you test with asynchronous compute both enabled and disabled, which leads to some interesting insights. More on those in a bit.
Better yet: Time Spy’s coming to all versions of 3DMark, including the Basic Edition that’s free for personal, noncommercial use. (You can download it on Futuremark’s site or by clicking the Download demo button on 3DMark’s Steam page.) Time to start your engines, folks—at least if you have Windows 10.
The paid-for Advanced Edition of 3DMark (which offers additional features and benchmarks) is rising in price from $25 to $30 thanks to Time Spy’s inclusion, however. If you already own 3DMark Advanced Edition, you can purchase Time Spy separately for $10. But if you buy before July 23 you’ll be able to upgrade for $5 or buy the complete Advanced Edition for $10.
Futuremark provided PCWorld early access to Time Spy to see how it handles. Let’s take a deeper look at this new DirectX 12 benchmark, and the performance results for a range of AMD and Nvidia cards.
Meet Time Spy
Futuremark developed Time Spy with input from AMD, Intel, Nvidia, and Microsoft, with an eye towards testing DirectX 12’s unique advantages. By giving developers lower-level access to hardware, DX12 offers reduced processor overhead, better multi-threading support, and the ability to get “closer to the metal” of graphics cards.
To drive home how much more DirectX 12 can process compared to DirectX 11, Futuremark provided a chart showing the average amount of processing per frame in Fire Strike versus Time Spy.
Beyond merely being built in DirectX 12, Time Spy also supports many of DirectX 12’s highlight features. That includes asynchronous compute, which allows compute and graphics rendering tasks to run simultaneously, rather than one after the other; explicit multi-adapter in the linked-node configuration, which essentially performs similarly to SLI or CrossFire, but allows the benchmark to pool the resources of the two graphics cards; “explicit synchronization between CPU, GPU, multiple GPUs, and multiple GPU queues;” and several other under-the-hood tricks like resource binding and data “heaps,” all running in a custom engine.
Time Spy’s actually broken up into a trio of tests running at 2560×1440 resolution in the official benchmark configuration. It starts off with two graphics-focused tests that stress your GPU in differing ways while minimizing the CPU load, to get as much of a pure GPU result as possible.
The first graphics test (pictured above) stars the titular time spy wandering through a museum that features artifacts and weapons from throughout time and space, complete with easter-egg-style nods to other 3DMark benchmarks and games like Halo, Far Cry, and Deus Ex. This first test “focuses more on rendering of transparent elements,” according to Futuremark.
The second graphics test pans and zooms throughout the same museum, showing different areas illuminated by different light sources than the first test. It “focuses more on ray-marched volume illumination with hundreds of shadowed and unshadowed spot lights.” The two graphics tests each receive separate scores, which are combined into an overall graphics score.
Then a third test hammers your CPU—the fans on our test system’s closed-looped cooler screamed during this section—while minimizing the GPU load. It features ice-like crystal clusters spreading throughout the screen using procedural generation, constantly transforming and growing. “The CPU test measures performance using a demanding combination of physics simulation, occlusion culling, and procedural generation,” Futuremark says.
Once everything’s done running, the CPU score and GPU score are combined to generate an overall score using a “weighted harmonic mean.”
So what does “weighted harmonic mean,” well, mean? Here’s how Futuremark describes it:
“For a balanced system, the weights reflect the ratio of the effects of GPU and CPU performance on the overall score. Balanced in this sense means the Graphics and CPU test scores are roughly the same magnitude.
For a system where either the Graphics or CPU score is substantially higher than the other, the harmonic mean rewards boosting the lower score. This reflects the reality of the user experience. For example, doubling the CPU speed in a system with an entry-level graphics card doesn’t help much in games since the system is already limited by the GPU. Likewise for a system with a high-end graphics card paired with an underpowered CPU.”
Got it? Good. Let’s see how several modern graphics cards compare in Time Spy.
Time Spy tested
We selected six cards to put Time Spy through its paces.
With AMD pushing DirectX 12 so hard, as it takes superb advantage of the dedicated async compute engine hardware inside Radeon graphics cards, testing the Polaris GPU-based $200 Radeon RX 480 was a no-brainer. That card’s competitive with Nvidia’s older Maxwell GPU-based EVGA GTX 970 FTW. We also wanted to test AMD’s older Hawaii GPU, so MSI’s R9 390X Gaming was added to the mix, along with its GeForce counterpart, the reference edition of the GTX 980. Finally, we also tested the Asus Strix Fury and Nvidia’s GeForce GTX 1070 Founders Edition to ensure AMD’s Fiji GPU and Nvidia’s Pascal GPU were each tested.
Why no Fury X? Because we were under time constraints and installing its integrated closed-loop water cooler into our test system’s Corsair Obsidian 750D case is a pain if you’re swapping out a bunch of cards.
Let’s see how things shook out. Focus mainly on the graphics (GPU) score.
First impressions: The power of the dedicated async shader hardware inside Radeon graphics cards shines in this test. Excluding the ridonkulously powerful GTX 1070, Radeon hardware dominates Nvidia’s GeForce cards in this DirectX 12-focused test, lending yet more credence to the idea that AMD’s cards have an advantage in next-generation graphics technologies. Even the $200 RX 480 thoroughly trounces the GTX 980. (Note that Nvidia tends to hold an advantage in DirectX 11 performance, however, which still accounts for the vast majority of game releases.)
The CPU scores are largely similar across the board, but we’re using a $1,000, eight-core Core i7-5960X. There’s a decent chance we’d see more variance in a system with a weaker processor, as “closer to the metal” APIs like DirectX 12 show larger performance increases in systems that are CPU-bound.
Here’s the thing, though: Time Spy leans heavily on asynchronous compute to “overlap rendering passes to maximize GPU utilization,” which naturally favors the dedicated hardware in Radeon graphics cards. While many DX12 games will feature async compute, it’s not a required feature of the API.
You can disable async compute in a custom run, however. We did so to see how AMD and Nvidia’s cards compare with that crucial feature turned off, and to see how the GTX 1070—which includes key new async compute features—performs in relation to the older GTX 970 and GTX 980. (Note that there is no overall score assigned in custom runs.)
And here’s a graph comparing just the graphics scores of the cards, with async compute both enabled and disabled.
Now isn’t that interesting.
Nvidia’s GTX 970 and GTX 980 see very minor graphics score decreases running Time Spy with async compute disabled, but remain static for the most part. Meanwhile, the GTX 1070 sees a larger decrease (while still dominating overall), indicating that the async improvements Nvidia built into the Pascal GPU indeed make a difference.
But the graphics scores for AMD’s cards positively plummet, driving home just how important the dedicated async shaders are for Radeon graphics cards—at least in DX12 games that take advantage of asynchronous compute. With async compute disabled, the Radeon cards tumble back down to relatively equal scores with their Nvidia counterparts, similar to what we’d see if testing these cards in DirectX 11.
Also compare the CPU scores between the two runs. It’s clearly dependent on the GPU’s async compute scheduling in some way, resulting in a fairly big difference when async compute is disabled.
Go forth and benchmark
But don’t necessarily take our word for it. Want to see how your system handles DirectX 12, or simply try to push your rig to the top of the Time Spy leaderboards? Go download Futuremark’s 3DMark suite—and expect to see Time Spy results show up in PCWorld’s graphics card reviews in the future. Get it? Time Spy? In the future?
Okay, I’ll stop now.