Standardized Metrics for Compute Marketplaces
Standardized Metrics for Compute Marketplaces
Time to move away from GPU count to FLOPS
Over the past year, there has been a surge in new compute networks, largely fueled by the AI hype cycle. At the same time, established compute networks, like Render, Akash, and Bittensor have thrived amidst this increased focus on computational resources, with their valuations soaring over 300% in the last year.
At its core, the intention of each network’s protocol is to incentivize contributors to direct compute resources towards the network. And the success of these networks—or their ability to effectively serve the demand-side—hinges on how well they can attract and sustain these compute resources. Therefore, one of the key metrics for these networks is the total amount of compute resources amassed.
Surprisingly, it’s not easy to actually find these metrics for many compute networks. Some don’t make this information readily available, while others have been caught red-handed inflating their numbers. Considering the revenue from these networks is merely a rounding error when compared to traditional cloud provider revenues, how else is an investor supposed to value these networks if they can't determine how much compute resources a network truly possesses?
GPU Count
Using GPU count to represent a network’s total compute resources isn’t the best metric either. If a network solely shows that it has 5,000 GPUs, for example, there’s no way to know the breakdown. They could all be low-end consumer GPUs, for all we know.
Additionally, compute networks are evolving beyond merely having GPUs on the supply end. We now have application-specific computing hardware like TPUs, LPUs, FPGAs, and other types such as Apple’s M2 and M3 chipsets being integrated into networks. Eventually, we'll likely see other consumer products connected to these networks, like Teslas, smartphones, and maybe even your smart fridge.
FLOPS
What is needed is a standardized metric that transcends the naive GPU count to represent a network’s compute capacity. The best way to do this is likely using Floating Point Operations per Second (FLOPS).
FLOPS is a well-established and widely recognized metric for measuring the raw computational performance of different hardware, whether it’s CPUs, GPUs, or other processors. Using FLOPS to represent a network's total compute capacity/resources seems like a perfect fit.
And a compute network within Bittensor (subnet 12) has done exactly this.
GPU counts can actually be misleading too. For example, a compute network with 2,000 RTX 4090s sits at 2,600 TFLOPS, whereas a network with 120 H100s has a capacity of 3,072 TFLOPS (using FP-64).
With the number of FLOPS, one can then derive more interesting metrics such as revenue per FLOPS capacity or be able to estimate a network’s inference performance potential.
Wouldn’t it be weird if storage networks didn’t show their total storage capacity and utilization rate? That’s the stage we’re at with compute networks at the moment.
As we continue to see an explosion of decentralized compute networks, it’s crucial to adopt standardized metrics like FLOPS to provide a clear, accurate picture of their computational power. It’s also important that this metric be verifiable on-chain. By doing so, we can foster greater transparency and trust, enabling investors, contributors, and users to make more informed decisions.