Inference Speed Matters So Much More Than You Think
文章目录
Everyone is obsessed with making AI smarter, yet most are drastically underestimating how valuable it is to make the intelligence we already have run faster.
"Yeah but Intelligence and accuracy come first, right? Surely nobody should pay a premium to get the wrong answer sooner!"
Yes, but...that does not mean intelligence is always the deciding factor.
There are essentially two ways to think about what faster inference looks like in practice:
-
Imagine two systems that can both do the job, except one finishes multiple times faster than the other
-
Imagine two identical models, but this time one is ran on infrastructure that delivers its capabilities substantially faster.
But here’s where it gets much more interesting:
Inference speed does not just determine how quickly you reach the finish line. It can change what you are capable of doing before reaching it too.
For example:
With the slower model, there may only be enough time to generate the initial answer. A faster model can use that same time budget to generate an answer, test and/or verify it, catch a mistake, and correct it all before responding. In that case, faster inference does more than just reduce the waiting time
It enables a more capable reasoning process within the same latency budget
Whether you need to move faster than everyone else or simply get more done per unit of time, ultra-fast inference is built for you. Below, I tie together agentic workflows, developer productivity, research velocity, infrastructure constraints, and why I think today’s ultra-fast inference market is still less than 1% of what it could become in the near future, and what that implies for the companies positioned to supply it!
Intelligence Is Only Half the Product
In February, Opus 4.6 Fast launched in Cursor with a speed improvement of up to 2.5× and list prices of $30 per million input tokens and $150 per million output tokens.
That was 6× the standard price!
The proposition was not that you were buying a smarter model. Anthropic defines fast mode as the same exact weights, just under a faster inference configuration. So basically:
You paid 6 times the price but to only go 2.5x faster
Anthropic didn’t disclose uptake, aka how many people willingly paid for it, and since I don’t trust the estimates I’ve seen enough to quote them, all you have to know is that some customers were willing to pay a substantial premium for speed alone, and they have been ever since.
Cognition made the point even more starkly in April. It launched SWE-1.6 with a free version served through Fireworks at about 200 TPS, while paying users could access a Cerebras-served version running at roughly 950 TPS. At first glance, that sounds irrational.
Why would would anyone in their right mind pay several times more for the same model, or worse pay when they can get the same model for free just on slower infrastructure?
Because for those users, **the token bill is not the expensive part of the workflow...
Their time is!**
A Chatbot Waits for You : An Agent Waits for Itself
This is probably the easiest way to understand why agents fundamentally change the value of inference speed.
The mental model most people still have for AI is basically ChatGPT:
You ask a question → the model answers → you read the answer → think about it → maybe open another tab → eventually ask something else.
The important thing here is that the human is naturally inserting massive amounts of time between model calls. Maybe you take 20 seconds to read the answer. Maybe 5 minutes, or maybe you come back 2 hours later.
So even if the model itself is a little slow, a lot of that latency gets hidden inside the much larger amount of time the user spends reading, thinking, deciding what to do next, doomscrolling on X, whatever lol.
Agents are completely different!
You give an agent a goal and it keeps working without waiting for you after every step. It might reason about what to do, search stuff online, inspect what it found, write code, run said code, see an error, figure out why it failed, change the code, run it again, check the result, and continue until the task is finished.
Anthropic’s own description of its research system looks pretty much like this: repeated planning, tool use, evaluation, refinement, delegation, and then more reasoning based on whatever happened in the previous step. Some of that work can obviously happen in parallel.
But here is the important part:
A lot of it cannot.
The agent cannot debug an error before the error exists. It can't inspect the result of a test before the test finishes. It can't decide what to do next based on information it has not received yet nor can it verify whether his fix worked before actually running the fix.
Call #2 sometimes literally cannot begin until call #1 is finished.
Those are called sequential dependencies, and unfortunately you can't pay2win, throwing another 50 subagents at the problem does not magically remove them.
That is the fundamental difference.
With a chatbot, the model is waiting for you. With an autonomous agent, increasingly, the machine is waiting for... the machine.
And once that happens, every second the model spends thinking on the critical path starts directly affecting how quickly the entire job gets done.
Let's take deliberately simple example:
100 sequential model calls, each depending on the previous result.
Endpoint A takes 10 seconds per call; endpoint B takes 2, but with the exact same outputs and reliability.
The model-only portion takes **16 minutes and 40 seconds for endpoint A
But only 3 minutes and 20 seconds for endpoint B**
Now image we extrapolate those 8 seconds on 500 calls instead of a 100
It becomes 83 minutes and 20 seconds versus 16 minutes and 40 seconds.
They are doing exactly the same work, with exactly the same model and exactly the same quality/inteligence. **The only difference is that one endpoint has a infrastructure that is simply 8 seconds slower at each step
And by the end of the workflow, that meer 8 sec per-call difference has compounded into more than an hour disadvantage.
**Obviously, in real workflows agents also have to wait for websites, tools, and external systems but you get the idea.
You cannot make a slow website load five times faster because the model generated the instruction to open it five times faster.
Speed Can Be Converted Into Intelligence
Now imagine you have a hard 60 second response-time budget.
With one system, infra, endpoint, whatever you want to call it, you might only have enough time to generate the initial answer.
But with a much faster system, you might have enough time to generate the answer but also to test it, critique it, find a mistake, fix it, and still return the final result inside the exact same 60 seconds.
That is where this gets much more interesting to think about because we are no longer just using speed to get to the final product faster. Instead we are using speed to do more thinking before the user gets the final answer.
Obviously, that extra compute still needs to be spent intelligently right. But the broader point, that more inference-time compute can improve the final result, is something that's been well established for years now.
On AIME 2024, OpenAI reported 74.4% accuracy for o1 using a single sample, versus 83.3% when using consensus across 64 samples.
So in other words:
Using the same model but different inference-time procedure they got different results, and more importantly, with more samples they got better results!
So it wasn't a hardware speed experiment, and obviously 64 samples can be parallelized and it obviously costs a lot more compute too...
But that's not really the point.
The point is that the intelligence you experience at the application layer is not determined by the weights alone. It also depends on what the system is able to do with those weights before it has to give you an answer.
Charlie Snell and collaborators at Berkeley and Google DeepMind found something similar from another angle. Their "adaptive test-time compute strategy" used inference compute more than 4× as efficiently as their best-of-N baseline in some settings. Basically it spent extra compute only on problems that needed it, reaching similar performance but requiring **over 4× less compute!
**They actually found cases where doing more search actually stopped helping, in fact they even found cases that unintuitively made it's performance worse! (source for the curious: arXiv)
Which is important to know because it proves that you can't magically make a model smarter by telling it to generate another say 10000 tokens of bullshit lol.
So you need a productive way to spend the extra compute, and this is exactly where fast inference becomes economically interesting:
Take AlphaSense.
In a September case study with Cerebras, it reported reducing p90 time to first token from 19.5 seconds to 2.3 seconds. If you don't know what p90 means bro google it my fingers hurt because I am explaining every little everything and typing it out.
Anyways, the more interesting number is that its "evidence-evaluation workflow" could review 3× as much evidence without increasing latency. .
(These are also customer/vendor reported numbers, I am always skeptical of their sincerity so take them with a grain of salt fyi)
But regardless, the mechanism is exactly the one I am talking about. The system suddenly had enough time to inspect substantially more information before the deadline. The model weights didn't change nor did the model suddenly become more intelligent.
The system simply became capable of doing something more ambitious with the same model inside the same amount of time.
That is why I think treating intelligence and inference speed as completely separate variables misses a significant part of the economics.
Faster inference gives you a choice:
-
You return the same answer sooner.
-
You use the time you just saved to search more, test more, verify more, reason more, and potentially return a better answer without making the user/agent wait any longer.
And in a lot of high-value applications, that second option is much more important than shaving a few seconds off a chatbot response, but don't take my word for it.
Moar Speed = Moar Money
Consider a simple coding workflow that generates 100,000 output tokens, reasoning included.
Using the 6× price difference we saw above, the inference bill goes from $2.50 at $25 per million tokens to $15 at $150 per million.
So you are effectively paying an extra $12.50 for the faster option/endpoint.
Now suppose it saves a developer 10 minutes of actual recoverable working time. At a loaded cost of $180/hour, those ten minutes are worth $30.
You spent $12.50 to recover $30.
The more expensive inference is literally cheaper in reality
If the dude waiting on the model is a quant researcher, an ML engineer, a software engineer, or basically anyone whose time costs substantially more than the tokens they are consuming, then optimizing for the cheapest possible inference can be economically stupid...
OpenAI’s Cerebras-powered Ultrafast preview actually included feedback from Jane Street’s AI-assistant team on how much the speed changed the way developers could work alongside the model. And yes, congrats to the eternally online: OpenAI is now using NVIDIA chips for Ultrafast, but it's really not the gotcha you think it is, and I’ll explain why later.
OpenAI obviously didn't disclose what Jane Street was paying, nor did it quantify the productivity gain, so I’m not going to invent a number or believe the hearsay, but if I was to believe it then the number would start with a 2 and then have 8 zeros behind it.
Okay so basically sometimes:
Expensive inference is better than an expensive person sitting around waiting for cheap inference. Got it, unfortunately even that understates the value, because the benefit does not necessarily end when the employee goes home.
Imagine two otherwise identical teams.
One can complete a useful build → test → inspect → revise cycle every 40 minutes.
The other can do it every 5.
Over four hours, one team gets roughly 6 shots.
The other gets 48.
Obviously 48 attempts does not mean eight times the progress. If your idea sucks, iterating on it 48 times may just help you discover that it now sucks a lil faster.
BUT
If iteration is actually the bottleneck, one team now has dramatically more opportunities to discover the broken assumption, find the bug, test another implementation, talk to a user, and eventually build something somebody will pay for.
Jane Street does not need Cerebras-speed inference **inside the actual trading loop, cause actually **even the so called hyper mega fast chips are still slow as fuck in comparison to their trading infra lol. Regardless, the more interesting use is upstream: helping researchers and engineers build, test, and improve the algorithms that eventually trade, much faster than they otherwise could.
Now suppose that team manages to launch a new strat 2 weeks earlier
Those 2 weeks are not just “time saved.” That's alpha collected that normally wouldn't have been harvestable. I hope that I don't have to explain how potentially valuable this shit could be for JS who made like $16B in Q1 lol
But the same concept works with tech companies too:
If they ship a new product a month earlier, they reach revenue a month earlier and suddenly they're collecting customer feedback while their competitor is still building. They are learning which features actually matter. They can feed those lessons into the next version before the other company has even shipped the first one.
And when the competitor eventually buys access to the exact same fast inference endpoint, they do not magically receive the month of customer learning you accumulated before them.
That is the part I think gets missed...
**Access to speed is copyable. The head start it creates might not be...
**Depending on the business, an early lead can compound through better data, faster iteration, customer feedback, revenue, distribution, or just through like more shots on goal type thing. If that lead becomes large enough, matching the same inference speed later does not mean the competitor has actually caught up whatsoever.
For a lot of industries, *not* having access to the fastest inference possible while their competitors do, is an gigantic disadvantage
To the point where the economics, especially if they remain as scarce as they currently are, might turn it into an asset that gets sold at auction to the highest bidders.
I guess now is a good as any:
You want to know why OpenAI is no longer offering Ultrafast on Cerebras chips for the public to use?
Cerebras is literally at capacity, completely supply-constrained right now
But much more importantly:
OpenAI is renting Ultrafast on Cerebras to Jane Street and few others for 200 MILLION UNITED STATES DOLLARS PER FUCKING MEGAWATT
So FYI:
Cerebras is contractually obligated to have shipped a total of 250MW by the end of 2026: we're in October
Rumor is that they took delivery of 100-125MW, potentially more by now
They're selling a megawatt for 200M to JS, and they, I would assume, currently have 100+
Yeah no need to be a mathematician to figure out how ridiculous the numbers look.
They're renting at roughly 7.5× annual revenue / lifetime hardware cost lmfao
Sure the premium is mostly thanks to the fact that it's running OpenAI's best model at lightning speeds.
But guys... read between the lines I don't know what you want me to say here lol
"THeY cAn oFfer ThEM foR So mUCh beCAuSe tHEy ONly HAve A LiTtLE suPPly tO OfFEr"
OpenAI buys the chips for roughly $7.5M/MW/year. Add the electricity cost, the data center costs, throw in the model development cost too while you're at it Total costs comes out, at most, at around 13-15M$ a year, and even with shipping, wiring etc you would struggle to get to >20M$ a year.
Now the reason why OpenAI can charge so much is because they can offer their most advanced model at up to 14x the normal speed.
Standard mode averages at 53 TPS, ultrafast on Cerebras can reach 750TPS.
That's a monumental difference!
(More on Cerebras below!)
Some Products Only Exist Below a Certain Latency
There is another category of value that is even more important. Sometimes faster inference does not make the product better.
It makes the product possible.
Take a deliberately simple voice-agent example. Suppose the system needs five sequential model calls before it can respond. At four seconds per call, you already have 20 seconds of model latency before adding tool calls, networking, speech recognition, text-to-speech, or anything else. At such speed the product is useless in prod...
But at 400 milliseconds per call, the same five steps take 2 seconds, and all of a sudden you have an entirely new industry waiting to be served! Those are not the same product with a slightly nicer loading animation.
And here is the thing:
The same thing applies anywhere intelligence has to sit inside a latency-critical loop.
Imagine a security system that wants an LLM to evaluate an action before allowing it to execute. If the model takes 10 seconds and the app has a one-second decision budget, it does not matter how intelligent the model is.
You literally can't put it there.
You're forced to do one of three things:
-
Use a different model/infrastructure
-
Remove the check from the critical path
-
Fall back to conventional software
-
Completely redesign the product
Generating an extremely intelligent explanation 10 seconds after the malicious action happened is far less interesting for obvious reasons.
CrowdStrike’s July partnership with Cerebras points in exactly this direction. The companies announced plans to use high-speed inference to help power Falcon AI Detection and Response with more capable models. Obviously that announcement doesn't prove that every cybersec decision should suddenly be handed to an LLM, but it illustrates the opportunity pretty well.
There is already intelligence that is good enough to create value but too slow to put where that value actually exists.
Make it fast enough, and you do not merely improve the existing application. **Instead you actually expand the set of places where intelligence can be inserted at all.
That's what makes all the difference!
Speed Matters Most Inside the "Intelligence Factories"
There is another place where I think inference speed may matter even more than it does for customers: inside the labs building the models.
We still talk about AI compute as if there are two clean buckets, but that's not exactly the case:
1. Training is where you build the model
2. Inference is where you use it. But that distinction is getting pretty blurry
A lot of model development now involves running the model over and over again.
In reinforcement learning for example, you give the model a problem, it generate a bunch of candidate solutions, scores them, turns that into a training signal, updates the model and rinse and repeat.
But here is the catch:
The generation part is also inference!
The same is true for synthetic data, rejection sampling, evals, agent environments, and increasingly the research process itself where models can basically write experiment code, inspect the results and suggest what researchers should try next.
So to be clear, inference is not just the thing you do once training is over. It's becoming part of training and research itself, which means inference speed can directly affect how quickly researchers can iterate.
There is already some evidence that this is a real bottleneck. AReaL, a system published in 2025, decoupled rollout generation from model training so the two could run without constantly blocking each other. In its experiments, it reported up to 2.57× faster training than the synchronous systems it compared against, on the same number of GPUs, while matching or improving final performance. (for the curious: arXiv)
If inference is 80% of the loop and you make it 5× faster, the whole thing gets about 2.8× faster, it's most definitely NOT negligible when you consider the ridiculous race going in between labs to one up their competitors.
**That being said, there is another reason this could matter infinitely more:
Recursive self-improvement (RSI)**
Basically that's when models helpi build better versions of themselves. And the reason it makes all the difference is because RSI is basically a big feedback loop.
A model helps improve the systems used to train, evaluate or modify the next version of itself. That improved system produces a better model, which can then contribute more useful work to the next round, and so on.
In that world, inference speed is not just latency anymore. It becomes part of the clock speed of the entire improvement loop.
Say the model is writing experiment code, generating synthetic data, proposing architecture changes, running agents through environments, analyzing failures and deciding what to try next, then every one of those steps involves inference somewhere along the way,
The faster you can close that loop, the faster any recursive process can run, it's really not rocket science.
If AI systems become increasingly capable of contributing to AI research itself, then ultra-fast inference becomes unusually valuable because it accelerates the very process that determines how quickly the next generation of models arrives.
I would argue that it's so valuable for frontier labs that OpenAI for example has a massive incentive not to offer most of it's supply but instead keeping it for themselves so that they can ship next gen models faster.
I would bet a lot of money that OpenAI is currently keeping >30% of their cerebras ships for themselves! Imagine how valuable a 2x speed edge is over who can ship the next model faster is.
And remember guys, speed compounds over time! The faster you ship a model the faster you can get to work on the next one etc etc
I believe that it is so important in fact that it makes me bearish on Anthropic because they have no way to compete! But that's a topic for another day :)
We Are Nowhere Near the Inference Speed Limit
Look guys, I’m not the sharpest tool in the shed but I am not giving away the actual edge of my startup lol. Plenty of ways to do it, look it up. I don't want to overshare on anything due to my excitement for said subject. Sue me
Just know this: things can get a lot better from here, and I fully expect them to.
Tokens per Second Is a Dogshit Metric
Small detour before I spend the next few sections glazing fast inference lol.
High TPS by itself does not mean much.
Say:
Model A does 2,000 tokens/sec
Model B does 200 tokens/sec.
You probably assume that Model A is 10× faster right?
Yeah well unfortunately, that's not necesarily the case...
Tokens are not a standardized unit, every frontier model has a different way to create tokens.
Say Model A needs 20,000 tokens to solve the task but Model B only needs 1,000, then Model A takes 10 seconds and Model B takes 5.
So congrats, your model with 10× higher TPS still took twice as long to actually get the job done.
What matters is not just how fast a model generates tokens, but how many tokens it needs to get to the answer.
A much better metric would be: GPU seconds per successful task done
Few Tasks Actually Require the Smartest Models
Not every task needs the smartest model available. In fact very few actually do...
If an agent just needs to decide whether a file is Python or JavaScript, using the most expensive frontier model is obviously overkill.
So the real question is:
What is the weakest model that can do this task reliably?
RouteLLM actually release an interesting paper on the subject. They basically showed that by sending easy requests to cheaper models and harder ones to GPT-4, it reported up to 85% lower cost while retaining roughly 95% of GPT-4 performance in one benchmark setup. Yes, GPT4 was a long time ago, but the concept remains stoopid.
Like it's obvious, once a model is smart enough for the task things like latency, cost, throughput and reliability start mattering more.
A hard proof might need your best model sure but choosing which database table to query probably doesn't right?
And once agents are making hundreds of these little decisions, calling the biggest model every single time becomes incredibly wasteful!
So I don't think the future is simply:
Best benchmark score wins
The better system may use frontier intelligence only where it is actually needed, and much faster/cheaper models everywhere else
The Latency Version of Jevons Paradox
The obvious way to think about faster inference is that you take today's workloads and just run them faster.
I think that misses what actually happens in certain cases:
When latency drops enough, people will simply do more work.
An agent that skips a verification step today because it adds say 45 seconds will probably stop skipping it if that same step instead took like 5 seconds. Or if generating 5 candidate solutions takes 10 seconds instead of 3 minutes, you'd stop generating one and start generating 5 or something.
It's pretty intuitive, if multiple agents can debate a problem without adding a noticeable delay, there is no reason to not make that the default and hence the workload itself changes!
That is basically the latency version of Jevons paradox!
Making intelligence cheaper makes people use more of it. Making it faster means they can also fit more intelligence into the same amount of time.
And I think that is actually one of the more important effect.
The value of faster inference is not just that the same task finishes sooner. It is that you can start doing things you previously would not have bothered doing at all.
You can actually already see the broader version of this in some datacenters. AI keeps getting more efficient per task, but aggregate power demand is still rising because the number and intensity of workloads are growing even faster.
Latency can work the same way.
That is why I think demand for fast inference could become surprisingly sticky. Not necessarily demand for any one provider, but if someone else offers the same speed for less money, people will obviously move...
But from experience I can tell you that once developers get used to building around low latency, they don't ever want to go back to normal inference!
The Demand Can Expand Faster Than We Can Build
This might be the most important part of the thesis for anyone trying to bet on it...
When I say demand for ultra-fast inference could become “effectively unlimited,” I obviously do not mean literally infinite demand at any price lol...
I mean something much simpler:
The amount of economically useful low-latency inference people could consume may grow much faster than the amount we can physically deploy.
And those are two very different growth curves:
Demand is software | Supply is hardware
You need:
- Chips
- HBM or other memory
- Networking
- Racks
- Cooling
- Transformers
- Substations
- Power
- Datacenters etc etc
And eventually somebody has to build all of that shit lmao and that's the issue...
You cannot simply decide on Monday that there is 10× more demand and have 10× more powered inference capacity by Friday. We are already seeing this mismatch right now.
Google said its model APIs were processing roughly 22 billion tokens per minute in Q2 2026, up from 16 billion just one quarter earlier, while also saying it remained supply constrained. That is a 37.5% increase in a single quarter.
Supply is also growing incredibly quickly and models are becoming dramatically more efficient.
But here is the more important distinction:
Not all compute supply is interchangeable.
A cluster optimized to process enormous batches cheaply overnight is obviously not useful for someone trying to run a frontier coding agent at ultra-low latency...
Low latency often means smaller batches, more dedicated capacity, more bandwidth per request, aka worse hardware utilization.
So the relevant supply is not:
“How many GPUs exist?”
It's like:
“How much frontier intelligence can I actually buy at the speed, quality and price my workload requires?”
Which means we could eventually have plenty of aggregate AI compute while still having a shortage of the thing customers like Jane Street, frontier labs, agentic coding companies and other latency-sensitive users actually want:
extremely fast, extremely capable inference.
And unlike software demand, the supply side has to move at construction speed.
The IEA is already pointing to bottlenecks in chips, memory, transformers, turbines and grid connections. Even if the money is available, a new datacenter or several hundred additional megawatts cannot just materialize because somebody increased an API rate limit.
But that demand for the most valuable forms of inference can change much faster than the physical infrastructure capable of serving it.
Software can create another billion requests almost overnight.
But you still have to build the fucking datacenter...
Why I Am Bullish on Cerebras
I think one reason a lot of people still don't understand the Cerebras bull case is simple:
They have never actually used inference this fast!
I PROMISE you that it's a Eureka moment for any dev with an agentic workflow. It's honestly hard to put into words how outrageously fast it actually is in comparison to literally anything else...
"Meh Groq also offers fast inference"
Yeah cool cerebras is like 3-7x faster depending on model and workflow.
I literally can't tell you enough how fucking ridiculously fast it is. Try it, yes it's a bit expensive, but just try it. Experience what 1000 TPS is actually like. It's the only way for you to actually comprehend why and how significant it is.
So after everything above, the obvious question is:
Who actually benefits if ultra-fast inference becomes as valuable as I think it will?
I do not need Cerebras to replace NVIDIA.
I don't even need it to be the fastest provider on every model forever.
I need it to remain one of the best ways to deliver frontier-level intelligence at extremely low latency, in a market where demand for that kind of inference is far, far too large for what the market can produce.
Cerebras attacks the problem very differently from conventional GPU systems.
Its WSE-3 puts 44GB of SRAM directly beside the compute on a wafer-scale processor. Large models still span multiple systems obviously but keeping assigned model weights close to the compute reduces a lot of the memory movement that makes autoregressive generation slow.
And I don't know what to tell you besides that it works...
OpenAI previewed GPT-5.6 Sol on Cerebras at up to 750 output tokens/sec, roughly 14× the usual processing. Cerebras also reported 5.6× faster end-to-end task completion on its GDP-Val comparison.
Again, vendor-reported, so take the exact number with a grain, a handful, of salt. But even if it's 10x instead of 14x, nobody comes close, and even if one did, it wouldn't matter because there would be far too much demand for those 2 parties to fill...
It is beyond me as to why the market is scared that the supply is currently concentrated to OpenAI. Yes you bunch of imbeciles, they can't build them fast enough so Cerebras bought it all, it doesn't mean there isn't more demand now does it?
Cerebras disclosed a 750MW agreement worth more than $20B, with capacity delivered in stages through 2028. Literally that alone tells you something important about the size of the demand.
Again, OpenAI is not the only category of buyer that could want it.
-
Frontier labs want faster research loops.
-
Agentic coding companies want lower decode latency.
-
Quant firms want faster research and iteration.
-
Security companies want intelligence inside time-sensitive workflows.
-
Defense cares about low-latency simulation, analysis, cyber and other real-time workloads.
The point is not that Cerebras can serve all of them today.
It very obviously cannot.
And that is where the current capacity constraint matters.
Reason 2: Cerebras Should Rent At a Huge Premium
I think people are looking at Cerebras pricing completely backwards.
They think of it like:
What does the hardware+shipping cost?
What does the electricity cost?
What does the datacenter cost?
Add a some margin and that's what the inference should cost right?
No it fucking shouldn't my dear GS junior analyst...
For the people who actually need this stuff, the value is in no way determined by what it costs Cerebras to produce a token.
It is determined by what waiting costs them!
If you're running some random background batch job, sure, go buy the cheapest inference you can find who cares.
But if you're a frontier lab, a quant fund, an agentic coding company, a defense company, whatever, and your workflow is genuinely blocked on model generation, then suddenly shaving tens of seconds or minutes off each cycle can be worth an ABSURD amounts of money.
That is why I think ultra-fast inference is most valuable on the smartest frontier models... at least for now. If some small model already runs at 400 TPS, taking it to 800 is nice, but it does not completely change what you can do with it.
Now take one of the smartest models available, where the actual bottleneck is that it spends forever reasoning, and make that run at several hundred TPS.
That is a completely different product.
And this is also why open-weight models matter so much for Cerebras. The moment Chinese labs release something genuinely competitive with Astra/Mythos, even if by then Ant and OAI have better models, Cerebras can serve that level of intelligence directly through its own cloud instead of depending on access to a closed model provider.
At that point, the value of its inference infrastructure goes up massively, because it is no longer just making weaker models faster. It can potentially offer some of the best intelligence available at ridiculous speed.
You get the smart models at fast-model latency.
Obviously that is way more valuable
Like OpenAI literally proved it with GPT-5.6 Sol Ultrafast on Cerebras: they're renting a MW for $200M for fuck sakes lol...
And once you move from buying individual API calls to renting dedicated Cerebras capacity, the argument gets even stronger.
In my opinion Cerebras should just stop offering API endpoints and instead rent per MW to businesses willing to pay the premium...
Then it becomes like buying guaranteed access to the fast lane.
Enough capacity that you can actually design your product around the assumption that the speed is always there without warrying about batching and all the other nonsense...
That should obviously rent for a premium.
And I think this gets even more extreme as models get smarter.
The strongest models are increasingly the ones used for the hardest, highest-value problems.
They also tend to use:
-
More search More reasoning
-
More verification
-
More agent steps
-
Basically more inference
So the smarter the model gets, the more painful slow inference can become.
Which means the value of accelerating the best model may actually increase faster than the value of accelerating some cheap small model!
That is the category I think Cerebras is really playing in:
Frontier intelligence at stupidly low latency.
And there is not enough of it. Which is exactly why I think the scarce capacity should clear at very high prices.
The customers who value it most will pay the most.
It's not rocket science...
And that is the part I think people miss when they try to value Cerebras like some normal compute rental business.
They're not just renting FLOPs.
They're renting the ability to get the same intelligence now instead of later.
What Is Actually Constraining Cerebras?
The Cerebras bull case gets misunderstood because people hear “supply constrained” and think:
“Okay bro just make more chips.”
Unfortunately mate nobody has defeated physics yet...
There are really 3 different constraints:
1. Manufacturing
Cerebras still needs wafers, components, assembly, testing, packaging and complete systems. It relies on TSMC for its wafers and has to physically scale the manufacturing operation. Management says it has already added manufacturing capacity through Flex, Sanmina and Rocket EMS and is targeting more than a 10× increase in manufacturing capacity during 2026.
It still doesn't mean they can instantly satisfy every customer asking for capacity, but it's not like they're stuck either is it.
2. Datacenter deployment
Even after you manufacture the system, you need somewhere to put it.
That means:
-
Power
-
Cooling
-
Networking
-
Racks
-
Transformers
-
Fiber
-
Commissioning
and actual powered datacenter capacity...
Cerebras said it had more than 600MW of datacenter capacity already live + contracted for delivery by the end of 2027.
The important words there are:
live OR contracted.
Signed future capacity is not the same thing as 600MW sitting there ready to sell tomorrow.
And if you followed the previous section of this article, you already know why hundreds of megawatts of new AI infrastructure take time to physically build.
3. How much useful inference each system can deliver
This is the more interesting constraint.
SRAM is insanely fast.
But it is capacity constrained.
Huge frontier models still need to be distributed across multiple Cerebras systems.
So even if the architecture is ridiculously fast, the economics still depend on how many systems it takes to serve each model and how much concurrent demand each deployment can handle.
That means Cerebras can increase supply in two ways:
build more systems
or
make each system capable of serving more useful inference.
The second one is where they need to work on, good thing is, they are. They now employ an entire division to take care of this and more through their "AI model service devision"
Can Cerebras Actually Escape the Constraint?
Yes!
At least there have a credible roadmap for doing it...
First the goal is obviously to manufacture more systems
Then to let partners finance and operate more of the infrastructure.
Like General Compute for example is deploying Cerebras systems and initially targeting agentic coding. AWS plans to integrate Cerebras decode into Bedrock.
Thankfully that means Cerebras doesn't have to personally build every datacenter used by every customer. Thank fuck for that, but we'll get back to this later...
But the really interesting route is actually by improving the economics of each deployment.
AWS's announced architecture separates the inference workload:
Trainium handles prefill while Cerebras handles decode.
That is actually quite clever:
Why burn the scarce wafer-scale capacity on parts of inference that another architecture could also handle perfectly well?
Just use the Cerebras where its latency advantage matters most.
AMD announced something that's conceptually similar where they are using Helios for prompt processing and Cerebras for generation.
At least until they get their capacity up to where it should be, customers and Cerebras for that matter should focus on:
The stages where speed is worth the most!
And then there is the roadmap:
Cerebras has disclosed plans for CS-6, which combines wafer-scale compute and SRAM with 3D-stacked DRAM.
In english that means it fits substantially more of the model per system and reduce the system footprint required for ultra-fast inference by roughly... an order of magnitude lol.
To be fair, it is a roadmap target, it's not in production yet.
But regardless, if they manage to retain most of the latency advantage while dramatically reducing the number of systems needed to serve a frontier model, that would change the economics substantially.
But honestly, the important part isn't even whether that “order of magnitude” number ends up being exactly right.
The thing I care about is the direction.
Right now Cerebras has basically built something that is extremely fucking fast but relatively expensive to supply, in large part because frontier models need to span multiple systems.
If future generations let them keep most of that speed while requiring materially fewer systems per model, then they can attack both sides of the equation at once: serve more demand with the same amount of hardware and lower the cost of supplying each unit of ultra-fast inference.
That would let them either keep prices high and massively improve the economics, lower prices and open up a much larger market, or realistically some combination of both.
And that is almost as interesting to me as whether CS-6 generates another 30% more TPS or whatever.
The dangerous version of Cerebras for competitors is not:
“Cerebras gets even faster.”
It's:
“Cerebras remains ridiculously fast while becoming dramatically cheaper to operate at scale.”
Which takes us to the most common bearish argument I've heard.
What If Someone Else Gets Just As Fast?
They will.
It’s not an if, it’s a when.
Will Cerebras still have the speed advantage by then? I think so, but I’m not writing another 15,000 words to explain why.
More importantly, even if it doesn’t, that really isn’t the end of the world...
Of course Cerebras is going to face serious competition. There is way too much money in ultra-fast inference for it to remain alone forever. NVIDIA is already attacking the same market with GPT-6 Astra Ultrafast on Blackwell, Groq is built around low-latency inference, Google has TPUs, AWS is building its own silicon, AMD is coming, and there will almost certainly be companies we barely care about today doing something ridiculous three years from now.
I just do not think that automatically kills the Cerebras thesis. ( and I also think their spot is safe until proven wrong anyways)
People seem to treat inference like there can only be one winner, where whoever posts the highest TPS number takes the entire market lol.
That is not how customers actually buy infrastructure. They care about some combination of:
intelligence + latency + price + availability + reliability
Being the absolute fastest is obviously valuable, but you don't suddenly become useless because somebody else is slightly faster.
Imagine one provider completes an agent workflow in 2 minutes, Cerebras does it in 3, and the normal alternatives take 15. Nobody serious looks at that and says Cerebras has no value because it came second lol. Of course it's bearish for the premium I mentioned before, but the market is big enough that it doesn't matter as long as capacity scales with it.
The real danger is more specific:
If competitors eventually offer the same frontier models at roughly comparable latency, materially lower prices, with enormous capacity and similar reliability, that would indeed be bad for Cerebras' pricing power.
That is the bear case.
Not:
“Groq is fast too bro.”
And the market does not even need one permanent winner anyway.
OpenAI alone contracted for 750MW of Cerebras capacity. That is one customer.
Now think about the rest of the potential demand: other frontier labs, quant firms, coding companies, cybersecurity, defense, research organizations, enterprise agents, and eventually entire categories of applications that only become practical once inference is fast enough.
Cerebras cannot possibly serve all of that by itself and neither can Groq. That's the point of this entire fucking article.
THERE IS JUST TOO MUCH DEMAND THAT CAN'T POSSIBLY GET FILLED!
NVIDIA cannot magically provide unlimited frontier inference at whatever latency every customer wants, and certainly not with GPUs...
That means another supplier becoming competitive does not automatically take demand away from Cerebras. In a supply-constrained market, it can simply serve demand that Cerebras was never physically capable of satisfying in the first place.
This is THE distinction.
THERE AREN'T ENOUGH SUPPLIERS CAPABLE OF DOING WHAT THEY'RE DOING, PERIOD.
Competition matters most when it destroys scarcity, not merely when it creates another fast option...
The real question is whether ultra-fast frontier inference eventually becomes so abundant, commoditized and cheap that customers stop paying meaningful premiums for it.I think that is a much harder outcome to reach than people assume and either way we are yeaaaars away from having to figure this out...
Because matching Cerebras is not just posting one impressive TPS number.
You need to serve the models customers actually want, at the context lengths they need, under real concurrency, with predictable latency, good reliability, attractive economics and enough physical capacity that customers can actually get access to it.
Cerebras has been working on this for a decade guys, that's 10 years by the way...
Okay, So How Does Cerebras Become a $1T Company?
I already explained what I mean by “effectively unlimited demand” above, so I am not going to repeat the whole supply-vs-demand argument again.
The $1T case is basically a bet that Cerebras can keep adding enormous amounts of capacity for years without coming remotely close to exhausting the number of customers who have an economically rational reason to buy it.
As mentioned in my past article, @PatrickToulme recently tried to put numbers around how stupidly large that gap could become.
His estimate is that the entire world currently has roughly 19 million H100-equivalents of AI compute across NVIDIA, TPUs, Trainium, AMD, Cerebras, Groq, etc.
So the question is:
What happens when AI usage stops meaning “one person talking to one chatbot” and starts meaning one person managing an AI that itself runs dozens of agents?
His model has the world's ~28 million software engineers each running a 50-agent swarm requiring around 47 million H100-equivalents.
That's already roughly 2.5× every AI chip on Earth, just for software developers.
Take half of the world's knowledge workers with the same 50-agent setup and his estimate jumps to roughly 830 million H100-equivalents.
About 44× the entire existing global fleet.
Some of you might laugh at those numbers
But that's because you don't know any better... actually think about it... But I promise you that they're really not all that absurd...
But regardless:
Cut his estimate by fucking 10× and you're still talking about demand several times larger than today's entire installed base...
And it gets even more interesting for Cerebras specifically because the inference people are willing to pay the most for is also some of the most compute-hungry inference to provide.
Higher per-user speed requires giving each request more of the hardware instead of batching everybody together.
In other words:
Fast inference consumes disproportionately more scarce capacity.
Like Patrick showed, even putting only 20% of that workload on the premium tier adds something like 330 million chips in his model.
Again, don't get religious about the exact number, it doesn't matter. I want you to realize that the market is in it's infancy.
98% of Americans aren't paying for AI...
You live on twitter in your little bubble, the normies literally do not give a shit or this the value! For now that is...
And for them, the interesting thing is what happens to the demand curve once millions of agents aren't merely running, but people start demanding that they run fast.
That is where I think a lot of the conventional Cerebras valuation work completely loses the plot...
They're trying to estimate Cerebras' share of some fixed inference TAM.
I don't think the TAM is fixed whatsoever lol
@GavinSBaker framing is much closer to how I think about this:
The high-frequency trading and cyber wars of the next few years increasingly get fought using low-latency inference infrastructure—SRAM-heavy systems like Cerebras and Groq, potentially sitting inside larger disaggregated systems alongside NVIDIA Rubin or whatever comes after it.
The reported Jane Street economics are the extreme early example of what that market could look like. The number people throw around is $200M/MW, or $200B/GW.
Whether that precise number survives scrutiny is honestly less interesting to me than why anyone would even contemplate paying something in that universe.
Because Jane Street isn't buying electricity.
They're buying whatever additional economic output that electricity running extremely fast frontier intelligence can create for them...
And Jane Street isn't some weird isolated use case either.
Take computer use.
Today an agent moves the mouse, clicks something, waits, inspects the new state, reasons again, clicks again, waits again...
Every tiny action can put another model call directly on the critical path.
At normal frontier-model latency it feels like you're supervising a very intelligent employee who freezes for 5 seconds after touching anything.
Make those calls close to instantaneous and suddenly the entire interaction is unrecognizable.
Same thing with putting LLMs directly into online recommendation and advertising systems.
Imagine:
Google serves an ad, recommendation, product ranking, whatever, and the intelligence needs to happen before the page loads, not 15 fucking seconds later.
Today a sufficiently capable frontier model is way too slow and expensive to insert everywhere inside those loops.
But...
Make frontier intelligence fast enough and suddenly you can imagine substantially more reasoning happening in real time before every recommendation, ad impression, transaction, security decision, search result, etc.
That's potentially an absurd amount of inference.
And an absurdly valuable market...
Voice is another obvious one, which I already covered above. The workaround today is often literally to sacrifice intelligence for responsiveness. Podium is already an Ultrafast customer using this kind of infrastructure in its voice stack.
And that's the pattern I care about:
speed does not just take demand that already exists and make it run faster. It keeps creating entirely new places where inference can be inserted.
Jane Street basically said the same thing themselves: the speed enables different ways of using the models.
That's the key.
They aren't going to be the last customer to discover that.
The $1T case therefore isn't really:
Cerebras eventually gets 20% of today's inference market.
That would be a pretty stupid way to model it imo.
It's that today's inference market ends up looking microscopic compared with the amount of machine work being performed once agents, computer use, real-time decision systems, AI research and eventually thousands of other applications start consuming inference continuously.
And anyways, remember what happens on the other side:
One person doesn't necessarily generate one inference stream anymore.
One person can trigger an orchestrator that spawns 10, 50 or 500 agents, all generating, verifying, testing and talking to other systems at the same time.
The unit of demand changes from the user to the amount of machine labor the user controls.
Patrick's wording on this is basically perfect: the unit of demand becomes the swarm, not the person.
Once you think about it that way, a $500B or $1T annual inference market starts sounding a lot less insane believe me.
And that's important because the financial requirement for a trillion-dollar Cerebras is still enormous.
Something like:
$100B revenue × 25% FCF margin = $25B FCF
At 40×:
$1T.
Nobody should pretend $100B of Cerebras revenue is around the corner, but the relevant question is whether there could eventually be enough demand to support that much revenue.
I think there very, very obviously will be.
The harder question is whether Cerebras can physically build enough capacity, maintain the performance advantage, and capture enough of the economics.
Because demand may genuinely be the least difficult part.
Bold take: Cerebras is too far ahead for them to be irrelevant in the next 5y. FYI, Cerebras can literally use Cerebras to create chips faster than their competitors can create their chips :) anyhow
Add to the industries I mentioned above: all the autonomous companies, robotics, training/RL, entertainment, simulation and whatever else people invent once intelligence becomes cheap and fast enough. Patrick's conclusion in his article is basically that manufacturers aren't fighting over a fixed pie because the plausible demand sits orders of magnitude above current supply.
And he is obviously right...
Hence, there can be ten massive suppliers and we can still spend years building nowhere near enough capacity.
The question I care about is much simpler:
Every time Cerebras adds another 10MW, then another 100MW, then another GW of ultra-fast capacity, is there somebody standing there willing to rent it at attractive economics?
Given where agents are going, how much compute genuinely fast inference consumes, and the number of industries where latency itself creates an economic advantage...
I think the answer could remain yes for a very fucking long time.
And that is the route to the ridiculous valuation.
Not stealing a fixed market from NVIDIA or idk what.
Building into a market that keeps getting larger faster than anyone can supply it.
I already wrote too much for a conclusion and a summary
You got the point hopefully
Thank you for reading, ask your questions below and give me your constructive criticism :))
