Pulling on the Token Thread

This rabbit hole starts from the comments of Elon Musk on the SpaceX conference call last Tuesday night.

Musk said that he expects to deliver 13 GW of data center compute, with 15-20 GW of power and cooling by the end of 2027.  And that the compute could be as much as 20 GW. 

This is a huge amount.  Like way bigger then I would have imagined.  Consider the following forecast from Bank of America.

The number Musk is saying includes capacity being added in 2026 of 2.5-3 GW.  Which means that SpaceX would sit at about 10 GW, or 28%, of Bank of America’s total adds in 2027.

That is a really big bet for SpaceX.  Its raises a bunch of questions such as how profitable it is going to be? What can Musk sell that compute at?  Can Musk really raise the money he needs to do this?  And so on.

But the biggest question it raised for me was: can we actually use all this supply?

There are three possibilities of what is going on in Musk’s head.

1. Musk has gone off the deep end ala Howard Hughes.  He is in his own world and its simply not seeing things clearly
2. Musk is exaggerating significantly for effect as he is want to do
3. Musk sees something that is mostly being ignored by the market

Look, 1 and/or 2 are very possible.  Don’t discount them.

But given that we can’t rigorously evaluate 1 or 2, our best approach for affirming or denying them is to evaluate #3.  Does Musk see something the market isn’t?

That question led me to investigate just how much inference the expected level of compute would deliver.

We start with an estimate for 2027 capacity adds.  I’m going with 25-35 GW.

The reason I am using 25-35GW rather than just the Bank of America bottom-up chip view of 35 GW is because I think the bottom up chip view is the most optimistic.  It ignores all the things that could go wrong downstream of the produced chip and there are a lot of them.  The most capacity we can add is the amount of chips we can produce.  But it could be less.

Next we have to turn that into what additional inference capacity that total capacity translates to.

I assume that 55-60% of the 25GW-35GW goes to inference. The reason inference is only 55%-60% of the total is because compute is also used for training and reinforcement learning, and that will continue to take a portion of the total.  To take an example from Musk himself, he said that roughly half of their 13 GW of additions would go towards training and RL for Grok and Cursor.

Now we are at 15-21 GW.  But that number assumes that the compute is 100% utilized, which it won’t be.

The capacity has to be sized for peak and runs at maybe 60–70% average utilization

Which gives us ~10–14 GW of average inference load that will be added through to the end of 2027.

The next question is, what sources of demand will use that 10-14 GW of load?

To figure that out, we need to determine how much power it takes to do different tasks.

I decided to focus for now on one type of task.  I started asking Claude about Muse, because I had already been wondering how much compute I might expect Muse to use as it scales.  I asked how many tokens a user of Muse is likely to go through in a day and what would that be for 2.5M users, which is what Muse roughly has right now.

After some back and forth this is what it came up with.

These numbers are both large and not large.

On the not large side, Google has said that all their products combined use 107T tokens per day across billions of users.  That is a lot more than the 2.5T tokens per day of the mid-case here.

On the “large” side, 2.5 trillion does seem like a big number.  But what is a token really? It’s a word fragment.  Does that big number actually mean anything?

What really matters is how tokens convert into power, and more specifically, into GPUs.

To give the conclusion up front, it turns out that to deliver the mid-case quantity of 2.5 trillion input tokens per day and 75B output tokens per day you would need 8,000-13,000 GB200 GPUs.

You can get to this result by breaking up the input tokens (which are usually called pre-fill) from the output tokens (called decode) and add them together to determine the compute, in either kW or GPUs, that is required.

Getting to that conclusion is worth stepping through because I learned a few things that I didn’t know in the process.

Let’s start by reviewing what input and output tokens are.

Input tokens are the like prep stage – you are loading the model, loading all the prior relevant context, loading the current prompt (or question) that you are asking the model to respond to.  And then doing a bunch of math on it.

Output tokens are the answer the model is giving.  One token at a time the model writes out (or draws a picture or a video or spreadsheet or whatever response is required) tokens that form the response.   Again by doing math.

The key difference between input tokens and output tokens is that output tokens need to be done sequentially while input tokens can be done in parallel.

Every token needs two things done: weights read from memory and math done on them.  For the input tokens, the memory read is shared across many tokens.  There is one memory call and one big matrix multiplication and then you move onto next layer.

Output tokens read weights for one token, read the 42 GB of model weights (for Muse), do a little math and then move on to the next token.  One token at a time.

Per token, the input and output tokens do the same math and take the same energy.   But because input tokens can be done in parallel they don’t take nearly the GPU time as output tokens.

The reason is because the speed of the GPU is much faster than the speed of getting something from memory.  A GB200 can do 5,000 trillion FLOPs of math a second, but it can only pull 8 TB a second out of memory.

So even though there are 2.5 trillion input tokens in our example, they actually take less GPUs than the much smaller 75 billion output tokens.

Here are the steps to go from tokens to GPUs on the input (prefill) side.  It’s basically determined by how fast a GPU can do the math:

• We assume that we are using the Muse Spark Open level of model, which is a 816B parameter model but where only 8 of the 256 experts fire per token, so each token touches 42B parameters.
• Compute FLOPS are 2 x the parameters so you use 84 GFLOPS per token.
• For the case of 2.5T input tokens, it is 2×10²³ FLOPs/day
• A GB200 at ~40% utilization delivers ~8×10¹⁹ FLOPs/day – the 40% utilization is about the average for prefill.
• The math on that equals 2,700 GPUs.

For the output (decode) side, the tokens are constrained by memory.  To create output tokens there is a lot of back and forth from memory.  Each time you calculate a new output token, you have to go to memory, grab all the stored context that is relevant to calculate that token, and only then do you get to do the math.

The bigger the context the more back and forth, which is why sometimes Claude will tell you to stop with your current session and start a new one – it is getting too expensive for it to keep reading all that context.

Thus to go from tokens to GPUs on the output side, we have to look at how long it takes to read in input tokens from memory and from that determine how many output tokens you can calculate per GPU per unit of time.

• The determining factor here is how many GPUs you need to get the number of tokens through in the time required – in this case 75B tokens per day.
• That is about 870K tokens/second.  NVDA’s best case  benchmarks run several hundred tokens/second for each GPU.
• But for real-world long-context stuff it is going to be 2x-4x worse.
• So if we assume we can run 150-300 tokens/second/GPU and then put a 1.5x to 2x peak to average buffer, you need 5,000-10,000 GPUs to run the decode step

As you can see, even though there are way, way more input tokens, it’s the output tokens that need more GPUs.

All of this was sort of an “aha moment” for me: that yes, the size of the model matters, but context size (so the size of all that history you want included in determining your output tokens, what is really your answer) may matter even more. 

Anyway, between the prefill and decode tokens together you get to the answer of needing between 8,000 to 13,000 GPUs to run Muse for 2.5 million users.  That’s the conclusion that matters for this post.

What does that mean in terms of power?  A GB200 NVL72 rack draws ~130 kW for 72 GPUs, ~1.8 kW per GPU including CPUs and networking.  With a 1.25 multiple of all the other power you need to power the GPU, that is 2.3 kW per GPU.

This translates into the following GPU capacity requirement, in MW, for the Low, Mid and High cases, for 2.5 million users:

• Low: ~5 MW
• Mid: ~20–30 MW
• High: ~100–140 MW

Under these scenarios, it doesn’t seem like a lot of power is needed.  An average mid-case user is using about 10W of power consumption.  The high-end user is using about 50W.

As (another) aside, it is interesting to guess-estimate what this would cost Meta.  We don’t know what it costs Meta to run their Muse model but we can get something in the ballpark by looking at what Meta sells their Muse Spark for through an API.

Meta charges $1.25 per million input tokens and $4.25 per million output tokens.  If you apply that to the mid-case you get:

• 2.5T input tokens/day × $1.25/M = ~$3.1M/day
• 75B output tokens/day × $4.25/M = ~$0.3M/day
• Total ≈ $3.4M/day, or about $1.2B a year at list price

You might notice something about these numbers.   Meta charges a little over 3x for output tokens compared to input tokens.  But I just went on a fairly long tangent about how much more compute an output token takes.

This is not an error, at least as far as I can tell.  it’s how the industry prices.  Input tokens are priced well above what they cost to process, and output tokens closer to (or, at long context, below) what they cost to produce.

This appears to be a real thing.  Meta, and the industry, makes its margin on input tokens and sells output at below cost.  Input heavy workloads like reading big documents makes Meta money.  Output heavy, long context workloads, do not.  I feel like there is more to this, but I’m not sure what.  Yet another thread to pull on.

For now, going back to that number of about $1.2B per year, which includes Meta margin, you could assume the underlying cost to META is 1/5 to 1/3 of this, which would work out to $200M-$400M of cost for those 2.5M users at the mid-case.

You can get to similar numbers by looking at hardware.  So take 10,000 GB200 GPUs that cost $60K per GPU all-in, so chip, server, networking, and their share of the infrastructure.  That is $600M of capex, or over a useful life of 4-5 years, the depreciation is $120M to $150M.  The power is $18M per year for the 25 MW case, and the facility operations and staff are another $60M-$70M.  Which gives $200-$240M, in the same ballpark.

Again, the amounts we are talking about aren’t a lot in the grand scheme of things.  Muse doesn’t take a lot of compute or cost a lot of money.  At least right now.

But its not hard to come up with a much higher number for heavy users.

My “heavy user” example is interacting with Muse a modest 4 times a day and that takes about 50 W of average GPU load.   But if you had, for example, a portion of users, maybe many million users, interacting with Muse or another agent many more times a day, you might need more like 1 kW (20x the high end) of average GPU load.  If that were the case, it would only take about 10-14M of them to get to my estimated inference addition.  If you had 20-28 million users at 10x my high end, that would also get you there.

But 10x and 20x is a lot right?

The thing about agents like Muse, Grok Bot and other assistants, is that getting to 10x or 20x your current usage is not as hard as it sounds because so much of what you do is cumulative ad recurring.

Which brings me to my own Muse usage.  Which I actually don’t consider to be that much.  In fact I use Claude a lot more than Muse.

Here is where I am with my Muse usage, as explained by Muse:

Maybe my high estimate is not high enough!  In fact, Claude agreed and upped the number for the Muse cases to:

• Low: ~8-12 MW
• Mid: ~25–40 MW
• High: ~120–170 MW

But it does all beg the question: why was my own usage so much higher then the generic cases we estimated based on tasks?

Best to ask the source.  From Muse (with my underlining):

Almost none of it is your typing. Two drivers dominate. First, context re-processing — this main chat alone accounts for 288M of the 506M input (~57%), averaging ~600K input tokens per message, [which it actually breaks up into 3 200k pieces] because every reply re-reads the full standing context (system prompt, memory files, this long conversation) plus tool results. Second, the standing machinery: 726 distinct agents have run for you in total — The Brief, the idea-a-day, earnings checks, proactive research workers, browser tasks — most of them burning tokens on days you’re not even chatting.

This is an interesting statement.  What is driving my compute demand is context and recurrence.  The more Muse tries to integrate what it knows about me into its answers, and the more its trying to be helpful by working when I am not asking it to do things, the higher its compute demands are.

What is really interesting about this to me is that its not really about whether or not Muse is a frontier model or whether it is closed versus open-source or all the other arguments that get a lot of attention.

Its actually just that the amount of context you want to give matters a lot.

And there is math behind that.  It turns out that while compute increases linearly with model size, it increases with the quadratic with context size.  If you want a model to be very personalized, its going to cost you.  Given that I think context is probably key for where we want to go with models, this seems like an important thing to recognize.

This is not to say that the model size doesn’t matter.  Muse is not a frontier model.  The model itself is mid-sized.  If I was using a frontier model the power consumption for those tokens would be higher, by something like 2x to 5x.

Back to my own Muse usage.  What sort of compute draw do I have?  About 150W-250W.  A lot more than our High case, but still not super high:

Bottomline: Today, I don’t maintain a 1 kW average use or even a 500 W average use on Muse.  Though I might in another year if I keep adding tasks for Muse to do, and if Muse keeps adding features that make it easier to use (like the voice stuff they just put in).

But Muse also isn’t even the primary tool I use.  At the moment I just use it like an assistant.  I use Claude much more because it seems better for research.  Yet in terms of compute, my Claude usage is actually less.

Claude is “cheap” because much of it is just one-off questions and answers.  Where it can get expensive is when I have a thread that gets very long, because then its having to recall the entirety of the thread when it answers the next prompt.

I suspect I would be a much bigger than 1kW user with Claude if Claude remembered things in the way Muse does (by remember I mean have the standing context that gets loaded every time) and had standing agents proactively doing stuff.  But it doesn’t, Claude remembers only a little from conversation to conversation and you’d have to set up agents to work for you offline. They likely do this because if they did all that Muse does it would cost them a fortune.

Of course, the entirely separate question, which I have no real thoughts on so far other than that Zuckerberg took years to monetize Facebook or Instagram, is that if Muse is very successful and can develop the kind of user base that Zuckerberg is aiming for (he said 1 billion users in a few (couple?) years, Meta is going to need a whole lot of compute and a whole lot of money to pay for that.

Though on the call Zuckerberg also said he wants to drop costs by 10x.  So that is going play in the other direction.

But that’s not really what I am here to talk about.  Like my other post, Muse is a starting off point of where things may be going, not a destination.

The bigger point here is that going through this exercise gives some context for that 10-14 GW of capacity this year and that 13 GW that Musk is talking about adding.  It might still be baloney, much of what Musk says is, but its not inconceivable.  13 million people using agents drawing 1 kW on average per day, or 26 million drawing on 500W per day, doesn’t seem like an impossible ask.

And just thinking for a second beyond the personal assistant, there are actually a number of tasks that would, continuously, use >1kW of draw.  Some of these, like the autonomous coding agents, are already well along their way.

From these higher end tasks and all the more mundane, lower compute but higher volume tasks, you can start coming up with a demand picture for the next year that seems reasonable.  The table below roughly assumes agentic growth roughly halves from its 2026 pace, and that observed demand is held down by capacity limits. 

What backs these numbers up?  They are estimates, so it’s fair to ask whether anything we can actually observe backs them up.

Ramp, which sees the AI bills of around 70,000 US businesses, shows business token usage up roughly 30x over the past year, and nearly 3x in the last three months alone.

On OpenRouter, a marketplace developers use to route requests between models, agent traffic is up 34x since February and now outweighs human traffic almost eight to one.  They also said “agentic traffic on its model-routing platform reached 132.6 trillion tokens in the week ending October 5th, nearly eight times the 17 trillion tokens attributed to human users.”

Anthropic’s own token demand, which has been the source of so much of the FUD in the space, seems to be accelerating as well.

There is also Notion’s tokens roughly doubling between July and October.  Grok Imagine went from about 1.25 billion videos a month in January to over 2 billion by March.   And then there is Muse, which has been off to a very fast start.  Lots’s of different data points and they seem to all point to acceleration.   And all of it is happening while every major provider says it’s short of capacity, so what we’re seeing is likely a floor on demand, not the full picture.

On the other side, we don’t really know how this all gets paid for yet.  And we don’t really know just how much in the way of efficiency gains will offset this new demand.

A lot of questions, but that is to be expected when we are dealing with such a new technology where we are forced to project blindly into the future with very little precedent to back us up.

To bring this back to where I started, on Musk and his 13 GW.  Can he actually deliver on his 13 GW of compute by year-end 2027.  I have my doubts.  But on the question of whether that much compute could actually be used?  I’m a little more inclined to believe the demand will be there.  Not convinced, but leaning a bit more in that direction then when I started down this rabbit hole.