Meta trained its 8 billion parameter Llama 3 model on more than fifteen trillion tokens. That works out to close to two thousand tokens for every parameter, while the Chinchilla paper from 2022 had put the compute-optimal ratio near twenty. The model was fed almost a hundred times past the point where the textbook says to stop, and Meta reported that it was still improving when they did.
That looks like waste until you ask whose money it is. Chinchilla answers a narrow question: for a fixed training budget, what model size and how many tokens give the lowest loss. It says nothing about what happens after the weights ship. A lab pays for training once. Every call afterwards is paid for again by whoever makes it, and a small model that was fed far more data is cheaper on each of those calls. Overtraining is a lab choosing to spend more of its own compute so that inference gets cheaper for everyone downstream.
Then in September 2024 OpenAI released o1 and turned the dial the other way. Its published charts showed accuracy on the AIME math exam climbing smoothly with the compute spent at training time, and also with the compute spent thinking at test time. The model gets better answers by producing more tokens before it answers, and those reasoning tokens are billed as output tokens even though the API never shows them to you.
I think that second part is the quiet change. With a classic model the price of an answer is close to a function of the question: some tokens in, a few hundred tokens out. With a reasoning model the price depends on how hard the model decided the question was, and you learn the number afterwards. Most vendors already price output tokens well above input tokens, so the part that grew is the expensive part. Training compute is a capital cost the lab carries. Test-time compute is an operating cost that lands on your invoice, per request, with variance.
The best evidence I know of for when that spending pays off is a 2024 paper on scaling test-time compute optimally, by Charlie Snell and coauthors from Google DeepMind and UC Berkeley. They compared giving a small model extra inference compute against using a model fourteen times larger. On easy and medium questions the small model that was allowed to think matched or beat the big one at equal FLOPs. On the hardest questions it lost, and a bigger pretrained model was the better buy.
My reading is that the two kinds of compute substitute for each other only in a middle band. If the base model already gets a question right some of the time, thinking longer converts luck into reliability. If it never gets the question right, a thousand attempts at nothing is still nothing. Anyone who has watched an agent retry a doomed tool call five times has seen the second case from the inside.
That turns the useful engineering question into a routing one. A large share of what an assistant handles is the easy kind: classify this ticket, pull the date out of this contract, rewrite this sentence. A long chain of thought on those buys nothing and adds seconds of latency that a person is sitting through. The multi-step plan and the ugly reconciliation deserve the budget. A router that sends the first group to a small fast model and the second group to a thinking model with a cap is unglamorous, and it is where I would look first for savings.
The cap matters more than people expect, because test-time compute inherits the behavior of the loop it runs in. A model that thinks for ten thousand tokens, calls a tool that fails, and thinks again has multiplied the bill by the number of retries. I wrote last week about the heat side of that loop, and the invoice side is the same arithmetic in dollars.
Where I would put the thinking budget is in config, next to the timeout and the retry count, with a name beside it the way a timeout has one. Somebody on the team should be able to say what it is set to today and why it was raised the last time it was.