Summarize this article with:
Next-Concept Modeling (NCM), from Stanford’s Hazy Research and Together AI, generates text from coarse structure to fine detail rather than token by token. It reports 10–31× faster generation than strong diffusion language models. This article explains how NCM works, what its benchmarks actually measure, how it compares with today’s LLMs, and what it means for teams building AI products.
Next-Concept Modeling (NCM) is a new text-generation framework from Stanford’s Hazy Research lab and Together AI, published on arXiv on October 9, 2026. Instead of predicting one token at a time, it builds text from coarse structure to fine detail. In the authors’ experiments, it matched or beat strong diffusion language models while generating 10–31× faster and training on roughly 4–5× fewer tokens.
Those numbers quickly turned into “NCM vs LLM” headlines. The comparison is worth making, just not the way most headlines make it. NCM’s results come from small research models benchmarked against diffusion models, not against GPT-, Claude- or Mistral-class systems. Here is what the approach changes, what the benchmarks really show, and what it means for teams building on language models today.
Three Ways to Generate Text
Most large language models are autoregressive. They predict the next token, append it to the sequence, and repeat until the response is complete. Prompt processing can run in parallel, but each output token depends on the ones before it, so decoding remains sequential.
Diffusion language models take a different route. They start from a corrupted sequence and refine many positions in parallel over several iterations. This reduces the number of sequential steps, but each iteration usually processes the full sequence, and quality tends to drop when the number of iterations is cut.
Next-Concept Modeling explores a third option. It does not predict individual tokens or repeatedly refine a full token sequence. Instead, it represents text as a hierarchy and generates that hierarchy from coarse to fine.
NCM does more than generate several tokens at once. It changes what is generated, a learned hierarchy of discrete codes rather than tokens, and the order in which the output takes shape.
How Next-Concept Modeling Works
NCM adapts an idea from image generation to text. That idea is next-scale prediction, popularized by VAR, which won a NeurIPS 2024 best paper award: generate a low-resolution version first, then add detail. Text is harder than images in this respect, because there is no natural way to “downsample” a sentence. The coarse representations have to be learned.
Stage 1: Learn a hierarchy. The researchers train a multiscale residual vector-quantized variational autoencoder (VQ-VAE) that converts text into discrete codes spread across several levels. Early levels hold only a few codes and capture broad structure. Later levels are longer, and each one encodes what the earlier levels left unexplained. A decoder turns the completed hierarchy back into text. Once trained, this tokenizer is frozen.
Stage 2: Learn to generate the hierarchy. A transformer learns to predict each level from the levels before it. At inference time, it generates the levels in order, but predicts all the codes within a level in parallel. The decoder then turns the result into the final text.
A Translation Example
The authors illustrate the process with a German-to-English translation. An autoregressive model would write the English sentence word by word. NCM’s first decoded level is barely readable. Each later level adds and corrects detail, and by the eleventh and final level the output closely matches the reference translation, with a BLEU score of 77.8. The improvement is not perfectly steady, since quality rises and dips between intermediate levels, but the overall movement is from rough draft to finished sentence. You can step through the example level by level on the Hazy Research blog.
Why It’s Faster: Fewer Steps, Cheaper Steps
NCM’s efficiency comes from two sources. First, it needs far fewer sequential steps than diffusion models to reach similar quality. Second, its early steps are cheap, because the coarse levels contain only a few codes, while a diffusion model processes the full sequence at every step.
What “Concept” Means Here
The word concept needs some care. In NCM, it refers to the information held in the coarse levels of a learned hierarchy of codes. It is not a claim that the model reasons like a human or works with human-readable ideas. The coarse levels appear to capture useful structure, which is an interesting architectural property, but it should not be read as a statement about intelligence.
It also helps to separate NCM from a similarly named project. Meta’s Large Concept Models (LCM), published in December 2024, treat each sentence as a “concept.” They predict the next sentence autoregressively in SONAR, an existing sentence-embedding space. NCM instead learns its own hierarchy of discrete codes and generates it level by level, in parallel within each level. The names are similar, but the architectures are quite different.
What the Benchmarks Show
The paper evaluates NCM on unconditional language modeling, machine translation, summarization and Sudoku. The comparisons are against strong diffusion language models at matched parameter scales.
The three speed figures come from three different experiments and should not be combined into one universal claim. The 21× result is from unconditional generation on LM1B, the 10× result is from OpenWebText, and the 31× result is from summarization on XSum.
The authors also measure wall-clock time rather than only counting generation steps. This matters because a step that processes a few codes costs much less than one that processes the full sequence, and step counts alone hide that difference.
The training-token savings are a separate result. Roughly 4–5× fewer tokens is notable, especially because it includes training the tokenizer. Still, it counts how many tokens the model saw, not compute or total cost. Hardware, data preparation and deployment all affect what a workload actually costs.
The Sudoku results stand out because every cell must satisfy row, column and box constraints at the same time. The authors suggest that coarse-to-fine generation helps with this kind of global planning, since later levels can revise the board as a whole. That makes structured tasks a promising direction to explore, but it does not show that NCM is a better general reasoner.
One more finding matters for practitioners. Because NCM samples quickly, the authors use the time they save to generate several candidates and pick the best one, using a technique called minimum Bayes risk (MBR) decoding. This improves translation and summarization scores while keeping total sampling time competitive with the diffusion baselines. Trading speed for more candidates is a pattern production teams already use with conventional LLMs.
NCM vs LLM: Where It Actually Stands
NCM is often discussed as a direct rival to GPT-, Claude- or Mistral-class systems. A more useful comparison looks at how each approach generates text, and how mature each one is.
The main limitation is scale. Most of NCM’s experiments use generators of roughly 100 million parameters, and the scaling runs up to 725 million parameters are described as preliminary. The design also has known constraints:
- It depends on a tokenizer that is trained separately and then frozen.
- It uses a fixed hierarchy depth, so even a short continuation goes through every level.
- The authors report occasional artifacts at the transitions between coarse levels.
The research team’s next steps address these points directly: further scaling, training the tokenizer and generator together, and adaptive computation that would let the model spend less effort on easier outputs.
In short, NCM shows that a different generation strategy can deliver large efficiency gains in specific settings. It has not yet shown a general advantage over frontier LLMs. That would require evidence at much larger scale, across many kinds of workloads, and under real serving conditions.
What NCM Means for AI Builders
The broader lesson is that language-model efficiency depends on more than parameter count. How a model represents information and how it produces its output can matter just as much. For teams evaluating any new approach, NCM included, the same four questions apply.
On availability specifically, the NCM GitHub repository and Hugging Face checkpoints make the work easy to experiment with. However, each checkpoint is trained for one task (XSum, WMT14, Sudoku, LM1B, TinyStories or OpenWebText), and none is served by an inference provider. It is a research release, not a drop-in replacement for a hosted model.
This gap between “works in a paper” and “works in production” is not unique to NCM. It is the same gap we described in The Gap in Shared Understanding of LLM Capabilities. The fastest method in a research paper is not automatically the right choice for every application, and the most powerful model is not always worth its cost either, as discussed in The greenest AI call is the one you did not oversize.
The practical response is to keep model choice flexible. If your application calls models through a single API, adopting a faster approach when it becomes production-ready means changing a configuration, not rebuilding your integration. With Eden AI, for example, you can list available models and their capabilities, route requests by price, speed or latency, and set up automatic fallback so that trying something new doesn’t put reliability at risk. Our guide to LLM routing covers the strategies in more detail.
NCM’s lasting contribution may be the question it raises more than the numbers it reports. It challenges the assumption that efficient text generation has to follow the path of next-token prediction. Whether it changes production AI will depend on what happens as the approach scales.
The next generation of AI systems may not be defined by a single generation strategy, but by how well each strategy is matched to the task it serves.
Further Reading
On Next-Concept Modeling
- Next-Concept Modeling: Fast, Parallel Coarse-to-Fine Text Generation: the original paper.
- NCM explained by Hazy Research: the authors’ walkthrough, with an interactive translation example.
- HazyResearch/ncm on GitHub and NCM checkpoints on Hugging Face: code, checkpoints and evaluation setup.
- Large Concept Models: Language Modeling in a Sentence Representation Space: Meta’s separate, sentence-level approach from 2024.
From Eden AI
- LLM Routing Explained: Best Strategies for Cost, Speed, and Quality
- The Gap in Shared Understanding of LLM Capabilities
- Eden AI documentation: one API for 500+ models, with provider routing and fallback built in.
