Where Discrete Diffusion Models Fit
Discrete denoising diffusion models (DDMs) work with discrete data. They serve as an alternative to autoregressive modeling (AR). Instead of extending a sequence step by step, the model generates in parallel and refines the result iteratively and globally.
The key difference from continuous diffusion is the structure of the state space. In the continuous case, the space is fixed. In the discrete case, it is constructed, and generation depends on that construction.
| Approach | Generation mode |
|---|---|
| Autoregressive modeling | Sequential generation |
| Continuous diffusion | Iterative refinement over a fixed state space |
| Discrete diffusion | Parallel generation and iterative global refinement |
The selection criterion is simple. DDMs are chosen where parallel generation and global refinement of the result are needed. But the quality of such generation is determined not only by the model itself.
State space: three components
A unified framework looks at discrete diffusion through the construction of the underlying discrete state space. The construction includes three elements.
- The tokenization scheme determines the units into which the data is split.
- The vocabulary topology determines how these units relate to one another.
- Domain-specific structural alphabets define the units and relations specific to the subject area.
These three decisions are not reducible to settings on top of a ready-made model. They define the very space through which diffusion proceeds. That is why tokenization and generation are considered within a single framework rather than separately.
Practical criterion: first describe the state space, then choose the model formulation.
A unified framework and three formulations
Existing formulations of discrete diffusion turn out to be different instantiations of a single solution space. The framework contains three of them.
- Transition-matrix — a formulation via the transition matrix.
- Masking / absorbing-state — a formulation via masking and the absorbing state.
- Score / ratio-based — a formulation via score and ratios.
The framework shows that these are not competing theories but variants of a single construction. The difference between them lies in the mode of notation, while what they share lies in the solution space to which they belong.
Criterion: comparing formulations makes sense within a single framework. The choice between them is secondary to how the state space is structured.
Cross-cutting trade-offs
Axes along which decisions are linked
The framework reveals shared trade-offs along several axes at once:
- training objectives;
- inference algorithms;
- scaling behavior;
- system optimization;
- evaluation protocols.
The axes are interconnected. A decision on the training objective constrains the choice of inference algorithm. System optimization and evaluation protocols depend on the same choice.
How to use this
Variants are compared across all axes at once. A gain on one axis does not remove the need to check the others. The framework also outlines directions for further research.
Order of decisions
- Describe the state space: the tokenization scheme, the vocabulary topology, the domain-specific structural alphabet.
- Choose a formulation within the framework: transition matrix, masking, score/ratio.
- Set the training objective and inference algorithm for this construction.
- Check the solution across the axes: scaling, system optimization, evaluation protocols.
Selection criterion: if the construction of the state space is not described, the formulation and inference scheme cannot be justified. The framework is described in the paper "Discrete Diffusion Models: A Unified Framework from Tokenization to Generation" (arXiv:2607.13431, v2 dated August 25, 2026).
