Some Thoughts on Self-Conditioning
A note on self-conditioning.
Diffusion
Self-conditioning was first introduced in Analog Bits. In brief, the training procedure in the original paper is as follows:
First, obtain a prediction using the same and :
then use it as a condition to make another prediction:
and finally compute:
At inference time, the procedure is:
meaning that the prediction from the previous timestep is used as the condition at the current timestep.
Thus, self-conditioning is not perfectly aligned between training and inference. Training uses the first prediction made at the current timestep, whereas inference uses the prediction retained from the previous timestep:
Of course, one could first make an unconditional prediction at every inference step and then use it for self-conditioning:
This would align training and inference, but it would require two forward passes per step. The common approach is therefore closer to an empirical trick: directly reuse the previous prediction to reduce computation.
Self-conditioned Flow Map Language Models via Fixed-point Flows interprets self-conditioning as a fixed-point iteration.
For fixed and , define:
Ideally, we iterate repeatedly:
and eventually converge to:
In other words, once the prediction has stabilized, feeding it back into the model should produce almost no further change.
However, the vanilla self-conditioning loss neither guarantees that this process will converge nor that the second prediction will be better than the first. It only requires:
but does not require:
The model may genuinely be performing refinement, or it may simply be copying its previous prediction.
My current understanding of self-conditioning is that it does not provide the model with new information. The condition is itself predicted by the model from , so it reveals nothing additional about the true .
Nor is it a critic or evaluator that judges whether the previous prediction is good before deciding how to revise it.
Instead, it gives the model a current guess. On the next prediction, the model does not have to start completely from scratch; it can continue from that guess.
The main problem it addresses is therefore not “insufficient information,” but “unstable predictions.”
For example, the current may be compatible with both token A and token B. Without self-conditioning, the model might reconsider its choice at every step:
With self-conditioning, if the previous step already leaned toward A, later steps are more likely to continue along A.
This does not truly reduce posterior uncertainty; rather, it reduces oscillation between different modes and makes the sampling trajectory more stable.
This effect may be especially pronounced for discrete data such as language.
If we use only embedding MSE and both token A and token B are plausible, the optimal prediction may be:
This embedding is neither token—it is simply the average of their embeddings.
Cross-entropy loss can instead represent the distribution directly:
It does not need to average multiple modes into an intermediate embedding and can preserve a discrete distribution instead.
Combined with self-conditioning, if the first step has already developed a slight preference:
a later step may move further toward:
Self-conditioning with cross-entropy may therefore let the model gradually settle on one token mode rather than remain at an average of several tokens.
However, cross-entropy alone does not force the model to choose one mode, and self-conditioning does not guarantee sharpening. More precisely, once a slight preference has emerged, self-conditioning makes that preference easier to preserve and amplify.
If the initial direction is correct, the prediction becomes increasingly stable; if it is wrong, the error may instead become locked in early.
I therefore currently think of self-conditioning as follows:
It provides no new information. Instead, it provides a current hypothesis that reduces mode switching.
For discrete data, cross-entropy prevents multiple discrete modes from being represented by a continuous average, while self-conditioning helps an established mode persist and continue to sharpen.
References:
If you have any thoughts you'd like to discuss, feel free to reach out: xsfeng@gatech.edu