TL;DR

DiffusionNFT fine-tunes a diffusion model on its own samples, sending each sample through an implicit positive or negative policy according to the external reward signal. In this blog I focus on supervision from data instead: the same two branches are fed with real (or preferred) images and with images generated by a model, and a KL keeps the model close to a frozen reference. The resulting objective is plain flow matching toward a modified target, so its optimum has a closed form at every noisy point. Flow matching, DiffusionNFT, RealNFT and guidance-free training are all special cases.

How I Got Here

I have to admit that I found the derivation of DiffusionNFT [1] hard to follow, even after writing my previous blog about its optimum. The paper reaches the optimal velocity through a posterior split of the old model into positive and negative parts, and it was not obvious to me why the two implicit policies are built the way they are.

I ended up at the same objects from a different direction. I had been working on DDO [5], which fine-tunes a diffusion model with a GAN-style logistic loss between real data and samples of a reference model. Its optimum extrapolates from the reference distribution toward the data, much like guidance, but the loss needs likelihood-ratio estimates and fresh samples from the reference model. I wanted a plain regression loss with the same target, and it turns out to take one line: regress a mixture of the trained model and the reference model onto the data. While discussing this loss with Claude, I got the hint that adding a negative branch on generated samples makes it almost identical to DiffusionNFT. Following that hint, the NFT derivation reduces to a short least-squares exercise. The same objective also covers RealNFT [2] and guidance-free training [3].

This blog follows that route:

1. start from DDO’s target and find a regression loss that reaches it;
2. add a negative branch, which turns the regression loss into an NFT objective on real and generated data, called (G) below;
3. solve (G) in closed form, which gives its optimum (★), and read off the special cases.<!–
4. go back to DDO: replacing the squared loss by a logistic link gives DDO, Diffusion-DPO and SPIN-Diffusion.–>

Setup

I use rectified-flow notation [6]. Noise $x_0\sim\mathcal N(0,I)$ is transported to a data sample $x_1$ along the straight path

so $t=0$ is pure noise and $t=1$ is data, and every model predicts the velocity $v = x_1 - x_0$. Conditioning on a prompt is implicit everywhere.

Three models appear:

• $v_\theta$, the model being trained;
• $v_{\mathrm{old}}$, a fixed model we improve upon, always used under stop-gradient. DDO calls it the reference model (the pretrained model, or the previous round); in DiffusionNFT it is an EMA of $v_\theta$. I call it the old model throughout;
• $v_{\mathrm{ref}}$, a frozen anchor for an optional KL term, for example the pretrained model.

I write $p_{\mathrm{data},t}$ and $p_{\mathrm{old},t}$ for the densities of $x_t$ under the data and under the old model.

Step 1: DDO’s Target

DDO turns the model itself into a discriminator: its logit is $\beta\log(p_\theta/p_{\mathrm{old}})$, and it is trained with the logistic loss between real data and samples of the old model,

up to a weight on the second term. Its optimum (their Theorem 3.3) is

With $\beta\lt 1$ this extrapolates from the old model toward the data, the same effect as guidance. The price is practical: $\log(p_\theta/p_{\mathrm{old}})$ has to be estimated through the ELBO, and every update needs samples from the old model.

Step 2: A Regression Loss with the Same Target

The loss is inspired by guidance-free training (GFT) [3]. GFT bakes classifier-free guidance into a single conditional model by regressing a mixture of its conditional prediction and its (stop-gradient) unconditional prediction onto the data. Replacing the unconditional prediction by the old model gives

At every noisy point the mixture converges to the data velocity $v_{\mathrm{data}} = \mathbb E[v\mid x_t]$, so solving for $v_\theta$,

The right-hand side of (5) is a guidance formula: $v_{\mathrm{data}}$ plays the good model, $v_{\mathrm{old}}$ the bad one, and $\omega$ is the guidance scale. Training with (4) therefore distills guidance into the weights, and the choice of $v_{\mathrm{old}}$ decides which guidance. With the model’s own unconditional prediction, (4) is GFT itself. With a frozen or earlier copy of the model, it is autoguidance [7] distilled into the weights. DSPO [4] has the same form, with preferred images as data, the pretrained model as $v_{\mathrm{old}}$, and a per-pair $\beta$ set by the reward margin of each pair.

The velocity is an affine function of the score, with the same coefficients for every model, so in terms of scores (5) reads

At the data end $t=1$, this is exactly DDO’s target (3), and (4) reaches it without likelihood ratios or samples from the old model. At intermediate $t$ the two are not identical: (6) is the score of the product of the two noised densities, whereas noising DDO’s target gives a different distribution, because noising and taking products do not commute. Classifier-free guidance has the same caveat [8].

Step 3: Adding a Negative Branch

DDO does not only pull the model toward real data; its second term pushes it away from samples of the old model. What is the regression counterpart of that term? The natural guess is to mirror the positive mixture through $v_{\mathrm{old}}$,

and to regress the mirrored mixture onto generated samples¹:

This is where DiffusionNFT shows up. Its implicit positive and negative policies are exactly (7), and its loss sends each of the model’s own samples through both of them, with weights $r$ and $1-r$ given by a reward. Loss (8), with real data as positives and the model’s own samples as negatives, is RealNFT from LingBot-Video [2]. So the regression version of DDO and DiffusionNFT are the same kind of objective; they differ only in where the samples and their labels come from.

The General Objective (G)

Putting the pieces together, the training data come from two sources:

• the data source $q_d$: real images, or the preferred images of a preference dataset;
• the generated source $q_g$: images produced by a model, such as the model’s own samples, samples of a reference model, or the rejected images of a preference dataset.

Each sample carries a label $r\in[0,1]$. Data samples always have $r=1$. Generated samples have either a constant label ($r=0$ for plain negatives) or a per-sample score such as a normalized reward, as in DiffusionNFT. With source weights $\lambda_d, \lambda_g\ge 0$, write

for the weighted average over both sources, where each expectation is over $t$, the data sample $x_1$ with its label, and the noise $x_0$. The objective is

Each sample is split between the two branches by its label: a share $r$ goes through the positive policy and $1-r$ through the negative one. The second term keeps $v_\theta$ close to the frozen anchor, with its own time weight $w_{\mathrm{KL}}(t)$.

With only the data source and no KL, (G) is the regression loss (4). With $r=0$ on the generated source, it is (8). With only the generated source ($\lambda_d = 0$) and reward labels, it is DiffusionNFT².

The Optimum (★)

Assume the network can represent any function of $(x_t,t)$, so (G) can be minimized separately at every noisy point. For one sample with target $v$, the derivative of the bracket in (G) with respect to $v_\theta$ is

So each sample pulls $v_\theta$ toward $v_{\mathrm{old}} + \frac{2r-1}{\beta}(v - v_{\mathrm{old}})$: a positive sample’s target is stretched by $1/\beta$ away from $v_{\mathrm{old}}$, and a negative sample’s target is stretched and reflected through $v_{\mathrm{old}}$. Let

be the posterior probability that $x_t$ came from the data source, where $p_d$ and $p_g$ are the densities of noisy data and noisy generated samples. Averaging (9) over all samples that could have produced $x_t$, adding the KL gradient $2\lambda_{\mathrm{KL}} w_{\mathrm{KL}}(v_\theta - v_{\mathrm{ref}})$, and setting the sum to zero gives

where $v_{\mathrm{data}} = \mathbb E_d[v\mid x_t]$ is the posterior mean of the data, and $\Delta_g$ is the label-signed posterior mean of the generated samples,

Without an external reward, all generated samples share one label, usually $r=0$ as in RealNFT and DDO, and then $\Delta_g = (2r-1)(v_{\mathrm{gen}} - v_{\mathrm{old}})$ with $v_{\mathrm{gen}} = \mathbb E_g[v\mid x_t]$. With $w_{\mathrm{KL}} = w$, the shrinkage is the constant $\kappa = \beta^2/(\beta^2 + \lambda_{\mathrm{KL}})$.

When do negatives matter?

Take the case without an external reward, so the generated samples are plain negatives. If they are exact samples of a self-consistent old model, then $v_{\mathrm{gen}} = v_{\mathrm{old}}$ and $\Delta_g = 0$. Without KL,

The negatives add no direction; they only lower the guidance scale from $1/\beta$ to $\alpha(x_t)/\beta$ where the model’s samples are dense. They push in a direction of their own only when they deviate from $v_{\mathrm{old}}$ (sampler error, sampling-time guidance, few-step samplers, drift over rounds, or samples from a different model), or when per-sample labels split them into good and bad parts, as in DiffusionNFT.

Special Cases

Each method below is one setting of (G), and plugging that setting into (★) gives its optimum. The regression loss (4) and its optimum (5) are the case with only the data source.

DiffusionNFT

Only the generated source ($\lambda_d = 0$, so $\alpha = 0$), consisting of the model’s own samples, labels from a reward, no KL. Write $m = \mathbb E_g[r\mid x_t]$ for the mean label and $v^{+} = \mathbb E_g[r\,v\mid x_t]/\mathbb E_g[r\mid x_t]$ for the reward-weighted posterior mean. Then (11) becomes $\Delta_g = 2m\,(v^{+} - v_{\mathrm{old}}) - (v_{\mathrm{gen}} - v_{\mathrm{old}})$, and if the old model is self-consistent, $v_{\mathrm{gen}} = v_{\mathrm{old}}$ and

This is DiffusionNFT’s optimal velocity, and $m(x_t) = \mathbb E[r\mid x_t]$ is the optimality posterior $p(o=1\mid x_t,c)$ from my previous blog. Note that $v^{+}$ is the velocity of the reward-reweighted samples (the paper’s $v^{+}$), not the implicit positive policy $\hat v_{\mathrm{pos}}$ (the paper’s $v_\theta^{+}$). At the optimum, $\hat v_{\mathrm{pos}}^\star = v_{\mathrm{old}} + 2m\,(v^{+} - v_{\mathrm{old}})$, so the two coincide only where $m(x_t) = \tfrac12$; elsewhere the positive policy overshoots $v^{+}$ ($m\gt\tfrac12$) or stops short of it ($m\lt\tfrac12$). DiffusionNFT centers its rewards at $\tfrac12$ within each prompt, which makes the reading “the positive policy fits the positive samples” hold on average.

RealNFT and real-versus-own objectives

RealNFT [2] uses real data and the model’s own samples as negatives, equal weights $\lambda_d = \lambda_g = 1$, an EMA as $v_{\mathrm{old}}$, and a KL to the frozen base (written with a factor $\tfrac12$, so their $\lambda_{\mathrm{KL}}$ is twice ours). Keeping a general constant label $r$ for the own samples, (★) gives

The first term is the regression optimum (5), discounted by how likely $x_t$ is to be real. The second term pushes away from the model’s own samples when $r\lt\tfrac12$, but, by (12), only by as much as those samples deviate from $v_{\mathrm{old}}$.

The Family at a Glance

References