Two Years on Forward-Forward, and the Answer Was No
I spent about two years on the Forward-Forward algorithm. The work ended up as a preprint, arXiv:2606.06539, and as a negative result: layer-local training does not hold up at real-data scale, and the benchmarks the field has been using to argue otherwise are measuring the wrong thing.
It was submitted to NeurIPS 2026 and rejected, as an earlier version was at ICCV 2025 before it. I am writing it up anyway, partly because I think the findings are right and partly because negative results that never leave a rejected PDF are how a field ends up believing things that are not true.
What Forward-Forward is, and why I cared
Backpropagation couples every layer to every other layer through the backward pass, and stores activations across the whole depth of the network to do it. Hinton's 2022 Forward-Forward algorithm replaces this with something strictly local: two forward passes, one on positive data and one on negative, with each layer trained on its own to produce high "goodness" (squared activation norm) on positives and low on negatives. No gradient crosses a layer boundary.
The appeal is structural. If training is layer-local, activation memory is O(1) in depth rather than O(L), and layers can in principle be trained independently. The problem was that FF had never been shown to work at any interesting scale -- the original result was roughly 60% on CIFAR-10 with MLPs, a 30-point deficit against BP.
A line of FF-CNN work has been closing that gap on 32x32 benchmarks. I wanted to know whether that trend meant anything, or whether it was an artifact of only ever testing on 32x32.
Building an instrument, not a method
The problem with auditing FF is that a weak result is ambiguous. If your FF network underperforms BP, you cannot tell whether that is a limit of layer-local training or a limit of your FF implementation. So the first job was to build the strongest FF instance I could, and treat it as a measuring device rather than as the contribution.
That instrument is DTG-FF, and it has three pieces:
Dynamic temperature goodness. Each layer learns a scalar temperature T_l, sigmoid-parameterized into a fixed range, and divides its goodness by it before the local readout. The learned temperatures use the full range meaningfully rather than collapsing: in a trained VGG8, layer 0 goes to the floor at 0.5, middle layers land between 1.4 and 1.9, and deep layers saturate at 2.0.
Decoupled three-path normalization. Each conv layer has three functional paths with different normalization. The goodness path gets none -- batch norm collapses the class-conditional variance in squared activation norms, which is exactly the signal the FF objective depends on. The inter-layer propagation path gets channel-wise LayerNorm, dropout, and detach. The classifier path gets GAP, BatchNorm1d, and a linear head. Prior work generally applied one normalization policy everywhere.
Multi-layer fusion. Per-layer GAP features are concatenated and fed to a classifier that is trained but does not backpropagate into the conv backbone. There is also a logit-sum variant with no trainable classifier at all, which keeps the whole pipeline strictly layer-local.
The reason to bother with the temperature at all shows up in the ablation: fixing T = 1 costs 0.72 pp on the concat variant and 1.34 pp on logit-sum, making it the single largest component effect. Removing inter-layer LayerNorm costs another 0.80 pp.
The right panel of that figure is the one I find more interesting. It shows how much the per-layer classifiers disagree with each other. Layer 0 disagrees with the deeper layers on 54-56% of test samples; layers 4 through 6 disagree with each other on only 3-6%. The mean off-diagonal disagreement is 25.1%. Late layers in an FF network are close to redundant, which is a hint about where the ceiling comes from.
The instrument works
Across nine real datasets, DTG-FF sets FF-family state of the art:
| Method | Arch | CIFAR-10 | CIFAR-100 | Tiny-IN | IN-100 |
|---|---|---|---|---|---|
| Hinton FF | MLP | ~60% | — | — | — |
| SoftHebb | CNN | 80.30% | — | — | — |
| DeeperForward | CNN | 88.72% | — | — | — |
| ASGE | VGG11 | 90.62% | 65.42% | — | — |
| DTG-FF (concat) | VGG8 | 91.33% | 67.28% | 48.17% | — |
| DTG-FF (logit-sum) | VGG8 | 91.79% | — | — | — |
| DTG-FF | VGG11 | — | — | — | 49.40% |
The Tiny ImageNet number is the one I was most pleased with: 48.17% against roughly 35% for the next-highest FF result I could find. And the ImageNet-100 run at 224x224 is, as far as I know, the first FF-family result at that resolution -- 49.4% in 200 epochs, 12.6 hours on a single RTX 4090.
That is the setup. Now the audit.
Finding 1: the gap does not close, it widens
The comparison that matters is not FF against other FF papers. It is FF against a BP baseline with the same backbone, the same depth, the same per-layer head count, and the same training recipe. I trained BP-VGG8-DeepSup for exactly this.
DTG-FF trails it by 2.40 pp on CIFAR-10 and 5.93 pp on CIFAR-100. Under strict aggregation matching, comparing logit-sum to logit-sum, the CIFAR-10 gap is 1.94 pp. Either framing gives the same shape: the gap grows with class count.
The gap is also stable. Re-running on a second seed gives DTG-FF 91.18/66.95% and BP-DeepSup 93.63/72.80% -- both methods reproduce within 0.4 pp, and the FF-BP gap itself moves by 0.05 and 0.08 pp. This is not noise.
At 224x224 the picture is worse. 49.4% against typical BP results above 75%. The 75% figure comes from a ResNet-50, so the precise number conflates algorithm with architecture, and a BP-VGG11 control is the obvious missing experiment -- a reviewer asked for it and was right to. But the conservative reading survives that caveat: FF at real resolution is nowhere near BP's operating range, and this is invisible if you only ever test on 32x32.
Finding 2: the synthetic benchmarks have the sign wrong
This is the part of the paper I would keep if I could only keep one thing.
FF work often validates on synthetic teacher-student tasks, sweeping the number of classes K to argue about scaling. I ran that sweep. On synthetic data, DTG-FF's advantage over an architecture-matched BP-DeepSup grows with K: from -0.23 pp at K=5 to +2.00 pp at K=50, with a pre-specified low-K versus high-K contrast of +1.37 pp (bootstrap 95% CI [+0.59, +2.15]).
On real images the same comparison reverses sign and widens with K: -2.42 pp at K=10, -5.89 pp at K=100.
At the matched point K=10, the synthetic regime predicts a +0.84 pp advantage for FF. The real-data outcome is -2.42 pp. Same class count, opposite sign, 3.3 pp apart.
The explanation is that K means two different things in the two regimes. In a synthetic sweep, K tracks output dimensionality and nothing else. On real images, K also tracks how hard the classes are to tell apart -- CIFAR-100's fine classes are visually harder than CIFAR-10's, not just more numerous. Synthetic K-sweeps confound the two and therefore overstate how well FF transfers.
To try to separate them, I ran a within-dataset probe: CIFAR-100 has both fine (K=100) and coarse (K=20) labels over the same images. The FF-BP gap shrinks from 11.35 pp on fine to 7.99 pp on coarse, a 3.36 pp differential in the predicted direction.
I want to be clear that this probe is weak. Both runs are non-converged at 50 epochs, and the fine gap drifts 5.42 pp by 400 epochs -- more than the 3.36 pp differential it is supposed to establish. It is direction-suggestive, not asymptotic. A reviewer flagged this and I agree with them; running both granularities to convergence is the fix and I did not do it.
Finding 3: the memory argument does not survive a fair baseline
The strongest defense of FF is not accuracy, it is the systems property: pipelined layer-local training has O(1)-in-depth activation memory, BP has O(L). I decided to measure whether that translates into anything.
Implementation matters a lot here. A naive forward-with-grad plus forward-without-grad split duplicates conv computation and loses to BP on both axes. A properly pipelined schedule -- single conv pass, autograd graph released by the local backward before deriving the next layer's input -- recovers the structural property. Within VRAM, pipelined DTG-FF hits 1.535 GB / 1726 imgs/s against BP-VGG8 at 1.589 GB / 1586 imgs/s. A small win.
The real test is the memory-cliff regime: VGG11 at 224x224 on a commodity 8 GB GPU, effective batch 128.
| Method | Phys. batch | Peak GB | imgs/s |
|---|---|---|---|
| Vanilla BP-VGG11 | 128 | 8.18 (host-spill) | 14 |
| BP + grad accumulation (64x2) | 64 | 4.18 | 157 |
| BP + activation checkpointing | 128 | 6.35 | 92 |
| DTG-FF | 128 | 7.90 | 138 |
FF beats vanilla BP at a batch size vanilla BP cannot actually hold. It loses to gradient accumulation -- a technique from 2016 that anyone can turn on in a line -- by 47% peak memory and 14% throughput. Activation checkpointing also fits in VRAM at a 1.5x slowdown.
So the honest conclusion is: the O(1) memory property is real, it is achievable, and it does not buy you anything that memory-optimized BP does not already give you more cheaply.
What I think is actually going on
The reading I offered in the paper is that many FF-family improvements are partial substitutes for the supervised cross-layer signal BP provides for free. Label overlay substitutes for the backward label gradient at every layer. BP-trained classifier heads substitute for end-to-end supervised gradient. Spatial goodness substitutes for a richer per-layer signal. Multi-layer fusion -- including mine -- substitutes post-hoc aggregation for in-training cross-layer flow.
Under that lens the residual 2.40/5.93 pp gap, the widening with K, the depth degradation, and the synthetic-real reversal are all the same phenomenon: incomplete substitutes for coordination that BP does natively.
I should be honest that this is interpretive, not causal. DTG-FF differs from BP-DeepSup along four axes at once -- detach boundary, goodness versus cross-entropy objective, fixed random projection versus learned head, and classifier supervision target. Isolating which one carries the load needs controls that swap one channel at a time, and I did not run them. A reviewer named this and I have no defense.
The reviews
Three reviewers: 5 (Accept), 4 (Borderline accept), 1 (Strong reject).
The 5 rated quality, clarity, and significance all excellent and asked one substantive question: the missing BP-VGG11 control on ImageNet-100. The 4 called the paper "unusually careful and self-critical" and asked for single-dimension ablations to fix the causal underidentification. The area chair's meta-review agreed the main argument was well-designed and that the paper was transparent about its own limits.
It was not accepted. Neither was the earlier version of this work, at ICCV 2025. It lives on arXiv as a preprint and has not been through peer review to publication.
Where this leaves the work
The preprint is on arXiv. I am revising it to add a comparison against HCL-FF, which appeared at CVPR 2026 after I submitted and may beat my FF-family numbers -- in which case the SOTA framing in the abstract needs to come down.
The findings themselves I still believe. FF does not scale on real data, the field's synthetic validation overstates it, and the memory argument does not survive gradient accumulation. Two reviewers thought that was worth publishing, and I still think it is.
If you work on FF or other local learning rules and you think any of the above is wrong, I would like to hear it.