Sakana AI trains 1000-layer networks without backpropagation
Sakana AI researchers presented PC-ALM, a layer-local training method that reaches within about 2 points of backpropagation on 1000-layer networks on MNIST.
Researchers at Sakana AI have introduced Augmented Lagrangian Predictive Coding (PC-ALM), a variant of predictive coding (PC) that keeps every weight update local to its layer while recovering credit signals aligned with backpropagation. The team reports training residual MLPs of up to 1000 layers on MNIST to within roughly 2 percentage points of backprop.
The idea
Backpropagation is a global algorithm: forward pass, backward pass and weight update each wait on the previous one. No known mechanism in the brain performs that kind of network-wide phase locking, which is why local learning methods keep attracting research. PC treats each hidden activation as an optimization variable and penalizes the mismatch with the prediction coming from the layer below. Supervision enters only at the output, and in deep, narrow networks the signal fades before reaching the input.
PC-ALM starts from the constrained formulation of training and attaches a Lagrange multiplier to each layer constraint while keeping PC's penalty term. Inference alternates two local steps: a primal gradient step on the activations and a dual step that accumulates the layer's prediction error. The team reads this as a per-layer PI controller, with the prediction error as the proportional term and the multiplier as the integral term. Setting the multiplier to zero recovers PC exactly.
Results
Sweeping residual MLPs with width and depth from 8 to 128 on Fashion-MNIST and MNIST, with an inference budget of T = 2L, PC-ALM matches backprop across every width, depth and activation (identity, tanh, ReLU), while PC falls off sharply in deep, narrow cells. In the repository's reference cell (width 32, depth 32, ReLU, Fashion-MNIST), test accuracy is 78.66% for BP, 68.13% for PC and 77.75% for PC-ALM, with gradient cosine to BP rising from 0.604 to 0.909.
The team says PC-ALM improves on PC across every benchmark tried, including ResNet-18 on CIFAR-10 and Tiny ImageNet. It is a training method rather than a model, and has so far been tested only on small image benchmarks. An MIT-licensed JAX reference implementation runs on CPU and reproduces the paper's width-depth grid. Details are in the MarkTechPost report.
