Working memory where it's not the digit that matters, but the magnitude
Holding a continuous variable in mind — an angle of rotation, a velocity, a cursor position, a pressure force — and smoothly adjusting it: this is perhaps the most honest test of working memory. Remembering "seven" is not hard, but retaining a magnitude without natural boundaries, one capable of shifting by an arbitrarily small fraction, is a task of an entirely different class.
With it, artificial networks have historically had a strained relationship. Models with continuous attractors — that is, with stable states forming not a handful of isolated points but a smooth surface — hang on by a thread: nudge the parameters slightly, and the behavior falls apart.
Why conventional recurrent architectures "quantize" state
Standard recurrent solutions, including GRUs and LSTMs, usually never learn continuous manifolds. Instead of a continuum, they break the state space into discrete point attractors: the network, as it were, rounds the input to the nearest "shelf." For classification this is convenient; for quantitative estimation it's a death sentence.
A team of researchers (Zhaotian Gu, Jie Su, Weiwei Wang, Chang Liu, Tianyi Qian, Dahui Wang) proposed treating this not by complicating the architecture, but with one specific computational trick. The work was published in Transactions on Machine Learning Research (August 2026); the preprint is available as arXiv:2608.01947 in the q-bio.NC, cs.AI, and cs.NE sections.
Division instead of subtraction: a minimal model
The idea came from neuroscience. Division (divisive normalization) is a canonical operation observed ubiquitously in cortical circuits, and the authors built around it a minimal, algebraically isolated recurrent network — the Recurrent Divisive Normalization Network, or RDNN for short. There's nothing extraneous inside: only the dynamics of division and whatever comes with it.
Then comes the move standard for computational neuroscience: dynamical systems analysis on classic working memory tasks. And here a biophysical constraint unexpectedly turns into an advantage. The network converges to stable slow manifolds, and with high precision: trajectories don't diverge but settle onto a narrow surface and move calmly along it.
What happens to the gradient
A separate thread of the work is an analysis of learning via backpropagation through time (BPTT). Division introduces a local scaling of the gradient that depends on current activity: the more strongly a block is excited, the more modest the parameter update it receives. The result is a built-in brake that engages exactly where it's needed.
The effect is observable: the network's effective rank noticeably self-compresses, and the recurrent dynamics end up locked in a tight low-dimensional subspace. At the same time, the authors emphasize the difference from explicit low-rank factorization — there, such a constraint often leads to optimization pathologies, whereas here it arises on its own, as a side effect of activity.
Ablations: without division, the manifold falls apart
The most telling part is the comparison with subtractive inhibition. Such a substitution can hold static memories: preserving a fixed point is no problem. But as soon as the input signals begin to change over time, the manifold "shatters" (manifold shattering) — continuity disappears exactly where it's needed most.
Hence a takeaway worth keeping in mind: division is not a biological curiosity inherited from the cortex, but a working computational mechanism. It's needed so the network learns high-precision continuous representations, not just neat discretization.
What to do with this in practice
The practical point is not to urgently rewrite all recurrent models. Rather, it's a reminder: normalization isn't only about training stability and isn't only about regularization. It's a way to set the geometry of the internal state space, and division has a clear advantage over subtraction here.
If a task requires holding a continuous magnitude and smoothly updating it — in control, in tracking, in any systems concerned with "how much" rather than "which one" — it's worth taking a closer look at schemes with multiplicative normalization. The cost is minimal, and in return the network stops rounding what shouldn't have been rounded.
