Dale meets Langevin: A Multiplicative Denoising Diffusion Model
Nishanth Shetty, Madhava Prasath, Chandra Sekhar Seelamantula
2025-10-06
Summary
This paper explores a new way to train machine learning models, drawing inspiration from how brains learn. It focuses on making the learning process more biologically realistic, specifically by mimicking a rule called Dale's law which governs how brain connections change.
What's the problem?
Traditional gradient descent, a common method for teaching computers to learn, doesn't align with how learning happens in the brain. Biological systems don't simply adjust connections in the same way that standard algorithms do, and this difference limits our understanding and potentially the effectiveness of artificial intelligence. Specifically, the way connections strengthen and weaken in the brain seems to follow different rules than those used in typical machine learning.
What's the solution?
The researchers developed a new optimization technique based on a mathematical concept called geometric Brownian motion, which describes random movements. They showed that this motion naturally leads to a specific distribution of connection strengths – a log-normal distribution – that’s consistent with Dale’s law. They then created a new method for training models, using a 'multiplicative update rule' that’s equivalent to the brain-inspired learning process. This method also improves how the model learns from data that is naturally positive, like pixel values in images, and allows for generating new images. They tested this on datasets like handwritten digits and fashion items.
Why it matters?
This work is significant because it's one of the first attempts to build a generative model – a model that can create new data – that’s directly inspired by how the brain learns. By using biologically plausible rules, the researchers hope to create more efficient and powerful AI systems, and gain a better understanding of the learning process itself. It opens the door to new types of machine learning algorithms that could be more adaptable and robust.
Abstract
Gradient descent has proven to be a powerful and effective technique for optimization in numerous machine learning applications. Recent advances in computational neuroscience have shown that learning in standard gradient descent optimization formulation is not consistent with learning in biological systems. This has opened up interesting avenues for building biologically inspired learning techniques. One such approach is inspired by Dale's law, which states that inhibitory and excitatory synapses do not swap roles during the course of learning. The resulting exponential gradient descent optimization scheme leads to log-normally distributed synaptic weights. Interestingly, the density that satisfies the Fokker-Planck equation corresponding to the stochastic differential equation (SDE) with geometric Brownian motion (GBM) is the log-normal density. Leveraging this connection, we start with the SDE governing geometric Brownian motion, and show that discretizing the corresponding reverse-time SDE yields a multiplicative update rule, which surprisingly, coincides with the sampling equivalent of the exponential gradient descent update founded on Dale's law. Furthermore, we propose a new formalism for multiplicative denoising score-matching, subsuming the loss function proposed by Hyvaerinen for non-negative data. Indeed, log-normally distributed data is positive and the proposed score-matching formalism turns out to be a natural fit. This allows for training of score-based models for image data and results in a novel multiplicative update scheme for sample generation starting from a log-normal density. Experimental results on MNIST, Fashion MNIST, and Kuzushiji datasets demonstrate generative capability of the new scheme. To the best of our knowledge, this is the first instance of a biologically inspired generative model employing multiplicative updates, founded on geometric Brownian motion.