backpropagation

Researchers at QLabs have published Dust, a method for pretraining transformer models without using backpropagation. The approach, described in a new research paper, would replace the gradient-based training that underpins virtually all modern deep learning. Interest is concentrated among machine learning practitioners debating whether such alternatives could realistically match conventional training at scale.

A technical article explaining the softmax function and how to compute its derivative has drawn attention among developers and machine-learning practitioners. Softmax converts a vector of raw scores into a probability distribution and is central to classification models and neural network outputs. The piece walks through the mathematics of its gradient, including the coupling between outputs, which often trips people up when deriving backpropagation by hand.

A research project called Dust claims a method for pretraining transformer models without backpropagation, the algorithm at the core of modern deep learning. If the results hold up, the approach could challenge assumptions about how large models must be trained, but independent verification and details of its performance are not yet established.