Gradient

Researchers at QLabs have published work on Dust, a method for pretraining transformer models without using backpropagation, the gradient-based algorithm that underpins nearly all modern deep learning. The approach, detailed in a new research paper, is drawing attention from machine learning practitioners debating whether alternatives to backpropagation could reduce the memory and compute costs of training large language models.

A technical article explaining the softmax function and how to compute its derivative has drawn attention among developers and machine-learning practitioners. Softmax converts a vector of raw scores into a probability distribution and is central to classification models and neural network outputs. The piece walks through the mathematics of its gradient, including the coupling between outputs, which often trips people up when deriving backpropagation by hand.

A student has entered a Kaggle competition as part of a college project, training a machine learning model on a dataset of over 100,000 rows with categorical fields. They are comparing three gradient boosting libraries — XGBoost, LightGBM and CatBoost — using F1 Macro as the evaluation metric, and report that XGBoost fell behind early.

Turion Space has undergone a rebrand, and Fast Company is examining how the company built a visual identity that reads as a serious space business without leaning on the usual clichés of the sector, such as starfields, gradients and futuristic tropes. The piece is drawing interest among designers and branding watchers as a case study in restrained, distinctive corporate design.