⬢github C · 625 ★ +622 since we first saw it · pushed 5 h ago · MIT
hyperionaidev/smolmaxxing
maxxing the ability to run different LLMs on the smallest devices possible
smolmaxxing is a set of llama.cpp patches plus tuning that lets large language models run on tiny, old hardware like TV boxes, old iPads, and budget phones. It combines 1-bit/2-bit custom kernels, vocabulary pruning, and flash streaming of mixture-of-experts weights, so devices with ~1 GB of RAM can run 8B–16B models and phones run 35B–80B ones.
Why now: It got attention on Hacker News ('I'm making LLMs work on small and old devices') thanks to surprising benchmarks: e.g. 16B MoE models on 1 GB-RAM devices and an 80B model on a mid-range phone.
Who it is for: Hobbyists and tinkerers who want to run local LLMs on old phones, TV boxes, single-board computers, or other low-RAM devices.
edge-aillama-cppllama-cpp-alternativellmlocal-llmlocal-llm-supporton-devicequantizationquantization-algorithms
Stars over our 117 snapshots: 3 to 625, since 1 d ago.
Where people talked about it
- Yhn I'm making LLMs work on small and old devices 51 min ago
API: https://socialmediatrends-api.osmike.com/v1/repos/hyperionaidev/smolmaxxing