search
AI inference hardware
Trends
- 1180B-parameter LLM runs locally on a laptop without a GPUโGPU ์์ด ์๋น์์ฉ ๋ ธํธ๋ถ์์ 180์ต ํ๋ผ๋ฏธํฐ LLM์ ๊ตฌ๋ํ๋ POCKET-Darwin-180B. 4๋นํธ GGUF ์์ํ๋ก 360GBโ111GB ์์ถ, ์ฝ $1,400 ํ๋์จ์ด๋ก ๋ก์ปฌ ์ถ๋ก ๊ฐ๋ฅ. # ai #
A project called POCKET-Darwin-180B is drawing attention for running a 180-billion-parameter language model on consumer hardware with no discrete GPU. Using 4-bit GGUF quantization, the model is compressed from roughly 360GB down to 111GB, enabling local inference on hardware costing about $1,400. Commenters in AI and open-source circles are highlighting it as a sign that frontier-scale models may soon run off the cloud.
- 2Power, Memory And Packaging Now Limit AI Chips, Not TransistorsโผPower, Memory And Packaging, Not Transistors, Now Limit AI Chips
Industry analysts say the bottlenecks holding back AI chip performance are no longer transistor scaling. Power delivery, memory bandwidth and advanced packaging have become the key constraints, shifting how chipmakers like Nvidia, AMD and TSMC approach next-generation AI hardware design and investment.
- 3UC Berkeley and FuriosaAI Propose HBF for LLM ServingโHBF for High-Throughput LLM Serving (UC Berkeley, FuriosaAI)
Researchers at UC Berkeley, working with chipmaker FuriosaAI, have published work on HBF, a memory approach aimed at high-throughput serving of large language models. The piece, carried by Semiconductor Engineering, focuses on how new memory architectures could ease the bandwidth and cost bottlenecks that limit LLM inference at scale. The work is being followed by readers tracking hardware innovation for AI infrastructure.