Mmastodon TechnologySoftware first seen 1 d ago, last 1 d ago, peak #6

Quantized AI model sizes tested against full-precision results

Original: How big is Jeb, and what do you lose with a smaller download? 27B: 55.6 GB in bf16, 29 GB as Q8_0. The Q8 gives the same

A developer has published size and accuracy figures for quantized versions of an AI model called Jeb, released in 27B, 9B and 4B sizes. The full 27B model weighs 55.6 GB in bf16 but only 29 GB in Q8_0 format, matching the full weights on 260 of 260 held-out test questions. The 9B and 4B versions match on 257 and 256 of 260 respectively, suggesting modest download savings cost little accuracy.

Why now: Practitioners are weighing whether smaller, cheaper quantized downloads sacrifice model quality, and these test results suggest the trade-off is minimal.

Jeb

Open on mastodon →

Evidence

• How big is Jeb, and what do you lose with a smaller download? 27B: 55.6 GB in bf16, 29 GB as Q8_0. The Q8 gives the same answer as the full weights on 260 of 260 held-out questions. 9B: 9.8 GB as Q8_0, same answer on 257 of 260. 4B: 4.6 GB as Q8_0, same answer on 256 of 260.… · jebadiah

API: https://socialmediatrends-api.osmike.com/v1/trends/2384299