⬢github Python · 1.4K ★ +385 since we first saw it · pushed 21 h ago · Apache-2.0
RunanywhereAI/trnrt
The open LLM inference engine for AWS Trainium: NKI megakernels, 1M-token context, OpenAI-compatible API. By RunAnywhere.
TrnRT is an open-source LLM inference engine for AWS Trainium chips. It serves an OpenAI-compatible API from a single trn2.48xlarge instance, using NKI megakernels for fast decode, supporting 1M-token context via sparse and linear attention, with an FP8 KV prefix cache and two engines per box. GLM-5.3-Flash is the first supported model.
Why now: It's trending as a newly released, most-starred repo, accompanied by a blog post with benchmarks: 3.57M tokens/minute on one Trainium2 box and faster decode via NKI megakernels.
Who it is for: ML infrastructure engineers serving open LLMs on AWS Trainium hardware who want an OpenAI-compatible stack.
awsglminferencellmllm-servingmegakernelsneuronnkiopenai-apitrainium
Stars over our 147 snapshots: 1K to 1.4K, since 1 d ago.
Where people talked about it
- ⬢github new repos, most starred 13 min ago
- ⬢github new repos, most starred 7 h ago
- ⬢github new repos, most starred 1 d ago
API: https://socialmediatrends-api.osmike.com/v1/repos/RunanywhereAI/trnrt