⬢github Python · 94.3K ★ +110 since we first saw it · pushed 46 min ago · Apache-2.0
vllm-project/vllm
A high-throughput and memory-efficient inference and serving engine for LLMs
vLLM is an open-source Python library for fast, memory-efficient inference and serving of large language models. It uses PagedAttention, continuous batching, prefix caching, quantization, speculative decoding, and distributed parallelism to serve 200+ Hugging Face model architectures across NVIDIA, AMD, Intel GPUs, TPUs, CPUs, and other accelerators, with an OpenAI-compatible API server.
Why now: not clear from the sources
Who it is for: Developers and teams who want to self-host or serve LLMs at high throughput in production.
amdblackwellcudadeepseekdeepseek-v3gptgpt-ossinferencekimillama
Stars over our 20 snapshots: 94.2K to 94.3K, since 5 h ago.
Where people talked about it
- ⬢github vllm-project/vllm 12 min ago
API: https://socialmediatrends-api.osmike.com/v1/repos/vllm-project/vllm