vllm
High-throughput, memory-efficient inference and serving engine for LLMs
vLLM is a fast and easy-to-use library for LLM inference and serving, with a Python API and an OpenAI-compatible HTTP server. USE=cpu/cuda/rocm pick a mutually-exclusive VLLM_TARGET_DEVICE; with none set the build is device-agnostic (API surface only).
homepage ↗homepage ↗homepage ↗ pypi: vllm github: vllm-project/vllm
Available in
| Overlay | Newest | Ebuilds | Last activity | |
|---|---|---|---|---|
| stuff GitHub ↗ | 0.25.1 | 5 | 2 d | details › |
Versions & arches
| Version | Overlay | amd64 | Committed | |
|---|---|---|---|---|
| 0.25.1 ≈ | stuff | amd64 testing | 5 d | view · download · history ↗ |
| 0.25.0 ≈ | stuff | amd64 testing | 15 d | view · download · history ↗ |
| 0.24.0 ≈ | stuff | amd64 testing | 22 d | view · download · history ↗ |
| 0.23.0 ≈ | stuff | amd64 testing | 22 d | view · download · history ↗ |
| 0.22.1 ≈ | stuff | amd64 testing | 22 d | view · download · history ↗ |
Use flags of 0.25.1
- cpu Build for CPU inference (VLLM_TARGET_DEVICE=cpu); pull torchaudio + numba
- cuda Enable NVIDIA CUDA support (computation on GPU)
- humming Pull dev-python/humming-kernels for the optional CUDA-only "humming" MXFP4 quantization backend (needs USE=cuda)
- rocm Build for AMD ROCm inference (VLLM_TARGET_DEVICE=rocm); pull HIP libs + torch{audio,vision}
- rust Build the optional vllm-rs Rust serving frontend (opt-in at runtime via VLLM_USE_RUST_FRONTEND=1)