llama-cpp

LLM inference in C/C++

llama.cpp is an inference engine for large language models written in plain C/C++, with no external runtime dependencies. It loads quantized models in the GGUF format and runs them on CPU, or with optional acceleration through CUDA, ROCm/HIP, Vulkan or OpenCL backends. The package provides llama-cli (interactive and one-shot inference), llama-server (an HTTP server exposing an OpenAI-compatible API, optionally with an embedded WebUI), llama-quantize, llama-bench and the ggml libraries used by many downstream projects. Models are not shipped with this package and must be downloaded separately, typically from Hugging Face. The CPU backend is tuned through the architecture's CPU_FLAGS: on x86 through CPU_FLAGS_X86, where the AVX-512 and AMX flags supported by the host processor have a large impact on inference throughput, and on ARM through CPU_FLAGS_ARM, which ggml gates at compile time via an accumulated -march string rather than per-feature build options. Note that ebuild is ported from the gentoo-zh overlay (sci-ml/llama-cpp).

homepage ↗ github: ggml-org/llama.cpp