ik_llama-cpp

llama.cpp fork with additional SOTA quants and improved performance

ik_llama.cpp is a fork of llama.cpp by Iwan Kawrakow, the author of most of the k-quants, carrying additional state-of-the-art quantization types (the IQK families) and CPU matrix-multiplication kernels that are substantially faster than upstream for quantized inference. It builds the same llama-cli, llama-server, llama-quantize and llama-bench programs as sci-ml/llama-cpp. To let both be installed at once, every binary is installed with an ik_ prefix (ik_llama-cli, ik_llama-server, and so on) and the shared libraries live in a private subdirectory of libdir, so nothing collides with sci-ml/llama-cpp. No headers or pkg-config files are installed: this package is an application, not a development target. Link against sci-ml/llama-cpp instead. Upstream publishes no release tags, so the package is pinned to a dated commit snapshot fetched as a commit archive. The CPU backend is tuned through CPU_FLAGS_X86. This matters more here than for sci-ml/llama-cpp, because the fork's quantization kernels are written for AVX2 and AVX-512; without those flags the generic fallback gives up most of the fork's advantage. Keywording is deliberately restricted to ~amd64. The SOTA quants are hand-tuned for AVX2/AVX-512, the ARM paths are far less exercised upstream, and no arm64 build of this snapshot has been verified. Note that this ebuild is adapted from the guru overlay (sci-misc/ik_llama-cpp).

homepage ↗ github: ikawrakow/ik_llama.cpp