llama.cpp

github.com · ML Frameworks

Open source

#10 in ML Frameworks

Plain C/C++ LLM inference engine that popularised GGUF quantised models running on everyday hardware.

#inference#c++#quantisation#gguf

llama.cpp is one of 20 ML framework tools on Axiomi. See all ML framework tools or how we check facts.

// guides

Guides that cover llama.cpp

What Is Safetensors? Safe Model Files Explained
What is safetensors? A plain guide to the Hugging Face model file format: why it is safer than pickle, how it compares to GGUF, and how to load or convert it.
Ollama vs vLLM: Which Should You Run?
Ollama vs vLLM compared: setup, model formats, speed for one user or many, hardware, Mac support and the signs it is time to move from Ollama to vLLM.
RTX 3090 BF16 at Full Rate: What It Means for AI
RTX 3090 BF16 runs at half rate, and a new Linux mod restores it on RTX 30 cards. What it speeds up for local LLMs, what it doesn't, the risks and setup.
How to Run DeepSeek V4.1 Flash Locally
Can you run DeepSeek V4.1 Flash locally? Real RAM, VRAM and disk needs by GGUF, Strata on one 12 GB GPU, the llama.cpp PR, vLLM, Ollama and DGX Spark.
SmolLM vs Llama: Which 3B Model Should You Run?
SmolLM vs Llama compared on benchmarks, licence, context, languages, memory and local setup, so you can choose between SmolLM3 and Llama 3.2 for your project.
SGLang vs vLLM on DGX Spark: Which to Use?
SGLang vs vLLM on DGX Spark compared on setup, memory, NVFP4 models, speed and clustering, so you can pick the right server for your GB10 desktop.

// related