rncz.net

Installing Llama.cpp on Windows 11 (AMD & Nvidia)

llmllama-cppwindowsgpuamdrx7700xtrx7800xtrdna3rx7900xtxvulkan

If you’ve been running local LLMs through LM Studio, you’ve probably noticed it has a backbone: Llama.cpp. It’s a fast C++ implementation of LLM inference that runs from the command line — no GUI required, and it’s the engine many other tools build on top of.

This post walks through installing Llama.cpp on Windows 11, choosing the right build for your hardware, and running your first model with llama-server.

1. Download Llama.cpp

  1. Go to llama-cpp.com and click the Download button.
  2. You’ll see builds for Windows, Linux, and macOS. Pick the Windows one — it links to the GitHub releases page with all available versions.

On the releases page, pick the build that matches your hardware:

  • CPU only → the regular build (no GPU suffix)
  • Nvidia GPU → the CUDA build
  • AMD GPU → the Vulkan build (pretty stable)

Download the archive and extract it somewhere convenient, e.g. C:\Users\<username>\Downloads\llama.

2. Open a terminal in the folder

  1. Rename the extracted folder to something short, e.g. llama.
  2. Open CMD (or PowerShell) from the Start menu.
  3. cd into the folder:
cd C:\Users\<username>\Downloads\llama
  1. Run cls to clear the screen, then list the tools:
dir

You should see the llama-*.exe tools — llama-server, llama-cli, and more.

3. Download a model

Head to huggingface.co and pick a small model to start with — for example a Qwen3.5 4B model from Unsloth.

  1. Go to the model’s Files and versions tab.
  2. Download the Q4_K_M quantization. It’s small enough that it will probably run on a GPU with just 4 GB of VRAM.

Save it in a folder you know, e.g. C:\Users\<username>\Downloads\.

4. Run llama-server

Go back to the terminal (still inside the llama folder) and start the server, pointing it at your model. Since the model lives one folder up, use ..\:

llama-server -m ..\unsloth-Qwen3.5-4B-Q4_K_M.gguf

That’s it — that’s all the magic needed to run a model.

5. Chat with your LLM

llama-server opens a built-in web UI (a ChatGPT-style chat) at localhost:8080. Open it in your browser and start talking to your model.

Stop the server any time with Ctrl+C.

Next steps

  • Experiment with other models and quantizations on Hugging Face.
  • Try llama-cli for quick one-shot prompts from the terminal.
  • Look into --ctx-size, --threads, and GPU layer offload flags (-ngl) to tune performance.
← All posts