rncz.net

Muse Glimmer 30B by Meta: The Best Local Coding Agent?

llmllama-cppmetamuse-glimmerlocal-aicoding-agentpiamdrx7900xtxdraft-model

Meta released Muse Glimmer — a 30-billion parameter model designed for local agents and coding. So it’s time to install it with Llama.cpp and see how good it really is.

Since I’m running a Radeon RX 7900 XTX with 24 GB of VRAM, I can only use the quantized GGUF version from Unsloth that fits fully in the VRAM of the GPU. This post walks through downloading the model, installing Llama.cpp, tuning the server for maximum speed, and connecting the Pi coding agent to it.

1. Download the model

Head to Hugging Face and find the Muse Glimmer 30B page from Unsloth. In the Files and versions tab, download two GGUF files:

  1. The main model — in my case the Q5_K_M quant, which works pretty fine with a context window of 130,000 on 24 GB of VRAM.
  2. The DFlash GGUF — a small draft model that gives you more speed when running llama.cpp.

Unsloth has really good documentation on the model page — it shows you exactly which quant you can run with your hardware. If you have a card with 12 to 15 GB of VRAM, you can run the 2-bit quant.

The model page also lists the recommended settings, which we’ll use later in the llama.cpp server command.

2. Install Llama.cpp

If you don’t know how to install Llama.cpp on your machine, watch my previous video on installing Llama.cpp on Windows, which explains the steps.

3. Run llama-server

After downloading the models, you should have two files: Muse Glimmer 30B GGUF and the quant you downloaded, plus the DFlash GGUF for extra speed.

Start the server, pointing it at your model:

llama-server -m /path/to/Muse-Glimmer-30B-Q5_K_M.gguf

I use --host 0.0.0.0 so I can access the server from another PC.

Open the URL (the IP address of the machine) and you can see Muse Glimmer is loaded. Say “hi” — I already get around 40 tokens per second, which is pretty nice.

If llama-server is running on the machine you’re using, you can just use localhost:8000.

4. Add the parameters

Now we add the parameters to run the model the way it’s meant to run:

llama-server \
  -m /path/to/Muse-Glimmer-30B-Q5_K_M.gguf \
  --draft-model /path/to/dflash-kquant.gguf \
  --spec-type draft-dflash \
  -ngl 999 \
  -c 131072 \
  --flash-attn on \
  -t 1 \
  -b 0.95 \
  -tk 64 \
  --host 0.0.0.0 \
  --port 8000 \
  -np 1
  • All GPU layers — the model runs entirely on the GPU.
  • Context of 131,000 — the large context window Muse Glimmer supports.
  • Flash attention on — speeds up attention and reduces memory usage; it works on both AMD and Nvidia GPUs.
  • Spec type: draft-dflash — this is the important part. The draft model is the DFlash GGUF we downloaded.

Run it, open the chat on port 8000, and say “hi” again. As you can see, I now get around 56.83 tokens per second — the DFlash draft model is doing its job.

5. Build a Python script

Let’s build a Python FastAPI script to test the server. As you can see, it gets me almost double the tokens per second — the DFlash speculative decoding really pays off.

6. Connect the Pi agent

Now it’s time to connect the Pi coding agent to the LLM.

In your user’s home directory (the same on Windows and Linux), there’s a .pi folder → agent → models.json. If it’s not there yet, don’t worry — you can create it yourself. This is what the JSON looks like:

{
  "providers": {
    "llamacpp": {
      "baseUrl": "http://<ip-address>:8000/v1",
      "models": [
        {
          "id": "Muse Glimmer 30B"
        }
      ]
    }
  }
}

Set the URL of your llama server — for me it’s the IP of the Linux machine, but if you’re running on the same machine, you can just use localhost. The model ID is Muse Glimmer 30B, the model we’re running.

7. Build something

Back on the Windows terminal (or on Linux, of course), create a project directory:

mkdir tic-tac-toe
cd tic-tac-toe
pi

Run /model and select Muse Glimmer. Say “hi” — bash is not configured yet, so the agent can’t run commands with it. But I can already ask it to build an HTML version of tic-tac-toe:

Can you make a two-player tic-tac-toe game in HTML, JavaScript, and CSS?

Wait a moment, and there we go. Open the files it created — and once you open index.html, you see the tic-tac-toe game. It looks pretty similar to the one we created with Qwen 27 billion parameters.

Next steps

  • Try Muse Glimmer 30B with different quantizations on Hugging Face.
  • Configure bash for your Pi agent so it can run commands, not just write files.
  • Keep a git repo and review the diffs to stay in control of the code the agent writes.
← All posts