PengyOS / Docs / Recipes / llama.cpp

Run a model with llama.cpp

llama.cpp runs language models on your own machine, on the CPU or a GPU, and serves them over the OpenAI-compatible API that Pengy speaks. Build it in the dev box, run it on the host.

ServerDesktop

Build it

pengyos-guest enter dev
sudo apt install libssl-dev
git clone --depth 1 https://github.com/ggml-org/llama.cpp ~/src/llama.cpp
cd ~/src/llama.cpp
cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_OPENMP=OFF
cmake --build build -j"$(nproc)" --target llama-server llama-cli
exit

Make the commands available outside the box:

ln -s ~/src/llama.cpp/build/bin/llama-server ~/src/llama.cpp/build/bin/llama-cli ~/.local/bin/

Run a model

-hf downloads a model from Hugging Face the first time and caches it in ~/.cache/huggingface. Start with a tiny one to check everything works:

llama-server -hf ggml-org/gemma-3-270m-it-GGUF --port 8080

In another terminal:

curl -s http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"messages":[{"role":"user","content":"Say hello."}]}'

For real work, and especially for Pengy's tool calls, use a bigger model that handles tools well, e.g. a Qwen3 at 4-bit quantization (-hf unsloth/Qwen3-8B-GGUF:Q4_K_M, about 5 GB). As a rule of thumb a 4-bit model needs a bit more memory than its file size; 8B models are usable on a modern CPU, and bigger ones want a GPU.

Without --host, llama-server listens only on the machine itself. To share it with other computers add --host 0.0.0.0 and open port 8080 in the firewall; it has no login, so do that only on a network you trust (or set --api-key).

Even on localhost, any web page open in your browser can send requests to it. On a Desktop, consider --api-key with a password of your own, and give Pengy the same key (/apikey).

Point Pengy at it

pengy-cli
/baseurl http://127.0.0.1:8080/v1
/models
/model unsloth/Qwen3-8B-GGUF:Q4_K_M

llama-server serves one model, under the name you started it with. More on the settings in Set up Pengy.

Keep it running

mkdir -p ~/.config/systemd/user
cat > ~/.config/systemd/user/llama-server.service <<'EOF'
[Unit]
Description=llama.cpp server
After=network-online.target

[Service]
ExecStart=%h/src/llama.cpp/build/bin/llama-server -hf unsloth/Qwen3-8B-GGUF:Q4_K_M --port 8080
Restart=on-failure

[Install]
WantedBy=default.target
EOF
systemctl --user enable --now llama-server
journalctl --user -u llama-server -f       # the first start downloads the model
sudo loginctl enable-linger "$USER"        # on a Server: keep it running after you log out

On a GPU

Vulkan (AMD, Intel, NVIDIA)

The Vulkan build uses the GPU driver PengyOS already ships, so it runs on the host like the CPU build. It works on AMD, Intel and NVIDIA graphics:

pengyos-guest enter dev
sudo apt install libssl-dev libvulkan-dev glslc spirv-headers
cd ~/src/llama.cpp
cmake -B build-vulkan -DCMAKE_BUILD_TYPE=Release -DGGML_OPENMP=OFF -DGGML_VULKAN=ON
cmake --build build-vulkan -j"$(nproc)" --target llama-server
exit
~/src/llama.cpp/build-vulkan/bin/llama-server --list-devices    # should name your GPU

Then use build-vulkan/bin/llama-server in place of build/bin/llama-server above. It puts as much of the model on the GPU as fits; a model bigger than the GPU's memory is split between the GPU and the CPU, and is slower for it.

CUDA (Desktop NVIDIA)

The NVIDIA edition's cuda box has NVIDIA's compiler and libraries. A CUDA build links those libraries, which aren't on the host, so it runs inside the box:

pengyos-guest enter cuda
sudo apt install libssl-dev
cd ~/src/llama.cpp
cmake -B build-cuda -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON
cmake --build build-cuda -j"$(nproc)" --target llama-server
build-cuda/bin/llama-server -hf unsloth/Qwen3-8B-GGUF:Q4_K_M -ngl 99 --port 8080

Update it

cd ~/src/llama.cpp && git pull
pengyos-guest enter dev -- cmake --build build -j"$(nproc)" --target llama-server llama-cli
systemctl --user restart llama-server
← GoSyncthing →