PengyOS / Docs / Recipes / llama.cpp
Run a model with llama.cpp
llama.cpp runs language models on your own machine, on the CPU or a GPU, and serves them over the OpenAI-compatible API that Pengy speaks. Build it in the dev box, run it on the host.
ServerDesktop
Build it
pengyos-guest enter dev
sudo apt install libssl-dev
git clone --depth 1 https://github.com/ggml-org/llama.cpp ~/src/llama.cpp
cd ~/src/llama.cpp
cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_OPENMP=OFF
cmake --build build -j"$(nproc)" --target llama-server llama-cli
exit
libssl-devlets llama.cpp download models over HTTPS. Without it the build still works, but-hfcan't fetch anything.-DGGML_OPENMP=OFFis what lets the result run on the host: llama.cpp uses OpenMP by default, and PengyOS doesn't ship its library (libgomp). llama.cpp's own thread pool replaces it.
Make the commands available outside the box:
ln -s ~/src/llama.cpp/build/bin/llama-server ~/src/llama.cpp/build/bin/llama-cli ~/.local/bin/
Run a model
-hf downloads a model from Hugging Face the first time and caches it in
~/.cache/huggingface. Start with a tiny one to check everything works:
llama-server -hf ggml-org/gemma-3-270m-it-GGUF --port 8080
In another terminal:
curl -s http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' \
-d '{"messages":[{"role":"user","content":"Say hello."}]}'
For real work, and especially for Pengy's tool calls, use a bigger model that handles tools
well, e.g. a Qwen3 at 4-bit quantization (-hf unsloth/Qwen3-8B-GGUF:Q4_K_M, about
5 GB). As a rule of thumb a 4-bit model needs a bit more memory than its file size; 8B models
are usable on a modern CPU, and bigger ones want a GPU.
Without --host, llama-server listens only on the machine itself. To share it with
other computers add --host 0.0.0.0 and open port
8080 in the firewall; it has no login, so do that only on a network you trust (or set
--api-key).
Even on localhost, any web page open in your browser can send requests to it. On a Desktop,
consider --api-key with a password of your own, and give Pengy the same key
(/apikey).
Point Pengy at it
pengy-cli
/baseurl http://127.0.0.1:8080/v1
/models
/model unsloth/Qwen3-8B-GGUF:Q4_K_M
llama-server serves one model, under the name you started it with. More on the settings in Set up Pengy.
Keep it running
mkdir -p ~/.config/systemd/user
cat > ~/.config/systemd/user/llama-server.service <<'EOF'
[Unit]
Description=llama.cpp server
After=network-online.target
[Service]
ExecStart=%h/src/llama.cpp/build/bin/llama-server -hf unsloth/Qwen3-8B-GGUF:Q4_K_M --port 8080
Restart=on-failure
[Install]
WantedBy=default.target
EOF
systemctl --user enable --now llama-server
journalctl --user -u llama-server -f # the first start downloads the model
sudo loginctl enable-linger "$USER" # on a Server: keep it running after you log out
On a GPU
Vulkan (AMD, Intel, NVIDIA)
The Vulkan build uses the GPU driver PengyOS already ships, so it runs on the host like the CPU build. It works on AMD, Intel and NVIDIA graphics:
pengyos-guest enter dev
sudo apt install libssl-dev libvulkan-dev glslc spirv-headers
cd ~/src/llama.cpp
cmake -B build-vulkan -DCMAKE_BUILD_TYPE=Release -DGGML_OPENMP=OFF -DGGML_VULKAN=ON
cmake --build build-vulkan -j"$(nproc)" --target llama-server
exit
~/src/llama.cpp/build-vulkan/bin/llama-server --list-devices # should name your GPU
Then use build-vulkan/bin/llama-server in place of build/bin/llama-server
above. It puts as much of the model on the GPU as fits; a model bigger than the GPU's memory
is split between the GPU and the CPU, and is slower for it.
CUDA (Desktop NVIDIA)
The NVIDIA edition's cuda box has NVIDIA's compiler and libraries. A CUDA build links those libraries, which aren't on the host, so it runs inside the box:
pengyos-guest enter cuda
sudo apt install libssl-dev
cd ~/src/llama.cpp
cmake -B build-cuda -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON
cmake --build build-cuda -j"$(nproc)" --target llama-server
build-cuda/bin/llama-server -hf unsloth/Qwen3-8B-GGUF:Q4_K_M -ngl 99 --port 8080
Update it
cd ~/src/llama.cpp && git pull
pengyos-guest enter dev -- cmake --build build -j"$(nproc)" --target llama-server llama-cli
systemctl --user restart llama-server