Quickstart

Complete the installation steps first. Use the default profile for normal use. It selects medium effort and adaptive thinking for Qwen3.8 Flash Next.

Select the files

Run bubo prepare first (see the installation steps). The prepare step writes bubo-package.json beside the model. The server finds the dense twin, the draft head, the MTP predictor and the vision projector through that file. You do not set file paths.

1. Open PowerShell in the bin folder of the package.
2. Set the path to the first model shard.

$model = "$env:LOCALAPPDATA\Bubo\models\huihui-qwen38-flash-next-ud-q4-k-xl-v1\Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf"

If you used --output ROOT, replace $env:LOCALAPPDATA\Bubo with ROOT. A flag or an environment value overrides the manifest. A broken manifest or a missing named file stops the server and names the file.

Start the optimal text setup

1. Start the server.

.\bubo.exe serve llm --model $model --served-model-name bubo

The bubo model identifier is explicit so the API example can use a fixed name. The default profile supplies survival control, automatic draft vocabulary, and prompt lookup with MTP. It enables bounded lenient acceptance inside thinking spans. It uses one session, 200,000 context tokens, and an 8.6 GB RAM cache. Disk prompt cache stays off.

2. Wait for event=load_completed in the log.

Send a first request

1. Open a second PowerShell window.
2. Set the request body.

$body = @{
    model = "bubo"
    messages = @(@{ role = "user"; content = "Explain what a local language model does in two sentences." })
    max_tokens = 512
} | ConvertTo-Json -Depth 5

3. Send the request.

$response = Invoke-RestMethod -Method Post -Uri "http://127.0.0.1:8080/v1/chat/completions" -ContentType "application/json" -Body $body

4. Display the answer.

$response.choices[0].message.content

The server speaks the OpenAI API. The default port is 8080, so the base URL is http://127.0.0.1:8080/v1. The model name is bubo. To use another name, set --served-model-name. Keep the local address for normal laptop use.

Send a request with curl

Windows includes curl.exe. Use the form for your shell.

In cmd, or in Git Bash:

curl http://127.0.0.1:8080/v1/chat/completions -H "Content-Type: application/json" -d "{\"model\":\"bubo\",\"max_tokens\":64,\"messages\":[{\"role\":\"user\",\"content\":\"Say hello.\"}]}"

In PowerShell, call curl.exe and not curl. In PowerShell, curl is an alias for another command. Pipe the body to curl.exe so that no quote needs an escape.

'{"model":"bubo","max_tokens":64,"messages":[{"role":"user","content":"Say hello."}]}' | curl.exe http://127.0.0.1:8080/v1/chat/completions -H "Content-Type: application/json" -d "@-"

Both forms return one JSON object. The reply text is in choices[0].message.content.

Send a request with Python

Install the OpenAI Python SDK.

pip install openai

Run this script.

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8080/v1", api_key="not-needed")
reply = client.chat.completions.create(
    model="bubo",
    messages=[{"role": "user", "content": "Say hello."}],
    max_tokens=64,
)
print(reply.choices[0].message.content)

The server checks no key unless you start it with --api-key. The SDK needs a key that is not empty, so any text works.

For every endpoint, request field and error, see the API reference.

Connect a client

Bubo speaks the OpenAI API. Set these values in a client that uses an OpenAI-compatible server.

Setting	Value
Base URL	http://127.0.0.1:8080/v1
Model name	The model’s id from GET /v1/models. bubo also works.
API key	Any text, for example not-needed. The server checks the key only when you start it with --api-key.
Context size	200000. This is the default context of the default profile. The largest is 262144.

Tool calling works. The server returns OpenAI-style tool calls, and the exact-output tests compared them with the reference. The agent speed numbers come from real coding-agent sessions.

Images

The vision projector from the manifest loads by default. Images need more memory. To save memory when you do not need images, start the server with --no-vision.

.\bubo.exe serve llm --model $model --no-vision --served-model-name bubo

Inspect the settings

1. Print the effective settings without a model load.

.\bubo.exe serve llm --model $model --print-effective-settings

This command prints JSON and exits before it starts a server.

Use exact output

1. Stop the current server with Ctrl+C.
2. Start the exact profile.

.\bubo.exe serve llm --model $model --profile exact --seed 1 --served-model-name bubo

The exact profile leaves out the MTP predictor, the dense twin and the draft head, even when the manifest names them. It refuses flags that turn on approximation or speculation (exit code 2). It disables speculation, behavior policies, and prompt caches. Each request processes its full prompt. Keep the prompt and request settings fixed for reproducibility. See profiles for the exact scope.

Fast restarts with the disk cache

The disk prompt cache is off by default. When you turn it on, Bubo keeps the computed prompt state of long conversations on the disk. After a server restart, or after the RAM cache drops a conversation, the conversation continues without processing the whole prompt again.

Measured on a 70 W Strix Halo laptop with a 30K-token session:

Measured	Value
Cold prompt processing, 30K tokens	About 25 s
Next turn, same server, from memory	0.9 s
Continue after a server restart, with the disk cache	About 2 s (1.8 to 2.4 s in two runs; about 30,400 prompt tokens restored)
First start with the cache (one-time model check)	74 s
Later starts with the cache	29 to 32 s
Size of one saved snapshot	About 954 MB for 30K tokens (about 31 KB for each token)

In an earlier test, a 109K-token session continued after a restart in 3.5 s instead of 88.8 s. A long saved session writes about 3 GB to the SSD. A 90-request agent replay wrote about 5.1 GB. Decode speed cost is about 2.5%. The cache writes to your SSD. Use these numbers to decide if the cache is right for your disk.

To turn it on, add the folder for the cache and a staging size to the server command:

.\bubo.exe serve llm --served-model-name bubo --cache-disk "D:\BuboCache" --cache-disk-staging-bytes 5368709120

The first start with the cache reads the eight model files once and checks them. Bubo saves a stamp. Later starts use the stamp and do not read the files again. Without a prepared model package the server refuses the cache.

• Bubo saves a snapshot about 30 s after the conversation goes idle.
• The disk budget is 8.6 GB by default. Change it with --cache-disk-bytes. A long session can need more.
• Add --cache-disk-staging-bytes 5368709120 (5.4 GB) for long sessions. Bubo stages each snapshot in RAM before it writes it. The automatic staging size follows the free RAM at start, and it can be too small for one snapshot. With about 7.5 GB free, the area was 941 MB, smaller than one 30K-token snapshot (954 MB). Bubo then dropped the snapshots and the restart processed the whole prompt again (25 s).
• Turn the cache off again with --no-cache-disk. The exact profile turns all caches off and ignores the disk cache.

See options for other controls.