Models supported
Bubo runs Qwen3.8 Flash Next, and it runs smaller models too. Flash Next is a 117 GB download and needs a 128 GB machine. The eight other models in the app are 16 to 23 GB. The Flash Next Coder is a 61 GB download and needs about 32 GB of GPU memory with the draft file, and about 38 GB for the running server at a short context.
Lower memory options
- Pick a smaller model. The app has eight models of 16 to 23 GB. Flash Next is 117 GB.
- Use the lower-memory build of Flash Next. At the default context of 200,000 tokens, Bubo estimates about 66 GiB of GPU memory for it instead of about 91 GiB, which is about 25 GiB less. It reads prompts at about 0.93 to 0.96 of the standard build’s speed. It writes at about 0.84 to 1.18 of that speed, depending on the work: C++ code is the slowest relative to the standard build, and prose the fastest. These are ranges from one run on one laptop at about 70 W. Its expert weights come from the ISTA IQ3_S file. Compared with the IQ3_S file itself, it needs about 4 GiB more GPU memory and reads a cold 32K prompt about 13 % faster (1,353 against 1,201.8 tokens/s). The app lists it as “Qwen3.8 Flash Next, HaloForge Labs lower-memory quant (GSQ-RCO)”, a download of 89.9 GB.
- On a 96 GB machine, use a lower-memory model. Use the lower-memory build of Flash Next, one of the ISTA files IQ3_S, IQ3_XXS and IQ2_XS, or the Flash Next Coder. Set Variable Graphics Memory (AMD Software: Adrenalin Edition > Performance > Tuning) to the largest value offered (72 GB if available, at least 64 GB), then restart Windows. At 72 GB, Bubo uses the default 200,000-token context for one session with the lower-memory build. At 64 GB it does too when about 16 GB of system memory is free, and picks a smaller context when less is free. The ISTA files and the Coder need less GPU memory than the lower-memory build: about 64 GiB for IQ3_S, 57 for IQ3_XXS, 50 for IQ2_XS and 41 for the Coder at 200,000 tokens and one session, measured on a 128 GB machine. To save more, set KV cache to Q8 (Server page in the app, or
--kv-cache-type q8_0): the context cache becomes 47 % smaller. These values are calculated: Bubo has not been tested on a 96 GB machine. - Use a shorter context. The default for Flash Next is 200,000 tokens, and the largest is 262,144. Use
--contextto change it. The app shows the largest context that fits now. - Use a smaller RAM cache. The default profile uses 8 GiB. In the app, change RAM cache size in Server setup, under Memory and caches. On the command line, use
--cache-ram-bytes. - Leave out what you do not need.
--no-twinfrees the dense twin’s storage.--no-visionleaves out the image projector. - Check the memory before you start. The app checks memory when you press Start. For Flash Next it shows an estimate and a recommended context. The command line gives the same:
bubo models estimate --model PATH. - The memory guard. Bubo keeps 4,096 MiB of memory free. A request that needs more is refused with HTTP 503 or 507, so the PC keeps running. Under Windows low-memory pressure, Bubo drops cache snapshots first. It stops a running request only as the last step.
In the app
The app downloads these models after you approve the files. Size is the full download, with the image projector and any drafting file.
| Model | What it is | Download | Images | Drafting |
|---|---|---|---|---|
| Qwen3.8 Flash Next (Huihui) | Mixture of experts, 4-bit | 116.6 GB | Yes | Yes |
| Qwen3.8 Flash Next (original) | Mixture of experts, 4-bit | 116.6 GB | Yes | Yes |
| Qwen3.8 Flash Next Huihui (lower memory) | Mixture of experts, mixed formats. About 66 GiB of GPU memory | 89.9 GB | Yes | Yes |
| Qwen3.8 Flash Next Coder (ISTA GSQ-RCO, 256 experts) | Mixture of experts, for code, ISTA-DASLab quantization. About 32 GB of GPU memory with the draft file | 61.2 GB | No | Yes |
| Qwen3.8 27B | Dense, 4-bit | 19.6 GB | Yes | Yes |
| Qwen3.6 27B | Dense, 4-bit | 18.5 GB | Yes | No |
| Cyber-Tiel Coder 35B-A3B | Mixture of experts, for code, 4-bit | 23.4 GB | Yes | No |
| Gemma 4 26B-A4B (Unsloth QAT) | Mixture of experts, 4-bit | 15.7 GB | Yes | Yes |
| Gemma 4 31B (Unsloth QAT) | Dense, 4-bit | 18.8 GB | Yes | Yes |
| Gemma 4 26B-A4B uncensored (Huihui, QAT) | Mixture of experts, 4-bit | 18.8 GB | Yes | Yes |
| Gemma 4 31B uncensored (Huihui, QAT) | Dense, 4-bit | 20.8 GB | Yes | Yes |
| DiffusionGemma 26B E38 (abliterated) | Diffusion model, 4-bit (NVFP4) | 20.1 GB | Yes | No |
Images on Gemma 4 reserve 2 GiB for the vision part.
Other model families in the code
Bubo contains the code of the other gufo model families. These are not tested on Windows, and the app does not download them. They run with upstream gufo settings, if at all.
DeepSeek V4 Flash, Qwen3-ASR 1.7B, Qwen3-TTS 1.7B, Qwen-Image-2.1, MiniMax H3.
Speed options
- Drafting (MTP). A small predictor guesses the next tokens and the main model checks them. On for Flash Next, Qwen3.8 27B and Gemma 4 when the package has a drafting file.
--no-mtpturns it off. - Dense twin. A 4-bit copy of the weights for faster generation. 1.2 GB. In the Flash Next download.
- Draft head. A 4-bit output head for drafting. 0.4 GB. In the Flash Next download.
- Fast prompt reading (experimental). Up to 16 % faster when a prompt has 4,096 or more new tokens, with no noticeable reasoning regression, by using a tuned number of experts while reading. Shorter prompts read as in normal mode. Decode unaffected. In the app: Prefill mode. On the command line:
--prefill-mode fast. For models with 512 experts. The Coder (256 experts) refuses it. - Smaller context cache (opt in).
--kv-cache-type q8_0, or "KV cache" on the Server page in the app, stores the context cache of Qwen3.8 Flash Next in 8-bit form. It is 47 % smaller: each extra 64K session needs 1.1 GB instead of 1.9 GB. It is lossy, and replies can differ slightly. - Prompt lookup. Copies tokens from earlier text when the text repeats. Helps code edits and quoted logs.
- RAM cache. Keeps past prompts in memory so the next turn skips them.
- NPU prefill. Optional. Off by default: after the prompt-reading speed-ups it adds less than 1 %. Turn it on with
--npu-prefill on. It needs AMD XRT runtime 32.0.20102.3931. With any other runtime, Bubo writes a warning, reports the NPU as faulted, and uses the GPU.
Decision API (experimental)
The decision API answers a question with probabilities, not text. You give it a prompt and a fixed list of options. It returns one probability for each option in one pass, and it writes no text. Use it when a program must choose: route a request, answer yes or no, or rate something on a scale.
This is experimental, and the interface can change. It follows the vLLM implementation: vLLM pull request 57250.
Licences
See Licences.