JustVugg/colibri

Maintainer? Show your league in your README

Card: your league's emblem, from Copper to Legendary

[![bestofgithub](https://bestofgithub.dev/badge/JustVugg/colibri.svg?style=card)](https://bestofgithub.dev/repo/JustVugg/colibri)

Flat: fits beside your other badges

[![bestofgithub](https://bestofgithub.dev/badge/JustVugg/colibri.svg)](https://bestofgithub.dev/repo/JustVugg/colibri)

Loading README…

Option	What it does
`--list`	every model against this machine, and why one does not fit
`--model ID`	install that model (the ids are in the tables below)
`--yes`	no questions: take the recommendation
`--dir DIR`	put the models on another disk (default `~/colibri-models`)
`--backend vulkan`, `cuda` or `cpu`	choose the engine build yourself; `--no-gpu` is `--backend cpu`
`--model-dir DIR`	use a model you already downloaded
`--reconfigure`	choose another model

What you see	What to do
it stopped during the download	run the same command again: it resumes from the bytes already on disk
`to use the GPU through ..., first run: <command>`	run that command, then the setup again: it rebuilds for the GPU and does not download again
`the ... build failed`, for example `Unsupported gpu architecture` when the installed CUDA toolkit no longer supports the card	the setup checks the toolkit against the card first and picks Vulkan by itself, saying why; if a build still fails it offers the next one (Vulkan, then the CPU). `./start-here.sh --backend vulkan` forces Vulkan; `--no-gpu` stays on the CPU
`needs N GB free for the download`	`--dir` with a folder on a bigger disk
on WSL, the model folder is under `/mnt/c`	keep it on the Linux disk (the default, `~/colibri-models`): `/mnt/c` is many times slower
you updated the checkout (`git pull`)	run `./start-here.sh` again: it rebuilds the engine when the sources changed, then starts it
anything else	`c/coli logs -n 50` shows the log of a server started in the background (one started in the foreground prints to its own terminal) and `c/coli logs --install` the setup's; open an issue with the last lines the setup printed

Model	`--model`	Download	RAM	GPU	Measured
Qwen3.6-35B-A3B: chat with thinking and tools	`qwen36-35b`	23 GB	10 / 20 GB	CUDA, Vulkan (integrated too)	6.0 tok/s on the CPU, 9.9 with Vulkan on the integrated GPU (A); 30.0 with CUDA (C)
Qwen3-Coder-30B-A3B: code and tool calls, no thinking	`qwen3-coder-30b`	19 GB	8 / 18 GB	CUDA, Vulkan (integrated too)	8.5-9.6 tok/s with every expert in RAM, 5.1 with 32 per layer (A, CPU)
Qwen-Image-2.1: text to picture, non-commercial licence	`qwen-image-2.1`	33 GB	12 / 18 GB	Vulkan	one 768x512 picture in 2 min 40 s (8 Zen 4 cores, CPU)

Model	`--model`	Download	RAM	GPU	Measured
DeepSeek V4 Flash REAP 150B: 132 of the 256 experts	`deepseek-v4-flash-reap`	85 GB	16 / 32 GB	CUDA, Vulkan
DeepSeek V4 Flash (284B): tools	`deepseek-v4-flash`	167 GB	16 / 32 GB	CUDA, Vulkan	0.93 tok/s with 32 GB (Ryzen 7 5800X), 1.24 with 63 GB (Ryzen 9 5950X), CPU only; 1.5-1.6 with CUDA (RTX 5080, 32 GB, two NVMe)
MiMo-V2.6 Flash (309B): vision and tools	`mimo-v2.6-flash`	172 GB	32 / 52 GB	Vulkan	2.34-3.37 tok/s (A, CPU)
Qwen3.8-Flash-Next (125B + 51B n-gram): vision and tools	`qwen38-flash-next`	186 GB	24 / 32 GB	CUDA, Vulkan (integrated too)	1.91-2.56 tok/s with 32-96 experts per layer; 3.99 with the optional int4 experts (A, CPU)
GLM-5.2 (744B): the reference model, with the MTP head	`glm-5.2`	429 GB	16 / 24 GB	CUDA, Vulkan	0.05-0.1 tok/s cold on a 25 GB laptop; 1.83 on a 128 GB Ryzen AI Max+ 395; 9.0-9.2 on 6x RTX 5090
GLM-5.3 (744B): the same engine, no MTP head	`glm-5.3`	419 GB	16 / 24 GB	CUDA, Vulkan
Inkling (975B): int4 experts, bf16 dense weights	`inkling`	514 GB	120 / 128 GB as downloaded; 25 GB after a dense conversion	CUDA, Vulkan	0.25 tok/s (Ryzen 9 7900, 187 GB, RTX A6000)
MiMo-V2.6 Pro (1.02T): vision and tools	`mimo-v2.6-pro`	564 GB	54 / 64 GB	Vulkan	0.66-0.79 tok/s (A, CPU)
Kimi K3 (2.8T): the largest	`kimi-k3`	1.56 TB	32 / 64 GB	CUDA, Vulkan	about 9.4 s per token, experts read at 6.3 GB/s

Model	Download, then on disk	RAM	GPU	Measured
OLMoE (7B): small, to learn the tools on	14 GB, 7 GB after conversion to int8	8 GB	Vulkan	22-23 tok/s (A, CPU)
Qwen3.8-27B (dense): text and images	56 GB, 51 GB after conversion	20 GB in int4, 30 GB in int8	Vulkan	3.45 tok/s in int4, 2.1 in int8 (16-thread CPU server)
GLM-5.3-Flash (321B): vision and tools	328 GB, converted shard by shard to 195 GB	25 GB	CUDA, Vulkan	about 20 s per token warm, 44 s cold (6 cores, 25 GB, ordinary disk)
DeepSeek V4.1 Flash (552B): vision and tools, no conversion but a one-off preparation	510 GB	about 18 GB plus the expert cache (24.8 GB peak with 8 per layer)	Vulkan	0.21-0.24 tok/s (16-thread CPU server holding 68% of the experts)

JustVugg/colibri

Get started in one step

If something goes wrong

Let your AI assistant set it up

Or by hand

What colibri is, and why

Which model for my machine

GPUs

No GPU needed

Vulkan: any GPU

CUDA: NVIDIA cards

Apple Silicon

System One: a decision with a probability

The dashboard

Use it from other apps

How it works

Benchmarks

Install by hand

Research, and how to help

Documentation

Repo layout

Supporting the project

Why "colibrì"

Acknowledgements

License

Model	Download, then on disk	RAM	GPU	Measured
Laya (Convai Innovations), English	0.85 GB	1.7 GB	CPU	219 ms for one question, 882 ms for three (B)
GLiNER2.5-Decide (fastino), English	1.95 GB	1.9 GB	CPU	294 ms for one question, 897 ms for three (B, under load)
Clef (Cloudflare): Qwen3.8-27B with a decision head, it also chats	55 GB, 52 GB after conversion	19 GB in int4 to 55 GB in f16	CPU	20.4 s per request in int8 (A)

	CPU	Vulkan, expert tier	Vulkan, tier and dense chain
Qwen3.6-35B-A3B, decode	6.0 tok/s	8.0 tok/s	9.9 tok/s
Qwen3.6-35B-A3B, a 512-token prompt	35.7 s	12.2 s	9.5 s
Qwen3.8-Flash-Next (int4 experts), decode	3.5 tok/s	3.8 tok/s	3.2 tok/s
Qwen3.8-Flash-Next, a 512-token prompt	43.6 s	38.7 s	30.1 s
OLMoE, decode (warm)	23.1 tok/s	12.8 tok/s	17.3 tok/s

hypothesis	evidence so far	experiment still needed
Routing history can place experts better than plain LRU	learned pins improve repeated workloads, but can overfit a prompt	held-out, cross-session A/Bs across coding, chat, multilingual, and long-context workloads
Multiple SSDs can turn independent bandwidth into decode speed	two independent NVMe drives measured +37.5% decode; a slower third drive was neutral after weighted striping (measurements)	reproduce across drive speeds, controller layouts, and cache states
A hardware-aware planner can approach each machine's best configuration automatically	RAM/VRAM budgets and several backends are detected today, and the setup picks the build	compare the generated plan with a controlled parameter sweep across laptops, workstations, NUMA hosts, and multi-GPU systems
Lossless or quality-bounded representations can reduce weight movement enough to matter	format and quantization ablations exist, with correctness/quality gates	reproduce quality, bytes moved, latency, and cost per useful token together, not compression ratio alone
Routing-aware speculation can pay before near-full residency	MTP and grammar drafts work, but MTP has also measured a 32% loss around 85% expert hit	map the break-even surface across acceptance, expert hit rate, batch union, and draft depth
CPU/GPU overlap can hide transfer and synchronization rather than merely move the bottleneck	CUDA, Metal and Vulkan wins exist, but fast CPUs, integrated GPUs and low residency can erase them	per-stage profiles and one-variable A/Bs across PCIe, unified-memory, and full-resident machines, and the first discrete-GPU numbers for the Vulkan tier and chain

topic	doc
The one-step setup and the manual install, every platform	quickstart.md
Setting up through an AI assistant, and the MCP server	AI_SETUP.md, MCP_SERVER.md
The API: OpenAI, Anthropic, tools, KV slots, dashboard	api.md
System One and the decision models	systemone.md, laya.md, gliner_decide.md, clef.md
Vulkan: the expert tier, the dense chain, cards without Resizable BAR	vulkan.md
CUDA, and Qwen3.6's CUDA tier	cuda.md, qwen36-cuda-tier.md
Apple Silicon, Windows	metal.md, windows.md
Tuning, the learning cache, prefetch, speculation	tuning.md
Several SSDs, several machines	multidisk.md, cluster.md
Benchmarks and how to measure	benchmarks.md, benchmarking.md
Every option and environment variable	SETTINGS.md, ENVIRONMENT.md
Grammar-forced drafts, and the experimental embedding ABIs	grammar-draft.md, segment-runtime.md, edge-runtime.md