m64_5deb4065b1cf / AGENT.md
qwen38-flash-next-llamacpp-gb10
An OpenAI-compatible chat endpoint for Qwen3.8-Flash-Next (Qwen's 125B-A6B mixture-of-experts
model; ISTA-DASLab's GSQ-RCO IQ3_S GGUF) on stock llama.cpp v1.0.0 (llama-server, CUDA 13, built
for the GB10's sm_121), with a 262144-token context (the model's full window). It only provides the model: run opencode (or any
OpenAI-compatible client) on your own machine and point it here.
The model runs locally; the container has no network access (unless the user served it
with --allow-external-network or --allow-internal-network, which it does not need).
On the GB10 the GPU and the CPU share one memory: the weights (shard 1, 55 GB) are memory-mapped and
offloaded whole, and the 29 GB PLE n-gram table (shard 2) is read lazily, one row per token, through the
page cache. Pictures are on (the BF16 vision tower), at least 1024 tokens each (--image-min-tokens 1024, which llama.cpp
recommends for Qwen-VL: below that small pictures lose fine detail). No speculative decoding: the GGUF has no MTP layer.
Timings and concurrency
Measured on a DGX Spark (GB10), 2026-11-15:
- Ready about 40 s after start with the weights in the page cache (the first start after a reboot reads ~84 GB from disk first).
- Decode ~30 tokens/s for one stream at short and 10K-token prompts, ~15 tokens/s after a 150K-token one; prompt ~1,000 tokens/s on long prompts (150K tokens read in 157 s).
- About 65 GB in use with a 150K-token prompt (MemAvailable stayed above 54 GB of 119 GB).
- airbench.ai with opencode at 64K context and before the picture setting, 2026-11-15: Full benchmark 40/49 (text 9/9, vision 11/19, mail 5/6, store 4/4, code 11/11).
- One request at a time (
-np 1); further requests wait.
Routes
All paths are relative to the capsule's URL (e.g. http://<tailnet-ip>:<port>), written
here as {{BASE_URL}}.
| Method | Path | What |
|---|---|---|
| POST | /v1/chat/completions |
OpenAI chat completions, streaming or not |
| GET | /v1/models |
one model, qwen3.8-flash-next-iq3_s-llamacpp |
| GET | /health |
200 when llama-server is serving, else 503 |
| GET | /status |
phase (ready when usable) |
| GET | /agent |
this document plus live status |
| GET | /human/ |
landing page: opencode setup and a quick-test chat |
POST /v1/chat/completions
Body: application/json, the OpenAI chat format. Accepted fields: messages, tools,
tool_choice, response_format, max_tokens, stream, seed (plus model, user, n: 1,
stream_options); anything else, temperature and top_p included, is refused with 400: the
sampling is the model's own (temperature 1.0, top-k 20, top-p 0.95). Only model is replaced. Tool
calls follow the OpenAI format. Messages may carry images (image_url parts with data: URLs). The
reply may include reasoning_content (the model's thinking) before content. stream: true returns
Server-Sent Events ending with data: [DONE].
curl -s {{BASE_URL}}/v1/chat/completions -H 'Content-Type: application/json' \
-d '{"messages":[{"role":"user","content":"Say hi"}],"max_tokens":200}'
curl -N {{BASE_URL}}/v1/chat/completions -H 'Content-Type: application/json' \
-d '{"messages":[{"role":"user","content":"Count to 5"}],"stream":true}'
opencode on your machine
Once: opencode auth login {{BASE_URL}}, then run opencode in any folder.
Or on a machine with m64: m64 qwen38-flash-next-llamacpp-gb10 --ui opencode.
Errors
- 503: still loading (or llama-server stopped); retry.
- 502: llama-server did not answer the request.
Access
On the tailnet (the default bind) there is no token: anyone on the tailnet can use the
endpoint. With a token (--host 0.0.0.0 etc.), send Authorization: Bearer <token>.