m64_f51e8491f6d5 / AGENT.md
qwen38-flash-next-llamacpp-gb10
An OpenAI-compatible chat endpoint for Qwen3.8-Flash-Next (Qwen's 125B-A6B mixture-of-experts
model; ISTA-DASLab's GSQ-RCO IQ3_S GGUF) on stock llama.cpp v1.0.0 (llama-server, CUDA 13, built
for the GB10's sm_121), with a 262144-token context (the model's full window). It only provides the model: run opencode (or any
OpenAI-compatible client) on your own machine and point it here.
The model runs locally; the container has no network access (unless the user served it
with --allow-external-network or --allow-internal-network, which it does not need).
On the GB10 the GPU and the CPU share one memory: the weights (shard 1, 55 GB) are memory-mapped and
offloaded whole, and the 29 GB PLE n-gram table (shard 2) is read lazily, one row per token, through the
page cache. Pictures are on (the BF16 vision tower), at least 1024 tokens each (--image-min-tokens 1024, which llama.cpp
recommends for Qwen-VL: below that small pictures lose fine detail). No speculative decoding: the GGUF has no MTP layer.
Timings and concurrency
Measured on a DGX Spark (GB10), 2026-11-15:
- Ready about 40 s after start with the weights in the page cache (the first start after a reboot reads ~84 GB from disk first).
- Decode ~30 tokens/s for one stream at short and 10K-token prompts, ~15 tokens/s after a 150K-token one; prompt ~1,000 tokens/s on long prompts (150K tokens read in 157 s).
- About 65 GB in use with a 150K-token prompt (MemAvailable stayed above 54 GB of 119 GB).
- airbench.ai Full benchmark, this configuration (256K,
--image-min-tokens 1024), 2026-11-15/16: pi 49/49 in 2 h 4 min; opencode 48/49 in 79 min (text 9/9, vision 18/19, mail 6/6, store 4/4, code 11/11). The earlier 64K version without the picture setting scored 40/49 with opencode (vision 11/19), and pi stopped when its 64K window filled. - One request at a time (
-np 1); further requests wait.
Routes
All paths are relative to the capsule's URL (e.g. http://<tailnet-ip>:<port>), written
here as {{BASE_URL}}.
| Method | Path | What |
|---|---|---|
| POST | /v1/chat/completions |
OpenAI chat completions, streaming or not |
| GET | /v1/models |
one model, qwen3.8-flash-next-iq3_s-llamacpp |
| GET | /health |
200 when llama-server is serving, else 503 |
| GET | /status |
phase (ready when usable) |
| GET | /agent |
this document plus live status |
| GET | /human/ |
landing page: opencode setup and a quick-test chat |
POST /v1/chat/completions
Body: application/json, the OpenAI chat format. Accepted fields: messages, tools,
tool_choice, response_format, max_tokens, stream, seed (plus model, user, n: 1,
stream_options); anything else, temperature and top_p included, is refused with 400: the
sampling is the model's own (temperature 1.0, top-k 20, top-p 0.95). Only model is replaced. Tool
calls follow the OpenAI format. Messages may carry images (image_url parts with data: URLs). The
reply may include reasoning_content (the model's thinking) before content. stream: true returns
Server-Sent Events ending with data: [DONE].
curl -s {{BASE_URL}}/v1/chat/completions -H 'Content-Type: application/json' \
-d '{"messages":[{"role":"user","content":"Say hi"}],"max_tokens":200}'
curl -N {{BASE_URL}}/v1/chat/completions -H 'Content-Type: application/json' \
-d '{"messages":[{"role":"user","content":"Count to 5"}],"stream":true}'
opencode on your machine
Once: opencode auth login {{BASE_URL}}, then run opencode in any folder.
Or on a machine with m64: m64 qwen38-flash-next-llamacpp-gb10 --ui opencode.
Errors
- 503: still loading (or llama-server stopped); retry.
- 502: llama-server did not answer the request.
Access
On the tailnet (the default bind) there is no token: anyone on the tailnet can use the
endpoint. With a token (--host 0.0.0.0 etc.), send Authorization: Bearer <token>.