# qwen38-flash-next-llamacpp-gb10 An OpenAI-compatible chat endpoint for **Qwen3.8-Flash-Next** (Qwen's 125B-A6B mixture-of-experts model; ISTA-DASLab's GSQ-RCO IQ3_S GGUF) on stock **llama.cpp v1.0.0** (`llama-server`, CUDA 13, built for the GB10's sm_121), with a 262144-token context (the model's full window). It only provides the model: run opencode (or any OpenAI-compatible client) on your own machine and point it here. The model runs locally; the container has no network access (unless the user served it with `--allow-external-network` or `--allow-internal-network`, which it does not need). On the GB10 the GPU and the CPU share one memory: the weights (shard 1, 55 GB) are memory-mapped and offloaded whole, and the 29 GB PLE n-gram table (shard 2) is read lazily, one row per token, through the page cache. Pictures are on (the BF16 vision tower), at least 1024 tokens each (`--image-min-tokens 1024`, which llama.cpp recommends for Qwen-VL: below that small pictures lose fine detail). No speculative decoding: the GGUF has no MTP layer. ## Timings and concurrency Measured on a DGX Spark (GB10), 2026-11-15: - Ready about 40 s after start with the weights in the page cache (the first start after a reboot reads ~84 GB from disk first). - Decode ~30 tokens/s for one stream at short and 10K-token prompts, ~15 tokens/s after a 150K-token one; prompt ~1,000 tokens/s on long prompts (150K tokens read in 157 s). - About 65 GB in use with a 150K-token prompt (MemAvailable stayed above 54 GB of 119 GB). - airbench.ai Full benchmark, this configuration (256K, `--image-min-tokens 1024`), 2026-11-15/16: pi 49/49 in 2 h 4 min; opencode 48/49 in 79 min (text 9/9, vision 18/19, mail 6/6, store 4/4, code 11/11). The earlier 64K version without the picture setting scored 40/49 with opencode (vision 11/19), and pi stopped when its 64K window filled. - One request at a time (`-np 1`); further requests wait. ## Routes All paths are relative to the capsule's URL (e.g. `http://:`), written here as `{{BASE_URL}}`. | Method | Path | What | | --- | --- | --- | | POST | `/v1/chat/completions` | OpenAI chat completions, streaming or not | | GET | `/v1/models` | one model, `qwen3.8-flash-next-iq3_s-llamacpp` | | GET | `/health` | 200 when llama-server is serving, else 503 | | GET | `/status` | phase (`ready` when usable) | | GET | `/agent` | this document plus live status | | GET | `/human/` | landing page: opencode setup and a quick-test chat | ### POST /v1/chat/completions Body: `application/json`, the OpenAI chat format. Accepted fields: `messages`, `tools`, `tool_choice`, `response_format`, `max_tokens`, `stream`, `seed` (plus `model`, `user`, `n: 1`, `stream_options`); anything else, `temperature` and `top_p` included, is refused with 400: the sampling is the model's own (temperature 1.0, top-k 20, top-p 0.95). Only `model` is replaced. Tool calls follow the OpenAI format. Messages may carry images (`image_url` parts with `data:` URLs). The reply may include `reasoning_content` (the model's thinking) before `content`. `stream: true` returns Server-Sent Events ending with `data: [DONE]`. ``` curl -s {{BASE_URL}}/v1/chat/completions -H 'Content-Type: application/json' \ -d '{"messages":[{"role":"user","content":"Say hi"}],"max_tokens":200}' curl -N {{BASE_URL}}/v1/chat/completions -H 'Content-Type: application/json' \ -d '{"messages":[{"role":"user","content":"Count to 5"}],"stream":true}' ``` ## opencode on your machine Once: `opencode auth login {{BASE_URL}}`, then run `opencode` in any folder. Or on a machine with m64: `m64 qwen38-flash-next-llamacpp-gb10 --ui opencode`. ## Errors - 503: still loading (or llama-server stopped); retry. - 502: llama-server did not answer the request. ## Access On the tailnet (the default bind) there is no token: anyone on the tailnet can use the endpoint. With a token (`--host 0.0.0.0` etc.), send `Authorization: Bearer `.