Chat · ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF · llama.cpp · GB10 · 80 GB VRAM
published by @dh7 on
m64 m64_5deb4065b1cfQwen3.8-Flash-Next IQ3_S on llama.cpp v1.0.0 (GB10). A local OpenAI-compatible chat endpoint for Qwen3.8-Flash-Next (Qwen's 125B-A6B MoE, ISTA-DASLab's GSQ-RCO IQ3_S GGUF) on stock llama.cpp v1.0.0 built with CUDA 13 for sm_121, 256K context. Reads images, streams, tool calls. Needs no network.
Lineage
- parent
- none declared
- derived from
- m64_ef24b3f62fa7, 50% shared files
- children
- none
Files
6 files- human/
- index.html9.6 KB
- AGENT.md3.7 KB
- capsule.toml1.4 KB
- Dockerfile2.6 KB
- proxy.py6.0 KB
- start.shexec2.0 KB
AGENT.md
qwen38-flash-next-llamacpp-gb10
An OpenAI-compatible chat endpoint for Qwen3.8-Flash-Next (Qwen's 125B-A6B mixture-of-experts
model; ISTA-DASLab's GSQ-RCO IQ3_S GGUF) on stock llama.cpp v1.0.0 (llama-server, CUDA 13, built
for the GB10's sm_121), with a 262144-token context (the model's full window). It only provides the model: run opencode (or any
OpenAI-compatible client) on your own machine and point it here.
The model runs locally; the container has no network access (unless the user served it
with --allow-external-network or --allow-internal-network, which it does not need).
On the GB10 the GPU and the CPU share one memory: the weights (shard 1, 55 GB) are memory-mapped and
offloaded whole, and the 29 GB PLE n-gram table (shard 2) is read lazily, one row per token, through the
page cache. Pictures are on (the BF16 vision tower), at least 1024 tokens each (--image-min-tokens 1024, which llama.cpp
recommends for Qwen-VL: below that small pictures lose fine detail). No speculative decoding: the GGUF has no MTP layer.
Timings and concurrency
Measured on a DGX Spark (GB10), 2026-11-15:
- Ready about 40 s after start with the weights in the page cache (the first start after a reboot reads ~84 GB from disk first).
- Decode ~30 tokens/s for one stream at short and 10K-token prompts, ~15 tokens/s after a 150K-token one; prompt ~1,000 tokens/s on long prompts (150K tokens read in 157 s).
- About 65 GB in use with a 150K-token prompt (MemAvailable stayed above 54 GB of 119 GB).
- airbench.ai with opencode at 64K context and before the picture setting, 2026-11-15: Full benchmark 40/49 (text 9/9, vision 11/19, mail 5/6, store 4/4, code 11/11).
- One request at a time (
-np 1); further requests wait.
Routes
All paths are relative to the capsule's URL (e.g. http://<tailnet-ip>:<port>), written
here as {{BASE_URL}}.
| Method | Path | What |
|---|---|---|
| POST | /v1/chat/completions |
OpenAI chat completions, streaming or not |
| GET | /v1/models |
one model, qwen3.8-flash-next-iq3_s-llamacpp |
| GET | /health |
200 when llama-server is serving, else 503 |
| GET | /status |
phase (ready when usable) |
| GET | /agent |
this document plus live status |
| GET | /human/ |
landing page: opencode setup and a quick-test chat |
POST /v1/chat/completions
Body: application/json, the OpenAI chat format. Accepted fields: messages, tools,
tool_choice, response_format, max_tokens, stream, seed (plus model, user, n: 1,
stream_options); anything else, temperature and top_p included, is refused with 400: the
sampling is the model's own (temperature 1.0, top-k 20, top-p 0.95). Only model is replaced. Tool
calls follow the OpenAI format. Messages may carry images (image_url parts with data: URLs). The
reply may include reasoning_content (the model's thinking) before content. stream: true returns
Server-Sent Events ending with data: [DONE].
curl -s {{BASE_URL}}/v1/chat/completions -H 'Content-Type: application/json' \
-d '{"messages":[{"role":"user","content":"Say hi"}],"max_tokens":200}'
curl -N {{BASE_URL}}/v1/chat/completions -H 'Content-Type: application/json' \
-d '{"messages":[{"role":"user","content":"Count to 5"}],"stream":true}'
opencode on your machine
Once: opencode auth login {{BASE_URL}}, then run opencode in any folder.
Or on a machine with m64: m64 qwen38-flash-next-llamacpp-gb10 --ui opencode.
Errors
- 503: still loading (or llama-server stopped); retry.
- 502: llama-server did not answer the request.
Access
On the tailnet (the default bind) there is no token: anyone on the tailnet can use the
endpoint. With a token (--host 0.0.0.0 etc.), send Authorization: Bearer <token>.
capsule.toml
schema = "m64/capsule@1"description = "Qwen3.8-Flash-Next IQ3_S on llama.cpp v1.0.0 (GB10). A local OpenAI-compatible chat endpoint for Qwen3.8-Flash-Next (Qwen's 125B-A6B MoE, ISTA-DASLab's GSQ-RCO IQ3_S GGUF) on stock llama.cpp v1.0.0 built with CUDA 13 for sm_121, 256K context. Reads images, streams, tool calls. Needs no network."kinds = ["chat"]authors = ["dh7"]license = "mit"[hardware]platform = "linux-aarch64"gpu = "NVIDIA GB10"gpu_count = 1# Unified memory: shard 1 (55 GB) offloaded whole, the PLE n-gram table (shard 2, 29 GB) read lazily through# the page cache, the picture reader (0.9 GB) and a 256K q8_0 KV cache (12 of the 48 layers have one).# Measured on gx10: ~65 GB in use with a 150K-token prompt.vram_gb = 80driver = ">=580.65"disk_gb = 90[runtime]sandbox = "container"# No image: the Dockerfile builds llama.cpp v1.0.0 (sm_121) in NVIDIA's PyTorch container and copies the server# onto ubuntu:24.04, both pinned by digest.gpu = trueshm_gb = 2internal_port = 9000health = "/health"expected_load_seconds = 900model_name = "qwen3.8-flash-next-iq3_s-llamacpp"user = "65534:65534"# ISTA-DASLab's GSQ-RCO shards and the picture reader, at the same revision as the Strata capsule.[[artifacts]]type = "hf"repo = "ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF"revision = "ed59f92082b1e93c0e96d60a8b11aab089b52f09"include = ["README.md", "IQ3_S/*.gguf", "mmproj-Qwen3.8-Flash-Next-BF16.gguf"]mount = "/weights/qwen38fn"