motif64

Chat · ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF · llama.cpp · GB10 · 72 GB VRAM

published by @dh7 on

m64 m64_ef24b3f62fa7
Download .tar.gz

Qwen3.8-Flash-Next IQ3_S on llama.cpp v1.0.0 (GB10). A local OpenAI-compatible chat endpoint for Qwen3.8-Flash-Next (Qwen's 125B-A6B MoE, ISTA-DASLab's GSQ-RCO IQ3_S GGUF) on stock llama.cpp v1.0.0 built with CUDA 13 for sm_121, 64K context. Reads images, streams, tool calls. Needs no network.

Lineage

parent
none declared
derived from
no published capsule shares half its files
children
none

Files

6 files

AGENT.md

qwen38-flash-next-llamacpp-gb10

An OpenAI-compatible chat endpoint for Qwen3.8-Flash-Next (Qwen's 125B-A6B mixture-of-experts model; ISTA-DASLab's GSQ-RCO IQ3_S GGUF) on stock llama.cpp v1.0.0 (llama-server, CUDA 13, built for the GB10's sm_121), with a 65536-token context. It only provides the model: run opencode (or any OpenAI-compatible client) on your own machine and point it here.

The model runs locally; the container has no network access (unless the user served it with --allow-external-network or --allow-internal-network, which it does not need).

On the GB10 the GPU and the CPU share one memory: the weights (shard 1, 55 GB) are memory-mapped and offloaded whole, and the 29 GB PLE n-gram table (shard 2) is read lazily, one row per token, through the page cache. Pictures are on (the BF16 vision tower). No speculative decoding: the GGUF has no MTP layer.

Timings and concurrency

Measured on a DGX Spark (GB10), 2026-11-15:

  • Ready about 95 s after start with the weights in the page cache (the first start after a reboot reads ~84 GB from disk first).
  • Decode ~30 tokens/s for one stream (31 on a short prompt, 30 after a 10K-token prompt); prompt ~120 tokens/s on a short prompt, ~1,090 tokens/s on a 10K-token one.
  • About 61 GB in use under load (MemAvailable stayed above 58 GB of 119 GB).
  • One request at a time (-np 1); further requests wait.

Routes

All paths are relative to the capsule's URL (e.g. http://<tailnet-ip>:<port>), written here as {{BASE_URL}}.

Method Path What
POST /v1/chat/completions OpenAI chat completions, streaming or not
GET /v1/models one model, qwen3.8-flash-next-iq3_s-llamacpp
GET /health 200 when llama-server is serving, else 503
GET /status phase (ready when usable)
GET /agent this document plus live status
GET /human/ landing page: opencode setup and a quick-test chat

POST /v1/chat/completions

Body: application/json, the OpenAI chat format. Accepted fields: messages, tools, tool_choice, response_format, max_tokens, stream, seed (plus model, user, n: 1, stream_options); anything else, temperature and top_p included, is refused with 400: the sampling is the model's own (temperature 1.0, top-k 20, top-p 0.95). Only model is replaced. Tool calls follow the OpenAI format. Messages may carry images (image_url parts with data: URLs). The reply may include reasoning_content (the model's thinking) before content. stream: true returns Server-Sent Events ending with data: [DONE].

curl -s {{BASE_URL}}/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"messages":[{"role":"user","content":"Say hi"}],"max_tokens":200}'

curl -N {{BASE_URL}}/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"messages":[{"role":"user","content":"Count to 5"}],"stream":true}'

opencode on your machine

Once: opencode auth login {{BASE_URL}}, then run opencode in any folder.

Or on a machine with m64: m64 qwen38-flash-next-llamacpp-gb10 --ui opencode.

Errors

  • 503: still loading (or llama-server stopped); retry.
  • 502: llama-server did not answer the request.

Access

On the tailnet (the default bind) there is no token: anyone on the tailnet can use the endpoint. With a token (--host 0.0.0.0 etc.), send Authorization: Bearer <token>.

capsule.toml

schema = "m64/capsule@1"description = "Qwen3.8-Flash-Next IQ3_S on llama.cpp v1.0.0 (GB10). A local OpenAI-compatible chat endpoint for Qwen3.8-Flash-Next (Qwen's 125B-A6B MoE, ISTA-DASLab's GSQ-RCO IQ3_S GGUF) on stock llama.cpp v1.0.0 built with CUDA 13 for sm_121, 64K context. Reads images, streams, tool calls. Needs no network."kinds = ["chat"]authors = ["dh7"]license = "mit"[hardware]platform = "linux-aarch64"gpu = "NVIDIA GB10"gpu_count = 1# Unified memory: shard 1 (55 GB) offloaded whole, the PLE n-gram table (shard 2, 29 GB) read lazily through# the page cache, the picture reader (0.9 GB) and a 64K q8_0 KV cache.vram_gb = 72driver = ">=580.65"disk_gb = 90[runtime]sandbox = "container"# No image: the Dockerfile builds llama.cpp v1.0.0 (sm_121) in NVIDIA's PyTorch container and copies the server# onto ubuntu:24.04, both pinned by digest.gpu = trueshm_gb = 2internal_port = 9000health = "/health"expected_load_seconds = 900model_name = "qwen3.8-flash-next-iq3_s-llamacpp"user = "65534:65534"# ISTA-DASLab's GSQ-RCO shards and the picture reader, at the same revision as the Strata capsule.[[artifacts]]type = "hf"repo = "ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF"revision = "ed59f92082b1e93c0e96d60a8b11aab089b52f09"include = ["README.md", "IQ3_S/*.gguf", "mmproj-Qwen3.8-Flash-Next-BF16.gguf"]mount = "/weights/qwen38fn"