motif64

m64_ef24b3f62fa7 / AGENT.md

72 lines, 3.3 KBraw

qwen38-flash-next-llamacpp-gb10

An OpenAI-compatible chat endpoint for Qwen3.8-Flash-Next (Qwen's 125B-A6B mixture-of-experts model; ISTA-DASLab's GSQ-RCO IQ3_S GGUF) on stock llama.cpp v1.0.0 (llama-server, CUDA 13, built for the GB10's sm_121), with a 65536-token context. It only provides the model: run opencode (or any OpenAI-compatible client) on your own machine and point it here.

The model runs locally; the container has no network access (unless the user served it with --allow-external-network or --allow-internal-network, which it does not need).

On the GB10 the GPU and the CPU share one memory: the weights (shard 1, 55 GB) are memory-mapped and offloaded whole, and the 29 GB PLE n-gram table (shard 2) is read lazily, one row per token, through the page cache. Pictures are on (the BF16 vision tower). No speculative decoding: the GGUF has no MTP layer.

Timings and concurrency

Measured on a DGX Spark (GB10), 2026-11-15:

  • Ready about 95 s after start with the weights in the page cache (the first start after a reboot reads ~84 GB from disk first).
  • Decode ~30 tokens/s for one stream (31 on a short prompt, 30 after a 10K-token prompt); prompt ~120 tokens/s on a short prompt, ~1,090 tokens/s on a 10K-token one.
  • About 61 GB in use under load (MemAvailable stayed above 58 GB of 119 GB).
  • One request at a time (-np 1); further requests wait.

Routes

All paths are relative to the capsule's URL (e.g. http://<tailnet-ip>:<port>), written here as {{BASE_URL}}.

Method Path What
POST /v1/chat/completions OpenAI chat completions, streaming or not
GET /v1/models one model, qwen3.8-flash-next-iq3_s-llamacpp
GET /health 200 when llama-server is serving, else 503
GET /status phase (ready when usable)
GET /agent this document plus live status
GET /human/ landing page: opencode setup and a quick-test chat

POST /v1/chat/completions

Body: application/json, the OpenAI chat format. Accepted fields: messages, tools, tool_choice, response_format, max_tokens, stream, seed (plus model, user, n: 1, stream_options); anything else, temperature and top_p included, is refused with 400: the sampling is the model's own (temperature 1.0, top-k 20, top-p 0.95). Only model is replaced. Tool calls follow the OpenAI format. Messages may carry images (image_url parts with data: URLs). The reply may include reasoning_content (the model's thinking) before content. stream: true returns Server-Sent Events ending with data: [DONE].

curl -s {{BASE_URL}}/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"messages":[{"role":"user","content":"Say hi"}],"max_tokens":200}'

curl -N {{BASE_URL}}/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"messages":[{"role":"user","content":"Count to 5"}],"stream":true}'

opencode on your machine

Once: opencode auth login {{BASE_URL}}, then run opencode in any folder.

Or on a machine with m64: m64 qwen38-flash-next-llamacpp-gb10 --ui opencode.

Errors

  • 503: still loading (or llama-server stopped); retry.
  • 502: llama-server did not answer the request.

Access

On the tailnet (the default bind) there is no token: anyone on the tailnet can use the endpoint. With a token (--host 0.0.0.0 etc.), send Authorization: Bearer <token>.