motif64
Log in

Chat · Qwen3 4B Q4_K_M · llama.cpp · NVIDIA · 6 GB VRAM

published by @dh7 on

m64 m64_53ac9668fd7c
Download .tar.gz

A small local model for opencode: Qwen3 4B (Q4_K_M GGUF) on llama.cpp's server, an OpenAI-compatible chat endpoint with tool calls, for opencode running on your own machine. About 6 GB of VRAM, 32k context. Needs no network.

Lineage

parent
none declared
derived from
no published capsule shares half its files
children
none

Files

5 files

AGENT.md

model-for-opencode

An OpenAI-compatible chat endpoint for Qwen3 4B (Q4_K_M GGUF), served by llama.cpp's llama-server on a local GPU. It only provides the model: run opencode (or any OpenAI-compatible client) on your own machine and point it here. Your files and tools stay on your machine; only the model calls come here. No network, no key.

Base URL: {{BASE_URL}}

Routes

Method Path What
POST /v1/chat/completions chat completions, streaming, tool calls
GET /v1/models one model: qwen3-4b
GET /.well-known/opencode the provider config opencode auth login reads
GET /status phase (ready when usable)
GET /human/ setup page for opencode 
curl -s {{BASE_URL}}/v1/chat/completions {{AUTH}} \
  -H 'Content-Type: application/json' \
  -d '{"messages":[{"role":"user","content":"Say hi"}],"max_tokens":50}'

Accepted fields: messages (required), tools, tool_choice, response_format, max_tokens, stream, seed. Others, like temperature, are refused with a 400. The model is always qwen3-4b.

opencode

Once, on your machine:

opencode auth login {{BASE_URL}}

opencode then loads this endpoint's settings from /.well-known/opencode on every start. Or write the provider yourself, in opencode.json:

{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "qwen-local": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "Qwen3 4B (capsule)",
      "options": { "baseURL": "{{BASE_URL}}/v1" },
      "models": { "qwen3-4b": { "name": "Qwen3 4B", "tool_call": true } }
    }
  },
  "model": "qwen-local/qwen3-4b"
}

When the capsule has a token (served with --host 0.0.0.0), add "apiKey": "<token>" to options.

Timings and limits

  • Loading: about 20 s once the weights are on disk. The first serve downloads 2.5 GB.
  • Context: 32768 tokens per request, shared by the prompt and the answer.
  • One request at a time; others wait their turn.
  • Qwen3 thinks before answering (<think>…</think>); put /no_think in a message to skip it.

capsule.toml

# model-for-opencode: a small local model on an OpenAI-compatible endpoint,# for an opencode (or any client) running outside the capsule.## What this example adds to hello-world:#   - a GPU: [hardware] says what it needs, [runtime] gpu = true;#   - [[artifacts]]: the weights, downloaded and hashed by m64, mounted read-only.#     Weights never go in the folder (5 MB per file, 10 MB in total);#   - a ready-made engine image (llama.cpp's server), pinned by digest.# It needs no network at run time and no secret: everything is local.schema = "m64/capsule@1"description = "A small local model for opencode: Qwen3 4B (Q4_K_M GGUF) on llama.cpp's server, an OpenAI-compatible chat endpoint with tool calls, for opencode running on your own machine. About 6 GB of VRAM, 32k context. Needs no network."kinds = ["chat"]authors = ["motif42"]license = "mit"[hardware]platform = "linux-x86_64"# Matched as a substring of the GPU's name, so any NVIDIA card passes; a# mismatch only warns. The image below also has an arm64 build: on a GB10,# set platform = "linux-aarch64" and gpu = "NVIDIA GB10".gpu = "NVIDIA"gpu_count = 1# 2.5 GB of weights + 2.4 GB of 8-bit KV cache at 32k tokens + CUDA's overhead.vram_gb = 6# The image is CUDA 12.8.driver = ">=570"disk_gb = 6[runtime]sandbox = "container"# ghcr.io/ggml-org/llama.cpp:server-cuda, build b12030 (llama.cpp commit# a91eab67d8389d979d29fdf6633b2edd0718eb4b), multi-arch index digest. Tags move;# a digest is exactly one image.image = "ghcr.io/ggml-org/llama.cpp@sha256:75037a842f2ec1d1b5317ce64078bbb4a2a134238e91a755c10ce31d8f90550a"gpu = trueinternal_port = 9000# llama-server answers 503 while it loads the model, 200 once it is ready.health = "/health"expected_load_seconds = 60# The front rewrites every request's "model" to this; start.sh gives# llama-server the same alias. It is also the name to use in opencode.model_name = "qwen3-4b"user = "65534:65534"# The model: Qwen's own GGUF build, at a fixed commit. m64 downloads only the# files in `include`, checks them, and mounts them read-only at `mount`. Pickle# formats (.bin .pt .pth .ckpt .pkl) are refused: GGUF and safetensors only.[[artifacts]]type = "hf"repo = "Qwen/Qwen3-4B-GGUF"revision = "bc640142c66e1fdd12af0bd68f40445458f3869b"include = ["LICENSE", "Qwen3-4B-Q4_K_M.gguf"]mount = "/weights/qwen3-4b"