llm-benchmark-harness¶

🇬🇧 English · 🇩🇪 Deutsch

llm-touchstone is a thin CLI that benchmarks a local LLM through an OpenAI-compatible endpoint — recording TTFT, prefill-tok/s and decode-tok/s with their distribution while a decoupled host sampler tracks macOS memory pressure and thermal throttling, then merges both into one Markdown table (+ CSV). It also evaluates answer quality (deterministic capture + optional LLM-as-judge) and can drive the whole loop from a local web control-center. Same code on an M1 (LM Studio) and an M5 (mlx-lm) — you only swap the config.

Features¶

Distribution, not the mean — TTFT as P50/P95, decode/prefill as median, consistency as CV%. On Apple Silicon latency jumps at the memory threshold; an average hides exactly that.
Decoupled host sampler — memory pressure, swap, power source and thermal throttling are recorded by a separate process and joined by timestamp, never estimated from the request thread.
Answer quality, not just speed — a use-case pack (packs/*.yaml: prompts, green/red flags, weighted dimensions, a safety knock-out) runs deterministically; a pluggable LLM-as-judge turns it into a weighted scorecard. A new use case is a new YAML, not new code.
Coding-agent tool calls, checked mechanically — touchstone tools runs a tools pack (packs/opencode-tools.yaml: opencode's real tool schemas, multi-call turns, code run against asserts, exact edits, long-context items) with no judge at all; tools-compare pairs two runs per item (exact McNemar), e.g. two quantizations or two reasoning_effort settings.
System prompts as a measured axis — every prompt runs once per system-prompt variant, so the same run tells you whether your prompt actually helps.
Engine-agnostic by construction — the only engine-aware file is client.py. LM Studio, mlx-lm, Ollama and llama.cpp are a config swap, never a code branch.
Resumable everywhere — eval, judge and the overnight queue all continue where they stopped; nothing is lost to an interrupted run.
Cross-machine comparison — aggregate rolls many bundles into one Hardware×Quality table, reporting the model delta (peak minus baseline) that actually compares across machines.
Overnight queue — several models sequentially, each with a fresh RAM baseline (unload → settle → eval → judge), with per-step watchdogs.
Optional web control-center — configure, start/stop, watch live, compare and export in the browser, as an out-of-process control-plane that never measures in its own process.

Requirements¶

macOS. The measurement core is portable, but the host sampler reads memory_pressure, pmset and powermetrics.
Python 3.12+ via uv (brew install uv).
A local OpenAI-compatible endpoint — LM Studio, mlx_lm.server, mlx-openai-server, Ollama or llama.cpp. If you don't have one yet, see jkaindl.de/uplink/llm-setup.
Optional — passwordless sudo for powermetrics. Without it the throttle flag stays off and throttled runs are not excluded (see How-to: set up a new machine).
Optional — the [gui] extra (FastAPI/uvicorn/Jinja) for the web control-center.

Install¶

git clone https://git.jkaindl.de/jkaindl/llm-benchmark-harness
cd llm-benchmark-harness
uv sync
uv sync --extra gui     # optional: web control-center

Quick Start¶

uv sync
# Latency benchmark — M1 → LM Studio (:1234), M5 → mlx_lm.server (:8080):
uv run touchstone run    --config config.m1.yaml
uv run touchstone embed  --config config.m5.yaml   # embedding throughput (separate)
uv run touchstone report --runs ./runs             # (re)build report.md from raw.csv

# Quality evaluation — run a use-case pack, then score it:
uv run touchstone eval   --pack packs/ndassist.yaml --config config.m5.yaml
uv run touchstone judge  --bundle runs/<ts>_eval_ndassist --judge-config judge.yaml
uv run touchstone aggregate --runs ./runs          # cross-machine Hardware×Quality table

# Web control-center (optional [gui] extra):
uv sync --extra gui
uv run touchstone gui                              # configure → start → watch → evaluate → compare → export

Usage¶

Point endpoint.base_url at your local server, set machine/models, and run. Each run sweeps a scenario × context-bucket matrix, discards one warmup per cell, measures a separate cold-start TTFT, and writes a report.md (whose columns paste straight into a comparison note) plus a raw.csv with one row per request.

Two processes stay decoupled by design: the latency runner (requests) and the host sampler (psutil + powermetrics + memory_pressure + pmset), joined afterwards by timestamp. Memory is never estimated from the request thread.

The harness reports distribution, not the mean: TTFT as P50/P95, decode/prefill as median, and TTFT consistency as CV%. Throttled and on-battery runs are flagged and excluded from the aggregates (kept raw in the CSV).

Quality evaluation (the second half). Performance doesn't say whether a model answers well. eval runs a use-case pack (packs/*.yaml — prompts + green/red flags + weighted dimensions + a safety knock-out + system-prompt variants) deterministically on the machine under test, capturing answers and tech-specs; judge then scores them with a pluggable LLM-as-judge into a weighted scorecard (aggregate rolls many bundles into one Hardware×Quality table). Generation and judging are two decoupled phases, both incremental and resumable. A new use case is a new YAML, not new code. Two packs ship today: ndassist (Neurodivergenz-Assistent) and buero (Büro-/Wissensarbeit-Assistent).

Tool calling (coding agents). Structured output doesn't need a judge. touchstone tools drives each item of a tools pack the way opencode drives an agent: every tool call gets a result back — a fixture file for read/cat, a listing derived from the item's own paths for ls/find, a plain receipt for anything with side effects — and the model keeps going until it stops calling tools or the turn budget runs out. Scoring is over the sum of all turns, so a model that looks around first and acts afterwards passes, exactly as it would in the real agent loop; the number of turns is reported, because each one costs a request. Checked mechanically: complete calls (a call truncated at the budget edge with empty arguments does not count), JSON arguments against the schema, finish_reason in any turn, code executed against hidden asserts in a sandbox, and edits applied with opencode's exact-match semantics. touchstone tools-compare A B pairs two runs per item and reports discordant pairs, an exact McNemar test, truncations, empty arguments and median reasoning tokens per side. Model knobs such as reasoning_effort go into the model's extra_body and are checked by a pre-flight request, so an invalid value fails loudly before the matrix instead of falling back silently.

Web control-center (touchstone gui). An optional local FastAPI server (the [gui] extra — build-free HTMX/Alpine, isolated from the measurement core) puts the whole workflow in the browser: see what each pack tests and how it's scored, configure and start/stop runs, watch them live, browse results, compare across machines, and export. It is an out-of-process control-plane — it spawns the same touchstone eval/judge subprocesses the CLI does, so the measurement loop stays decoupled and runs/ remains the single source of truth. Bound to 127.0.0.1, one measurement at a time.

macOS note: throttle detection shells out to sudo powermetrics. Allow passwordless sudo for it or the throttle flag stays off (runs are then not excluded as throttled). See AGENTS.md → Gotchas.

Documentation¶

Docs are in German; the entry point is docs/README.md.

Tutorial — one full pass: install → measure → evaluate → read the scorecard
How-to Guides — build your own pack · set up a new machine · run the overnight queue · use the web control-center
Reference — metric definitions, CSV schema, config keys
Explanation — why distribution over mean, the two-process split, why the GUI is out-of-process
Decisions — 15 ADRs: context, alternatives, consequences

License¶

Code: AGPL-3.0 — see LICENSE; commercial dual-license option in LICENSING.md. Documentation/text: CC BY-SA 4.0 — see LICENSE-DOCS.

Arbeitsregeln (AGENTS.md)¶

Conventions for AI agents (Claude Code, Codex, …) working on this repository.

Project character¶

llm-touchstone is a thin benchmark harness for local LLM endpoints. It drives a fixed prompt set through an OpenAI-compatible endpoint, records TTFT / prefill-tok/s / decode-tok/s with their distribution, and — in a decoupled process — samples macOS memory pressure and thermal throttling, then merges both into one Markdown table (+ CSV). The whole point: the same code runs on an M1 (LM Studio, :1234) and an M5 (mlx_lm.server :8080 / mlx-openai-server :8000); only the config changes.

Non-negotiable design decisions (from the architect brief): - OpenAI-compatible is the only interface. No engine-specific code in the hot path (the single thin adapter is client.py). - Two decoupled producers, merged by timestamp: latency runner + host sampler. Memory is never estimated from the request thread. - Persisted output is Markdown + CSV, nothing else. No DOCX/PDF/HTML artifacts. (The opt-in eval --web live monitor serves HTML transiently for viewing; it persists nothing as HTML — the bundle stays jsonl/csv/md.) - Distribution over mean: TTFT P50/P95, decode/prefill median, TTFT CV% for consistency. - Warmup discarded, cold-start TTFT reported separately; throttled/battery runs flagged and excluded from aggregates (kept raw).

Architecture principles¶

Data flows through one shared contract (models.py); keep modules from drifting apart.

models.RAW_CSV_COLUMNS is the single source of truth for the CSV schema and is asserted against RunRecord at import — change one, change both.
Pure logic (stats, padding, merge, metric derivation, host-tool parsers) is unit-tested; I/O (runner stream, sampler loop) is dependency-injected so it stays testable without a live server or sudo (stream_once(clock=…, wall=…), run_benchmark(sampler=…)).
Don't add engine-specific branches outside client.py. Don't drive Obsidian/Smart Composer/Local GPT — the endpoint is measured, not the UX layer.

Commands¶

uv sync                                        # venv + deps (+ dev group)
uv run touchstone run    --config config.m1.yaml # M1 → LM Studio
uv run touchstone run    --config config.m5.yaml # M5 → mlx_lm.server / mlx-openai-server
uv run touchstone embed  --config config.m5.yaml # embedding throughput
uv run touchstone report --runs ./runs           # (re)generate report.md from raw.csv

uv run touchstone eval   --pack packs/ndassist.yaml --config config.m5.yaml  # qualitative run → bundle (tech-specs auto)
uv run touchstone eval   --pack packs/ndassist.yaml --config config.m5.yaml --resume runs/<ts>_eval_ndassist  # nach Abbruch weiter
uv run touchstone judge  --bundle runs/<ts>_eval_ndassist --judge-config judge.yaml  # LLM-as-judge → filled scorecard (resumebar)
uv run touchstone judge  --bundle runs/<ts>_eval_ndassist --judge-config judge.yaml --web  # + live judge monitor (score dist / red-flags / master preview)

uv sync --extra gui                            # install the optional web-UI deps (fastapi/uvicorn/jinja2)
uv run touchstone gui                            # local web control-center: configure→start→watch→evaluate→compare→export

uv run touchstone queue --queue queue.example.yaml          # Nacht-Queue: mehrere Modelle sequenziell eval→judge
uv run touchstone queue --queue queue.example.yaml --check  # nur die LM-Studio-Modell-Wechsel-Kette verifizieren (kein Matrix-Lauf)
uv run touchstone queue --queue queue.example.yaml --resume runs/<ts>_queue  # nach Abbruch weiter (fertige Einträge übersprungen)

uv run touchstone tools --pack packs/opencode-tools.yaml --config config.m5-lmstudio.yaml --models-json '[{"id":"<id>","quant":"<q>"}]'  # Tool-Calls/Code deterministisch
uv run touchstone tools-compare runs/<A> runs/<B> --out cmp.md   # Paarvergleich je Item + exakter McNemar

uv run pytest -q                               # tests (no server/sudo needed)
uv run ruff check . && uv run ruff format .    # lint + format
uv run mypy touchstone/                          # strict type-check
uv run --extra tokenizer python -c "..."       # exact per-model tokenizer (transformers)

Conventions¶

Workspace standards (maintainer-local): The binding Leitkonvention lives in _docs/CONVENTIONS.md (profile python-uv) in the maintainer's multi-project workspace, ../_docs relative to this repo — not part of this repository, ignore if absent in your clone. Model: comply-or-explain.

Project-specific: - Python 3.12, uv only (committed uv.lock). Ruff (line-length 100) + mypy strict. - German is fine in prompts, report output and commit descriptions; code/identifiers English. - runs/ is gitignored — report.md is pasted by hand into the SSOT test note. - The shipped configs (config.*.yaml) must always validate (tests/test_config.py guards it). - Eine neue Gotcha gehört in die passende Regeldatei unter .claude/rules/ (oder in eine neue mit paths:), nicht in diesen Kern; das Warum nach docs/internal/decisions.md, der Aufbau nach docs/internal/architecture.md. Der Kern bleibt unter 12.000 Einheiten (kontext_lint im _docs-Repo misst es).

Gotchas¶

Throttle detection needs sudo. sampler.py spawns sudo -n powermetrics. Without passwordless sudo for it, the throttle flag stays False and runs are not excluded as throttled (they're just unflagged). Either grant NOPASSWD for powermetrics or accept that throttling won't be detected on that machine.
engine_version is often unknown. The OpenAI-compatible API doesn't expose mlx_lm/mlx/LM-Studio build versions. Set engine / engine_version in the config to get them into the report header.
usage may be missing even with stream_options.include_usage. The runner then falls back to a heuristic completion-token count; prefer servers that emit usage.
VLM needs a vision model. prompts/vlm_sample.png is a text-bearing document page (swap in your own for representative load), but the vlm scenario only works against a vision-capable model on the endpoint — a text-only model rejects the image content.

Context index¶

Before you change touchstone/sampler.py, merge.py, report.py, models.py or runner.py, or add a RAM/CSV field: .claude/rules/metrics-and-ram.md (columns: docs/reference/metrics-and-schema.md).
Before you change how client.py / runner.py handle reasoning, thinking or finish_reason: .claude/rules/reasoning-capture.md.
Before you change judge.py, scorecard.py, the judge backend or the judge-model picker: .claude/rules/judge-backend.md (ADRs: docs/decisions/0009-reasoning-only-unscored-preflight.md, docs/decisions/0010-judge-runaway-guard.md).
Before you change preflight.py, qualrun.py, config.py, ModelSpec knobs, extra_body, remote endpoints or eval resume: .claude/rules/eval-run-and-endpoints.md.
Before you touch webmon.py, events.py, judge_events.py, tail.py or loadview.py (eval --web / judge --web): .claude/rules/live-monitor.md.
Before you change anything under touchstone/gui/ (control plane, compare, meta-report, reports.jsonl, model picker): .claude/rules/gui.md (method: docs/explanation/design-decisions.md, docs/decisions/0007-holistische-dimensionen.md, docs/decisions/0014-gui-control-plane.md).
Before you change sampling, config*.yaml, queue.example.yaml or anything that talks to LM Studio (JIT loads, lms load, penalties): .claude/rules/lmstudio-and-sampling.md.
Before you change toolbench.py, packs/opencode-tools.yaml or packs/opencode-tools-context/: .claude/rules/tools-pack.md.
Before you change runqueue.py or the queue command: .claude/rules/night-queue.md (guide: docs/how-to/nacht-queue-fahren.md, ADR docs/decisions/0015-nacht-queue.md).
To find a module or understand the qualitative/live-monitoring split: docs/internal/architecture.md. Why hosting and mirrors are as they are: docs/internal/decisions.md.

Memory¶

Projekt-Memory under ~/.claude/projects/-Users-Shared-code-llm-benchmark-harness/memory/ (index: MEMORY.md). Session-Handoff under .remember/ (gitignored). Vault cockpit: $VAULT/25_Coding/llm-benchmark-harness/ (status/tasks/decisions; $VAULT = etablierter CORE-META-14-Platzhalter für den Obsidian-Vault des Maintainers).

SDD-Artefakte (seit 2026-07-16): Cockpit, nicht Repo — Specs/Plans/Task-Reports leben im Coding-Cockpit des Maintainers ($VAULT/25_Coding/llm-benchmark-harness/_SDD/, CORE-META-14, maintainer-lokal). Sie tragen Arbeitskontext (Vault-Pfade, Schwester-Repo-Interna), der in einem public Repo niemandem nützt. Das Repo behält die Design-Essenz in dieser Datei + CHANGELOG.md.
Alt-Bestand: docs/superpowers/{specs,plans}/ ist eingefroren — nichts Neues dort ablegen.
Nie im Repo: absolute Pfade außerhalb des Repos (/Users/…, Vault-Pfade) — Platzhalter nutzen ($VAULT/…, ~/…, repo-relativ). Herkunftsnachweise als Repo-Name + Datei:Zeile sind dagegen erwünscht.

Hosting¶

origin = git.jkaindl.de (primär): https://git.jkaindl.de/jkaindl/llm-benchmark-harness
Auth: Forgejo-Token ~/.forgejo-token, GitHub-Token ~/.github-token (HTTPS, nicht in .git/config).

Abweichungen von der Leitkonvention¶

CORE-META-03/04 — Doku-Einstieg docs/README.md (Hero/Landing + Diátaxis-Karte), die Entscheidungs-Ebene docs/decisions/ (ADRs + Index, je Kontext/Alternativen/Auswirkungen), docs/reference/ + docs/explanation/ (Narrativ, verlinkt die ADRs) sowie seit 2026-07-28 docs/tutorial.md + docs/how-to/ (4 Guides + Index) sind erstellt. Die READMEs (de/en) erfüllen den readme-spec.json-Tier python-cli — readme_lint.py --strict ist grün. Noch offen: Hero-Bild.

Einstiegspunkte¶

Art	Name	Ziel
`python-script`	`touchstone`	`touchstone.cli:app`

Modul-Abhängigkeiten¶

graph LR
    touchstone___main__ --> touchstone_cli
    touchstone_cli --> touchstone_aggregate
    touchstone_cli --> touchstone_client
    touchstone_cli --> touchstone_config
    touchstone_cli --> touchstone_embed
    touchstone_cli --> touchstone_events
    touchstone_cli --> touchstone_gui_app
    touchstone_cli --> touchstone_gui_bundles
    touchstone_cli --> touchstone_gui_control
    touchstone_cli --> touchstone_gui_judge_meta
    touchstone_cli --> touchstone_hostinfo
    touchstone_cli --> touchstone_judge
    touchstone_cli --> touchstone_judge_events
    touchstone_cli --> touchstone_models
    touchstone_cli --> touchstone_pack
    touchstone_cli --> touchstone_preflight
    touchstone_cli --> touchstone_qualrun
    touchstone_cli --> touchstone_report
    touchstone_cli --> touchstone_result_schema
    touchstone_cli --> touchstone_results
    touchstone_cli --> touchstone_runner
    touchstone_cli --> touchstone_runqueue
    touchstone_cli --> touchstone_scorecard
    touchstone_cli --> touchstone_toolbench
    touchstone_client --> touchstone_runner
    touchstone_client --> touchstone_toolbench
    touchstone_embed --> touchstone_config
    touchstone_embed --> touchstone_merge
    touchstone_embed --> touchstone_models
    touchstone_embed --> touchstone_prompts
    touchstone_embed --> touchstone_sampler
    touchstone_gui_app --> touchstone_aggregate
    touchstone_gui_app --> touchstone_config
    touchstone_gui_app --> touchstone_gui_bundles
    touchstone_gui_app --> touchstone_gui_compare
    touchstone_gui_app --> touchstone_gui_configs
    touchstone_gui_app --> touchstone_gui_control
    touchstone_gui_app --> touchstone_gui_glossary
    touchstone_gui_app --> touchstone_gui_judge_compare
    touchstone_gui_app --> touchstone_gui_judge_meta
    touchstone_gui_app --> touchstone_gui_live
    touchstone_gui_app --> touchstone_gui_meta_report
    touchstone_gui_app --> touchstone_gui_packs
    touchstone_gui_app --> touchstone_gui_report_md
    touchstone_gui_app --> touchstone_gui_trash
    touchstone_gui_app --> touchstone_pack
    touchstone_gui_app --> touchstone_webmon
    touchstone_gui_bundles --> touchstone_gui_control
    touchstone_gui_bundles --> touchstone_judge
    touchstone_gui_bundles --> touchstone_merge
    touchstone_gui_bundles --> touchstone_pack
    touchstone_gui_bundles --> touchstone_qualrun
    touchstone_gui_bundles --> touchstone_results
    touchstone_gui_bundles --> touchstone_scorecard
    touchstone_gui_compare --> touchstone_gui_bundles
    touchstone_gui_compare --> touchstone_merge
    touchstone_gui_compare --> touchstone_models
    touchstone_gui_compare --> touchstone_pack
    touchstone_gui_compare --> touchstone_qualrun
    touchstone_gui_compare --> touchstone_results
    touchstone_gui_compare --> touchstone_scorecard
    touchstone_gui_compare --> touchstone_stats
    touchstone_gui_configs --> touchstone_client
    touchstone_gui_configs --> touchstone_config
    touchstone_gui_configs --> touchstone_judge
    touchstone_gui_control --> touchstone_config
    touchstone_gui_control --> touchstone_pack
    touchstone_gui_judge_meta --> touchstone_gui_glossary
    touchstone_gui_judge_meta --> touchstone_gui_report_md
    touchstone_gui_judge_meta --> touchstone_results
    touchstone_gui_live --> touchstone_events
    touchstone_gui_live --> touchstone_judge_events
    touchstone_gui_live --> touchstone_tail
    touchstone_gui_meta_report --> touchstone_aggregate
    touchstone_gui_meta_report --> touchstone_gui_glossary
    touchstone_gui_meta_report --> touchstone_gui_report_md
    touchstone_gui_packs --> touchstone_pack
    touchstone_gui_report_md --> touchstone_gui_glossary
    touchstone_gui_report_md --> touchstone_result_schema
    touchstone_judge --> touchstone_config
    touchstone_judge --> touchstone_pack
    touchstone_judge --> touchstone_results
    touchstone_merge --> touchstone_models
    touchstone_preflight --> touchstone_config
    touchstone_preflight --> touchstone_runner
    touchstone_qualrun --> touchstone_config
    touchstone_qualrun --> touchstone_merge
    touchstone_qualrun --> touchstone_models
    touchstone_qualrun --> touchstone_pack
    touchstone_qualrun --> touchstone_preflight
    touchstone_qualrun --> touchstone_prompts
    touchstone_qualrun --> touchstone_results
    touchstone_qualrun --> touchstone_runner
    touchstone_qualrun --> touchstone_sampler
    touchstone_report --> touchstone_models
    touchstone_report --> touchstone_stats
    touchstone_result_schema --> touchstone_scorecard
    touchstone_runner --> touchstone_config
    touchstone_runner --> touchstone_merge
    touchstone_runner --> touchstone_models
    touchstone_runner --> touchstone_prompts
    touchstone_runner --> touchstone_sampler
    touchstone_runqueue --> touchstone_client
    touchstone_runqueue --> touchstone_config
    touchstone_runqueue --> touchstone_preflight
    touchstone_sampler --> touchstone_models
    touchstone_scorecard --> touchstone_pack
    touchstone_scorecard --> touchstone_results
    touchstone_scorecard --> touchstone_stats
    touchstone_webmon --> touchstone_events
    touchstone_webmon --> touchstone_judge_events
    touchstone_webmon --> touchstone_loadview
    touchstone_webmon --> touchstone_tail