Roundhouse — the llama-systemd-unit renderer & watcher

This is the part every existing tool gets wrong, and the part that decides whether you trust Roundhouse. The recon numbers make the constraint vivid: mixperten.service is 103 lines, of which 9 are the command. The rest is a lab notebook — KLD tables, a McNemar test, a decision record signed "ENTSCHEIDUNG markus 2026-08-11", and a warning about which comment is now decoration. Any mechanism that regenerates the file from a parsed model will eventually normalize, reorder, or drop something. So:

Mechanism: parse everything, rewrite nothing, splice bytes

1. Concrete-syntax parse. Read the file as lines. Locate ExecStart= and its backslash continuations; tokenize the command with systemd's exact quoting rules (single/double quotes, \ continuations including the no-space form llama-server\ in the NPU unit, %% specifiers). Every token keeps its byte range in the original file. Same for plain Key=Value lines.
2. Edits are splices. Changing -c 65536 to -c 131072 replaces exactly those bytes. Adding a flag inserts into the line the operator picks in the preview. Removing a flag deletes its tokens and nothing else. Comments, blank lines, ordering, indentation: untouched by construction, because they are never parsed into a model that gets serialized back.
3. Readback verification — refuse rather than corrupt. After splicing (to a temp file): re-parse and assert the argv equals the intended argv; then systemd-analyze verify the unit; then after daemon-reload, compare systemctl --user show -p ExecStart — systemd's own parser — against intent. Any mismatch aborts the write and shows both parses. This is the guard against the tokenizer being subtly wrong about an edge case.
4. Atomic write with compare-and-swap. Hash the file at read time; refuse to write if it changed underneath (you hand-edit these files, and will continue to — that must stay safe).
5. Git replaces .bak. git init in ~/.config/systemd/user/; the seven existing .bak* files become the initial history (dated commits reconstructed from their timestamps, then the files removed). Before any Roundhouse write, uncommitted hand edits are auto-committed as manual edit (imported) — hand edits are first-class, never clobbered, and drift is visible exactly as your Push philosophy wants.

The stale-prose problem, honestly

The comments reference the values ("65536 kept from the previous unit to change one thing at a time"). A byte-perfect splice preserves the prose and thereby makes it a liar. No tool can fix that; a good tool makes it impossible to do unknowingly: on every edit, grep the comment block for the flag name and the old literal value; hits appear in the review step and must be either edited or explicitly acknowledged. Never auto-edit prose — a tool that rewrites your measurement notes is a tool you'd rightly uninstall the same day.

 TWEAK qwen3.6-coding.service                                        step 2 of 3 — review
 ─────────────────────────────────────────────────────────────────────────────────────────
   -c   65 536 → 131 072     KV q8_0 on this arch: +~2 GiB (estimate — no measured peak
                             at this ctx) · projected peak 19.9 / 30.6 GiB ✓ fits
 ─────────────────────────────────────────────────────────────────────────────────────────
 DIFF — only these bytes change
   −   -c 65536 -t 4 -fa on -ctk q8_0 -ctv q8_0 \
   +   -c 131072 -t 4 -fa on -ctk q8_0 -ctv q8_0 \

 ⚠ PROSE MAY NOW BE STALE — 1 comment line mentions the old value:
   # …supports 262144 ctx; 65536 kept from the previous unit to change one thing at a time
   [ edit this comment ]   [ acknowledge — leave as is ]

 PREFLIGHT   ✓ readback parse = intent     ✓ systemd-analyze verify clean
             ✓ port unchanged              ✓ projected 19.9 < 28.0 warn line
 commit: [ ctx 65536→131072 — headroom from the 1.1 GiB smaller model            ]
                                    [ back ]   [ apply + restart ]   [ apply, restart later ]

Drop-ins

You have none today (verified). Policy: read effective config via systemctl --user cat; if a drop-in ever overrides a key, that key renders read-only with "override lives in foo.service.d/x.conf" and Roundhouse edits the fragment only. It never creates drop-ins: for user-owned units in ~/.config, drop-ins would split one file of truth into two files of truth for zero benefit. That's a fight with systemctl edit deliberately not picked.

Named failure modes

• Tokenizer wrong on an edge case (quoted JSON like '{"enable_thinking":false}', \-no-space, %%). Caught by readback-vs-intent; outcome is a refusal, not a corruption. Residual risk: refusing too often. Acceptable.
• Stale prose acknowledged and forgotten. Mitigation is the ack being recorded in the commit message trailer. Residual risk real; it exists today with hand edits too.
• Concurrent hand edit. Caught by compare-and-swap; the UI re-renders and asks again.
• Git repo confusion if you ever put that directory under another repo/stow scheme. Detect a pre-existing repo and join it instead of nesting.
• Semantic edits beyond splicing (e.g. restructuring a command). Out of scope by design: the escape hatch is the raw-file editor with the same verify + commit pipeline and none of the parameter UI.

"Active" is nearly meaningless here: a unit is active (running) for the full 72 s a 15 GiB model takes to load, and inactive can be the correct, healthy state of a kernel-gated unit. Roundhouse's status is a ladder, and the UI always shows the rung, not a boolean:

Memory fit

Memory is the failure mode that matters, and your own data says formulas lie: the 35B Q4_K_M "fits" on paper, then ARM repack residency pushed 30.5/31.6 GiB and prefill halved while decode looked fine. And the KV formula needs per-architecture knowledge anyway (the hybrid Qwen has KV on 11 of 41 blocks only). So:

• Measured beats modeled. Roundhouse records cgroup memory.peak per (unit, model-file hash, ctx) into a small sqlite. Predictions come from the last measured peak when one exists — labeled "measured" — and from file-size + KV estimate only as a fallback, always labeled "estimate".
• Warn line at 28 GiB projected (the measured collapse zone starts ~30.5 of 31.6); hard confirm above it. Co-residency is summed: big slot + always-on peaks + a fixed system reserve.
• Prefill-collapse tripwire: if a unit's journal-observed prompt-processing rate drops well below its own history while RSS is near the ceiling, badge it "memory-pressure suspected" — that exact signature cost you a day once; the UI should name it.

• Port interlock, gate-aware. Claims are computed from parsed --port across all units, then evaluated per scenario: active conflict (two running — should be impossible, flagged red), armed conflict (two enabled whose conditions can be simultaneously true — the :8086 case on an NPU kernel — flagged on the port board and at enable time), latent (disabled claimant — a footnote, like mixperten's :8085). Enabling or starting into an armed conflict requires typing past a named warning, never a silent failure at boot.
• Alias interlock, same machinery. Clients route by --alias; two live units answering the same alias on different ports is a subtler lie than a port clash.
• The retired trap. [RETIRED …] in Description is the convention (already your practice). Retired units render folded and grey, keep their full notes (they're the best-documented decisions you have), and their start/enable actions are demoted behind an explicit "this unit is marked RETIRED 2026-08-12 and claims :8085, owned by qwen3.6-coding — start anyway?" Never deleted, never hidden.
• STANDBY is not broken. Kernel-gated units get their own section, neutral color, and the condition spelled out in words. No red, no "failed", no restart storm — the whole point of the ExecCondition pattern, finally reflected in a UI.
• OOM prevention = the §4 fit check, enforced at three doors: starting a unit, switching the big slot, and any edit that raises -c / changes model / changes KV quant. Above the warn line the action needs an explicit override; the override is recorded in the commit/journal so the 3-a.m. question "why is the box swapping" has an answer.
• Co-fit notes stay human. The "stop the small models first — they do not co-fit" knowledge lives in comments today. v0.1 does not invent a metadata language for it; the summed-peaks check covers the arithmetic. If it earns its keep later, a single opt-in comment directive (# roundhouse-slot: big) — in the file, so truth stays in the file — is the ceiling of cleverness allowed here.

Switch flow — turning the table

 SWITCH big slot   qwen3.6-coding  →  qwen3-30b
 ─────────────────────────────────────────────────────────────────────────────────────────
 INTERLOCK   :8085 free after stop ✓ · alias overlap none ✓
             fit: measured peak 21.3 + always-on 3.0 + reserve 1.5 = 25.8 / 30.6 GiB ✓
 ─────────────────────────────────────────────────────────────────────────────────────────
  1 ▣ stop qwen3.6-coding        done · 2 s
  2 ▣ start qwen3-30b            active · 1 s
  3 ▶ model loading              /health 503 · 48 s elapsed (last load on this box: 72 s)
  4 ▷ ready
  5 ▷ smoke test (optional)      one short real completion, wall-clock timed, logged
 ─────────────────────────────────────────────────────────────────────────────────────────
 [ abort — restores qwen3.6-coding ]        enable-on-boot follows the switch: [x] yes

Q1

Who owns the truth? Two truths, two owners, one boundary. llm-proxy on hossenfelder owns catalog truth: what exists, what it costs, what it can do. Roundhouse owns process truth on the serving host: what is actually up, on which port, with which ctx, in which state — because that truth is literally the systemd state it renders. The boundary is a one-way street: Roundhouse publishes process truth (a read-only /status JSON: alias, port, rung, ctx, last-measured t/s), and llm-proxy may consume it to mark its 4 [local] entries live or dead instead of guessing. Roundhouse never pushes routing decisions and never reads the catalog to decide anything. Consumers keep talking to the gateway, not to Roundhouse.

Q2

Push or Pull? Answered in §1: Push for process config (boot must not depend on the network), Pull for selection metadata (a request already implies the gateway is reachable). Roundhouse is the Push horn done right — files in git, drift visible, and the "generator" temptation explicitly rejected because your units are documents, not build artifacts.

Q3

May the measurement loop influence the ladder? Close the loop through the operator, not around him. bullpen-evals verdicts flow into a recommendation inbox: "rung 2 solved 14/15 tasks that escalated to rung 3 last month — demote the default?" One click applies it; the applied change is a git commit (or an llm-proxy config change) with the evidence linked. What makes self-reconfiguration undebuggable is not the feedback — it's unattributed change. Your own unit files show the standard: every decision in them has a date, a measurement, and a signature. Hold the loop to the same standard and it stays debuggable; let it write silently and in six months no one can say why the ladder looks the way it does. Fully automatic re-rating is a later, separate decision to make after a few months of watching how often you just click "apply".

Shape: one Python file (matching the /opt/llm-proxy.py house style) + one static page, SSE for live updates, sqlite for measured peaks, talking to systemd via the user D-Bus/systemctl --user. Runs as — of course — a systemd user unit, roundhouse.service, on a free port (8090 is unclaimed today; the port board will confirm its own). No node toolchain, no build step, nothing to babysit.

• v0.1Read-only, one week, already useful. Scan + parse all units, Platform + Stall views, status ladder, operator's-notes rendering, port board with the :8086 finding visible, kernel-gate STANDBY treatment, memory gauge. Zero write path = zero risk while trust builds. This alone beats every existing tool for this fleet, because nothing else can render these units.
• v0.2Operations. Start/stop/restart/enable with port+alias interlocks; the Switch flow with the loading ladder; git init + .bak absorption; journal tail in the Stall view.
• v0.3The round-trip. Parameter tweak with splice, diff preview, readback verification, stale-prose flagging, commit messages. The hard part, shipped only after v0.1/v0.2 have proven the parser against all 23 real files.
• v0.4Memory intelligence. Peak-RSS sqlite, measured-vs-estimate fit predictions, prefill-collapse tripwire, per-request t/s history from the journal.
• v0.5Roundhouse hooks. /status JSON for llm-proxy; bullpen-evals recommendation inbox. Gated on the /v1/models enrichment landing first — that remains the true smallest Roundhouse step, and it happens in llm-proxy, not here.

Fleet scope: v0.1–v0.4 are boltzmann-only on purpose. The design generalizes (a host column, N unit dirs), but multi-host before the round-trip is trustworthy would multiply risk, not value.

1. Auth: LAN-only bind like the llama-servers, or tailnet/basic-auth? It can stop and start your models, so "whatever :8085 does" may not be enough.
2. Git location: repo directly in ~/.config/systemd/user/, or a mirror repo with the unit dir symlinked into it? Direct is simpler; say if that directory has other plans.
3. Naming: Roundhouse, or something else in the railway family? The metaphor is load-bearing enough in the UI copy ("turn the table", "standing by") that the name should be settled before implementation.