The Measurement

A skill is instructions we hand to the agent. We wish to know two things about it, and the whole apparatus exists to stop us from confusing them:

• Did it fire? When a request arrives that the skill is for, is the skill reached for — and, just as important, is it left alone when the request only looks like its work?
• Did it help? Once fired, did the guidance change the result for the better — or would the model have done the same thing unaided?

They come apart in the cruelest way. A skill can fire flawlessly and change nothing — a costume worn over the model's own instincts. Under the first metric it looks radiant. Under the second it is hollow. Merge the two numbers into one and the hollow skill borrows the radiant one's score, and you never find out. So we do not merge them. Ever.

Those two questions are not first answered by a measurement. They are answered by the author, out loud, before a line of the body is written — nine of them, and the record of an expedition that skipped them is a skill that measures well and means nothing:

WHY IT EXISTS
  1a  Name ONE live occasion where it should fire. Not a category.
  1b  What comes out in that case WITHOUT it? Show it, don't describe it.
  1c  And WITH it? Show that too.

HOW IT IS USED
  2a  What has to be in place for it to work at all?
  2b  What exists at the end? Show it literally.
  2c  What checks that? Name a command — or say where to look.

THIS ONE, OR THE NEIGHBOUR
  3a  Take one query it must fire on. Who else answers it? Name them.
  3b  If someone did — which of you wins, and under what condition?
  3c  What work looks like yours but isn't? Name it, and whose it is.

Question 1b is the one that kills the skill. If the output is already fine without it, the skill should not exist — you say so and you stop. Everything downstream assumes 1b showed a real wound.

And the questions are not paperwork — each answer becomes a load-bearing part of what is later measured. 1b/1c become the eval's trigger queries. 2a/2b become the ## Contract. 2c becomes the acceptance check and rubric.json. 3b becomes the boundary sentence in the description. 3c becomes the negative eval cases — the ones that must stay quiet. The interview you answer up front IS the rubric you are graded against later; the measurement only reads back what the questions already decided.

Triggering is measured by a self-contained instrument, score-description.py. It reads the skill's rubric.json — an array of natural-language queries, each labelled with whether the description should make the skill fire — and it runs each one as a nested agent, watching only whether the Skill tool activates.

[
  {"query": "convert the px in this stylesheet to rem", "should_trigger": true},
  {"query": "make these sizes scale with the base font",  "should_trigger": true},
  {"query": "fix the border-width on this chart svg",     "should_trigger": false}
]

At least six cases, at least one of each polarity, because a test set that only proves the skill fires cannot prove it stays quiet. The score reads as a fraction — 16/18, an accuracy of 0.889 — and it is anchored to two hashes: one over the description it measured, one over the queryset. Edit either and the record marks itself stale, because the thing this eval measures is the description, and a rewritten description must not keep an old score.

One rule outranks the fraction. A negative case that fires is a real failure, whatever the skill's type. A missed positive can be forgiven in an advisory skill; a description broad enough to grab work that was never yours cannot. The narrator learned to fear the false positive more than the false negative — the former is a skill that will fire on a thousand unrelated prompts, forever, quietly.

The second question needs a different run entirely — you cannot learn whether guidance helped by watching whether a tool fired. The first rite for it took one task per skill, run once with the guide and once without, judged by a model. It produced a tidy number for every skill, and every one of those numbers rested on a single task run once. One task cannot tell a real effect from zero. The numbers were struck from the record, and each skill now reads not measured until it is measured properly.

The rite that replaced it keeps the honest step and drops the single task. A quality eval is fifteen or more real tasks, built with Claude Code's /claude-api build-eval. Each task runs as an agent in a small fixture repository, once with the skill installed and once without, several times each, and a grader reads what the agent actually left behind. Without the second arm you cannot separate "the skill was read" from "the skill changed something", and the difference between those two is the entire reason the document exists.

"acceptance": {
  "eval_type": "quality-with-vs-without",
  "cases": 16, "reps": 3,
  "with_skill":    { "mean": 0.83, "ci_half_width": 0.14 },
  "without_skill": { "mean": 0.19, "ci_half_width": 0.14 },
  "effect": 0.65,   // with minus without
  "noise": 0.35,    // half-width of that difference, over cases
  "verdict": "helps"
}

The verdict is computed from two numbers and nothing else: an effect larger than its noise helps, one smaller than minus the noise hurts, and everything between is within noise. Nobody gets to talk a result up. It is deliberately not run in the gate. It spawns agents; agents are expensive and non-deterministic; a merge must not wait on a jury. The gate only reads the record the run left.

A skill runs on someone else's machine. So a third confirmation rides in the same per-skill file, written not by the scorer but by cerberus:security-scan: an eight-point consumer-safety walk — prompt-injection, exfiltration, secrets, dangerous commands, obfuscation, external fetches, credential access, privilege escalation — one verdict per point, and the whole passes only when none is flagged.

Its freshness key is broader than the trigger's, and on purpose: it hashes the skill's installer-facing material — SKILL.md, references/, and scripts/ — because a script edit changes what reaches a consumer even when the description never moves. Consumer safety has no skip flag. None. You do not get to push a skill whose safety scan is stale.

Here is the horror, and it is worse for being ordinary. For months, nothing wrote the measurement at all. The scoring had been moved to a new instrument when the marketplace replaced its upstream harness; the saving of the result did not move with it. Runs happened. They landed in *-workspace/ directories that .gitignore swallows as scratch. They evaporated. There was no evidence any skill had ever been measured.

And the gate said so, every time, honestly:

⚠ eval present but never measured (push allowed — refresh when you can)

Forty-two skills carried that line. It was true. It was ignored. Because a warning that appears on everything distinguishes nothing — it becomes background, weather, the hum of the fan. The mechanism whose breakage is reported as a warning breaks silently. It had been there all along. Warning. Twice a push. Seen by no one.