Agent Skill Model-Release Regression Harness
Check whether an existing agent skill still improves the same fixtures after a model release without expanding tools, permissions, cost, latency, or output-contract risk.
What problem does Agent Skill Model-Release Regression Harness solve?
A skill that improved output on last month's model can lose its advantage, break an output contract, expand tools or permissions, or become too expensive or slow after a model release while still appearing to work in a single demo.
Use it to
- Retest a production skill after a major US or non-US model release
- Detect when a skill loses uplift even though the candidate baseline improves
- Block a rollout when tool or permission boundaries expand
- Compare cost and latency against explicit release thresholds
- Preserve CANNOT_ASSESS until a compatible runner produces evidence
What it returns
- CLEAR, REVIEW, or HOLD posture across the supplied release pair
- Per-case STABLE, IMPROVED, CANNOT_ASSESS, or regression classification
- Output-contract, tool, permission, cost, latency, and skill-uplift findings
- Coverage counts, unresolved runs, Markdown report, JSON report, and deterministic evidence receipt
A representative input and result
Access and approval boundaries
- Terminal permission to run the bundled local Python harness.
- Read access to one user-supplied UTF-8 JSON observation matrix.
- Write access only to the selected local JSON and Markdown report paths.
- No browser, network, model-provider account, API key, credential, marketplace, messaging, or publication permission is required.
Known limitations
- The harness evaluates supplied observations and does not call or certify a model.
- CLEAR covers only the supplied fixtures, scoring contract, evidence references, and thresholds.
- A changed or biased evaluator can make numeric comparisons misleading; keep the scoring contract fixed across all four observations.
- The package does not include provider-specific runners, Foundry adapters, live dashboards, or automatic release-note retrieval.
- Testing, publication, indexing, installs, views, reactions, and owner verification are not qualified demand or revenue.
Questions
Does this call OpenAI, DeepSeek, Qwen, Anthropic, or another provider?
No. It analyzes recorded observations from a separate compatible runner and requires no API key or model credits.
Why are four observations required per case?
They separate model improvement from skill improvement. A candidate model can score better alone while the skill itself adds less value or makes the result worse.
What happens when the candidate run is missing?
The case remains CANNOT_ASSESS and the report posture is REVIEW unless another supplied case independently requires HOLD.
Does CLEAR prove the skill works on the new model?
Only for the supplied fixtures, outputs, thresholds, and evidence references. It is not a general compatibility or safety certification.
Can it detect tool or permission drift?
Yes. It blocks disallowed tools or permissions and separately flags expansion beyond the prior skill-assisted boundary even when the expanded items were pre-allowed.
Can I use Promptfoo, LangSmith, or my own runner?
Yes. Map the recorded results to the documented JSON contract. The package stays provider-neutral and does not replace those runners.
Does it measure demand for my skill?
No. Run evidence, publication, installs, views, and indexing are operational signals. Qualified buyer demand and revenue require separate external evidence.
Agent Skill Model-Release Regression Harness keeps proof and approval boundaries visible.
The listing includes the tested package, realistic samples, declared permissions, and known limitations.
Get Agent Skill Model-Release Regression Harness on Agensi