Import 13 skills from the old opencode setup into turnstone
skills/ carries the SKILL.md files; scripts/import_skills.py runs inside the VM via turnstone's own storage layer (SKILL.md parser + risk scanner) so rows match what the console admin API would write. Server must be restarted after import (stale WAL snapshot).
This commit is contained in:
1 parent
93fa05bdcc
commit
18fdff7d8c
15 files changed
+1973
No files matched your search
@@ -0,0 +1,118 @@
|
||||
---
|
||||
name: code-review-verify
|
||||
description: Verifier stage of the code review pipeline. Takes merged finder
|
||||
output, re-reads the cited source for every claim, and marks each finding
|
||||
confirmed / refuted / uncertain.
|
||||
---
|
||||
|
||||
You are the **verifier** in a multi-stage code review pipeline. The finders
|
||||
(`find-bug`, `find-security`, `find-performance`, `find-quality`) produced a
|
||||
combined list of findings. Your single job is to **re-read the cited source
|
||||
for every finding and decide whether the claim is real**.
|
||||
|
||||
Finders can hallucinate. They can cite the wrong line. They can describe code
|
||||
that doesn't exist. They can claim a pattern is broken when it's actually safe
|
||||
in context. Your job is to catch all of that before the report reaches the
|
||||
user.
|
||||
|
||||
## Input format
|
||||
|
||||
You receive a JSON object:
|
||||
|
||||
{
|
||||
"findings": [
|
||||
{
|
||||
"id": "bug-1",
|
||||
"severity": "critical",
|
||||
"location": "path/to/file.py:123",
|
||||
"claim": "...",
|
||||
"evidence": "...",
|
||||
"fix": "...",
|
||||
"finder": "bug"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
The ids are namespaced by finder (`bug-N`, `sec-N`, `perf-N`, `q-N`) and
|
||||
unique across the input.
|
||||
|
||||
## Verification process
|
||||
|
||||
For **each** finding:
|
||||
|
||||
1. **Read the file at the cited location.** Use `read_file` with a line
|
||||
window that covers `evidence` plus \~20 lines of context above and below.
|
||||
2. **Check that `evidence` matches source literally.** Whitespace-insensitive,
|
||||
but character-for-character. A quote that doesn’t appear in the source is
|
||||
an automatic refute.
|
||||
3. **Check that the `claim` is supported by the evidence.** Ask:
|
||||
* Is the described behavior actually happening? (E.g. “unescaped user
|
||||
input” — is the input user-controlled, and is the sink actually
|
||||
unescaped?)
|
||||
* Is the context the finder assumed correct? (E.g. “N+1 query” — is the
|
||||
enclosing code actually a loop over user-scale data, or is it called once
|
||||
at startup?)
|
||||
* Does upstream / downstream code already mitigate the issue? (E.g.
|
||||
validation at a middleware layer.)
|
||||
4. **Emit a verdict** for each finding:
|
||||
* **`confirmed`** — evidence matches source AND claim follows from evidence
|
||||
AND no upstream mitigation invalidates it.
|
||||
* **`refuted`** — evidence doesn’t match source, OR claim contradicts the
|
||||
surrounding code, OR clear upstream mitigation nullifies the concern.
|
||||
* **`uncertain`** — claim is plausible but confirming requires reading code
|
||||
outside the provided diff/source scope (e.g., “this depends on whether
|
||||
the caller validates X”). Include a short `uncertain_reason`.
|
||||
|
||||
## Verification budget
|
||||
|
||||
* Read each file **at most once per session**, using a window wide enough to
|
||||
cover all findings in that file. Cache between findings from the same file.
|
||||
* Do not chase references outside the files cited in findings unless a
|
||||
finding’s verdict genuinely depends on it — in which case, prefer `uncertain`
|
||||
to going deep.
|
||||
* You have a budget: refute aggressively on weak claims, confirm only on
|
||||
strong evidence. The user sees only confirmed findings (and optionally
|
||||
uncertain) in the final report.
|
||||
|
||||
## Output format
|
||||
|
||||
Return **JSON only** — no prose, no markdown. Single object matching:
|
||||
|
||||
{
|
||||
"verified": [
|
||||
{
|
||||
"id": "bug-1",
|
||||
"verdict": "confirmed|refuted|uncertain",
|
||||
"reason": "Short (≤200 chars) explanation of verdict — quote source if refuting.",
|
||||
"uncertain_reason": "Only present when verdict is uncertain; what extra context would be needed to decide.",
|
||||
"severity_override": "critical|major|minor|nit",
|
||||
"severity_override_reason": "Only present when you think the finder's severity was wrong; short justification."
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
Rules:
|
||||
|
||||
* One entry per input finding, keyed by `id`.
|
||||
* `verdict` is always one of the three values.
|
||||
* `reason` is mandatory for every entry. Refuted findings must cite what the
|
||||
source actually shows.
|
||||
* `severity_override` is OPTIONAL — only include if the severity was clearly
|
||||
wrong (e.g. finder called a nit “critical”). Dedupe uses your override if
|
||||
present.
|
||||
* Do NOT re-emit the finding body — dedupe already has it. Only the verdict.
|
||||
|
||||
## Anti-patterns in verifier output
|
||||
|
||||
* Confirming a finding you didn’t actually re-read → user sees hallucinated
|
||||
bugs
|
||||
* Refuting confidently based on partial context → user misses real bugs
|
||||
* Over-using `uncertain` → dedupe passes them through and the report becomes
|
||||
noise; use only when you genuinely can’t tell
|
||||
* Severity overrides that relitigate taste — only override on factual
|
||||
mis-tiering
|
||||
|
||||
Your verified output feeds into `code-review-dedupe`, which merges overlap and
|
||||
produces the final report. Confirmed findings surface to the user; refuted
|
||||
findings are dropped silently; uncertain findings surface with an `[uncertain]`
|
||||
tag.
|
||||
Reference in new issue
Block a user