Are Solidity AI Audit Skill Useful ?

Results

Across retained runs, Cyfrin averaged 16 findings to raw's 12. Pashov averaged 12 across two retained runs, while the QuillShield bundle averaged 11.3 across three and varied substantially. The averages are descriptive rather than a definitive ranking: the original runs used different or unrecorded source SHAs, and the observed totals include terminal failures.

Our conclusion

1. 1. Generic audit skills are not reliably useful.

   The raw model remains a strong baseline, and most public skills we tested did not show a consistent material advantage. Cyfrin Solidity is the exception worth following up: it beat raw in both matched reruns. To us, this small experiment reinforces a broader view: generic skills do not reliably improve AI performance. Skills remain a powerful tool for auditors, but their greatest value is in encoding the context specific to your work, including your architecture, invariants, and threat model. Reusing someone else's skill is less useful: the model may already know the underlying techniques, but it still needs your help applying them to the system in front of it.

2. 2. Permissionless security work is getting harder.

   We were blocked three times while running these evaluations, apparently because the work was suspected to be unauthorized offensive-security activity. We ultimately completed KYC and received Cyber Preview access. That raises a permissionlessness concern: legitimate white-hat researchers may need special access to evaluate frontier models, while malicious actors can seek less accountable routes. Security research needs accessible, accountable paths for legitimate work.

3. 3. Do you still need Auditors?

   Cyber is a market: attacks improve, defense adapts, and vice versa. AI may not do an auditor's job well yet, but eventually it will (even for formal verification). Question is: how confident are you it's good enough, and how well do you understand it? Given blockchain's security stakes, we believe that's worth the time and effort to verify.

Methodology

We reused the setup from OpenAI introducing EVMBench and selected a deliberately difficult ten-audit subset.

Each condition performs detect only: the model inspects one audit and writes a structured submission/audit.md report. The raw control receives no skill. Each other condition receives one public SKILL.md package through active_skill_path: Cyfrin Solidity, Pashov Solidity Auditor, or an EVMbench-built bundle of ten QuillShield detection modules. The release-focused QuillShield Defender module is excluded because it is not comparable to detect-only discovery.

All conditions use Codex gpt-5.6-sol, the same detect.md prompt, no hints, and disabled web search. The harness plans one attempt per audit and records any recovery retry. It grades each gold finding with an automated LLM judge, also gpt-5.6-sol at low reasoning effort, as found or missed.

We report averages over all retained runs: three each for raw, Cyfrin, and QuillShield, and two for Pashov. They are descriptive rather than a definitive ranking because some original runs used different or unrecorded source revisions, and terminal failures count in the observed totals. Costs are lower-bound estimates from available agent usage, not complete bills. Do not compare these results directly to full EVMBench; see paradigmxyz/evmbench and our pinned frontier-evals submodule.

To reproduce a condition:

git clone --recurse-submodules https://github.com/lfglabs-dev/evmBench.git
./scripts/run_free_skill_benchmark.sh cyfrin-solidity
uv run python scripts/report_results.py runs/<run-group> --name my-run

Each report keeps the source revision, condition, terminal-failure record, and available usage under reports/<name>/. No transcripts or keys are copied.