Introducing Verification Scores: Evidence-Based Trust for Agent Tools
Every tool on AgentNode now has a verification score computed from real sandbox execution — not self-reported badges. Here is how it works and why it matters.
The problem with verified badges
Most package registries have a single binary: verified or not. That tells you almost nothing. A tool that needs an API key to run is not broken — it just cannot be fully tested without credentials. A tool that requires Chromium is not defective — it needs a system dependency the sandbox does not have.
Binary verification conflates we could not test it with it does not work. That is misleading.
Our approach: Earned scores, not granted badges
Starting today, every tool on AgentNode gets a verification score from 0–100, computed from real evidence:
Install (15 pts) — Does
pip installsucceed cleanly?Import (15 pts) — Do all declared entrypoints load?
Smoke test (25 pts) — Can we call the tool with schema-generated inputs and get a result?
Tests (15 pts) — Does the package test suite pass? Real tests score higher than auto-generated ones.
Reliability (10 pts) — Does the same input produce a successful result across multiple runs?
Determinism (5 pts) — Are results consistent? Lightly weighted — legitimate non-determinism exists.
Contract (10 pts) — Is the return value serializable and structurally valid?
Warnings (−2 each, max −10) — Deprecation warnings, unsafe patterns.
Scores map to tiers:
Gold (90–100) — Everything checks out, including multi-run stability.
Verified (70–89) — Core functionality confirmed.
Partial (50–69) — Works but cannot be fully tested (needs credentials, system deps, etc.).
Unverified (0–49) — Did not pass enough checks.
The key insight: Not testable is not broken
Tools that need API credentials get partial smoke credit (12/25) instead of zero. Same for tools requiring Chromium, FFmpeg, or binary file inputs. We classify why something is inconclusive — needs_credentials, missing_system_dependency, needs_binary_input — and score accordingly.
A stub package that raises NotImplementedError gets zero smoke credit. That is the difference.
Continuous verification
Scores are not static. Every package is automatically re-verified every 30 days. If a dependency update breaks something, the score drops and you see it before your agent does.
Search ranking
Verification scores now factor into search results. Higher-scoring tools rank higher — but not at the expense of relevance. A perfect-score PDF tool will not outrank a relevant web scraper just because of its score.
What is next
This is the scoring foundation. We are watching the distribution, tuning the weights, and collecting feedback. If you publish a tool and think the score is unfair, tell us — that is exactly the signal we need.