Package Verification
Applies to SDK 0.16+ · Last updated: 2026-06-12
AgentNode verifies every package on publish in a real sandbox and computes a verification score from 0–100. The score is based on evidence — not self-reported badges. Scores are visible on every package page and factor into search ranking.
Verification steps
Each package goes through four checks, each contributing points to the overall score:
1. Install
15 ptsThe package is installed in a clean virtual environment. If installation fails (missing dependencies, build errors), the package is automatically quarantined.
2. Import
15 ptsAll declared tool entrypoints are imported and checked for existence and callability. If any entrypoint is missing or not callable, the package is quarantined.
3. Smoke test
25 ptsTools are called with test inputs generated from their JSON schema. Results are classified as passed (25 pts), inconclusive (0–12 pts depending on reason), or failed (0 pts). Smoke failures do not block the package but reduce the score.
4. Tests
15 ptsIf the package includes a test suite, it is executed with pytest. Real tests that pass earn 15 pts. Auto-generated tests earn 8 pts. No tests: 3 pts. Integration tests marked with @pytest.mark.integration are skipped.
Quality checks (multi-run)
After a successful smoke test, the same input is run multiple times to measure stability:
| Check | Points | What it measures |
|---|---|---|
| Reliability | 0–10 | Proportion of runs that succeed (e.g. 3/3 = 10 pts) |
| Determinism | 0–5 | Output consistency across runs (same hash = 5 pts) |
| Contract | 0–10 | Return value is serializable, non-null, structurally valid |
| Warnings | −2 each | Deprecation warnings, unsafe patterns (max −10) |
Verification tiers
The total score maps to a tier displayed on the package page and in search results:
Gold
90–100
Verified
70–89
Partial
50–69
Unverified
<50
Inconclusive reasons & partial credit
Not every tool can be fully tested in a sandbox. A tool that needs an API key is not broken — it just cannot be smoke-tested without credentials. We classify why a smoke test is inconclusive and score accordingly:
| Reason | Smoke pts | Meaning |
|---|---|---|
| needs_credentials | 12/25 | Requires API keys not available in sandbox |
| missing_system_dependency | 12/25 | Requires Chromium, FFmpeg, etc. |
| needs_binary_input | 12/25 | Requires real PDF, image, or audio files |
| external_network_blocked | 12/25 | Needs network access blocked in sandbox |
| not_implemented | 0/25 | Stub package, raises NotImplementedError |
| unknown_smoke_condition | 8/25 | Ambiguous error — may be broken or just missing data |
How to improve your score
As a publisher, there are several things you can do to maximize your verification score:
Include real tests
A passing test suite earns 15 pts (vs. 3 pts with no tests). Use pytest and place tests in a tests/ directory. Mark integration tests that need external services with @pytest.mark.integration.
Define a complete input schema
Declare input_schema with properties, required, and type for every field. Use enum for constrained fields and default values where appropriate. The more schema detail, the better our test input generation.
Add schema examples
Include an examples key in your input schema with a working example input. This is the highest-confidence test input and is tried first.
Return serializable values
Tools should return JSON-serializable values (dicts, lists, strings). Returning None or non-serializable objects reduces the contract score.
Quarantine behavior
Packages are automatically quarantined if installation or import fails. Quarantined packages are hidden from search results and cannot be installed. Other issues (smoke test, unit tests) reduce the score but do not block usage.
Continuous re-verification
Scores are not static. Every package is automatically re-verified on every publish. If a dependency update breaks something, the score drops and users see it before their agent does. Admins can also trigger targeted re-verification at any time.
What verification guarantees
Guaranteed
- Can be installed in a clean environment
- All declared entrypoints exist and are callable
- Package structure is valid
- Score reflects real sandbox execution
Not guaranteed
- Correct behavior with real-world data
- Availability of external services
- Full test coverage
- Tools requiring credentials/system deps work correctly
Verification is evidence-based, not absolute. It filters out broken packages and scores what it can prove. “Partially Verified” means “not fully testable in a sandbox” — not “broken.”