Test beyond the happy path

Test the tests and the gate

A passing test helps only if it can catch the relevant defect. Try a deliberate faulty variant in a separate lab copy. Distinguish a hook running, warning and blocking; a missing check is not a pass.

A door alarm differs from a lock.

Running is different from blocking

A post-action hook cannot prevent an action that already happened. A crash, missing check or timeout must stay visible rather than become a pass.

A door alarm differs from a lock that keeps the door closed.

Practice

Complete submitOnce according to its contract.

Edit only submitOnce in model.mjs, preserving original tests. Same requestId/canonical input replays the receipt; changed input throws REQUEST_CONFLICT; do not mutate state. Run quality and earlier checks; report the single-process/in-memory limit.

Check your work

Replay, conflict and distinct request IDs are tested.

Function checks and agent evaluation are distinguished.

References

• Demystifying evals for AI agents — Anthropic Engineering
• Testing Agent Skills with Evals · Dominik Kundel, Gabriel Chua — OpenAI Developers
• Claude hooks — Claude Code Docs
• Anthropic agent evaluations — Anthropic Engineering
• OpenAI iterative repair loop — OpenAI Docs
• CMU assessment alignment — CMU Eberly Center

• Back to the AI Builder roadmap
• Understand AI agents
• Shape the idea
• Set up your workspace
• Give a clear task
• Keep project rules clear
• Design a usable journey
• Build the first feature
• Check access rules
• Find and fix a bug
• Prepare a release
• Observe and respond
• Add tools when needed
• Split work and hand it over