Developer productivity · AI agents · 2026
Making API tests the AI agent's verify step
As AI coding agents took on more of our engineering work, they produced changes faster than humans could verify by hand—and unit tests alone don't exercise live services or real data. I started the team's Bruno API test repository and built the guardrails for agents to run it safely: read-only defaults, runtime secret injection, and a single pass/fail verdict. Running the suite against a local or sandbox deployment is now the final gate before I open a pull request.
Context
The team's services (product feeds, product search, a GraphQL API and an admin tool) had their API calls in Postman collections. In early 2026 I moved those collections into one Bruno repository in git. Bruno stores each request as a plain text file and has a CLI, so the same tests run from the desktop app, from a shell script or from an agent.
What made it safe to hand to an agent
- Tags as the trust boundary. On day one I tagged about 80 requests as
unsafe: anything that writes, deletes, launches a job or generates a feed. The default run excludes them, so an agent's default run against live nonprod and production services is read-only. Mutating runs are a deliberate choice, made against a sandbox. - Secrets come from the password manager at run time. A wrapper script fetches credentials and hands them to the CLI, so the agent runs the script instead of handling the secret.
- One environment per sandbox. Besides dev, prod and local, the repository has environment files for its sandboxes and feature work. Each new endpoint deployed to its own sandbox and passed a Bruno suite before merge.
- Instructions the agent reads. The repository carries an agent guide: how to run each collection, which tags to exclude, how the services authenticate. The agent skills I wrote for the team call the same commands, turning service startup and verification into one routine.
- A verdict, not a log. For the search re-platform I added a script that runs every search core through the API, on nonprod, production or both, and prints one pass or fail. An agent, or a person at the end of a long day, can act on that without reading output.
What it caught
- A dead core behind a 200. The API returns HTTP 200 with empty data when a backing search core goes down, so a status-code check passes on a dead core. The smoke test checks the data that comes back, and during testing on the new AWS cluster it flagged a core that was down. I rebuilt the core with the cluster's deploy tooling, and the next run passed.
- A write that reports success. The suite also catches writes that return HTTP 200 without mutating state. The Solr 9 gate suite, which coding subagents built under my direction, verifies that the index actually changed after a write instead of trusting the response. That suite covered schema parity, plugins, reads, the write path and the import jobs, and every gate passed in the lab.
- New features, verified where they run. New API features, such as a spec-sheet API and the RetailMeNot search work, shipped with Bruno suites that passed on their sandbox before merge.
What didn't work at first
- Secrets that the desktop app could see and the CLI couldn't. Bruno keeps secret variables in the desktop app's local store, not the repository. A token request that worked in the app failed in an agent's CLI run. Fetching the token inside the run, from the password manager, fixed it, and the agent guide now documents that pattern.
- Tags only protect what's tagged. The largest collection is fully tagged; some older ones still hard-code a full URL. The agent guide says to read a request's URL before running anything unfamiliar.
My part: I started the repository, migrated the team's Postman collections, set the tagging and secrets conventions, wrote most of the suites and the agent guide, and trained the team on the CLI and on using Bruno for fast iteration. Teammates add their own suites, including ones for merchant and affiliate APIs. The team's per-pull-request environment build was a teammate's. AI coding agents (Claude Code) helped write the newer suites and run them under my direction; I review the results.
What I carry forward
An agent is only as trustworthy as the check at the end of its loop. Give it a test it can run safely against real services, make the safe run the default, and make the result one word. Then the human review is about the design, not about whether the endpoint works.