How to Evaluate a Tool That Claims to Be Agent-Native

"Agent-native" is not a verifiable claim. The five questions below are, and every one of them can be settled during a trial rather than argued about in a meeting: can my agent call it, where does the output live, can the agent read the result back, what happens when something fails, and where exactly is the human.

Gartner named the problem before most of these products shipped. It called the failure mode agent washing: agentic and agent-native language applied to what is actually rule-based automation, a chatbot, or process automation with a model in front. Its research line estimated that only a small fraction of vendors claiming agentic capability were building genuinely agentic systems, and forecast that a large share of agentic AI projects would be scrapped before the end of 2027.

That is not a reason to be cynical about the category. It is a reason to evaluate the mechanism instead of the adjective. What follows is vendor-neutral by construction, and it works equally well on the tool you already bought.

Question 1: Can my agent call it?

The whole architecture rests on this. If the only way to make the product do something is for a person to click it, then an agent cannot operate it, whatever the homepage says.

What to ask. Do you publish an MCP server, a command-line interface, or agent skills? Is it documented publicly, with the exposed tools and their parameters listed? Can I install it into my own agent without a sales conversation?

What a good answer sounds like. A URL to install docs, a list of exposed tools, and a one-line install command you can run during the call. The vendor should name the agents it has been tested with rather than gesturing at a count.

Red flags. A demo of their console offered in place of an answer. "We have an API" with no agent-callable surface on top of it, which means you are being asked to write the integration. An MCP server that wraps only authentication and reporting, with the real work still in the console. A roadmap date instead of a link.

The ten-minute check. Install it into your own agent, in your own repository, and ask the agent to do the simplest real task the product claims to do. Watch whether the agent gets a usable response or asks you to open a browser.

Question 2: Where does the output live?

Every tool produces an artefact: a test, a config, a report, a policy. Where that artefact is stored decides who can maintain it later.

What to ask. What does the tool produce, and where does that file end up? If it lives in your cloud, can I export it? Does the export still run after export, or is it a read-only record of something that used to run?

What a good answer sounds like. The artefact is a file in your repository, in a format you can read in a pull request. Export is not a special request because there is nothing to export: it is already yours.

Red flags. "Everything is in the platform, so there is nothing to manage." An export that produces a JSON dump nothing can execute. A format the vendor describes as proprietary. A migration-off story beginning "our team can help with that", which means services engagement rather than file copy.

Why this one matters more than it looks. An artefact in a vendor database is not editable by an agent in any meaningful way, so the agent-native claim collapses behind it. It is also the entire lock-in question in one check, which is why the answer is often the vaguest in the evaluation.

Question 3: Can the agent read the result back?

Triggering an action is half a loop. An agent that can start work but cannot find out what happened has to hand the task back to a person, which is the condition the tool was supposed to remove.

What to ask. When my agent invokes this, what comes back? Show me the raw response, not the dashboard. Does a failure return structured detail or a status code and a link? Can the agent retrieve run artefacts such as logs, screenshots, and traces through the same callable interface?

What a good answer sounds like. Someone opens a terminal and shows you an actual response payload, containing what happened, in text, with enough specificity that a model could act on it without a screenshot.

Red flags. "You will see it in the dashboard." A response that is a run ID and a URL. Beautifully designed reporting that exists only as rendered HTML. Results delivered by webhook to your server but unreadable by the agent that started the run.

This is the property most often missing in products that pass question 1. Callable actions are easy to bolt onto an existing console. Making the outcome legible without a person interpreting a chart is a deeper change, because it means the reporting was designed for a reader that cannot see.

Question 4: What happens on failure?

Failure handling is where the difference between an assistant and an operator becomes visible, and it is the question most demos are structured to avoid.

What to ask. Break something on purpose during the trial. What does the tool tell me? Does it distinguish the tool being wrong, the configuration being wrong, and the system under test being wrong? What does it do next without being asked?

What a good answer sounds like. A diagnosis with a proposed fix, plus a statement of which category the failure falls into. Critically, the tool must be able to conclude that the fault lies outside itself. A system whose only possible outcome is a green run will eventually manufacture one.

Red flags. A red mark and a timestamp. A failure message that only says something did not match. Auto-repair that silently rewrites the artefact with no diff. Any answer where every failure resolves to "we retried and it passed".

The economics are worth stating plainly. The expensive part of a failure is rarely the fix. It is the ambiguity: a person sitting with a red result at nine in the morning, deciding whether it is a real bug, a stale artefact, an infrastructure hiccup, or config drift. A tool that shortens the distance between a failure and the right resolution path is doing the work. A tool that only reports the failure has moved the triage cost, not removed it.

Question 5: Where is the human?

Every serious definition of agent-native includes human supervision, so the question is never whether a person is involved. It is where in the sequence they sit.

What to ask. Walk me through a full cycle and name every point where a person is required. Which of those are approvals and which are operations? What can the system merge, deploy, or delete unapproved?

What a good answer sounds like. The human appears at review and approval, near the end, on a named gate: a pull request someone merges, an artefact someone reads. The vendor should be comfortable saying out loud what the product will not do on its own.

Red flags, in both directions. A human required in the middle of the loop, approving each step, means the agent is not the operator and you have bought a faster console. A system with no human gate anywhere is the opposite problem: something that can merge its own changes will eventually merge a change that makes the check pass without making the software correct. The good shape is autonomy in the middle and supervision at the edge.

The scorecard

What to ask for in the demo

Requests, not questions, since a request is harder to answer with a slide.

1. Install it into my agent, on my repository, while I watch. Not a sandbox, not a prepared account.
2. Show me the raw response from one tool call, in a terminal.
3. Show me the artefact it produced, in my editor, as a file.
4. Break something and show me what the tool says about it.
5. Show me the point where a human is required, and what happens if that human is on holiday.
6. Show me what "we stop paying you" looks like: what still runs, what I keep, in what format.

The last is the most informative and the least asked. A vendor confident in the architecture answers it quickly, because the honest answer is short.

What this checklist deliberately does not test

Two limits worth being explicit about, because a checklist that oversells itself is the same problem in a different coat.

It tests operability, not accuracy. Every question above can be satisfied by a tool that is agent-callable, file-producing, readable, diagnostic, and reliably wrong about your application. Quality cannot be judged from a demo: run a pilot on your own product, with your own flows. For testing tools specifically, those dimensions are covered in how to evaluate AI test generation tools.

It does not reward breadth. A long integration list or a wide feature matrix tells you little about whether the loop closes. A tool that does one bounded thing precisely and reports back clearly is usually worth more inside an agent workflow than a broad platform with a thin callable surface, because the orchestration happens in your agent, where the context about your codebase already lives.

There is also a real cost to insisting on this architecture: fewer products qualify. If a console-first tool is genuinely the best fit for a workflow a person operates anyway, this is the wrong instrument for that decision. It is the right one when the intent is for your agent to own the work.

Shiplight, against its own checklist

It would be unreasonable to publish a checklist and dodge it.

Can your agent call it? Yes. Shiplight installs into your coding agent as a browser MCP server plus skills, with a copy-paste install for Claude Code, Cursor, VS Code, Codex, Gemini CLI, Claude Desktop, Zed and others. Commands run as /shiplight verify, /shiplight create-yaml-tests, and /shiplight fix, and local browser automation and authoring need no Shiplight account or token.

Where does the output live? In your git repository, as plain YAML written from intent rather than brittle selectors. You read it in a pull request like a spec, and you can change it by editing the file. Each run transpiles that YAML to Playwright, so nothing is trapped in a proprietary runtime and there is a real exit.

Can the agent read the result back? Yes. Runs return structured results the agent consumes directly, along with screenshots and traces available in the local debugger for the cases where a person wants to step through.

What happens on failure? /shiplight fix reproduces the failure, diagnoses the root cause, and proposes a repair. When the application itself is broken rather than the test, it reports the bug instead of editing the test to make it pass, which is the behaviour question 4 exists to catch. Deterministic steps cache their resolved locator, and on a miss the step re-resolves at run time and the run continues, so tests adapt without waiting on a commit.

Where is the human? At review and at merge. The tests are readable YAML you change like a spec, and in CI Shiplight will diagnose, fix, rerun, and open a pull request. It never auto-merges.

Two honest limits. Authoring runs through your coding agent's own subscription and costs nothing from us, but executing a test file needs an LLM key either way: ours, which spends credit, or your own, which your provider bills. And Shiplight covers web applications, so if your critical flows are native mobile or desktop, this is not the tool for that part of your estate. Every core capability is on the free plan, including bringing your own LLM key; Pro is $60 a month and Enterprise is contract priced, with SOC 2 and a dedicated CSM.

FAQ