Agent-Native CI: What Changes When the Agent Triages Its Own Failures

Continuous integration was designed around a person. The build goes red, a webhook fires, a message lands in a channel, and someone opens the run, scrolls the log, finds the assertion that failed, decides whether it means anything, and either fixes it or reruns it. Every part of that pipeline, the notification format, the log layout, the artefact retention window, exists because a human was going to be the reader.

That assumption is now optional. An agent can fetch the run logs through the CI provider's API, download the artefacts, look at the screenshot taken at the moment of failure, read the network activity in the trace, form a diagnosis, and act on it. What changes is not the speed of the fix. It is which step a human is doing.

A notification is not a diagnosis

The default output of a failed CI run is a notification: a red X, a workflow name, a link. It transfers the fact of a failure and nothing else, so everything expensive happens after the click.

A diagnosis is a different artefact. It names the failing test, states what the run actually did versus what the test expected, points at the evidence, and proposes a resolution path. Something closer to: the checkout test failed at the "order confirmation appears" step; the screenshot shows a 500 page; the trace shows POST /api/orders returning 500 with a null constraint violation on promo_code; this is an application bug and no test change will fix it.

The difference between those two outputs is roughly twenty minutes of a senior engineer's morning, multiplied by however many red builds landed overnight. The second output has been rare not because it is hard to imagine, but because producing it required someone to look at pictures, correlate them with a stack trace, and know the codebase.

Artefacts matter more than logs when the agent is the reader

Most CI tooling optimises the log. Coloured output, folded sections, a summary block at the end. That optimisation was for human scanning, and it does not carry much information.

Here is the log line a browser test failure usually produces:

Error: expect(locator).toBeVisible() failed
Locator: getByRole('heading', { name: 'Order confirmed' })
Expected: visible
Timeout: 30000ms exceeded

That text is compatible with at least six different root causes. The heading could have been renamed. The order could have failed server-side. A cookie banner could be covering it. The test user's session could have expired. The seed data could be missing. The deploy under test could have been the previous commit.

The artefacts separate those cases immediately:

• The screenshot at the moment of failure usually resolves it in one look. An error page, an empty state, a modal covering the target, a login screen where the app should be. This is the single highest-value artefact and it is the one most pipelines forget to upload on failure.
• The trace carries the network calls, the console errors, and the DOM at each step. A 500 response in the trace ends the argument about whether the test is at fault.
• The step-level history shows how far the run got. A test that failed at step 2 of 14 is a different story from one that failed at step 13.

An agent reads all of these: the screenshot as an image, the trace as structured data, the log as one input among several rather than the only one. That inverts an old priority. For agent-readable CI the artefact upload matters more than the log formatting, and it has to run on failure, which means unconditionally rather than only on success.

One practical detail follows. Full traces and video recordings are large, and the agent does not read the video at all. Uploading a slimmed artefact for the triage path cuts most of the weight without removing anything the agent uses, while the full report still goes to the dashboard for humans who want to scrub through a recording.

The expensive part is the ambiguity, not the fix

Ask a senior engineer what the worst part of a red Monday build is and the answer is rarely "writing the fix". It is not knowing which of four things happened:

The classification is the work. Once you know which row you are in, the action is usually obvious and often small. That is why triage automation is worth more than fix automation: getting from "red" to "this is a config problem, BASE_URL points at staging and the deploy went to preview" removes the part that cost the time.

It is also why the classification has to be evidence-bound rather than pattern-bound. A rule that says "timeout means flaky, retry it" will retry a real outage sixteen times. Every classification should be traceable to something in the artefacts, and when the evidence is thin, the honest output is "cannot classify, here is what I saw".

The loop, concretely

Here is the shape of it, using the way it runs in GitHub Actions since that is the environment where the pieces line up.

1. The test workflow runs and goes red. It uploads two things: the full report to a dashboard for humans, and a slimmed report artefact for the triage path. Both uploads run whether or not the tests passed.
2. A second workflow triggers off the first one's completion. In GitHub Actions this is the workflow_run trigger, watching the test workflow by its exact name. This decoupling matters: triage is not a step inside the test job, so it cannot slow down or destabilise the job that produces the signal.
3. A read-only diagnosis job runs first. It pulls the run logs and downloads the artefacts, and the agent produces the classification and the diagnosis. This job needs no write permissions and no application credentials, which keeps the privileged surface small.
4. The diagnosis is posted where the team already looks. A Slack message or a PR comment carrying the failing test, the classification, the evidence, and the proposed action. For an application bug, a configuration problem, or an infrastructure failure, this is the terminal state. The agent stops. A human picks it up with the ambiguity already removed.
5. Only for a classified test issue does a second job attempt a fix. This one is privileged: it needs to write to the repository, and it needs the same environment credentials the test workflow had, because it is going to rerun the failing test to prove the fix.
6. The fix is proven by a rerun, on the same commit. A test change that has not been rerun is a guess. A rerun against a different commit proves nothing about the failure you started with.
7. The result arrives as a pull request with the diagnosis attached, so the reviewer reads why the test changed rather than reverse-engineering it from the diff.

The whole loop runs unattended and ends at a pull request. It does not end at a merge.

What stays human: the merge gate

There is one rule in this design that is not a preference. The agent never merges.

The reason is structural rather than cautious. The purpose of a test suite is to be an independent check on the code. An agent that can both change the test and merge the change has removed the independence, because the check and the thing being checked now share an author with no reviewer in between. Everything else in the loop can be automated without touching that property. Merging is the one step that cannot.

Practically this means the pull request goes through the same review the team uses for any other change: required reviewers, required checks, branch protection. The agent is a contributor, and contributors do not approve their own work.

Guardrails, or: how an agent could hide a regression

The obvious failure mode is an agent that makes red go green by weakening the test. It is worth being specific about the ways that happens and what stops each one.

Editing application code to satisfy a test. The strongest guard is a path allowlist: the fix job may only write to the directories that hold tests and test fixtures, enforced by the job configuration rather than by instruction. An agent that cannot write to src/ cannot make a failing test pass by changing the product.

Deleting or loosening assertions. This is the subtle one. Replacing a check for "Order confirmed" with a check that the page is not blank technically fixes the red build and destroys the test's value. Two things help. First, the diff arrives as a pull request, so a human sees it. Second, treat any diff that removes an assertion, widens a timeout, or converts a specific expectation into a general one as requiring explicit human attention, and say so in the review checklist. Assertion removal is a code smell with a signature, and it is worth grepping for in review.

Classifying an application bug as a test issue. The classifier's default has to be conservative: when the app returned an error or produced incorrect state, report the bug and stop, even if a test change would make the run pass. This is a design decision in how the triage agent is prompted and evaluated, not something the CI configuration can enforce, so it deserves auditing. Sample the last month of triage outputs occasionally and check that the application bugs were called application bugs.

Retry loops that grind through budget. Cap the attempts: one diagnosis, one fix, one rerun, then a human gets it. An agent that retries indefinitely converts a test failure into a billing incident.

The privileged job itself. The fix job runs with write access to the repository and live credentials for the rerun. Pin the workflow it calls to an immutable tag rather than a moving branch, and keep the diagnosis job read-only so the common path never touches the privileged one.

A worked example

Shiplight does exactly this loop, so it is a fair illustration of the shape rather than a hypothetical.

The coding agent authors end-to-end tests as readable YAML that lives in your repository. In CI, the test workflow runs those tests and uploads both a full report for the dashboard and a slimmed artefact for triage. A separate triage workflow triggers on the test workflow's completion, reads the run logs and the artefacts, and posts a diagnosis. For failures it classifies as test issues it applies the fix, reruns the test to prove it, and opens a pull request. When the application itself is broken, it reports the bug instead of editing the test. It never auto-merges. The fix job is constrained to an explicit list of top-level directories it may edit.

Two honest limits. This triage pipeline is built on the workflow_run trigger and reusable workflows, so it is a GitHub Actions capability specifically, not a generic CI feature. And running tests needs an LLM key, ours or your own, on any plan including the free one. Authoring through your coding agent runs on that agent's own subscription and needs no Shiplight account, but execution is a separate thing and it is worth being clear about which is which.

What this does not fix

Agent triage removes the ambiguity tax. It does not remove the need for tests worth trusting. A suite of flaky tests fed into an automatic fixer produces a stream of pull requests that each make one flaky test slightly more permissive, and the end state is a green suite that checks nothing.

The ordering that works: get the suite deterministic first, then automate triage. Triage automation is an amplifier, and what it amplifies is whatever your tests already were.

FAQ