CI/CD for Agent-Written Code

CI/CD for AI-generated code needs one stage that most pipelines do not have yet: a PR-time behavioral gate that proves each pull request behaves correctly in a real browser, not just that its unit tests compile and pass. When a coding agent writes the change, the code and the tests that cover it can be authored in the same pass, by the same model, from the same misreading of the requirement. A green unit suite under those conditions confirms that the code does what the agent thought it should do. It does not confirm that the feature works.

That gap is the organizing problem of this guide. The rest of your pipeline can stay the same. What changes is that you add a verification stage between "tests pass" and "safe to merge," and you define merge criteria in terms of observed behavior rather than exit codes. This matters more as the share of agent-authored PRs climbs, because the failure mode is no longer a syntax error a linter catches. It is a plausible-looking change that ships a broken checkout flow with a full green check.

The stages below assume a standard trunk-based flow: a coding agent opens a PR, CI runs on the pull request, branch protection requires certain checks, and a merge queue serializes merges into the main branch. Each point is a place to add behavioral verification, and each stresses differently under agent-heavy volume. We walk the stresses first, then the stages to add, then GitHub Actions setup with a config example.

How an AI-heavy PR flow stresses the pipeline

The first stress is volume. Teams that adopt coding agents report going from single-digit pull requests on a busy day to 30 to 50 per day, one analysis putting the average jump from 8 to 35 PRs per day (Autonoma). Pipelines sized for a human review cadence buckle: if a suite runs 45 minutes and the team merges 40 PRs a day, that is roughly 30 hours of CI capacity needed daily just to hold the queue steady.

The second stress is batch size, and it is the one that actually breaks quality. The 2024 DORA report found that AI adoption came with an estimated 7.2% reduction in delivery stability even as it raised throughput, and it tied the effect to larger changesets: AI makes it easier to write more code, and larger batches carry more risk (Google Cloud, DORA 2024). Agent-written PRs also tend to touch more files than a human would, because the agent does not restrict itself to the minimum surface area a change needs.

The third stress is review attention. In the same research, 39% of respondents reported little to no trust in AI-generated code, yet review capacity has not scaled with PR count. The human reviewer who used to be the behavioral check is now spread across three times the pull requests, so something automated has to hold the line the reviewer used to hold.

None of this is solved by adding runners. More capacity lets you run the same shallow checks faster. The question is what to run, and where, so a broken behavior cannot reach the main branch behind a passing build.

The stage most pipelines are missing: behavioral verification

A typical PR pipeline runs lint, type-check, unit tests, and maybe an integration test against mocked dependencies. Every one of those can pass on a change that renders a blank page. They verify the code's internal consistency. They do not exercise the product the way a user does.

Behavioral verification closes that gap by driving the actual application: loading the real UI, performing the user action, and asserting on the observed result. For agent-written code this is the layer that catches the class of defect agents produce. When the agent writes both the feature and its unit test, the two can be wrong together in a way no amount of unit coverage will reveal. An end-to-end check written from user intent, and maintained independently of the change under review, is what disagrees with a confidently wrong PR.

This is why the quality gate for AI pull requests is built around real-browser end-to-end tests rather than more unit coverage, and why teams automating testing in AI-native pipelines treat the agent-native E2E layer as distinct from data, retrieval, and model-judge checks. The point of the stage is a test that can fail even when the code and its unit tests agree with each other.

Pipeline stages to add for agent-written code

Three additions turn a standard pipeline into one that can be trusted with agent-authored volume.

1. A PR-time behavioral gate

Run a focused set of end-to-end tests on every pull request, triggered on the pull_request event so the check is associated with the PR and can be made required. Keep this subset fast, ideally under 10 minutes, because feedback that lands more than about 15 minutes after a push arrives after the author has lost context. Cover the critical user paths and the flows the changed files touch, not the whole regression suite. This gate answers one question: does the product still behave correctly with this change applied?

2. Merge criteria defined by behavior

A required status check only blocks a merge if branch protection requires it, and only blocks reliably if it runs on pull_request rather than push alone. All required checks must pass against the latest commit SHA before a PR can merge (GitHub Docs). Add the behavioral gate to the required set and define "green" as the browser-level checks passing, not merely that the build compiled. This is the line that keeps a plausible-looking but broken change out of the main branch.

3. A merge queue as the final behavioral checkpoint

At high PR volume, a check that passed against an out-of-date base branch is not proof the merged result works. A merge queue groups each PR with the latest base branch and the changes ahead of it, then requires the checks to pass on that combined result before merging in first-in-first-out order (GitHub Docs). It creates temporary branches to validate each group and removes any PR whose required checks fail. For your behavioral gate to count here, its workflow must also trigger on the merge_group event, so it runs against the queued combination and not only the isolated PR. This is where two agent-written PRs that each pass alone but conflict behaviorally get caught before they land together.

Concrete setup: GitHub Actions with a behavioral gate

The general three-tier structure of E2E in CI, smoke on PR, full suite on merge, extended runs nightly, is covered in the E2E testing in CI/CD setup guide and the GitHub Actions E2E testing walkthrough. What is specific to agent-written code is wiring the behavioral gate to run on both pull_request and merge_group, and pointing it at tests in your repository rather than an external service.

A minimal workflow looks like this:

# .github/workflows/behavioral-gate.yml
name: Behavioral Gate
on:
  pull_request:
    branches: [main]
  merge_group:          # run against the queued combination too

jobs:
  verify:
    runs-on: ubuntu-latest
    timeout-minutes: 10
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
      - run: npm ci
      # Run the repo-owned YAML E2E tests against a preview URL
      - run: npx shiplight test --suite critical-paths
        env:
          BASE_URL: ${{ steps.preview.outputs.url }}

Two properties make this suitable as a merge gate for agent PRs. First, the tests are checked into the same repository as the code, so a PR that changes behavior and the test that guards it are reviewed in the same diff, and there is no vendor cloud that has to stay in sync with your branch. Second, the same npx shiplight test command runs locally, so the coding agent can run the gate before it opens the PR, and the pipeline runs the identical suite to enforce it. To make the check block merges, add "Behavioral Gate" to the required status checks in branch protection, and, if you use a merge queue, confirm the merge_group trigger is present so the check reports on queued groups.

Where the verification comes from

The behavioral stage needs tests that are cheap enough to keep current at agent speed and honest enough to disagree with a confident PR. Shiplight is built for this position in the pipeline. It installs into the coding agent as an MCP server and a set of skills, giving the agent eyes and hands in a real browser: a /shiplight verify command to confirm a UI change looks right during implementation, a /shiplight create-yaml-tests command where the agent walks the app and writes end-to-end coverage, and a /shiplight fix command that reproduces a failure and root-causes it, reporting a real bug instead of quietly editing the test when the app is what broke.

The tests are readable YAML written from intent rather than brittle selectors. They live in your git repository and run locally with npx shiplight test, or on Shiplight-hosted CI runners for enterprise teams that want the same YAML on managed infrastructure with SOC 2 Type II and a 99.99% uptime SLA. They are Playwright-compatible and run alongside existing Playwright tests, so adding the gate is additive rather than a rip-and-replace. Self-healing happens in a real browser, and heals surface as reviewable PR diffs rather than silent rewrites, which keeps the gate trustworthy as the UI moves. Teams have reached reliable coverage of critical flows in days, with a first suite of around 300 tests realistic inside the first week.

The through-line is continuous verification of AI code: when the agent writes most of the change, the pipeline earns its trust by observing behavior at PR time and at the merge queue, on tests the team can read and the agent can maintain.

Key Takeaways

• Unit tests passing is not a merge signal for agent PRs. When the agent writes the code and its tests together, both can be wrong together. Add a behavioral stage that can disagree.
• Gate on behavior, not exit codes. Run end-to-end checks on pull_request, make them required in branch protection, and define "green" as the browser-level result.
• Use the merge queue as the last behavioral checkpoint. Trigger the gate on merge_group so it validates the queued combination, catching PRs that pass alone but conflict together.
• Keep the gate fast and repo-native. Under 10 minutes, tests in your git repo, the same command locally and in CI, so the agent can pre-check before opening the PR.

Frequently Asked Questions

How do you set up CI/CD for AI-generated code?

Keep your existing lint, type-check, and unit stages, then add a behavioral verification stage between "tests pass" and "safe to merge." Run a focused end-to-end suite on the pull_request event, add it to your required status checks in branch protection, and, if you use a merge queue, also trigger it on merge_group. Point the gate at tests that live in your repository so the change and its coverage are reviewed together.

Why are unit tests not enough for agent-written code?

A coding agent often writes the feature and its unit tests in the same pass, from the same interpretation of the requirement. If that interpretation is wrong, the code and the tests are wrong together and the suite still passes. A real-browser end-to-end check exercises the product the way a user does, so it can fail even when the unit tests agree with the code.

How does a merge queue help with high PR volume from agents?

A merge queue validates each pull request against the latest base branch plus the changes ahead of it, in first-in-first-out order, merging only when the required checks pass on that combination. That catches two agent PRs that each pass in isolation but break when combined. For your behavioral check to apply, its workflow must trigger on the merge_group event.

Where should behavioral tests for AI-generated code live?

In your own git repository, alongside the code. Repo-owned tests are reviewed in the same diff as the change they guard, run identically on a developer machine and in CI, and avoid a separate vendor cloud that has to stay in sync with your branches. Shiplight authors these as readable YAML that runs with npx shiplight test and is Playwright-compatible.