Automate test runs on pull requests with Shiplight AI in a Jenkins pipeline

Pull requests are where product quality either becomes a habit or becomes a scramble. If your team only runs end-to-end coverage after merge, you are effectively asking reviewers to approve UI behavior they cannot reliably observe. If you run UI tests on every PR but those tests are brittle, you get the opposite failure mode: noisy pipelines, flaky failures, and a gradual loss of trust in “red” as a meaningful signal.

Shiplight AI is built for the reality of fast-moving, AI-native development: verify UI changes in real browsers during development, convert those verifications into durable regression tests, and keep them running with near-zero maintenance through intent-based execution and self-healing behavior. Shiplight also fits cleanly into existing CI systems, including Jenkins, because you can run Shiplight YAML tests locally via a Playwright-compatible CLI and still push rich evidence and metadata into Shiplight Cloud when you want centralized visibility.

This post walks through a practical Jenkins setup that automatically runs Shiplight tests on pull requests, publishes artifacts, and optionally reports the run to Shiplight Cloud for team-wide dashboards and AI summaries.

What you are automating, exactly

There are two common ways teams operationalize Shiplight in CI:

• Run Shiplight locally in CI for PR gating. Jenkins executes your Shiplight YAML tests using the Shiplight CLI (npx shiplight test). The build fails when tests fail, just like any other gate.
• Keep execution local, but report results to Shiplight Cloud. After the run, Jenkins uploads the report (screenshots, videos, traces, git metadata) with npx shiplight report when REPORT_TO_CLOUD=true and SHIPLIGHT_API_TOKEN are present. This gives you a central place to triage and trend results without forcing your team to move execution into a separate system.

Both approaches work well together: fast, deterministic gating in Jenkins, plus high-quality evidence and cross-run visibility in Shiplight Cloud.

Prerequisites for a reliable Jenkins PR gate

Before you touch a Jenkinsfile, make sure these basics are in place:

• Shiplight tests exist in your repo. Shiplight’s CLI runs YAML tests locally via npx shiplight test, alongside any existing Playwright .test.ts files.
• Your Jenkins agents can run Playwright browsers. Shiplight’s local runner builds on Playwright. Ensure your agent image or node has the dependencies needed to run Chromium (and any other browsers you use).
• Node.js meets the Shiplight CLI requirements. Shiplight’s docs call out Node.js >= 22 for the CLI.
• Secrets are stored in Jenkins credentials, not in git. Depending on how you author tests, you may need:
  
  
  • An AI provider key such as GOOGLE_API_KEY or ANTHROPIC_API_KEY for AI-driven actions and VERIFY assertions.
  • SHIPLIGHT_API_TOKEN if you want to upload results to Shiplight Cloud.

If you already have a Multibranch Pipeline job building PRs, you are in a good place. If not, set Jenkins up so it actually builds pull requests (not just branches) and exposes PR context variables to the pipeline.

A Jenkinsfile you can drop into a repo

The pipeline below is designed for Multibranch jobs and runs only on change requests. It installs dependencies, runs Shiplight tests, archives the HTML report, and optionally uploads results to Shiplight Cloud.

Adjust paths to match where your test project lives (monorepos often keep tests under a subdirectory like ./tests/e2e).

pipeline {
agent any

options {
timestamps()
ansiColor('xterm')
}

environment {
// Keep browsers and caches inside the workspace to reduce surprises across agents.
PLAYWRIGHT_BROWSERS_PATH = "${WORKSPACE}/.playwright-browsers"
}

stages {
stage('Checkout') {
steps {
checkout scm
}
}

stage('Install test dependencies') {
when { changeRequest() }
steps {
dir('tests') {
sh '''
node --version
npm ci
npx playwright install chromium
'''
}
}
}

stage('Run Shiplight tests on PR') {
when { changeRequest() }
steps {
dir('tests') {
withCredentials([
string(credentialsId: 'ANTHROPIC_API_KEY', variable: 'ANTHROPIC_API_KEY'),
string(credentialsId: 'GOOGLE_API_KEY', variable: 'GOOGLE_API_KEY'),
string(credentialsId: 'SHIPLIGHT_API_TOKEN', variable: 'SHIPLIGHT_API_TOKEN')
]) {
sh '''
# Run the suite. Shiplight writes an HTML report to shiplight-report/ by default.
npx shiplight test
'''
}
}
}
}

stage('Upload results to Shiplight Cloud (optional)') {
when {
allOf {
changeRequest()
expression { return env.SHIPLIGHT_API_TOKEN?.trim() }
}
}
steps {
dir('tests') {
sh '''
# Upload the existing local report directory to Shiplight Cloud.
REPORT_TO_CLOUD=true npx shiplight report
'''
}
}
}
}

post {
always {
// Archive the Shiplight HTML report and artifacts for easy retrieval from Jenkins.
archiveArtifacts artifacts: 'tests/shiplight-report/**', allowEmptyArchive: true

// If you use the HTML Publisher plugin, you can also publish the report as a build page.
// publishHTML(target: [
// reportDir: 'tests/shiplight-report',
// reportFiles: 'index.html',
// reportName: 'Shiplight Report',
// keepAll: true,
// alwaysLinkToLastBuild: true,
// allowMissing: true
// ])
}
}
}


Why this structure works in real Jenkins environments

• It is PR-specific. when { changeRequest() } keeps PR gating focused. You can add separate stages for mainline nightly runs later.
• It is artifact-first. Jenkins remains the enforcement point, but you still get a portable, reviewable evidence bundle (shiplight-report/).
• It keeps secrets out of the filesystem. Shiplight can read a .env locally, but CI is a better place to inject environment variables from a secret store.

Making PR runs faster without weakening the gate

A good PR gate is fast enough that developers do not route around it. Shiplight helps by reducing the classic UI automation tax (selector churn and fragile assertions), but you still want smart pipeline ergonomics:

• Shard the run when suites grow. Shiplight supports Playwright shard flags on shiplight test, and shiplight report --merge ... can merge shard reports into a single upload when you report to cloud.
• Use Playwright’s storage state for login. Shiplight’s docs outline a standard Playwright setup project pattern that logs in once, saves storageState, and reuses it across tests. That keeps PR runs both faster and more stable.
• Keep your tests intent-first. Shiplight’s value shows up when your tests describe user outcomes instead of brittle mechanics. That is what makes PR checks survive UI refactors without turning every PR into a test repair project.

Where Shiplight fits if you already have Playwright in Jenkins

If you already run Playwright tests in Jenkins, Shiplight is not a replacement for disciplined engineering. It is an upgrade to how reliable your UI checks can be under constant UI change.

Shiplight’s local runner is Playwright-based, and YAML tests can live alongside existing .test.ts files. That means adoption does not have to be a “big switch.” Many teams start by adding Shiplight YAML coverage for the flows that create the most PR pain: authentication, navigation, billing, onboarding, and the core revenue path. As those stabilize, the PR gate becomes something your team trusts again.

A pull request gate that stays trustworthy

The goal is not “more tests.” The goal is a PR workflow where:

• Developers get fast feedback in Jenkins.
• Reviewers can rely on evidence, not hope.
• Teams stop paying the maintenance penalty that makes UI testing feel like a tax.

Shiplight gives you a practical way to run durable, intent-based UI tests on every pull request in Jenkins, and to graduate into richer cross-run visibility when you want it, without rewriting your CI system.