The best way to get instant PR feedback and block merges with Shiplight in GitLab CI

Fast teams do not lose time because they lack tests. They lose time because the feedback arrives too late, or arrives without enough evidence to trust it.

In GitLab, merge requests already give you the right control point: a single place where code review, design review, and release risk converge. The goal is simple:

• Run UI verification automatically on every merge request
• Return actionable results quickly, with evidence
• Block merges when the verification fails

Shiplight AI fits this workflow cleanly because it runs on top of Playwright while adding a higher-level verification layer: YAML tests, intent-based actions, self-healing locators, and a human-readable HTML report with per-step artifacts.

What instant PR feedback means in GitLab

If you want GitLab to behave like a real quality gate, you want merge request pipelines, not “best effort” branch pipelines.

GitLab’s merge request pipelines are specifically designed to run when a merge request exists, and GitLab can be configured to treat their status as the merge blocker. The critical mechanic is that your .gitlab-ci.yml must include rules that match merge request pipeline events, typically CI_PIPELINE_SOURCE == "merge_request_event".

Once you have that, you can enable the project setting that prevents merging when the pipeline is not green. In GitLab’s merge checks, the “Pipelines must succeed” option blocks merge requests from being merged if the required pipeline fails (and, depending on configuration, even if no pipeline ran).

That is the foundation. Shiplight then makes the feedback fast and reviewable.

Why UI gates usually fail in practice

Most GitLab CI UI gates fail for one of three reasons:

1. They are slow, so developers treat them as background noise.
2. They are flaky, so teams stop trusting red builds.
3. They do not explain themselves, so the only “debug tool” is re-running the job and scrolling logs.

Shiplight is designed to change the economics of that loop. Its CLI runs YAML tests alongside regular Playwright tests, and it generates a report you can review like evidence rather than like a console transcript.

A GitLab CI pattern that delivers fast Shiplight feedback on every MR

Shiplight’s CLI is an npm package (shiplightai) invoked via npx shiplight <command>. It is meant to be installed as a project dependency (not globally).

Shiplight also documents a straightforward CI shape: install dependencies, install Chromium, run npx shiplight test.

Below is a practical .gitlab-ci.yml starting point you can adapt. It is intentionally conservative: one job, one responsibility, and artifacts that reviewers can open.

stages:
- qa

workflow:
rules:
# Create a merge request pipeline for MR feedback
- if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
# Also run on main branch pushes (optional, but recommended)
- if: '$CI_COMMIT_BRANCH == "main"'
- when: never

shiplight_ui_gate:
stage: qa
image: node:22
rules:
- if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
cache:
key: ${CI_COMMIT_REF_SLUG}
paths:
- node_modules/
script:
- npm ci
- npx playwright install chromium
# Run your Shiplight YAML tests (and any Playwright tests) as a gate
- npx shiplight test
artifacts:
when: always
expire_in: 7 days
paths:
- shiplight-report/
- test-results/


Why this works:

• It triggers where review happens. Merge request pipelines exist specifically for MR feedback and are created via CI_PIPELINE_SOURCE == "merge_request_event".
• It matches Shiplight’s supported execution model. Shiplight’s CLI is built for npx shiplight test, and Shiplight explicitly calls out CI usage with npm ci && npx playwright install chromium && npx shiplight test.
• It produces reviewable output. After a run, Shiplight’s HTML report lives at shiplight-report/index.html, and is designed to include per-step screenshots, videos, and traces. Saving the report folder as a GitLab artifact gives reviewers something concrete to inspect.

How to make the feedback feel instant instead of eventually

Speed is not just runtime. It is also how quickly you can narrow the scope of what runs.

Shiplight forwards arguments after shiplight test to Playwright, which means you can do targeted execution without inventing a second toolchain. That enables a pragmatic gating strategy:

• Run a small “PR gate” subset on every merge request (critical paths only).
• Run broader suites after merge, nightly, or on merge trains depending on your release model.

Common patterns teams use:

• Run by folder or project: npx shiplight test my-saas-app/
• Run by name/grep for a gate tag: npx shiplight test --grep "critical"
• Tune parallelism: npx shiplight test --workers 4

This is the simplest way to keep PR feedback tight without weakening your overall regression posture.

Blocking merges the right way in GitLab

To actually prevent risky changes from merging, configure GitLab to treat your Shiplight job as a required signal:

• Enable the merge check that enforces successful pipelines before merge (“Pipelines must succeed”).
• Make sure your merge request pipeline reliably exists by using merge request pipeline rules (CI_PIPELINE_SOURCE == "merge_request_event"), not only push pipelines.

If you use merge trains, GitLab can run pipelines on the merged result as part of the train, which is useful when integration risk is high and multiple MRs are queued.

Where Shiplight adds leverage beyond just run Playwright

You can run Playwright in GitLab CI today. The question is whether your team will still trust and maintain that gate after six months of UI churn.

Shiplight is built to keep PR gates stable as the UI evolves, while keeping results easy to review:

• YAML tests that read like user intent (instead of brittle wiring)
• Self-healing locators and AI-assisted verification in the CLI workflow
• An HTML report designed for fast triage, with step-level artifacts

And if you want to go further than CI artifacts, Shiplight Cloud adds centralized execution, dashboards, and team workflows without changing how developers author tests.