AI Testing Tools and Data Security: SOC 2, API Keys, and What Actually Leaves Your Environment

AI testing tools handle data security very differently from one another, so the useful question is not "is this tool secure" but "which of my data does it transmit, store, and show to a model." Every AI-powered testing product touches some combination of five sensitive data types: page structure, screenshots, source code, test credentials, and the prompts sent to language models. A SOC 2 Type II report tells you the vendor's controls over that data were audited over time; it does not tell you which data leaves your environment in the first place. This guide gives you both halves: how to trace the data flows, and how to read the compliance claims.

If you are running a security review of an AI test automation tool, or preparing to pass one as an engineering team, this is the checklist.

The five data flows in AI testing

Traditional test frameworks were libraries: your code, your infrastructure, nothing transmitted. AI testing tools add cloud services and model inference, which creates data paths that a security review has to trace one by one.

Flow 1: DOM and page structure

To generate or heal tests, a tool reads the page's DOM or accessibility tree. That structure contains more than layout: form labels, table contents, error messages, and user data rendered into the page. Ask whether DOM snapshots are processed transiently or stored, and where.

Flow 2: Screenshots and recordings

Visual testing, vision-model fallbacks, and run debugging all involve screenshots, and sometimes full video. These are the highest-risk artifacts because they capture exactly what a user would see, including real names, balances, and internal admin views if tests run against realistic data. Retention policy and storage location matter more here than anywhere else.

Flow 3: Source code

Some AI testing tools read your application source to plan tests or diagnose failures. Tools that integrate with coding agents may have code access implicitly through the agent. Establish whether the testing vendor's cloud ever receives source, or whether code stays inside your development environment and only test files are involved.

Flow 4: Credentials and test data

E2E tests log in. That means test-account passwords, API tokens, and seeded data exist somewhere the tool can reach. The spectrum runs from worst case, credentials stored in a vendor database in recoverable form, to best case, credentials injected at runtime from your own secret manager into a test that runs on your own machines and never transmits them.

Flow 5: Model inference payloads

Whatever the tool sends to a large language model or vision model is a data flow of its own: which provider receives it, under what retention terms, and whether it can be used for training. This flow exists even in otherwise self-hosted deployments, which is why it deserves its own line in a review. It also connects directly to the deployment question covered in our guide to on-premise and private cloud AI testing.

What SOC 2 Type II actually means

SOC 2 is an attestation framework from the AICPA in which an independent auditor examines a company's controls against the Trust Services Criteria, most commonly security, availability, and confidentiality.

The Type matters:

• Type I says the controls were suitably designed at a single point in time.
• Type II says the controls were designed and operated effectively over an observation period, typically 3 to 12 months. An auditor sampled evidence across that window.

Type II is the meaningful bar for a vendor you will use continuously, and it is what most enterprise security questionnaires require. Three caveats keep it honest:

• Scope is everything. The report covers specific systems and services. A vendor's SaaS can be in scope while a newer product line or a VPC deployment option is not. Read the system description.
• It is not a data-flow disclosure. A vendor can hold a clean SOC 2 Type II while receiving your screenshots, source, and credentials. The report says they control that data responsibly, not that they never see it.
• Subprocessors carry it forward. Model providers and cloud hosts appear in the report as subservice organizations. Check that the AI provider the tool uses is listed and how the vendor monitors it.

Other frameworks come up in specific industries: ISO 27001 internationally, HIPAA business associate agreements in healthcare, PCI DSS where payment flows are tested. For sector-specific requirements, see AI testing for regulated industries.

Data exposure by architecture

The single biggest security variable is not any vendor's policy but the tool's architecture. This table compares what each data type typically does in a cloud-hosted platform versus a local-first, repo-based tool.

Neither column is automatically right. A cloud platform with a strong SOC 2 Type II report and configurable retention can be a fine choice for testing staging environments with synthetic data. The local-first column exists because some teams cannot accept the left column at all, and because minimizing the data that leaves reduces the review to a short list of narrow, auditable flows.

Bring-your-own-key patterns

"Can I use my own API keys" has become a standard security question for AI tooling, and vendors answer it in three distinct ways:

• BYO model key. You supply your own API key for the language or vision model, so inference happens under your existing agreement with the model provider, with your negotiated retention and zero-training terms. The testing vendor never proxies the payload, or proxies it without persisting it.
• BYO endpoint. A step further: you point the tool at your own model endpoint, such as a cloud provider's hosted model service inside your account. Inference stays within infrastructure you already govern.
• Vendor-held keys with attestation. The vendor calls models with its own keys and covers the flow in its SOC 2 scope and subprocessor list. Simplest to operate, weakest for teams with strict data-processing terms.

When a tool runs inside a coding agent, there is a fourth pattern worth noticing: the testing tool inherits the agent's model access. If your organization already approved a coding agent and its model provider, a testing layer that works through that same agent adds no new model relationship to review.

Security review checklist for AI testing tools

Send these to the vendor in writing:

1. For each of the five data types above: transmitted, stored, or neither? Where, and for how long?
2. Do you hold a SOC 2 Type II report, what is its scope, and can we read it under NDA?
3. Which model providers do you use, are they listed as subprocessors, and is our data excluded from training?
4. Can we bring our own model API keys or endpoint?
5. How are test credentials stored, and can they be injected at runtime from our secret manager instead?
6. Can screenshots and artifacts be stored only in our own infrastructure?
7. What runs locally versus in your cloud, and what breaks if we block egress?
8. What deployment options exist for our internal, non-internet-reachable applications?

A vendor that answers all eight crisply is a vendor whose security team has been through this before. Evasive answers on 1, 3, or 5 are the common failure points.

How Shiplight approaches data security

Shiplight's answer to most of the checklist is architectural: the core workflow runs in your environment, so the sensitive data mostly never leaves.

Tests are readable YAML files in your own git repository, and they execute locally or in your CI with npx shiplight test. The browser runs on your machines, credentials are injected from your environment at runtime, and run artifacts land where your CI already puts artifacts. The MCP plugin that gives coding agents browser automation and test authoring runs locally and requires no Shiplight account or token, so nothing about authoring depends on a vendor cloud. Your tests and your application stay yours, in the literal sense that they are files and processes you control.

Because tests live in the repo, every change to them is a pull request: a human-reviewable, permanently logged diff, which doubles as the audit trail security teams ask for.

For teams that want managed pieces, Shiplight offers hosted CI runners and private cloud or VPC deployment on enterprise plans, with SOC 2 Type II, a 99.99% uptime SLA, and a dedicated customer success manager. Honest scope note: Shiplight is web-focused, and teams wanting a fully outsourced QA service, or native mobile coverage, should look at other categories.

Frequently Asked Questions

What AI testing tools are SOC 2 compliant?

Many established AI testing vendors hold SOC 2 attestations, including Shiplight, which maintains SOC 2 Type II. Rather than working from a name list, which goes stale quickly, verify three things per vendor: that the report is Type II rather than Type I, that its scope covers the specific product and deployment you are buying, and that the AI model providers appear as subprocessors. Any vendor with a genuine report will share it under NDA, and most publish their posture through a trust portal. Treat "SOC 2 in progress" as "not SOC 2 attested yet."

How do AI testing tools handle data security and API keys?

It varies by architecture. Cloud-hosted platforms typically store your tests, credentials, and run artifacts in their cloud and call AI models with vendor-held API keys, relying on SOC 2 controls and subprocessor agreements to protect the chain. Local-first tools keep tests as files in your repo, execute in your own environment, inject credentials at runtime from your secret manager, and limit what reaches a model to narrow, documented payloads. For API keys specifically, the strongest patterns are bring-your-own-key and bring-your-own-endpoint, which keep model inference under agreements you already negotiated.

Can I use my own API keys with an AI testing tool?

With some tools, yes. Support falls into three patterns: supplying your own model API key so inference runs under your provider agreement, pointing the tool at your own model endpoint inside your cloud account, or no BYOK at all, with the vendor calling models on its own keys. Tools that operate through your coding agent effectively inherit the model access you already approved for that agent, which means no new key relationship. If BYOK matters to your security team, ask the vendor which pattern they support and whether it covers every model call, including self-healing and visual analysis, not just test generation.

Do AI testing tools send my source code to the vendor?

Some do, some never touch it. Tools that generate tests by analyzing your codebase may upload source or embeddings of it to their cloud; tools that work from the rendered application read the DOM and screenshots instead. Repo-based tools that integrate with coding agents keep code analysis inside your development environment, where the agent already operates. This is a per-vendor question worth asking explicitly, because the answer often surprises teams that assumed "testing tool" meant "no code access."

What data does self-healing send to an AI model?

Self-healing typically transmits the failed step's intent, the relevant DOM or accessibility-tree fragment, and sometimes a screenshot, so the model can find the element's new location. The security-relevant details are payload minimization, whether the fragment is trimmed or the whole page goes up, retention at the model provider, and auditability of what was sent. Tools that surface heals as reviewable diffs, as described in our guide to self-healing test automation, also give you a human checkpoint on what the model changed.

Is a SOC 2 report enough to approve an AI testing vendor?

No. SOC 2 Type II attests that the vendor's controls operated effectively over an audit window; it does not describe which of your data the product transmits, and it may not cover the exact deployment you buy. Pair the report with a data-flow review using the five flows in this guide, confirm subprocessors, and match the deployment model to your requirements. For teams with regulatory obligations, layer on the sector requirements covered in AI testing for regulated industries.