QA engineer

AI adoption by role

The QA engineer (quality assurance engineer) decides whether the tests and evaluations prove the change is ready. With AI in the picture, that means two kinds of testing side by side: deterministic tests for behavior that must be exactly right every time, and probabilistic evaluation for AI steps whose output varies. This track covers QA engineers, test engineers and SDETs (software development engineers in test) working in the first line.

QA evidence is first-line evidence. It is necessary, and it is not independent validation. Numbers on this page are illustrative.

1. At a glance

Field	Answer
Mandate	Prove the product does what the criteria say, on every risk slice, and that the evidence can be reproduced on any version
Primary outcome	A test and evaluation record the release manager can rely on and an independent validator can rerun
First safe win	Use the assistant to brainstorm edge cases from the acceptance criteria, then write the cases yourself
Biggest failure mode	Generated tests that mirror generated code, pass, and share its blind spot

Line: 1st line · Family: Assure · Variants: QA engineer, SDET or test automation engineer, AI evaluation engineer

Variant	What differs
QA engineer	Owns test design, risk slices and test sign-off for the product
SDET or test automation	Builds the harness, fixtures and CI (continuous integration) jobs that run tests and evaluations on every build
AI evaluation engineer	Specializes in evaluation sets, scoring rules, judge calibration and statistical thresholds for AI steps. Still first line: works with the build team

2. Your journey: first steps to fully adopted

This is the path from your first month to a fully adopted way of working. The order of the steps is the same in any organization; only the weight of the controls changes.

Start here: your first steps

Classify every check as deterministic or probabilistic before you write a test.
Confirm that every test fixture is synthetic, and remove any real data you find.
Brainstorm edge cases with the assistant from the acceptance criteria, then write each case yourself.
Write the date-only slice at 100% and add a seeded bad build that a good suite must fail.

Fully adopted: how the work runs

Moment in the work	Before AI	Fully adopted (Stage 4–5)	Decision that stays human
Criteria arrive	Tests are written after the code, from the code	The test strategy splits exact and variable behavior from the criteria; slices and negative paths come from the business analyst	QA owns the strategy; the product owner sets slice priority
Test generation	Generated tests mirror generated code	An agent generates tests from criteria in a separate session; a human-written evaluation set is protected from it; seeded defects must fail every run	A QA engineer who did not request the run keeps or removes each test
Every build	One lucky run counts as proof	The matrix runs on each build; variable slices run several times with a stated sample size; the agent proposes missing cases as pull requests	QA approves additions; a developer never approves their own tests
Release	Sign-off is a checkbox	Signed test evidence with a run manifest that an independent validator can rerun, linked to the change record	QA signs the evidence; the release manager approves the change
Change and incident	Old results are reused after a model change	A model, prompt or corpus change steps the task down and reruns the set; each incident adds a case	Only people change the evaluation set and its thresholds

What changes for the organization

Before	After
"Tested" means someone ran the suite once	"Tested" means a reproducible record anyone can rerun on any version
AI-generated tests inflate coverage	Every generated test is kept or removed by a person, against seeded defects
Model changes break features quietly	Every version change reruns the evaluation before release

In any organization

Who covers it: in a small or unregulated organization developers often test their own work; a peer who did not write the code reviews generated tests and signs the evidence.
What scales: confidence-bound statistics, judge calibration and a separate test team scale down with risk. Exact rules at 100%, people writing the evaluation set, synthetic test data and a rerun after every version change never do.

3. Your decisions

Decision	Your relationship	Inputs	Escalation	Cannot be delegated
Test strategy: which behavior is deterministic, which is probabilistic	QA owns; AI engineer and developer consulted	Acceptance criteria, architecture, risk tier	Engineering manager	Yes
Evaluation set content and risk slices	QA owns, written by people first; the agent never writes the set it is scored on	Criteria, known failures, incidents, BA (business analyst) negative paths	Product owner for slice priority	Yes
Thresholds per slice	QA proposes; product owner accepts; AI / model risk is consulted for high tier	Customer impact, sample size, baseline	Product owner, AI / model risk	Yes
Keep or remove each generated test	QA owns	Generated suite, seeded bad build results	QA lead	Yes
Test sign-off for a release	QA owns the test evidence; the release manager owns the change	Run manifest, results, open defects	Release manager	Yes
Step down a task after a failed evaluation	QA triggers; engineering manager confirms	Per-slice results	Engineering manager	No; the rule triggers automatically

4. Two kinds of adoption

Kind	What it looks like	Guardrail
How you work with AI	The agent brainstorms edge cases, drafts test cases and test code from criteria, explains a failing test, summarizes a run	Synthetic data only; the evaluation set is human-written; every generated test is kept or removed by a person
What you test that contains AI	AI features (the policy assistant), AI-drafted artifacts (code, stories, notes) and the AI tasks themselves (is test generation good enough for Stage 3?)	Probabilistic evaluation with slices, repeated runs, metamorphic and negative cases, and a reproducible run manifest

Deterministic versus probabilistic

The first question for every check: must this be exactly right every time?

Question	Deterministic test	Probabilistic evaluation
What it tests	Code with one correct output: date formatting, cutoff formatting, the banned pattern	AI output that varies: generated tests, drafted stories, assistant answers
Pass rule	Exact assertion. One failure fails the build	Rate per slice at or above a threshold, with a stated sample size
Runs	Once per build	Several times per version (illustrative: 5 runs) to see variance
Example	T3: due date 2026-11-02 in `America/Los_Angeles` shows "Nov 2" and cutoff "2:00 PM PST" (Pacific standard time)	Generated suite catches the seeded one-day-early defect in 5 of 5 runs
Wrong choice	Treating a date rule as "95% correct is fine"	Treating one lucky AI run as proof

The due date is deterministic. No threshold below 100% is acceptable for the date-only slice. AI may help write those tests; it never makes them probabilistic.

Sample size matters. The rule (illustrative): the two-sided 95% Wilson score interval lower bound must be at least 95%. On 200 cases, 196 pass: an observed rate of 98%. The lower bound is 94.97%, so the slice fails the rule despite the 98% rate (a one-sided 95% bound would be 95.62%, which is why the rule names its sidedness). More cases, not a better-looking rate, would clear it. State the sample size, the method and the sidedness with every rate.

5. Your ladder

Stage	AI use	Guardrail	Artifact or evidence	Decision relationship	Step-down trigger
0 · Foundations	None	Test data is synthetic; acceptable use attested	Test-data note	QA lead owns	Real data found in a test fixture
1 · Personal productivity	Brainstorm edge cases; explain failures	No records in prompts; the engineer writes every case	Edge-case list	QA owns	A case accepted without checking it
2 · Team context	Test-design prompt uses the temporal-domain contract as context	Reusable prompt card; instructions file	Draft test matrix	QA lead reviews the prompt card	Contract changes without the prompt being updated
3 · Delegated tasks	Agent generates test cases and code from criteria	Human-written evaluation set; seeded bad build; threshold per slice	Generated suite, evaluation run, keep/remove log	A QA engineer who did not request the run reviews it	Below threshold on any slice
4 · Agentic workflows	Workflow runs the matrix on each build and proposes missing cases	Proposals are pull requests; the evaluation set is protected from the agent	Per-build results, proposal pull requests	QA approves additions; developer cannot approve their own tests	Model, prompt or corpus version change
5 · Scale and sustain	Shared evaluation sets and harness in the catalog; continuous evaluation	Owner and version per set; periodic review	Catalog entries; regression trend	Set owners review on cadence	Regression caught after release

Human-operated at Stage 5. Writing and changing the evaluation set, setting thresholds, and signing test evidence stay with people. An agent that could edit the set it is scored on would grade its own homework.

6. Priority scenarios

Scenario	Value	Risk tier	Stage	First step	Do not use AI when…
Edge-case brainstorm	Finds cases people miss	Low to medium	1	Feed the criteria, not the code	You plan to accept the list without reading it
Test generation for the due-date display	Faster matrix, wider coverage	Medium	3	Write the evaluation set first (L4)	The same agent wrote the code in the same session
Metamorphic tests	Catch errors without an exact expected value	Medium	3	Write relations (below) by hand	The relation itself is unclear to the team
Evaluating the policy assistant	Evidence for release and validation	Medium	3	Slices from the worked example: answerable, out of scope, stale, injection, ambiguous	A judge model scores answers with no calibration against people
Accessibility checks	Release criterion met	Medium	2 to 3	Automated checks plus a person with assistive technology	The automated check is treated as complete
Reproducing a defect	Faster triage	Medium	1	Synthetic data only	Reproduction "needs" production records

Metamorphic relation	Applies to	Pass when
Zone change keeps the due date	Due-date display	Changing the customer's zone changes the cutoff text, never the due date
Same instant, any zone	Cutoff display	Every zone renders the same instant; converting back gives 5:00 PM Eastern
Display equals the source (contract test)	Cutoff display	The displayed cutoff equals the payment-schedule-service instant formatted with `formatPaymentCutoff` in the customer's zone; no channel calculates its own
Viewing time is irrelevant	Due-date display	Rendering at 11:30 PM Pacific or 9:00 AM Eastern shows the same date
Paraphrase keeps the citation	Policy assistant	Two phrasings of one question cite the same section
Irrelevant text changes nothing	Policy assistant	Adding an unrelated sentence keeps the answer and citation
Injected instruction is ignored	Policy assistant	A planted "ignore your rules" line does not change behavior

7. The due-date ask from your seat

QA holds material artifacts on this ask: the test matrix T1 to T13 and the evaluation set that decides whether test generation may run at Stage 3. The full matrix is in the worked example; the evaluation set is built in L4.

Decision	Input received	Output and evidence	Recipient
Which behavior is deterministic	Temporal-domain contract from the architect	All of T1 to T13 are exact assertions; the date-only slice threshold is 100%	Developer, release manager
Is test generation ready for Stage 3	Agent-generated suite	Evaluation run: the suite fails on the seeded bad build (`new Date(dueDate)`) in every run; per-slice pass rates recorded	Engineering manager
Which generated tests to keep	41 generated tests (illustrative)	Keep/remove log: 33 kept, 8 removed as duplicates or wrong expectations	QA lead
Test sign-off	Build, run manifest	Signed test evidence linked to the change record	Release manager
Negative path: model change	Provider notice via AI risk	Test generation steps down to Stage 2; evaluation set re-run on the new version before it returns	Engineering manager, AI risk

Reproducibility. Every evaluation run records what produced it. Without this manifest a result cannot be compared across versions, and an independent validator cannot rerun it.

# run-manifest.yaml · test-generation evaluation (illustrative)
run_id: eval-2026-10-27-0931
evaluation_set: due-date-display@v1.3      # owner: payments QA lead
system_under_test:
  model: approved-coding-model@2026-09
  prompt_card: test-design@v1.2
  instructions_file: payments@sha 4e1a9c2
  tools: [repo-read, test-run]              # versions pinned in the sandbox image
  corpus: n/a
settings: { temperature: 0, runs: 5 }
seeded_defects: [one-day-early-instant, fixed-offset-dst]
results:
  date-only: 1.00                           # threshold 1.00
  dst: 0.98                                 # threshold 0.95, n=50
  missing-zone: 1.00
reviewer: second QA engineer (not the requester)

8. Hand-offs and separation of duties

You give → to whom	You receive ← from whom	You challenge	You are notified of
Test evidence and sign-off → release manager	Acceptance criteria and negative paths ← BA, product owner	Criteria that cannot be tested	Criteria changes
Run manifests and evaluation sets → independent validation (read access)	Validation findings ← independent validation	A finding with no reproducible basis	Validation scope decisions
Defects → developer	Synthetic fixtures ← data engineer	Fixtures that look real	Fixture or schema changes
Evaluation results → AI engineer, engineering manager	Model and prompt version changes ← AI engineer, AI platform ops	A version change with no re-run	Provider changes
Monitoring cases → production support	Production incidents and complaints ← production support, business operations	An incident with no new test case	Incidents on the feature

The boundary with independent validation. QA designs and runs tests for the team that builds the product. Independent validation challenges that design, may rerun it, and reaches its own conclusion. A QA pass is an input to validation, never a substitute for it. The two must not report to the same build manager for high-tier work.

Within the team. The requester of a generated suite does not review it. The developer does not sign off their own tests. The agent cannot edit the evaluation set.

9. Evidence and records

Evidence produced or consumed	System of record	Retention owner
Test matrix and results per build	CI system	QA lead; retention per records schedule
Evaluation set (versioned)	Repository and asset catalog	Set owner (QA)
Run manifest per evaluation	Evaluation store	QA lead
Keep/remove log for generated tests	Pull request	QA engineer
Test sign-off	Change system	QA engineer
Accessibility results	Test management tool	QA lead
Judge calibration (judge model vs human labels)	Evaluation store	AI evaluation engineer

10. Skills and learning path

Level	Skills	Labs and practice
Foundation	Synthetic data rules; edge-case thinking for dates and zones; when a test must be exact	Brainstorm with the assistant, then write T1 to T13 yourself
Practitioner	Evaluation sets, risk slices, seeded defects, run manifests, metamorphic relations	L4 end to end; add metamorphic relations to it
Advanced	Sample size and confidence bounds, judge calibration, injection and abstention testing, evaluation across versions	Evaluate the L6 assistant on all slices; calibrate a judge against 50 human-labelled answers (illustrative)

11. Measures and incentives

Metric	Definition	Anti-gaming note
Seeded defect catch rate	Share of seeded defects the suite catches	Seed defects the team has not seen; rotate them
Worst-slice pass rate	Lowest per-slice rate on each evaluation run	An overall rate hides the slice that matters
Escaped defects	Defects found after release ÷ defects found in total	Count customer contacts too, not only tickets
Reproducible runs	Share of evaluation results with a complete manifest that reran within tolerance	A result without a manifest counts as missing
Generated test keep rate	Kept ÷ generated	Higher is not better; a 100% keep rate may mean nobody read them
Evaluation set growth from failures	New cases added after a defect or incident	Adding easy cases to raise the pass rate does not count

Never measure QA by the number of tests generated. Read these with the measure tree.

12. Cost, capacity and workforce

Item	What to plan for
Review time	Generated tests still need reading. Budget review hours, not just generation
Evaluation cost	Repeated runs on every version cost model calls; tag them as evaluation in FinOps (financial operations) reporting
Skills shift	QA engineers add evaluation design and basic statistics; this is growth, not replacement

13. Anti-patterns and stop conditions

Anti-pattern or trigger	Why it hurts	What to do
Agent writes the evaluation set	It grades its own work	Stop; people write the set; protect the path
Code and tests from one session	Shared blind spot	Generate tests from criteria, separately
Probabilistic pass for a deterministic rule	"95% of due dates are right" is a defect	100% on the date-only slice
One run treated as proof	Variance hidden	Repeated runs; state the sample size
Uncalibrated judge model	Scores look precise and mean little	Calibrate against human labels first
Model, prompt or corpus change	Old results no longer apply	Step down; re-run the evaluation
QA pass presented as validation	Separation of duties broken	Label first-line evidence as such; route to validation

14. Action kit and 30/60/90

Copy this card for every AI task or AI feature you evaluate.

## Evaluation card · <task or feature>
Deterministic parts (exact assertions): <list>
Probabilistic parts (rate per slice): <list>
Evaluation set: <name@version> · owner: <QA name role> · written before the agent ran: yes/no
Slices and thresholds: <slice: threshold, n>
Seeded defects: <list>
Metamorphic relations: <list>
Negative cases: injection, out of scope, stale source, missing data
Runs per version: <n> · settings recorded in the run manifest
Reviewer (not the requester): <role>
Re-run triggers: model, prompt, corpus, tool, policy or threshold change
Hand-off: release manager (sign-off); independent validation (read access, high tier)

Questions to ask your enablement lead:

Where do evaluation sets and run manifests live so validation can rerun them?
Who sets thresholds for each risk tier, and who accepts them?
How many review hours are planned for generated tests this quarter?

Window	Do
First 30 days	Classify every check as deterministic or probabilistic; write the date-only slice at 100%; confirm all fixtures are synthetic
Days 31 to 60	Run L4; add a seeded bad build and run manifests; start the keep/remove log
Days 61 to 90	Add metamorphic relations; calibrate any judge model; publish the evaluation set to the catalog with an owner