Production support and SRE

AI adoption by role

Production support keeps services running and leads the response when they fail. AI (artificial intelligence) helps here in two ways. It speeds up triage and summarization for the people on call. And it is part of some products the team now operates, which can fail in new ways: drift, stale retrieval, tool errors and runaway cost.

Humans keep incident command, consequential remediation, customer communication and closure. Counts and durations on this page are illustrative.

1. At a glance

Item	Answer
Mandate	Detect, contain, recover and learn, for ordinary services and for AI products, with evidence preserved
Primary decision	Declare, command and close incidents; choose containment such as degrade, disable a tool or pull the kill switch
First safe win	AI drafts an incident timeline and a triage summary in which every statement links to a log line, alert or ticket; the operator verifies each link
Biggest failure mode	An AI summary is trusted without checking its sources, or an agent runs a consequential fix in production without a human decision

Line: 1st line · Family: Release and run · Variants: SRE (site reliability engineering), incident commander, L2 and L3 (second- and third-level) support, AI product operations

Role variants

Variant	What differs
SRE	Owns reliability engineering: service level objectives, alerting, capacity, automation, the kill-switch mechanism. Writes and tests runbooks
Incident commander	Leads one incident: declares severity, assigns roles, approves containment, owns the timeline and closure. Does not debug while commanding
L2 and L3 support	L2 works the ticket: reproduces, checks known issues, applies documented fixes. L3 is the engineering team: diagnoses code or configuration and proposes fixes through change control
AI product operations	Watches AI-specific signals: quality, drift, retrieval freshness, tool failures, cost, overrides and complaints. Routes findings to the technical owner and AI / model risk

2. Your journey: first steps to fully adopted

Production support moves from AI-drafted summaries that people verify to a shared triage agent that proposes; incident command stays human at every stage.

Start here: your first steps

List every AI product you operate with a named owner, its signals and a fallback runbook.
Confirm that logging and retention cover traces, prompts and outputs, so evidence survives an incident.
Use an approved tool to explain an error message, with no production data outside approved tools.
Connect a read-only runbook search that cites the runbook or past incident behind every answer.

Fully adopted: how the work runs

Moment in the work	Before AI	Fully adopted (Stage 4–5)	Decision that stays human
Alert	During an alert storm, the on-call engineer sorts alerts by hand	A shared triage agent groups alerts and links each group to its signals; grouping never hides a high-severity alert	Declaring the incident and its severity
Triage	The first minutes go to reading dashboards and chat	A triage summary arrives with a source link per claim; the operator opens the links and verifies	Whether the summary is right
Diagnosis	Engineers search logs and old tickets from memory	The workflow runs pre-approved, read-only diagnostics and finds matching past incidents	Which cause to act on
Remediation	A fix is typed into production under pressure	The workflow proposes a runbook step, such as a flag off or a tool disabled; nothing runs without a recorded human approval	Containment and every consequential fix, by the incident commander
Post-incident review	Written days later, from memory	A draft review with timeline, causes and owned actions, each linked; new cases join the evaluation set	Closure, and the material-change or revalidation hand-off

What changes for the organization

Before	After
AI products run with no AI-specific signals or fallback	Every AI product has an owner, signals, thresholds and a rehearsed fallback
Incident time goes to finding information	Incident time goes to verifying and deciding, with command kept clear
Lessons fade after closure	Each incident adds test cases and runbook updates

In any organization

Small or unregulated: the developers on call cover this role, and one named person commands each incident.
What scales down: severity schemes, rota size and game-day frequency shrink. A human incident commander, approval before consequential remediation, preserved evidence and a fallback never do.

3. Your decisions

Decision	You own, advise or challenge	Inputs	Escalation	Cannot be delegated
Declare an incident and its severity	Own (incident commander)	Alerts, reports, AI triage summary with links	Business owner for customer-impacting severity	Yes
Containment: degrade, fallback, disable a tool, kill switch	Own (incident commander, with the technical owner)	Runbook, blast radius, customer impact	Business owner if a customer-facing service goes to manual fallback	Yes
Consequential remediation in production	Own the approval; L3 or engineering proposes	Proposed fix as a reviewed change; rollback plan	Release manager for an emergency change	Yes
Customer communication	Advise; business owner decides, compliance approves wording	Impact assessment, draft notice	Legal and compliance	Yes
Evidence preservation and legal hold	Own the preservation steps; legal decides a hold	Logs, traces, prompts and outputs in scope	Legal, records management	Yes
Closure	Own (incident commander)	Recovery confirmed, post-incident review booked, actions owned	Business owner for high-tier incidents	Yes
Hand-off for material change or revalidation	Own the hand-off	Post-incident findings	AI / model risk, independent validation	Yes

4. Two kinds of adoption

Kind	What it means for production support	Example
How you work with AI	AI groups alerts, drafts a triage summary and timeline, searches runbooks and past incidents, drafts the post-incident review. Every statement carries a source link. The operator verifies before acting	AI links a spike in "due date confusion" contacts to one notification-service release; the on-call engineer opens the linked deploy log and confirms
What you operate that contains AI	Products with a model, a retrieval corpus or tools in them. They need AI-specific monitoring and runbooks alongside the usual ones	The internal policy assistant: citation failures, stale policy documents and abstention rate are watched every day

AI product operations adds signals that ordinary monitoring does not watch.

Signal	What it shows	Example threshold (illustrative)	First response
Quality and groundedness	Answers that are wrong or not supported by their citation	Sampled citation failure rate above 2%	Page the technical owner; consider degrade
Drift	Inputs or outputs change shape against the evaluation baseline	Abstention rate moves 50% from its weekly mean	Re-run the evaluation set
Retrieval freshness	The index serves withdrawn or old document versions	Any withdrawn version cited	Re-index; block the stale document
Tool failures	Errors, timeouts or unexpected side effects from tool calls	Error rate above 1% for 15 minutes	Disable the tool; switch to fallback
Cost and rate	Token spend or calls per user beyond plan, a sign of loops or abuse	Spend twice the daily budget by noon	Rate limit; check for denial of wallet
Overrides and complaints	Staff reject outputs, or customers complain	Override rate doubles in a week	Route to the business owner and AI / model risk
Trace availability	Traces missing means evidence is missing	Any gap over 5 minutes	Treat as a control gap, not a logging nuisance

5. Your ladder

Stage	AI use	Guardrail	Artifact or evidence	Decision relationship	Step-down trigger
0 · Foundations	None. You confirm logging, retention and the incident process cover AI products	Logging with retention rules; incident categories name AI failure types	Updated incident process; runbook index	Owner of the operational readiness check	Not applicable yet
1 · Personal productivity	AI explains an error message or a public runbook pattern. No production data in prompts outside approved tools	Approved tool; acceptable use; data classification	Your own notes	Owner of every action you take	Production data pasted into an unapproved tool
2 · Team context	AI searches the team's runbooks and past incidents through an approved, read-only connector	Instructions file says: cite sources, never propose an untested command	Triage notes with links	Owner; L3 is consulted on fixes	A summary cites a runbook that does not exist
3 · Delegated tasks	An agent drafts the triage summary, timeline and post-incident review draft, each claim linked	Evaluation set of past incidents; trace kept; read-only access	Draft summary and timeline	Incident commander is owner of the timeline; agent output is input only	Missed or invented events in evaluation; operators stop checking links
4 · Agentic workflows	A workflow runs pre-approved, reversible diagnostics and can propose a containment step from a runbook	Least privilege; diagnostics read-only; containment needs a human approval click; kill switch on the workflow itself	Proposed action, approval record, trace	Incident commander approves; the agent never executes consequential actions alone	Any action outside the allow-list; an approval given without reading
5 · Scale and sustain	Shared triage agent and runbook library across teams; continuous evaluation	Asset owner; periodic review; recovery tested	Catalog entries; evaluation trend	Catalog owner approves assets; incident command stays with you	A model or provider change for the triage agent

Stays human at Stage 5: incident command, consequential remediation, customer communication and closure. An agent may propose a runbook step. A person decides, and the decision is recorded. Step down on a model change, a failed evaluation, an incident caused by the triage agent, or a control gap; see the adoption ladder.

6. Priority scenarios

Scenario	Value	Risk tier	Stage	First step	Do not use AI when…
Triage summary with source links	Faster first understanding (illustrative: 10 to 15 minutes saved per incident)	Medium	3	Require a link per claim; reject summaries without them	No source can be linked; then the operator reads the raw signals
Incident timeline draft	Accurate record without a scribe	Medium	3	Feed only the incident channel and alert history	The timeline is the legal record and nobody has verified it
Runbook search	The right procedure, found fast	Low to medium	2	Index only approved, owned runbooks	The runbook is stale or unowned
Post-incident review draft	Consistent reviews; actions not lost	Medium	3	Template with causes, actions, owners, evidence links	It assigns blame or invents a cause
Alert grouping	Less noise during a storm	Medium	3	Shadow mode first: compare with human grouping	Grouping would hide a safety or customer-impact alert
Automated remediation	Speed	High for consequential actions	4	Only reversible, pre-approved diagnostics	The action changes customer data, money or access

7. The due-date ask from your seat

For the due-date change, production support operates the result. AI is not in the runtime path, so there is no AI product to run here. The team uses AI only for triage.

Role card	Value
Decision	Accept the release into operation; declare and command an incident if channels disagree
Input received	Change record, phase plan, rollback by flag per channel, the channel-consistency monitor
Output and evidence	Monitor results per phase; any incident ticket with timeline, containment and post-incident review
Recipient	Release manager, business owner, change system of record

A short negative path shows the split of work. After the mobile phase, the consistency monitor finds 12 mismatches (illustrative). The AI triage summary links them to one mobile build that still calls an old helper. The on-call engineer confirms the link in the build log.

The incident commander turns the mobile flag off, which is the rollback in the change record. The fix goes back through change control. The AI drafted the summary; people declared, contained and closed.

For the companion internal policy assistant, the team runs AI product operations: citation failures, stale documents, abstention rate, overrides from contact-center staff. See the worked example for its trust boundary.

8. Hand-offs and separation of duties

You give → to whom	You receive ← from whom	You challenge	You are notified of
Incident record and timeline → business owner, change system	Change record and phase plan ← release manager	A release with no tested rollback or kill switch	Releases and AI component changes
Containment decision → engineering on call	Runbooks and kill-switch mechanism ← SRE and the technical owner	A runbook that has never been rehearsed	Provider outages and model change notices
Material-change and revalidation hand-off → AI / model risk, independent validation	Override and complaint data ← business operations	An AI triage summary with unlinked claims	Security events touching AI products, from security
Preserved evidence → legal, records management, security	Hold notices ← legal	A request to delete logs during an open incident	Audit requests for incident samples
Customer-impact facts → business owner, compliance	Approved customer wording ← compliance	Customer notices drafted before impact is confirmed	Complaint spikes, from the contact center

Separation of duties: the person who proposes a production fix is not the one who approves it, except under an emergency change that is reviewed afterwards. The triage agent proposes; it never approves or executes consequential actions.

9. Evidence and records

Evidence produced or consumed	System of record	Retention owner
Incident ticket, timeline, severity, closure	Incident management system	Production support, under the incident records schedule
Containment and kill-switch actions	Incident ticket, linked to change records	Incident commander
Traces, prompts and outputs in scope	The log store the records decision named	Records management, with engineering; reachable by legal hold
AI triage summaries used in the decision	Attached to the incident ticket, marked AI-drafted, with the verifier named	Production support
Post-incident review and actions	Incident system, linked to the AI inventory entry	Business owner
Runbook versions and rehearsal records	Runbook repository	SRE
Monitoring signals and thresholds	Observability platform, configuration versioned	SRE and technical owner

10. Skills and learning path

Level	You can…	Labs and practice
Foundation	Verify an AI summary against its sources; know which data may enter which tool	Lab L1 (team instructions file for the on-call assistant)
Practitioner	Read AI-specific signals: drift, citation failure, tool errors, cost; run an AI runbook	Lab L6 (policy assistant: watch citations and abstention)
Advanced	Design runbooks and approval gates for agentic diagnostics; run AI incident game days	Lab L5 (delegated task behind an approval gate) and Lab L4 (evaluation sets)

Practise with game days (illustrative: one per quarter): a stale policy document, a tool that returns errors, a provider outage, and a cost spike from a retry loop.

11. Measures and incentives

Metric	Definition	Anti-gaming note
Time to contain	From declaration to containment, by severity	Do not delay declaring to keep the number low; measure detect-to-declare too
Summary link accuracy	Share of sampled AI summary claims whose link supports the claim	Sample randomly; the verifier is not the person who used the summary
Runbook rehearsal coverage	Share of AI runbooks rehearsed in the last cycle	A rehearsal needs a record and a result
Near misses reported	AI-related near misses logged per month	Rising reports can mean better culture; never set a target to cut them
Repeat incidents	Incidents with the same cause within 90 days	Closing early and reopening under a new ticket counts as a repeat

12. Cost, capacity and workforce

Item	What to plan
Capacity	AI triage saves minutes; on-call load still needs a rota sized for human command and verification
Cost	Cost monitoring for AI products sits with you and platform FinOps (financial operations); a cost spike is an operational signal
Workforce	L2 work shifts from searching to verifying. Keep hands-on diagnosis practice so skills do not fade

13. Anti-patterns and stop conditions

Anti-pattern	Why it hurts	Stop condition
Summary trusted, sources skipped	Automation bias turns a wrong guess into a wrong action	A decision made on an unlinked claim: review in the post-incident review
Agent runs the fix	A consequential action with no accountable decision	Any agent write to production outside the allow-list: disable the workflow
Logs cleaned during an incident	Evidence lost; a hold may be breached	Any deletion request during an open incident: refuse, escalate to legal
No fallback for the AI product	The process stops when the model does	AI product with no manual fallback runbook: no Stage 3 for it
Signals without owners	Drift and stale retrieval go unnoticed	An AI signal with no named owner: treat as a control gap
Closing without hand-off	The cause returns after the next model change	AI incident closed with no material-change or revalidation decision

14. Action kit and 30/60/90

RUNBOOK: AI PRODUCT DEGRADE AND KILL SWITCH (internal policy assistant)
Trigger: citation failure rate above threshold, stale document cited, tool errors, or cost spike
1. Declare. Incident commander named. Severity set. Timeline started.
2. Preserve. Snapshot traces, prompts, outputs and index version. Tag for possible hold.
3. Degrade. Turn on "answers with citations only"; raise abstention; post a staff banner.
4. Disable tools. Turn off any tool showing errors or unexpected side effects.
5. Kill switch if needed. Assistant off; staff use the manual policy lookup (fallback).
6. Notify. Business owner, technical owner, AI / model risk; security if abuse is suspected.
7. Recover. Fix through change control; re-run the evaluation set before turning back on.
8. Close. Post-incident review; new test cases; material-change or revalidation hand-off.
Never: delete logs, change customer data, or send customer notices from this runbook.

Questions to ask your enablement lead:

Which AI products do we operate today, and does each one have a named owner, signals and a fallback?
Is there an approved triage assistant with read-only access to our runbooks and incident history?
Which traces and prompts are records, and how do we preserve them during an incident?

Window	Goal	Done when
30 days	See AI products	Every AI product listed with owner, signals, thresholds and a fallback runbook
60 days	Triage safely	Triage summaries require a link per claim; first evaluation set of past incidents built
90 days	Rehearse	One AI game day run; kill switch rehearsed; summary link accuracy reported

Related: operating model for the incident workflow, governance first for the monitoring and kill-switch control row.