Security

AI adoption by role

Security sets the threat model for AI (artificial intelligence) systems, tests it, and responds when it fails. AI changes the attack surface. Text becomes an attack path, retrieved documents can carry instructions, and an agent with tools can act with more rights than the task needs.

This track covers the second-line security function that sets standards and challenges, and the first-line security engineering that builds controls into products. Counts and durations on this page are illustrative.

1. At a glance

Item	Answer
Mandate	Every AI system has a current threat model, authorization at every tool call, tested detection and containment, and a red-team record
Primary decision	Accept the threat model and the residual security risk, with conditions, or block until controls exist
First safe win	AI drafts a threat model from the architecture diagram and the tool permission list; the security engineer reviews and corrects every threat and control
Biggest failure mode	The model is treated as the security boundary: a system prompt says "do not do X" and nothing in code enforces it

Line: 2nd line (and 1st-line security engineering) · Family: Govern · Variants: security architecture, AppSec (application security), security operations, red team

Role variants

Variant	What differs
Security architecture	Sets patterns: identity for agents, trust boundaries, egress rules, secrets isolation. 2nd-line standard owner
AppSec	Reviews code and designs; runs SAST (static application security testing) and dependency checks; reviews tool permission lists. Often 1st-line engineering
Security operations	The SOC (security operations center): detection rules, alert triage, containment, forensics, credential rotation
Red team	Attacks the system as an adversary would; reports abuse paths and proves fixes. Independent of the build team

2. Your journey: first steps to fully adopted

Security moves from AI-drafted threat models that an engineer corrects to shared, evaluated security assistants; threat-model acceptance and containment stay human at every stage.

Start here: your first steps

Publish an AI threat-model template and a security standard that names approved tools, egress and logging.
Turn on secrets scanning and an egress allow-list for every approved AI tool.
Use an approved tool to explain a public attack technique, with no secrets or unpatched findings in the prompt.
Write a security instructions file, reviewed like code, so AI code review cites your real rules.

Fully adopted: how the work runs

Moment in the work	Before AI	Fully adopted (Stage 4–5)	Decision that stays human
Threat model	Written late, or never, for AI features	An assistant drafts the threat model from the architecture and the tool permission list; the security engineer corrects every line	Threat-model acceptance and residual risk
Review	Reviewers check code, but not prompts, tools or corpus	AI review flags tool permission, protected-path and injection issues on every PR (pull request); red-team test sets from the catalog run before release	Suppressing a finding; the security sign-off
Detection	Detection rules ignore prompts, tool calls and spend	Detection covers injection attempts, out-of-scope tool calls, exfiltration canaries and spend spikes; AI groups and enriches the alerts	Which alerts are real incidents
Response	Analysts gather context by hand	A workflow runs read-only investigation queries and proposes containment; nothing isolates a host or rotates a credential without approval	Containment and credential actions, by security operations
After the incident	The abuse path is fixed once and forgotten	The new abuse path joins the red-team set and the threat-model template	Whether the threat model must be reopened as a material change

What changes for the organization

Before	After
The system prompt is treated as the security boundary	Authorization at every tool call is enforced in code, outside the model
AI features reach production without a threat model	Every medium- and high-tier AI use case has a current threat model and a red-team record
Near misses stay hidden	Near misses are reported, thanked and turned into test cases

In any organization

Small or unregulated: a senior developer or the platform engineer covers security, with an outside review for high-risk features.
What scales down: red-team depth, SOC tooling and formal sign-off shrink. Least privilege, authorization at every tool call, no secrets in prompts and human containment decisions never do.

3. Your decisions

Decision	You own, advise or challenge	Inputs	Escalation	Cannot be delegated
Threat model accepted	Own (security architecture)	Architecture, data flows, tool list, trust boundaries	Chief security officer or equivalent for high tier	Yes
Tool permission list and agent identity	Own the standard; challenge each list	Tool manifest, action boundaries, workload account design	Security architecture	Yes
Egress and data exfiltration controls	Own	Data classes, allowed destinations, provider regions	Privacy for personal data	Yes
Red-team scope and residual risk	Own (red team scopes; security accepts residual)	Threat model, past findings	Business owner accepts residual business risk	Yes
Security sign-off for release	Own for the security lens only	Threat model, test results, open findings	Release manager holds the release	Yes
Containment during a security incident	Own (security operations, with the incident commander)	Detection, scope, runbook	Business owner, legal	Yes
Security material-change trigger	Own	New tool, new data source, new provider, wider agency	AI governance function	Yes

4. Two kinds of adoption

Kind	What it means for security	Example
How you work with AI	AI helps review code and designs, draft threat models, explain alerts, group SOC alerts and summarize investigations. Each output is checked by a security analyst who makes the call	AI flags that a pull request gives the coding agent write access to a production config path; AppSec confirms and asks for a protected-path rule
What you secure that contains AI	Products and internal tools with models, retrieval and tools. Each needs an AI threat model, authorization at every tool call, and red teaming	The internal policy assistant: indirect prompt injection through a policy document, tested before release

The AI threat model extends the governance threat table. Each threat has a control that works without the model's cooperation.

Threat	What it looks like	Control enforced outside the model	How it is tested
Direct prompt injection	A user types instructions that override the system prompt	User text treated as data; narrow tool set; output checks	Red-team prompt set; refusal and action logs
Indirect prompt injection	A document, ticket or web page the agent reads carries hidden instructions	Source allow-list; retrieved text never triggers a tool by itself; approval gate for actions	Seeded documents with hidden instructions in the evaluation set
Data exfiltration	Restricted data leaves in an output, a link, an image address or a tool call	Egress allow-list; output filters; data classification check before the prompt	Canary data and exfiltration test cases
Poisoned content	An approved source is edited to change answers	Owned, versioned corpus; change review on documents; citation checks	Corpus diff review; stale and altered document tests
Insecure tool use	The model builds a tool call with injected parameters	Schema validation; parameterized calls; no shell or raw query from model text	Fuzzed tool arguments
Excessive agency	More tools, rights or autonomy than the task needs	Least privilege; action boundaries; human approval for consequential actions	Permission review; attempted out-of-scope actions
Identity confusion	The agent acts with a person's broad rights, or mixes two users' contexts	Workload accounts; per-request user context; authorization checked per call against the end user	Cross-user test cases
Supply chain	A model, plug-in, MCP (Model Context Protocol) server or package is replaced or compromised	Approved list; pinned versions; signed artifacts; SBOM (software bill of materials)	Dependency diff; provider change notice review
Denial of wallet	Loops, abuse or huge inputs run up model cost	Rate limits; spend caps per user and use case; input size limits; alerting	Load and loop tests; budget alert drill
Cross-tenant leakage	One team's or customer's data appears in another's answer through a shared index, cache or memory	Tenant isolation in retrieval and caches; per-tenant keys; filter by entitlement at query time	Cross-tenant canary queries

Authorization at every tool call is the rule that holds the rest together. The policy enforcement point checks, at execution time, that this end user, through this agent, may take this action on this resource. A permission granted at session start is not enough.

5. Your ladder

Stage	AI use	Guardrail	Artifact or evidence	Decision relationship	Step-down trigger
0 · Foundations	None. You approve tool configurations, logging, egress rules and the AI threat-model template	Approved tools; secrets scanning; egress allow-list	Security standard for AI; threat-model template	Owner of the standard	Not applicable yet
1 · Personal productivity	AI explains a CVE (Common Vulnerabilities and Exposures) entry or a public attack technique; drafts a note	Approved tool; no secrets, logs or vulnerability details of unpatched systems in unapproved tools	Your own notes	Owner of every finding	Sensitive findings in an unapproved tool
2 · Team context	AI reviews code and designs against the team's security instructions and patterns	Instructions file reviewed like code; read-only repository access	Review comments with rule references	Owner of the finding; developer fixes	Comments that cite rules that do not exist
3 · Delegated tasks	An agent drafts threat models, triages SAST findings, and groups SOC alerts	Evaluation set of past findings and known false positives; trace kept	Draft threat model; triaged finding list	Owner; agent output is input only	Missed known threats; analysts stop reading drafts
4 · Agentic workflows	A workflow enriches alerts, runs read-only investigation queries and proposes containment	Least privilege; read-only by default; containment needs a human approval; kill switch on the workflow	Investigation record; approval record	Security operations approves containment; the agent never isolates hosts or rotates credentials alone	Any write action outside the allow-list
5 · Scale and sustain	Shared, evaluated security assistants and red-team test sets in the catalog	Asset owners; periodic review; its own threat model	Catalog entries; test-set trend	You remain decision owner	A model or provider change for the assistant

Stays human at Stage 5: threat-model acceptance, residual-risk acceptance, containment decisions and credential actions. An assistant can find a weakness. A person decides what to do about it. Step down after a model change, a failed evaluation, a security incident or a control gap; see the adoption ladder.

6. Priority scenarios

Scenario	Value	Risk tier	Stage	First step	Do not use AI when…
Threat-model first draft	Every medium and high use case gets one early	Medium	3	Template plus the threat table above as context	The architecture is not yet written down
AI-assisted code review	More issues found before merge	Medium	2	Security instructions file; comments only, no commits	The finding would be closed by the AI without a human
SAST and dependency triage	Less time on false positives	Medium	3	Evaluation set of labelled past findings	Suppressing a finding needs a decision; a person makes it
SOC alert grouping and enrichment	Faster triage during alert storms	Medium	4	Shadow mode against analyst grouping	Grouping could hide a high-severity alert
Red-team prompt generation	Wider abuse coverage	Medium	3	Generate variants of known attacks; humans curate	The generated set replaces human red-team thinking
Automated containment	Speed	High	4	Human approval click on every action	The action isolates production systems or rotates shared credentials

7. The due-date ask from your seat

The due-date change has a small security footprint. The display change adds no new data or tools. The security work is on the coding agent that drafts it, so this is a compact role card.

Role card	Value
Decision	Accept the coding agent's threat model and tool permission list for this repository
Input received	Tool manifest: read repository, write branch, run tests in sandbox; no production access; egress allow-list
Output and evidence	Threat model note; protected-path rule for deploy configs; secrets scanning on; review of the near miss below
Recipient	Security register; the team's instructions file; release manager

The near miss: the agent proposed copying production customer records into an unapproved tool "to reproduce the bug". The egress policy blocked the tool. The developer refused, used synthetic data and reported the near miss.

Security confirmed the block worked, added the case to the red-team set, and thanked the reporter. See the negative paths on the worked example.

For the internal policy assistant, security tests indirect prompt injection through policy documents and confirms that the assistant has no tools that act.

8. Hand-offs and separation of duties

You give → to whom	You receive ← from whom	You challenge	You are notified of
Threat model and conditions → technical owner, architect	Architecture and data flows ← architect	A trust boundary drawn inside the model	New tools, data sources or providers
Tool permission standard → AI engineer, platform engineer	Tool manifests ← AI engineer	A permission granted at session start instead of per call	Model and provider changes
Security sign-off → release manager	Test and red-team evidence ← QA (quality assurance) and red team	A release with open high findings	Incidents on AI products, from production support
Detection rules and runbooks → security operations, production support	Logs and traces ← platform engineer	Missing security logs for tool calls	Near misses from developers and staff
Security lens decision → AI inventory, risk and compliance	Privacy constraints ← privacy	Exceptions with no compensating control	Audit sample requests

Separation of duties: the red team is independent of the team that built the system. The person who writes a detection rule does not approve its suppression. The security assistant never closes its own findings.

9. Evidence and records

Evidence produced or consumed	System of record	Retention owner
AI threat model, versioned	Security register	Security architecture
Tool permission list and approvals	Configuration repository, linked from the inventory	Technical owner; security approves
Red-team findings and fixes	Security register	Red team
Security test results in CI (continuous integration)	CI system, linked to the PR (pull request)	Engineering
Security logs for tool calls and authorization decisions	Security log platform	Security operations
Incident investigations	Security incident system	Security operations; legal for holds
AI-assisted analysis used in a decision	Attached to the finding or case, marked AI-drafted, analyst named	Each analyst's team

10. Skills and learning path

Level	You can…	Labs and practice
Foundation	Explain each threat in the table; see why a system prompt is not a control	Lab L1 (team instructions file: what it does and does not enforce)
Practitioner	Write an AI threat model; review a tool permission list; build injection test cases	Lab L3 (advisory hook and CI check) and Lab L6 (policy assistant trust boundary)
Advanced	Red-team agentic workflows; design per-call authorization; run AI incident drills	Lab L5 (delegated task behind an approval gate)

Practise with drills (illustrative: once a quarter): a poisoned policy document, an exfiltration attempt through a link, and a spend spike from a loop.

11. Measures and incentives

Metric	Definition	Anti-gaming note
Threat-model coverage	Share of medium and high AI use cases with a current threat model	Current means reviewed since the last material change
Per-call authorization coverage	Share of agent tool calls checked by a policy enforcement point	Measure from logs, not from design documents
Red-team findings closed with retest	Findings fixed and proven by a retest	A fix with no retest stays open
Time to contain	From detection to containment for AI security events	Measure detect time too; late detection hides here
Near-miss reports	AI security near misses reported per month	Never target a fall; thank reporters

12. Cost, capacity and workforce

Item	What to plan
Capacity	Threat models for every medium and high use case need reviewer time; AI drafts help, review time remains
Cost	Red teaming and security logging for tool calls cost money; budget them per high-tier use case
Workforce	Analysts need AI attack literacy; pair AppSec with AI engineers on the first threat models

13. Anti-patterns and stop conditions

Anti-pattern	Why it hurts	Stop condition
System prompt as the control	Injection bypasses it	A security claim that rests only on prompt wording: block until enforced in code
Broad agent identity	The agent acts with a person's full rights	Agent using a personal account: stop the workflow
Authorize once per session	Later calls skip the check	Any tool call without a per-call authorization log: fail the release
Unbounded spend	Denial of wallet	AI use case with no spend cap: no Stage 3
Shared index across tenants	Cross-tenant leakage	Retrieval without entitlement filters: block
AI closes findings	Nobody decided	Finding closed with no human name: reopen

14. Action kit and 30/60/90

AI THREAT-MODEL CHECKLIST (per use case, per material change)
[ ] Trust boundaries drawn; the model is inside one, never the boundary itself
[ ] Inputs listed: user text, retrieved documents, tool outputs, memory; each treated as untrusted
[ ] Tools listed with least-privilege scope; consequential actions need human approval
[ ] Authorization checked at every tool call, for this end user, on this resource
[ ] Egress allow-list; output filters for restricted data; canary tests
[ ] Corpus owned and versioned; changes reviewed; citation checks on
[ ] Supply chain: model, plug-ins, MCP servers, packages pinned and on the approved list
[ ] Spend caps and rate limits per user and per use case
[ ] Tenant isolation in index, cache and memory
[ ] Security logs for every tool call and authorization decision
[ ] Red-team set run; findings fixed and retested; kill switch rehearsed

Questions to ask your enablement lead:

Which AI use cases are medium or high tier today, and which of them lack a current threat model?
Do our agents run under workload accounts with per-call authorization, or under people's accounts?
Is there an approved security review assistant, with an evaluation set, that my team may use?

Window	Goal	Done when
30 days	Set the standard	AI threat-model template and the checklist above published; tool permission standard agreed with platform
60 days	Cover the inventory	Threat models for every high and most medium use cases; per-call authorization gaps listed with owners
90 days	Prove it	First red-team round on one high-tier use case; detection rules for tool-call abuse live; one drill run

Related: governance first for the threat rows in the control crosswalk, operating model for incidents, templates for the threat-model row.