The data engineer decides how data reaches an AI step, and proves where it came from. This track covers two jobs that share one pipeline but hold different decisions. The data engineer and the analytics engineer build and run the pipeline. The data steward and the data owner decide definitions, permitted purpose, access and quality thresholds.
A model, a retrieval index or an evaluation set is only as trustworthy as the data behind it. Numbers on this page are illustrative.
1. At a glance
| Field | Answer |
|---|---|
| Mandate | Only authoritative, classified, permitted and current data reaches a model, a retrieval index or a test, with data lineage to prove it |
| Primary outcome | A data flow a risk partner can trace from source to answer, approved by the data owner for this purpose |
| First safe win | Use the approved assistant to draft documentation and data-quality rules from schemas and metadata, never from records |
| Biggest failure mode | AI quietly "repairs" bad data (for example, guesses a missing time zone) instead of the gap going to the data owner |
Line: 1st line · Family: Data & platform · Variants: data engineer, analytics engineer, data steward, data owner
| Variant | What differs |
|---|---|
| Data engineer | Builds ingestion, indexing and reconciliation pipelines. Owns pipeline code, tests and run evidence |
| Analytics engineer | Builds the data models, metrics and dashboards that monitoring and the adoption scorecard read. Owns metric definitions in code |
| Data steward | Owns the definition, classification and quality rules of a data element. Recommends whether it fits a purpose. Does not build the pipeline |
| Data owner | The accountable business owner of a dataset. Approves access and permitted purpose, and accepts or rejects quality exceptions |
2. Your journey: first steps to fully adopted
This is the path from your first month to a fully adopted way of working. The order of the steps is the same in any organization; only the weight of the controls changes.
Start here: your first steps
- Register every source that feeds AI with its class, owner and permitted purpose in the source register.
- Draft documentation and quality rules with the approved assistant from schemas and metadata only, never from records.
- Write the data rules section of the team instructions file: authoritative sources, banned conversions, data classes.
- Replace any production sample in tests with synthetic fixtures, checked by a real-value scan.
Fully adopted: how the work runs
| Moment in the work | Before AI | Fully adopted (Stage 4–5) | Decision that stays human |
|---|---|---|---|
| New data need | Someone exports a table or a shared-drive folder | The source register names the authoritative source, class and permitted purpose before any pipeline exists | The data owner approves the purpose; privacy is consulted |
| Pipeline build | Hand-written pipeline; lineage drawn from memory | An agent drafts pipeline changes, quality checks and lineage as pull requests; generated checks must catch seeded defects | A second engineer approves; the steward approves rule changes |
| Retrieval index | Built once from whatever was at hand | Rebuilt in staging on each approved policy notice, with an index manifest (version, hash) and an evaluation run | The policy office confirms content; the release manager approves promotion |
| Data gap found | A script or AI "repairs" missing values silently | The gap goes to the data owner as an exception with an expiry; the fallback is labelled on screen | The data owner accepts or rejects the exception |
| Run and retire | Withdrawn documents stay retrievable | Continuous freshness checks; deletion propagation proved with a seeded document; data lineage in the catalog | Records management sets retention; data owners review on cadence |
What changes for the organization
| Before | After |
|---|---|
| Nobody can say which data a model saw | Every AI answer traces back to an authoritative, approved source |
| Data gaps are patched quietly | Data gaps are owned decisions with an expiry date |
| Tests copy production data | Tests run on synthetic fixtures; no customer data sits in any test |
In any organization
- Who covers it: in a small or unregulated organization one engineer may build the pipeline and also act as steward; the business owner of the data still approves new purposes and accepts gaps.
- What scales: catalog tooling, review cadence and a formal exception register scale down to a tracked list. No records in prompts, authoritative sources only, gaps routed not repaired, and withdrawn content removed never do.
3. Your decisions
"Cannot be delegated" means the decision stays with a named person: an AI step may prepare the inputs, never make the call.
| Decision | Your relationship | Inputs | Escalation | Cannot be delegated |
|---|---|---|---|---|
| Which source is authoritative for a data element | Engineer advises; steward recommends; data owner owns | Service catalog, data catalog, architecture decision | Data owner; architect for service boundaries | Yes |
| Whether data may enter a prompt, index or evaluation set | Engineer advises; steward classifies; data owner approves the purpose; privacy is a consulted control partner | Classification, intake card, privacy assessment | Privacy office | Yes |
| Pipeline design: ingestion, provenance, freshness SLA (service level agreement) | Data engineer owns | Source change rate, consumer needs, platform golden path | Architect, platform engineer | No; AI may draft, a reviewer approves the pull request |
| Quality thresholds per slice and what happens below them | Steward owns; engineer implements | Consumer impact, historical profiles | Data owner | Yes |
| Accepting a quality exception (a known gap, such as missing zones) | Engineer challenges; data owner owns | Profiling result, customer impact, expiry | Data owner; risk partner when customers are affected | Yes |
| Deletion and retention propagation into indexes, caches and copies | Engineer owns the mechanism; records management sets the rule | Records schedule, legal hold notices | Records management | Yes |
| Removing a withdrawn document from a retrieval index | Policy office decides; engineer executes and verifies | Policy version notice | Policy office | No; automated on notice, verified by test |
4. Two kinds of adoption
| Kind | What it looks like | Guardrail |
|---|---|---|
| How you work with AI | The assistant explains an unfamiliar schema, drafts SQL (structured query language) and transforms, writes data-quality rules and lineage notes, summarizes a profiling report | Schemas, metadata and synthetic samples only. No records in prompts. Every change lands as a reviewed pull request |
| What you build that feeds AI | Retrieval corpora and indexes, evaluation datasets, synthetic test fixtures, monitoring datasets, data products a model reads | Lineage, provenance, permitted purpose, minimization, freshness, deletion propagation, reconciliation |
The second row is the larger risk. A model inherits every defect of its data, and a retrieval index turns a stale document into a confident, cited answer.
5. Your ladder
| Stage | AI use | Guardrail | Artifact or evidence | Decision relationship | Step-down trigger |
|---|---|---|---|---|---|
| 0 · Foundations | None | Data classification rules; named steward and owner per source; access and logging | Source register: each source with class, owner, permitted purposes | Data owner approves; privacy consulted | A source with no named owner |
| 1 · Personal productivity | Explain schemas; draft queries and docs from metadata | Metadata only; secrets and PII (personally identifiable information) block on prompts | Reviewed query, documentation page | Engineer owns; peer reviewer | A record pasted into a prompt (a near miss, reported) |
| 2 · Team context | Assistant reads the team's data rules: authoritative sources, banned conversions, classes | Data section of the instructions file, reviewed like code | Instructions file "data rules" section | Steward reviews the data rules | Steward rejects a definition the file states |
| 3 · Delegated tasks | Agent drafts pipeline changes, quality checks, lineage docs and synthetic fixtures as pull requests | Evaluation set: generated checks must catch seeded defects; fixtures must contain no real values | Pull request, evaluation run, lineage record | Second engineer approves; steward approves rule changes | A generated check misses a seeded defect |
| 4 · Agentic workflows | On an approved policy version notice, the workflow rebuilds the index in staging, runs the evaluation and opens a promotion change | Least privilege: read approved sources, write the staging index only; promotion by a person | Index manifest, evaluation report, change record | Policy office confirms content; release manager approves promotion; AI risk notified | Evaluation fails; source schema changes; embedding model changes |
| 5 · Scale and sustain | Shared ingestion golden path; lineage in the catalog; continuous freshness checks | Pipeline ownership and funding; periodic review | Catalog entries for pipelines; freshness dashboard | Data owners review on cadence | New source or new purpose (material change) |
Human-operated at Stage 5. Approving a new purpose for a dataset and accepting a quality exception stay with the data owner at every stage. AI may draft the impact summary. It never decides.
6. Priority scenarios
| Scenario | Value | Risk tier | Stage | First step | Do not use AI when… |
|---|---|---|---|---|---|
| Catalog documentation from schemas | Catalog coverage rises without a documentation sprint | Medium | 1 | Export metadata only; the steward reviews each entry | The schema itself is classified restricted |
| Data-quality rules from a profile summary | More checks, found earlier | Medium | 3 | Evaluation set of seeded defects the rules must catch | The rule encodes a business definition the steward has not approved |
| Policy assistant index | Grounded, cited answers for staff | Medium | 3 to 4 | Ingest only approved, versioned documents with a manifest | The source is email, chat, a shared drive or a draft |
| Synthetic test fixtures for QA (quality assurance) | No customer data in any test | Medium | 3 | Generate from the schema and the rules; scan for real values | "Realistic" seems to need real records |
| Lineage for impact analysis | Faster, evidenced impact analysis | Medium | 2 to 3 | Assistant reads the lineage graph and code, cites each edge | The lineage graph has gaps: run a deterministic scan, do not let AI guess edges |
| Channel reconciliation | Mismatches found before customers find them | Low (no AI in the check) | n/a | Deterministic comparison against the source | Always for the comparison itself. AI may only explain a mismatch a query found |
| Filling missing values (zones, dates) | None that justifies the risk | High if proposed | n/a | Route the gap to the data owner | As a silent repair, ever. A deliberate proposal goes through intake as High |
7. The due-date ask from your seat
The data engineer changes nothing in payment-schedule-service or profile-service. Both are read-only, authoritative sources for this change. The role's work is evidence: where the data comes from, how good it is, and whether the channels agree after release.
| Decision | Input received | Output and evidence | Recipient |
|---|---|---|---|
| Which datasets carry the due date, and as what type | Impact analysis question from the architect and BA (business analyst) | Lineage record: payment-schedule-service is the source; four display channels read it; the warehouse copy stores a date type. A copy stored as a timestamp would be a separate finding for its data owner, not part of this change | Architect, BA |
| How complete is the zone preference | Fallback rule (state of residence) from the PO (product owner) and compliance | Profile completeness by state: share of profiles with no IANA (Internet Assigned Numbers Authority) zone. The steward sets the threshold; the data owner accepts the known gap with an expiry | PO, data owner |
| Synthetic fixtures for the test matrix | Test cases T1 to T13 from the QA engineer | Fixtures generated from the schema: zones, DST (daylight saving time) weeks, a missing zone, an issued statement. A scan confirms no real account values | QA engineer |
| Channel consistency monitor | Monitoring signal defined by the team | A deterministic daily job: 1,000 sampled accounts, due date on each channel compared with payment-schedule-service. Zero mismatches expected | Production support, PO |
# data-contract.yaml · payment due date · v1 (illustrative)
element: contractual_due_date
source: payment-schedule-service # authoritative, read-only for this change
type: date # date-only value; never converted to an instant
steward: payments data steward
owner: payments business owner
consumers: [web, mobile, statement-service, notification-service, warehouse]
quality:
- rule: not_null
- rule: no_time_component
element: payment_cutoff
type: instant # shown in the customer's IANA zone at display time
zone_source: profile-service
fallback: state_of_residence_zone # labelled on screen; gap accepted by owner until 2027-03-31Companion: the policy assistant index. Here the data engineer holds a material artifact. The index is the trust boundary's source list: if a document is in it, the assistant can cite it. The engineer ingests only approved, current documents from the policy library, records a version and hash for each, and removes withdrawn versions on the policy office's notice.
# index-manifest.yaml · policy assistant · build 2026-10-28 (illustrative)
corpus: approved-policy-library
embedding_model: approved-embedding-v3 # a change here is a material change
documents:
- id: POL-PAY-014
version: 7
sha256: 9f2c...e1
status: current
approved_by: policy office
- id: POL-PAY-014
version: 6
status: withdrawn
removed_from_index: 2026-10-21 # deletion propagation verified by test
checks:
injection_scan: passed # text that reads like instructions is flagged for the owner
stale_slice_eval: passed # withdrawn version is never citedThe QA engineer and AI engineer run the evaluation against this manifest. Independent validation can rebuild the same index from it. See the worked example for the evaluation set and the trust boundary.
8. Hand-offs and separation of duties
| You give → to whom | You receive ← from whom | You challenge | You are notified of |
|---|---|---|---|
| Lineage and impact evidence → architect, BA | Authoritative source decision ← architect, data owner | A "source" that is really a copy | Schema changes in a source |
| Synthetic fixtures → QA engineer | Test data needs ← QA engineer | Any request for a production sample | New test slices |
| Index manifest and build → AI engineer, release manager | Approved policy versions ← policy office | A document with no owner or no version | Withdrawn or superseded policies |
| Quality findings → data owner | Quality thresholds ← data steward | A threshold with no slice | Incidents tagged as data issues |
| Deletion evidence → records management | Records schedule, legal holds ← records management | A tool whose default retention differs from the schedule | A legal hold placed or released |
| Reconciliation results → production support, PO | Monitoring thresholds ← PO | A monitor that samples only one channel | Channel mismatch alerts |
Separation of duties. The engineer who builds a pipeline does not approve its promotion. The steward who writes a quality rule does not accept an exception to it; the data owner does. The agent that drafts a pipeline change never holds rights to merge it or to write to a production index.
9. Evidence and records
| Evidence produced or consumed | System of record | Retention owner |
|---|---|---|
| Source register (class, owner, permitted purpose) | Data catalog | Data steward |
| Lineage record per pipeline | Data catalog or lineage store | Data engineer |
| Index manifest (document, version, hash, added, removed) | Asset catalog and evaluation store | Data engineer; retention set by records management |
| Quality results by slice | Data-quality platform | Data steward |
| Exception record (accepted gap, compensating control, expiry) | Exception register | Data owner |
| Deletion propagation log | Records system | Records management |
| Traces of AI-drafted pipeline work | Trace store behind the AI gateway | Per the records schedule |
10. Skills and learning path
| Level | Skills | Labs and practice |
|---|---|---|
| Foundation | Data classification, acceptable use, metadata-only prompting, recognizing a near miss | Write the data rules section in L1; scenario check: "a colleague pastes ten account rows to debug a query" |
| Practitioner | Provenance, versioned indexes, synthetic data generation, evaluating generated checks against seeded defects | Build fixtures for L4; build and version the index in L6 |
| Advanced | Poisoning and injection defenses at ingestion, deletion propagation, embedding change as material change, slice design with the steward | Extend L6 with a withdrawn document and a planted instruction; add a banned-conversion rule for SQL to the L3 CI (continuous integration) check |
Stewards and owners need decision literacy more than tooling: practice reading a lineage record and an exception request, and deciding.
11. Measures and incentives
| Metric | Definition | Anti-gaming note |
|---|---|---|
| Lineage coverage | Share of AI-fed datasets with lineage to an authoritative source, verified by the steward | Auto-generated lineage nobody checked does not count |
| Index freshness | Share of policy changes live in the index within the freshness SLA | Measure from the policy office notice, not from build start |
| Deletion propagation time | Time from withdrawal or deletion to removal from index, caches and copies | Prove it with a seeded document, not by reading config |
| Worst-slice quality | Pass rate of quality rules on the weakest decision-relevant slice | An average hides the slice that hurts customers |
| Gaps routed, not repaired | Share of quality gaps sent to the data owner | Rewarding "zero exceptions" rewards silent repair |
| Fallback rate (shared with the PO) | Share of cutoffs shown with the fallback label | A drop after someone "fills" zones is a red flag, not a win |
The balanced metric tree is on measure.
12. Cost, capacity and workforce
| Item | What to plan for |
|---|---|
| Compute | Embedding and re-indexing on every approved policy change; versions kept for reproduction. Cost is tagged per index through the platform's FinOps (financial operations: cost visibility and control) showback |
| Steward capacity | The bottleneck is rarely the pipeline. It is the steward's time to approve definitions and the owner's time to decide exceptions. Fund it |
| Skills shift | Data engineers add retrieval, evaluation and provenance skills; analytics engineers add AI-specific monitoring metrics |
13. Anti-patterns and stop conditions
| Anti-pattern or trigger | Why it hurts | What to do |
|---|---|---|
| AI fills missing values | A hidden model now drives a customer outcome | Stop. Route the gap to the data owner. A deliberate proposal goes through intake as High |
| Real records "for realism" | A prohibited data flow | Stop. Report the near miss. Use synthetic fixtures |
| Index built from a shared drive | Unknown provenance; easy to poison | Stop. Approved sources with owners and versions only |
| Withdrawn document still retrievable | The assistant cites stale policy with confidence | Block promotion; fix deletion propagation; add the case to the stale slice |
| Embedding model or chunking change without re-evaluation | A material change shipped as a config tweak | Step down; re-run the evaluation; notify AI risk |
| Source schema change | Lineage and contracts silently go stale | Re-run impact analysis before the next build |
| Lineage drawn from memory | Evidence that proves nothing | Generate from code and run metadata; the steward verifies |
14. Action kit and 30/60/90
Copy this card for every dataset that will reach a prompt, an index or an evaluation set.
# data-readiness-card.yaml (blank; filled version above)
dataset:
authoritative_source: # service or system, not a copy
classification: # per the organization's scheme
permitted_purpose: # approved by the data owner, with date
minimization: # fields excluded and why
residency: # where it is stored and processed
lineage_record: # link
freshness_sla:
deletion_propagation: # mechanism + last test date
quality_rules_by_slice: # link; steward approved
open_exceptions: # id, owner, expiry
injection_scan: # for documents entering an index
steward:
owner:Questions to ask your enablement lead:
- Which approved tool may read our schemas, and is metadata classified differently from records?
- Where do index manifests and lineage records live so validation and audit can find them?
- Who funds steward time for AI use cases, and how do I escalate a stalled decision?
| Window | Data engineer and analytics engineer | Steward and owner |
|---|---|---|
| First 30 days | Fill the source register for every AI-fed source; write the data rules section of the instructions file | Name a steward and owner for each source; confirm permitted purposes |
| Days 31 to 60 | Ship synthetic fixtures with a real-value scan; version the first index with a manifest | Set quality thresholds per slice; decide open exceptions with expiry dates |
| Days 61 to 90 | Prove deletion propagation with a seeded document; publish the ingestion golden path to the templates and catalog | Review the first freshness and worst-slice report; agree the periodic review cadence |
Related tracks: platform engineer, AI engineer, QA engineer, independent validation, risk and compliance. Controls behind this page: governance first.