Data foundations for AI: quality, ownership, lineage, access controls, retention, labelling

Most AI projects don’t fail because the model is “not smart enough”. They fail because the data feeding the system is inconsistent, unauthorised, outdated, or impossible to explain when something goes wrong.

If you are implementing AI in a real organisation-especially anything customer-facing or decision-adjacent-you need data foundations that are boring, explicit, and enforceable. Otherwise you get the worst combination: impressive demos and fragile production behaviour.

This is the practical checklist: data quality, ownership, lineage, access controls, retention, and labelling. Not as a theoretical data governance programme, but as the minimum scaffolding required to run AI without creating new risks and costs.

Data quality: define “fit for purpose”, not “perfect”

Teams often aim for “high quality data” without defining what that means. In AI, “quality” is context-dependent:

• For RAG, quality is about whether documents are current, readable, well-structured, and correctly attributed.
• For analytics or forecasting, quality is about consistency, completeness, and stability over time.
• For automation, quality is about precision and clear edge cases, because mistakes cause downstream actions.

Start by defining quality in operational terms:

• Freshness: how old can data be before it becomes misleading?
• Completeness: which fields must exist to be useful?
• Accuracy: what are acceptable error rates, and how will you detect them?
• Consistency: are values standardised (dates, currency, identifiers, naming)?
• Duplication: can the same entity appear multiple times with different representations?
• Structure: do you have headings, metadata, and stable identifiers, or is it a pile of PDFs?

Then choose the cheapest mechanism that enforces those constraints:

• validate at ingestion,
• quarantine bad inputs,
• and maintain a small set of quality metrics you can trend.

“Perfect data” is an endless project. “Fit for purpose” is measurable.

Ownership: if nobody owns it, nobody fixes it

AI systems create a new kind of accountability pressure: people will ask “Why did it answer that?” and “Where did that come from?” If your data is ownerless, every incident becomes a blame ping-pong.

You need clear ownership at two levels:

1. Data domain ownership (business responsibility)

   • Who decides what the data means?
   • Who decides when it’s obsolete?
   • Who approves changes to definitions?

2. Platform ownership (technical responsibility)

   • Who owns ingestion pipelines?
   • Who owns indexing/embedding?
   • Who owns access controls and audit logs?

Make this explicit. Put owners on the hook for:

• defining what “current” means,
• providing authoritative sources,
• and agreeing change windows for major updates.

Without it, AI outputs will drift and nobody will be authorised to correct them.

Lineage: you must be able to answer “where did this come from?”

Lineage sounds like enterprise bureaucracy until you have a production incident:

• A customer gets an answer that references outdated policy.
• A salesperson is shown pricing guidance that was superseded last month.
• An internal assistant repeats something that should never have been indexed.

At that point you need fast, concrete answers:

• Which source document was used?
• Which version?
• When was it ingested?
• What transformations happened?
• Which embedding/index build produced the retrieval result?

Minimum viable lineage for AI workloads:

• Stable document IDs (not filenames).
• Source system and location (e.g., SharePoint site, wiki space, ticket system).
• Version or last-modified timestamp.
• Ingestion timestamp and pipeline version.
• Chunk IDs and offsets (if using RAG).
• Ability to trace an answer back to the exact chunks retrieved.

You don’t need a perfect data catalogue on day one. You do need the ability to audit and remediate quickly.

Access controls: treat AI like a new data distribution channel

A common failure mode: an AI assistant becomes the easiest way to access information, and it quietly bypasses the access model that existed in the underlying systems.

If a user can ask, and the system can retrieve, you have created a new path to disclosure.

Minimum requirements:

1) Enforce permissions at query time

Do not rely on “we only indexed approved content” unless the corpus is static and tightly curated. In most businesses it won’t be.

You need a clear policy:

• either per-document ACLs carried into the index and enforced at retrieval time,
• or segmented indexes (per tenant, per department, per security boundary),
• or both for high-risk domains.

2) Separate tenant data

If you operate a multi-tenant product, assume adversarial behaviour. Ensure:

• per-tenant isolation at storage and retrieval,
• no cross-tenant embeddings or shared indexes unless you have a strong reason and strong controls,
• strict auditing.

3) Log access for audit, without leaking data

You want to know:

• who queried what,
• what documents were retrieved,
• and whether the response included restricted content.

But you must balance that with privacy and sensitive-data exposure in logs. This usually implies:

• storing references/IDs and metadata,
• redacting or hashing sensitive fields,
• and limiting retention of raw prompts/responses unless explicitly required.

Retention: AI multiplies copies of data unless you control it

AI implementations create new artefacts:

• extracted text,
• chunks,
• embeddings,
• indexes,
• cached prompts and responses,
• evaluation datasets.

If you don’t define retention, you will accumulate an ungoverned secondary data estate.

Decide, explicitly:

• How long do you retain raw documents pulled into the pipeline?
• How long do you retain extracted text?
• How long do you retain embeddings and indexes?
• How long do you retain prompt/response logs?
• How do you honour deletion requests and retention policies?

Key principle: retention must match the most restrictive policy applicable to the underlying data. If HR data must be deleted after X, your index must also delete after X. “We forgot the embeddings” is not a defence.

Operationally, you need:

• deletion propagation (source delete triggers index delete),
• rebuild strategies (when policies change),
• and proof that deletion actually happened.

Labelling: if you don’t label it, the model will treat it all the same

Labelling here is not only ML training labels. For AI systems-especially RAG and assistants-labelling is metadata that tells the system what content is and how it should be used.

Useful labels include:

• Sensitivity: public / internal / confidential / restricted
• Domain: HR, legal, finance, engineering, product
• Authority: draft / approved / superseded / archived
• Applicability: region, customer segment, product version
• Validity window: effective dates, expiry dates
• Source type: policy, FAQ, ticket, email thread, meeting notes

Why this matters:

• Retrieval can filter out drafts and superseded documents.
• Responses can cite only “approved” sources for high-stakes questions.
• You can route certain queries to stricter policies or models.
• You can stop the system from using low-authority content as if it were fact.

Labelling is one of the cheapest ways to improve quality and reduce risk, and it often beats model upgrades.

Putting it together: the minimum house standard for AI data

If you want a practical baseline your organisation can adopt, make it something like:

1. Every indexed document has an owner, source system, and last-modified timestamp.
2. Every chunk can be traced back to a document ID and location.
3. Access controls are enforced at retrieval time, not by hope.
4. Retention and deletion apply to derived artefacts (chunks, embeddings, indexes, logs).
5. Content is labelled for sensitivity and authority, and retrieval respects it.
6. Quality is measured with a small set of metrics and a quarantine path for bad inputs.

If you cannot do all of that immediately, do it in order:

• access controls and lineage first (risk),
• then retention (compliance/cost),
• then quality and labelling (results).

Closing thought

Model selection gets attention because it’s visible. Data foundations decide whether your AI system is trustworthy, maintainable, and affordable.

If you want “good enough” AI in production, be ruthless about the basics: data you can explain, data you are allowed to use, and data you can delete when you must.