Your LLM Isn't Hallucinating. Your Data is Stale

When a GenAI assistant gives a wrong answer, the first thing people blame is the model. Sometimes that's fair. More often, in my experience, the model did exactly what it was told. It read what we gave it and repeated it faithfully. The problem was what we gave it.

An internal policy assistant told a staff member they could carry over ten days of annual leave. That had been true until March. The policy was updated, the new PDF went into the S3 bucket, and nobody re-synced the knowledge base. The assistant quoted the old policy perfectly, with a citation. It looked grounded because it was grounded, in a document that was no longer true.

That's the failure mode this post is about. It isn't a model problem, and no amount of prompt engineering fixes it. It's a data quality problem, and we already have the language and frameworks to deal with it.

Grounded is not the same as true

Amazon Bedrock Guardrails has a contextual grounding check, and it's genuinely useful. It scores whether a response is grounded in the reference source and relevant to the query, and blocks anything below the thresholds you set (AWS docs).

Read that carefully, though. It checks the answer against the retrieved source. If the source is out of date, a faithful answer passes with a high score. Grounding checks catch the model inventing things. They can't catch your data being wrong. That job sits upstream, with you.

DAMA just gave this problem a name

The 2024 revision of DAMA-DMBOK 2.0 added Currency as a recognised data quality dimension (DAMA). It now sits alongside long-standing dimensions such as accuracy, completeness, consistency and timeliness.

I think that addition is overdue, and AI is the reason it matters now. A dashboard with last month's numbers is annoying, but people usually notice the date in the corner. A RAG assistant with last month's policy speaks with complete confidence, and it doesn't show you a date unless you design it to.

In DAMA terms, stale RAG answers are a Data Quality failure with a Data Governance root cause. Somebody has to own how current each source must be. If nobody does, "current enough" gets decided by whenever someone last remembered to press sync.

CDMC turns it into a control

This is where the EDM Council's CDMC framework earns its place again. Three of its 14 key controls apply directly to RAG (EDM Council).

Control 12, data quality measurement. Quality has to be measured, not assumed. For a knowledge base source, freshness is one of the measures that matters most.

Control 14, lineage. You need to be able to trace an answer back through its chunk and source document to the system it came from. Without lineage, you can't tell which answers a stale source has contaminated.

Control 11, retention and purging. When a document is withdrawn, its embeddings have to go too. A vector store is a copy of your data, and copies outlive their originals unless something removes them.

DAMA tells you currency matters. CDMC tells you to measure it, trace it and purge it, and to be able to prove you did.

Where staleness gets in on AWS

In a typical Bedrock knowledge base setup there are three places data goes stale, and they fail independently.

1. The source lags. The extract job failed on Tuesday and nobody noticed.

2. The sync doesn't run. Bedrock knowledge bases don't watch your bucket. Each time you add, modify or remove files, you have to sync the data source for them to be re-indexed (AWS docs). The good news is that syncing is incremental: only changed documents are re-parsed, re-chunked and re-embedded, and deleted documents are removed from the vector store.

3. Old content lingers. If a data source is deleted with its deletion policy set to RETAIN, the ingested content stays in the vector store and keeps being used for retrieval until you explicitly resync (AWS docs).

How I'd build it

Start with a freshness agreement, not a tool. For every source feeding the knowledge base, the data owner writes down how current it must be: "HR policies within 24 hours of approval", "product pricing within one hour". This is the DAMA part, and it's the part that's usually missing.

Turn the agreement into a rule. AWS Glue Data Quality has a DataFreshness rule type that checks the gap between now and the values in a date column (AWS docs):

Rules = [

    DataFreshness "last_approved" <= 24 hours,

    Completeness "policy_id" > 0.99,

    IsUnique "policy_id"

]

For files rather than tables, the same documentation shows how to add the S3 file modification time as a column and check freshness against that.

Sync on an event, not on a calendar. Don't schedule a nightly sync and hope. Run the quality gate when new data lands, and start the knowledge base ingestion job only when the gate passes. If it fails, hold the sync and alert the owner. A stale-but-known answer is better than a fresh-but-broken one.

Put currency into the metadata, then filter on it. Bedrock knowledge bases support metadata filters at retrieval time, including equals, greaterThan and andAll conditions (AWS API reference). Tag each document with a status and an effective date in its metadata file, then retrieve only what's current:

{

  "andAll": [

    { "equals": { "key": "status", "value": "current" } },

    { "greaterThanOrEquals": { "key": "effective_year", "value": 2026 } }

  ]

}

This is also the cheapest fix for superseded policies. Old versions can stay for audit, but they never reach the model.

Use grounding as the last line, not the first. Keep the contextual grounding check on. It catches real hallucinations. Just don't mistake it for a freshness control.

Show the date. Have the application display the effective date of the sources it used. Users forgive "as of March". They don't forgive a confident answer that turns out to be old.

What worries me

Anomaly detection won't save you here. Glue Data Quality's ML anomaly detection is good, but the DetectAnomalies rule works on metrics such as row count, completeness, uniqueness and column values. DataFreshness isn't on its list of supported rules (AWS docs). Freshness needs an explicit rule and an explicit threshold, which means an owner has to set one.

Null dates fail the rule. DataFreshness fails for rows with NULL values (AWS docs). That's the right behaviour, but it'll surprise you on legacy data, so decide on purpose whether a missing date means "stale" or "exempt".

A failed rule doesn't necessarily stop anything. Check what your pipeline actually does when a quality rule fails. If the failure only writes a score to a dashboard, the stale data still flows into the knowledge base.

Re-embedding has a cost. Every changed document is re-chunked and re-embedded on sync. Sources that get rewritten wholesale every night, even when little actually changed, burn embedding tokens for nothing. AWS can update metadata without re-embedding in some cases (AWS docs), so separate content changes from metadata changes where you can.

Sovereignty still applies. As I covered in my last post, Bedrock called from the Auckland Region runs through cross-region inference (AWS ML Blog). Fresh data processed in the wrong place is still a problem.

The short version

A RAG assistant is only as current as its last sync. DAMA now has a word for this, Currency. CDMC makes it a control you can prove. AWS gives you the parts: freshness rules, incremental sync, metadata filters and grounding checks. What it can't give you is someone who owns how fresh each source has to be. Name that person first.