Your AI answer is only as fresh as its oldest dependency

The answer looked perfectly reasonable

Consider a support assistant asked whether an order can still be cancelled. It retrieves the cancellation policy, explains the rules clearly, and says yes. Meanwhile, the warehouse has already handed the parcel to the courier. The answer is well written, correctly cited, and useless.

Nothing in this example requires an exotic model failure. The policy describes what may happen. The order record describes what has happened. Those are different questions, and a search over policy documents cannot settle both. Yet the interface can easily present them as one seamless answer.

I would put this example in front of a team before discussing embeddings. Where would each fact come from? How old could it be? What should the assistant say if the warehouse system is unavailable? The answers to those questions will shape the architecture more than another round of prompt editing.

One question, two kinds of evidence

Retrieval-augmented generation supplies a model with relevant external material. That is useful for finding the right policy. It does not turn an indexed document into a live operational record. Microsoft’s RAG overview also treats source access and authorization as retrieval concerns, rather than things a model can sort out after seeing the data.

For this assistant, I would split the evidence path. Retrieve policy text from a permission-filtered index. Read order state through a narrow, authenticated application API. Return only the fields needed for the decision, not a whole customer record. The model can explain the result; it should not invent the order status or decide which tenant it belongs to.

Download Excalidraw source ↗Click diagram to enlarge

Text version

Reference path · what is allowed?

1. Policy documents: Version + effective date
2. Authorized retrieval: Only material this reader may see
3. Policy evidence: Relevant passage + source link

Operational path · what happened?

1. Authenticated request: Customer identity established by the app
2. Order service: Narrow read with tenant checks
3. Current state: Status + version + observed time

Both paths feed the answer composer. A missing operational read means no confident cancellation claim.

There is more than one clock

A document has a publication time. The index has an ingestion time. A cache has a refresh time. An operational response has an observation time. Calling all of these ‘updated at’ hides precisely the distinction an operator needs when an answer goes wrong.

Suppose a policy was indexed this morning but stopped applying yesterday. The index is fresh; the policy is not applicable. Conversely, a policy published six months ago may still be the current rule. Age alone is not a reason to discard it. You need both an applicability rule and a freshness rule.

These rules belong in the product requirements. An explanatory policy answer can use a known published version. A statement about a particular shipment needs a successful operational read. A cancellation attempt needs the order service to recheck eligibility at write time, because the world can change between reading and acting. The read architecture in the first diagram deliberately does not authorize a write.

Download Excalidraw source ↗Click diagram to enlarge

Text version

Illustrative answer contract

1. Old index: The source changed after ingestion. Re-index; show the evidence version.
2. Unavailable service: Order state cannot be verified. Explain the gap; offer a human route.
3. State changed: The read was valid, but the write is no longer allowed. Recheck inside the action.

A small test set with a bad attitude

I would start evaluation with a handful of deliberately awkward cases: a policy superseded yesterday, a duplicate document with a different effective date, an inaccessible account, an order that changes status during the conversation, and a service timeout. For each case, write down the evidence the assistant is allowed to use and the claim it must not make.

Then inspect the retrieved evidence separately from the final prose. If the wrong version arrives at the model, better wording is not a repair. If the right evidence arrives and the answer overstates it, the answer policy needs work. Keeping those failures separate makes the next engineering task much less ambiguous.

The model still matters. A weak model can misread perfectly good evidence. But I would rather debug that problem with a trace showing source versions, retrieval time and tool results than with a screenshot of a confident paragraph. Keep sensitive values out of those traces unless there is a justified, access-controlled need to retain them.

The useful promise is modest: here is what the system knows, here is when it checked, and here is what it could not verify. For an operational assistant, that is a better starting point than sounding certain.