RAG in practice: getting your agent to answer with today’s data

Where this comes from

The guide is vendor material, so it talks about the vendor’s tools. The logic, though, works with any tech stack, and that is what we care about here.

RAG without the jargon

A language model was trained on a huge amount of public text, up to a cutoff date. It does not know your returns policy, this month’s price list or the history of the customer on the other end of the chat.

RAG is how you hand it that knowledge at the moment of the question. It works like this:

1. Company documents are split into chunks that make sense on their own: a paragraph of a policy, a contract clause, a support ticket.
2. Each chunk becomes a vector, a mathematical representation of its meaning, and goes into a vector database.
3. The customer’s question becomes a vector too, and the database returns the chunks closest in meaning. This is not keyword search: “my suitcase never showed up” finds the passage on “lost baggage”.
4. Those passages go into the request to the model as context, and it answers from them.

The alternative would be retraining the model itself on company data (fine-tuning). The guide points out that this is expensive, needs specialist skills and can make the model forget what it already knew. With RAG the model stays as it is, and the knowledge changes when the business does.

The problem almost nobody sees: data goes stale

Most RAG rollouts load the documents once and move on. Three weeks later the policy has changed, the inventory is different and the agent keeps answering from the old snapshot.

The guide’s central point is that the knowledge base has to be fed continuously, with every change flowing in as it happens, not in weekend batch loads. Without that, RAG just swaps a hallucination for an outdated fact, which is nearly as bad.

A RAG agent is only as good as the data that reaches it, and when it reaches it. Yesterday’s correct data can still be today’s wrong answer.

Not everything should become a vector

A common mistake is dumping everything into the vector database. The guide is clear about what makes sense:

• Worth vectorizing: free text and documents, such as policies, contracts, manuals, emails, customer reviews and reports that mix tables with prose.
• Not worth vectorizing: things with no meaning on their own, like Social Security or card numbers and lists of product codes, or aggregate metrics (“how many visitors clicked the banner”). Those stay as regular database queries.

One practical tip from the guide: when a structured field matters, translate it into something closer to language. A “price band” field with “cheap” or “expensive” helps the model understand what a bare number does not. And personal data needs to be filtered or masked before it enters the flow, for GDPR and similar laws as much as for common sense.

The four steps of RAG that works

1. Fill the knowledge base. Connect the sources (systems, documents, support history), clean the data, chunk it, create the vectors and keep it all current as the business changes.
2. Build the context for each question. At request time, combine the right passages from the vector database with the customer’s data from other systems, such as the booking, the plan or the balance.
3. Break the question into steps. “Can I use my miles to upgrade and bring a second bag?” is three questions: seats and price, baggage rules, miles balance. An agent runs each lookup against the right system and only then asks the model for the answer. First, it checks whether the same question was already answered, so you do not pay twice.
4. Check before answering. The model’s answer goes through a separate check against business rules: does the refund amount match the policy? Does the ticket number exist? If not, it does not go out.

The fourth step is the one most often missing from rushed rollouts. The guide suggests handing it to a different team, closer to compliance, and keeping it separate from the rest of the system, so the rules can change without touching the agent.

Fast or right: a business decision

Every extra lookup costs time. The guide gives an honest example: an answer in one second that is right 90% of the time, or in ten seconds that is right every time. There is no single right setting.

• A VIP customer asking for a refund: worth the wait to get it right.
• An internal team looking up a procedure: fast and almost always right is fine.
• Most people will happily wait a few seconds for a support answer they can trust.

What matters is making that choice on purpose, case by case, and measuring it afterwards. It is not a technical call; it belongs to whoever knows the customer.

Three examples from the guide

• Retail product recommendations. Inventory, sales, reviews and loyalty data feed the knowledge base, and the assistant recommends based on each customer’s profile.
• Reading the market in real time. News and financial data stream in continuously, so “how did the market react to the earnings report?” gets answered with what came out minutes ago.
• A jobs portal. Résumés are read and summarized, profiles are enriched and the system matches candidates and openings, for both sides.

These are only examples. The same logic works for customer support, legal, procurement, HR or finance: if the answer depends on a company document or a piece of data that changes, it is a case for RAG.

What this changes in practice

The lesson for anyone bringing AI into operations is that the model is the easy part. The work is in connecting the right sources, keeping the data live, deciding what becomes a vector, breaking down hard questions and checking the answer before it reaches anyone. All of it with swappable parts: the guide itself notes you can change the model, the vector database and the embedding model with no vendor lock-in.

• The AI Analyst maps where each answer the agent gives comes from and which rules it must never break.
• The Forward Deployed Engineer builds the flow inside your operation: connectors, continuous updates, personal-data filtering, a check before answering and human sign-off where mistakes are expensive.
• Training teaches your team to keep the knowledge base current, which is how the agent learns what changed.

Support, contracts, internal policies and catalogs are only examples. If a task repeats and depends on looking up company information, it can be automated, and with RAG done right the agent answers with today’s data.