What does it mean for AI to learn from your business?
Using AI for six months does not guarantee it improves. The useful question is what changes: its source knowledge, its working procedures, or the model itself.
Consider a business that has used an AI support assistant for six months. Employees have corrected its answers, explained product changes, and shown it how to handle exceptions. Thousands of conversations are sitting in a database.
The assistant still makes a mistake someone corrected last week.
That can happen even when every conversation has been saved. Experience becomes useful only when something carries the lesson into the next relevant task. The system might need a current source document, a revised procedure, or training that changes the model's behavior. Those are different interventions with different costs and limits.
When a supplier says its AI learns from your business, the useful question is what actually changes, and how anyone establishes that the change helped.
What changes after a correction?
Suppose the support assistant gives a customer outdated setup instructions. An employee replaces them with the correct steps. There are several possible explanations for the mistake.
The current instructions may exist, but the assistant retrieved an older version. It may have found the right document and ignored the condition that applies to this customer's device. Or it may lack a reliable way to carry out the troubleshooting procedure at all.
Saving the employee's final answer does not identify which of those problems occurred. The correction is evidence to investigate. It is not yet a general rule.
We find it useful to separate three places an improvement can live:
These layers interact. A better procedure may tell the assistant to check a document's version before answering. A better retrieval system may make that procedure easier to follow. When facts change after training, the system needs a way to obtain them.
Anthropic's context-engineering account describes ways to manage the information available to an agent, including retrieval and structured notes across sessions. These mechanisms can improve what the system does without updating the underlying model weights. That distinction matters when deciding where a correction belongs.
Keep changing facts connected to their sources
In the support example, setup instructions change when a product changes. Keeping an authoritative document current lets the business inspect, correct, and withdraw that information directly.
The next question is whether the assistant uses it. If an old document remains in the index, or two versions look equally authoritative, adding another document may leave the original problem untouched.
For this failure, we would first check the full path: which document was retrieved, which version it represented, and which steps the assistant selected. The repair might be better version metadata, an indexing change, or a requirement to cite the governing instructions.
Training on the corrected answer would address a different surface. It might teach a useful response pattern, but it would not remove the obsolete source. When the product changes again, the business would face the same maintenance question.
This is why a growing conversation history is a weak measure of learning. The useful result is that the system consistently reaches the right information when it needs it.
A repeated correction can become a procedure
Now suppose the assistant has the right instructions but repeatedly skips a necessary check before recommending a reset. That looks more like a procedural problem.
The business could encode when the check is required, what evidence establishes that it passed, and what the assistant should do if the result is uncertain. The system can then test whether that procedure reduces the original mistake without causing unnecessary checks elsewhere.
Our article on learning from work develops this approach through maintained skills. The important point here is where the improvement lives: an inspectable working method, supplied to the model when relevant.
A written procedure can still fail: the assistant may apply it to the wrong task, or a tool may return incomplete information. A procedure also needs review when a product feature it depends on changes. Making the rule and its dependencies explicit gives the team somewhere to investigate. They can compare the requirement, the current source, and the observed failure.
Updating the model creates a different learning problem
In Why Nobody Has Solved Continual Learning, Charlie O'Neill of Baseten focuses on repeatedly updating model weights. He distinguishes a successful training run from a learning process that must keep working as new tasks and information arrive.
His discussion examines tradeoffs between supervised fine-tuning, which trains on examples of desired behavior, and reinforcement learning, which uses rewards to improve behavior through attempted tasks. He also distinguishes narrow, stable tasks from broader agents facing changing work. Near the end, he notes that frontier labs already run slower improvement loops; his concern is the difficulty of frequent updates. Talk, 9:42–19:00
The practical complication is that improving the model on new material can change behavior elsewhere. An empirical study of continual instruction tuning observed forgetting across knowledge, reasoning, and reading-comprehension tasks in its tested models. That establishes a real failure mode, not an inevitable outcome for every training method or present-day model.
For a business, this makes model training a development and release process. It needs suitable data, a training method, a comparison with the current system, and a way to detect damage to behavior that already worked. A stream of approved conversations supplies only part of that process.
Measure the lesson and what it might displace
For the support assistant, an improvement claim should name the failure it addresses. The team might expect fewer answers based on obsolete instructions, fewer skipped prerequisites, or less reviewer correction on a particular troubleshooting task.
Those should be checked separately. A model can become more fluent at explaining a procedure while still selecting the wrong version. An aggressive prerequisite check can reduce skipped steps while sending too many ordinary cases for review.
We would compare the revised system with the current one on fresh examples of the targeted failure and on important cases it already handles well. Record the relevant source versions, instructions, tools, and model. Otherwise a gain caused by a new document can easily be credited to a training run that happened at the same time.
The evaluation method needs its own scrutiny. If the reviewer accepts plausible but incorrect steps, optimizing for that review will preserve the wrong lesson. Give the agent a definition of done explains how to turn expected behavior into checks and keep acceptance criteria under separate control.
After six months, the support team should be able to identify corrections that stopped recurring, show which change produced the improvement, and explain how that change stays current. That is a more useful promise than a system that merely accumulates experience.