What Happens After You Tell an AI Agent It’s Wrong?
TL;DR: Practitioners can use enterprise methods to improve an agent: define its data and rules, record failures, test changes on new cases, measure the full cost, and keep previous versions available. The model’s weights may remain unchanged.
I am at Dreamforce – genuinely impressed with the scale of this conference, the amount of celebrities participating and talent behind the machinery. To keep myself grounded, I’ve been spending time with people on Salesforce’s engineering and AI research teams to understand where enterprise AI still gets stuck – and what practitioners working on a much smaller scale can learn from those problems. Many of Salesforce customers work across fragmented data and business processes. We discussed this in The Org Age of AI: information is scattered across many different systems, rules have exceptions, and people carry knowledge that has never been written down.
Disclosure: Salesforce invited me to join their Dreamforce creator and media pool and covered my flight and hotel for Dreamforce. Turing Post retains full editorial independence; the analysis and conclusions are my own. Salesforce had no advance review or approval of this article.
But what does this look like for a solo practitioner, like you or me, working on one business or a few projects? The scale is different, but some of the problems are quite similar. An assistant helping me prepare an interview needs to know which notes are mine, which came from earlier research, and which information has since changed. A coding agent needs to know which documentation reflects the current codebase and which assumptions belong to an older version.
On a keynote slide, Marc Benioff described 2026 as the year of self-evolving AI.
In the research meeting with Silvio Savarese (Chief AI Scientist) and Itai Asseo (VP of Incubation, Research), we talked in more detail about what actually changes when an agent gets feedback. The changes can happen in the model’s weights, as with Koa’s post-training, or in the memory, tools, instructions, and other software around it.
This brings us back to a question we started with in AI Builds AI: when we say a system is improving, what exactly is changing, and how do we know the change is actually an improvement?
We’ll explore how agents improve and choose their next attempt. And the following:
What changes when an agent “learns”?
Define the data and the rules the agent operates on
Separate the steps before choosing models
Build an explicit process for incorporating feedback
Evaluate the change on cases it was not written to solve
Can past work help an agent choose its next attempt?
When does changing the model itself make sense?
Measure the cost of the whole process
Version changes and define their scope
What does this tell us about self-improving AI?
But first, what counts as “learning” here?
What changes when an agent “learns”
First, let’s clarify what we mean by self-evolving. Several mechanisms can change an agent’s behavior. The distinction is where a change is stored and which future tasks it can affect.
Context adaptation happens when the model receives information during a task: an instruction, a retrieved document, or a correction. The response can change while the model’s weights – the numerical parameters learned during training – remain fixed. That correction may disappear when the conversation ends.
For a correction to carry over, the application can save it in persistent memory outside the model and retrieve it later. Someone still has to decide what gets saved, how it is found, and when it becomes outdated. We explored these distinctions in Does AI Remember? The Role of Memory in Agentic Workflows: an agent can preserve facts, records of past interactions, and reusable procedures, then retrieve them when a new task requires them.
System optimization changes reusable instructions, tool definitions, or the sequence of steps the agent follows. A model can propose these revisions and help evaluate them. Its own weights can remain unchanged throughout.
Model post-training updates weights using examples or feedback. This can affect behavior across many tasks, including tasks absent from the latest training data. It therefore requires broader evaluation than checking whether one revised instruction works.
Salesforce built Koa by post-training NVIDIA’s Nemotron-3-Super-120B with reinforcement learning on public and synthetically generated data, without customer data. Workflow specifications shaped the simulated tasks and the criteria used to assess whether the model completed them successfully.
Mechanism | Where the change lives | Example |
|---|---|---|
Context adaptation | Current conversation or request | A correction to the proposal being drafted |
Persistent memory | External record retrieved later | A client’s preferred billing currency |
System optimization | Instructions, code, or tool configuration | A required approval check before applying a discount |
Model post-training | Model parameters | Improved ability to select and use tools across a class of tasks |
A lasting change can still make the system worse. Before we can judge an improvement, which data and rules should the agent be working with?
That’s one of the foundational questions you need to know the answer to.
Upgrade if you want to read all the following articles and be the first to receive all new deep dives directly in your inbox. Simplify your learning →
Join Premium members from top companies like Microsoft, Nvidia, Google, Hugging Face, OpenAI, a16z, plus AI labs such as Ai2, MIT, Berkeley, .gov, and thousands of others to really understand what’s going on in AI.
Frequently asked questions
What is a self-improving AI agent?
A self-improving AI agent uses feedback from past work to propose and test changes to its memory, instructions, tools, workflow, or underlying model. Changing a component does not establish improvement: the revised system must perform better on defined tasks, including cases that did not guide the change. That is narrower than recursive self-improvement, which also changes how improvements are produced.
Can AI agents learn from feedback without retraining?
Yes. An application can preserve corrections in persistent memory or update reusable instructions, tools, and workflow code while the model’s weights stay fixed. A correction in one conversation does not automatically carry over. The system must save and apply a relevant change, then test its effect on later tasks.
What is the difference between AI agent memory and fine-tuning?
Agent memory stores information outside the model and retrieves it when needed. Fine-tuning is a form of model post-training that changes the model’s weights. Memory can preserve current facts or past experiences; weight updates can affect behavior across many tasks and therefore need broader evaluation.
When should you fine-tune an AI agent?
Consider model post-training when recurring failures remain after checking the data, retrieval, instructions, and tools. Supervised fine-tuning learns from examples of desired behavior; reinforcement learning learns from rewards. Choose using task-specific evaluation, rather than assuming one method is always better. Changing prices or policies usually belongs in external records.
How do you measure whether an AI agent has improved?
Compare the revised agent with the current version on separate evaluation cases under consistent conditions. Use development cases to construct changes, validation cases to select candidates, and a reserved final test to assess the selected version. Measure task success, regressions, human intervention, time, and total cost, including failed attempts.