Modern AI: A Brilliant Mind with Peculiar Memory
The first time you give a modern AI agent a real task, it feels like science fiction come true. It builds a working website, researches a whole market, analyses a year of sales figures, often better than you would, and always faster. Within days, working without it feels like working without the internet.
Weeks pass, and something starts to feel off. You find yourself explaining the same things again and again. You correct a mistake, only to watch the agent make it again. You ask it to develop a better work habit; it writes itself a note, follows it for a while, then slips back into its old ways. Eventually, you catch yourself wondering: how can something so brilliant be so bad at learning? It can solve problems that would take you days, yet it struggles to learn from the experience of working with you. It never really gets to know your work the way a human colleague would.
The reason is simple to anyone who knows how today's AI is trained: its learning ends in the lab, before it ever meets you. First, it gains broad knowledge by reading enormous amounts of text. Then, it learns to be helpful from people's ratings of its answers. Finally, it learns to carry out complex tasks by practising on millions of tasks whose results can be automatically verified. But once it meets you, it stops learning. Instead, it takes notes. AI companies now equip their assistants with written memories: notes about you, your preferences, and your corrections, which the assistant reads when working with you. That helps, much as a manual helps a new hire. But unlike a human colleague who gradually learns the job, the assistant must keep consulting its notes.
A job manual works well until it gets too long, which can easily happen in real-world work. Imagine an AI assistant working alongside an estate agent. Over time, its notes accumulate observations: buyers who ask about heating bills usually make an offer; listings should lead with photos taken in morning light; the word "cosy" should never appear in a listing; and one building manager is always late with paperwork. Sale after sale, the notes pile up into the thousands, and every new sale involves dozens of them. Eventually, there are so many notes that more and more of them silently slip from the assistant's attention.
Research backs this up: studies show that beyond a certain point, giving AI models more instructions or text makes them increasingly likely to quietly ignore some of what they were told. Organising notes and retrieving them through an index helps, but doesn't solve the problem. The assistant still has to find the right information at the right moment, and that often fails. In a way, note-based memory resembles the expert systems of the 1980s: AI built around thousands of hand-written rules that had to be combined to handle the messy real world. Modern AI took off precisely when it stopped following rules and started learning.
What's missing is learning that changes the AI itself, not just the notes it reads. In technical terms, this means changing the model's weights, the internal parameters that determine how it behaves. This is called continual learning, and leading AI researchers like Ilya Sutskever and Andrej Karpathy consider making it work one of the field's big open problems. With continual learning, the assistant's training doesn't end when it meets you; it continues through the work you do together. The corrections you make don't become rules it must remember to check; they become second nature. The estate agent's assistant no longer has to search through thousands of notes before each sale. Like a seasoned agent, it just knows which buyers are serious, how a listing should read, and whose papers to chase.
Training AI agents in the lab requires vast amounts of data and computing power. So can continual learning really work for each person? Yes, because we don't start from scratch. The big model trained in the lab stays as it is, supplying everything it already knows. What the AI learns from working with you goes into small models that learn alongside it. Lab training is expensive for two reasons: the models are enormous, and they learn from an internet's worth of text and millions of practice attempts. Learning your work needs neither. The small models can run on a laptop, learn from the record of your work together, and improve without waiting for the next version of the big model. Like a new hire, your AI assistant arrives with a wealth of knowledge and skills, thanks to the large model trained in the lab. The small models learn the rest on the job: which information to recall and when, what outcomes to expect, what you prefer, and what tends to work in familiar situations. What they learn is small compared with everything the large model already knows, but it's what makes the assistant good at your job.
We are already seeing this work in our own AI assistants. We use them every day, and they are starting to feel more like colleagues who know our work habits. At first, we put our corrections into notes and instructions, but the assistants followed them inconsistently. One instruction told them to let us know what they were doing before starting a long task. They often went silent anyway, and we kept having to remind them. Now, part of our system has learned this habit from working with us, incorporating it into its own model weights. Since then, we've seen nearly a tenfold reduction in the unwanted behaviour. Other annoyances have quietly become rarer too. We are now starting our first pilot outside the company.
But as AI starts learning from experience, two questions become critical. First, can we keep it safe? As AI grows more capable and learns on the job, will it learn to do what we actually want, or discover ways to get results at any cost, even by cheating or causing harm? Our next post explains why today's methods of training AI can be dangerous, and how a different approach to learning could make AI safer. Second, who owns what it learns? If your assistant becomes an expert at your work, will that expertise belong to you or to the company that built it? That question determines whom AI will ultimately work for, and it is the subject of our third post.