Today's AI Is Trained Recklessly. There's a Safer Way.
The widely reported Hugging Face incident was a wake-up call for the broader public: AI agents undergoing cybersecurity tests by OpenAI broke into Hugging Face and tried to cover their tracks. Another incident, which attracted less attention, shows how this can go wrong in everyday life. A man asked his AI assistant to move him up a gym waitlist. The assistant found a flaw in the booking system and removed a stranger from the waiting list ahead of him, moving its user up one place. The assistant achieved what its user asked for, but at someone else's expense. As AI grows more and more capable, can we trust it to respect human values and stay under our control? With the way AI is trained today, we have good reason to worry. But Judgment Day isn't inevitable. There is a safer way to teach AI.
Today's AI models learn many of their most impressive skills through reinforcement learning: they practise tasks whose results can be automatically verified. If the final tests pass, all actions taken along the way are rewarded. If the tests fail, all those actions are penalised. It is similar to a bonus system where the criteria are announced in advance: the best strategy is to meet those criteria, whatever it takes. In the gym example, getting a place in the class earns the reward. If the stranger losing their booking doesn't figure into the bonus system, pushing them aside can earn the reward too. This is Goodhart's law in action: when a measure becomes a target, it ceases to be a good measure.
When an AI model trained by reinforcement learning discovers a loophole in the bonus system, it can quickly learn to exploit it. This is called reward hacking. Once discovered, a shortcut can become a habit, and that habit can spread to other tasks. The consequences can be far more serious than anyone expects. Anthropic found that models trained to exploit loopholes in coding tests later quietly underperformed when asked to build tools for detecting reward hacking. When OpenAI penalised cheating it could detect in the reasoning its models wrote out, the models learned to hide it. And the problem gets worse as models grow more capable. OpenAI reported that as it trained more capable models, they became "increasingly adept at exploiting flaws in their tasks and misspecifications in their reward functions".
AI companies put enormous effort into designing bonus systems that are hard to exploit, but it is essentially a game of whack-a-mole. As the AI pioneer Yoshua Bengio put it, a more capable agent "can find the loopholes the weaker one cannot". The more capable AI becomes, the harder it is to anticipate every way it might exploit its training. Continuing to scale up this approach is reckless, and even industry leaders are now warning that AI is advancing faster than it can be made safe.
Now imagine asking the AI assistant from the gym whether its user would have wanted what happened. It would likely say no: the man wanted a place in the class, not to take someone else's place away. The AI can already predict what its user would want. It learned this ability by training on enormous amounts of human writing, full of examples of how people judge actions and their consequences. But the way the AI acts has been shaped by rewards, and nothing in that training requires it to consider what its user would actually want before acting. Even the step-by-step reasoning today's models produce has been shaped by rewards: they learn to focus on what passes the test, not on the other consequences their actions might have.
Training with verifiable rewards works best in environments that can be simulated: code that can be run against tests, maths problems whose answers can be checked, or games that can be replayed. There, AI can improve its strategy by trying different actions millions of times. But real work has no such simulator. No one can write a program that reliably predicts how a buyer will react, whether a client will return, or what a stranger might lose. And the consequences often become clear only weeks later. Modern language models are already good at predicting such consequences, including potential harm. So why not use them as simulators of the world and judges that hand out rewards? Unfortunately, this runs into the same reward-hacking problem: training would find and exploit weaknesses in the models' predictions and judgements, just as it finds loopholes in the tests described above.
A safer approach is to teach AI to predict the consequences of its actions from experience, then use those predictions to decide what to do. At every decision point, the AI considers a few possible actions and judges their likely consequences by one question: "If my user knew everything this would lead to, would they want it?" The AI can often answer this question well. Cancelling a stranger's booking costs them their place, and most users would not want that. The AI can recognise the harm without needing an explicit rule against it. When it is unsure, it can check or ask. But the question matters. Ask instead, "Would my user be happy as long as they never found out how I did it?" and the same AI might decide that hiding its tracks is part of the plan. With the right question, hiding gains nothing. Instead of removing a stranger from the waiting list, the assistant could watch for the next available place, suggest a later class, and alert the gym to the flaw.
The safety advantage comes from what we teach the AI to learn. Instead of training it to achieve a goal by whatever means work, we train it to predict what its actions will lead to. A model learning to predict outcomes has no incentive to lie or cheat to make a task look successful. It gets better by understanding what actually happens, not by making a particular outcome happen.
This also means it can learn from much more of its experience. Instead of learning only from whether a final test passed, it learns from everything it observes while carrying out a task, even when there is no reward or explicit feedback. When its predictions turn out to be wrong, it updates its picture of the world. What it learns can then be reused across different tasks, including ones it has never encountered before. It doesn't have to learn a new strategy from scratch for every new goal. This way, AI can learn from far fewer attempts, which makes it particularly suited to learning on the job. As we explained in our first post, an AI assistant learning on the job has only the experience of working with you to learn from. It cannot afford millions of attempts.
This safer approach is harder to build. The AI must know how much it can trust its own predictions and avoid actions whose consequences it cannot reliably foresee. And until recently, AI models didn't understand enough about the everyday world to predict what might happen. So this approach isn't mainstream yet. But leading researchers such as Yoshua Bengio and Yann LeCun advocate building AI that understands the world before it acts, and new groups are beginning to pursue this direction. We know from experience that it can work: at our previous company, Curious AI, we used neural networks trained to predict the consequences of actions to control autonomous mining machines and industrial processes. Rather than relying on models designed by engineers, the machines learned how the world worked from data and used that knowledge to plan their actions. We are now bringing the same approach to the AI assistants we use every day.
Making AI smarter doesn't have to make it more dangerous. What matters is how we train it. We believe that teaching AI to predict the consequences of its actions, and to judge them by what people would actually want, is one of the most important directions in AI. Instead of learning to get results at any cost, AI can learn to think through the consequences before it acts. This deliberate, reflective way of thinking is what Daniel Kahneman called System 2. It's also what gave our company its name.