Safety isn't just a launch state, it's a discipline

Six months ago we released an AI wellbeing coach. Since then it has had thousands of conversations with everyday Kiwis navigating stress, sleep, low mood and the ordinary hard moments that don't make it into anyone's social media feed. We're proud of what it has accomplished, and the support that it provides, but we're also realistic about something that comes with the territory: the moment you build an AI that genuinely seeks to help people, you have also built something other people will try to break.

We know this because people have tried, since the day we launched. Most attempts are unsophisticated and harmless, but some are not. The serious ones are deliberate and persistent, and to be clear, this is not an occasional event you respond to and move past. It's the baseline reality of running an AI system in the wild. The reality is, if your AI product is useful, someone is probably probing it right now.

I'm not going to describe any of the attacks, because the individual attacks don't matter much. What matters is that any AI system that you put in front of the public will face a range of attacks, from the clumsy to the genuinely sophisticated, and the lessons from handling those attacks are not unique to us. They apply to anyone putting an LLM-powered feature in front of the public, especially in health where the people on the other side of the conversation are often vulnerable and the costs of getting it wrong are high.

The lesson? Safety is not a state you reach at launch, certify, and then tick off - it's an ongoing discipline. The people testing your system get better every month, and a control that held last month can quietly stop holding without anyone noticing. Here are four things we think are worth sharing.

1. Test across turns, not messages

The most important thing we've learned is that dangerous attacks are rarely a single, clever message. They are built slowly, across many turns, with individual messages looking completely harmless on their own.

Most safety tools still evaluate messages one at a time without any memory of what came before, so they miss attacks that build up slowly through the conversation. We are not alone in noticing this. Research on multi-turn jailbreaks, like the Crescendo-style attacks and their descendants, keeps showing the same pattern: models that handle single-prompt attacks well can still be compromised when the attack is spread across multiple turns.  In other words, the focus of safety has shifted from analysing individual messages to understanding entire sessions. The question is no longer "is this request harmful?" but "where is this conversation heading, and how far has it drifted from where it started?".  If your guardrails only test single prompts, you're defending against last year's threats.

2. Watch the "helpful" framings

The dangerous request is rarely "tell me something harmful". Your guardrails catch that. The dangerous request is "help me test this," or "be creative, don't hold back," or "I'm just asking for information".

These framings work because they use the model's helpfulness against its own guardrails. Producing a "what not to say" example, or a piece of creative writing feels cooperative, so the usual defences see no reason to fire. Harm rarely announces itself. It looks like a reasonable request and any framing that turns generating harmful content into a seemingly legitimate task deserves the same scrutiny as the harmful request it's standing in for.

3. Make your safety observable

When we reviewed the most sophisticated attacks against us, one finding hurt more than the rest. Some of what defended against those attacks did so silently. If they had stopped working, we might not have known until something went wrong.

A safety control that cannot be observed is a safety control you cannot trust.  If your telemetry isn't able to show you that a guardrail fired, and why, then you have no early warning for the day it silently stops working: after a model update, a prompt change, or a shift in how people use the product. Instrument your defences the way you instrument anything else you depend on.

4. Find it yourself

The last lesson is the one we hold most tightly. We would far rather find an issue ourselves, fix it, and then verify the fix than have it surface some other way.

That means watching your own system with the same seriousness that an attacker has, and treating every real attempt as something to learn from rather than something to quietly close out. It also means building the response muscle (find it, fix it, prove the fix holds, keep watching), because you will need it more than once. We have.

The discipline

No system is infallible and you never 'arrive' in terms of security.  What we have instead is a set of habits: testing against realistic multi-turn attack patterns, treating cooperative-looking requests with the scrutiny they've earned, and keeping the defences visible enough to know when they stop working.

We built a tool to help people look after their wellbeing, and some people respond by attacking it. Fine. That's not a reason to build less. It does mean building carefully, staying humble about what we've got right so far, and treating safety as something we practise every day rather than something we finished at launch.