The Forest School App has a camera in it, and for about a day that camera was the most dangerous code I'd written all year. Not because it crashed. Because it worked too well.
I'd asked for a simple thing: let a leader photograph what the children are making and attach it to the session record. The AI built it in minutes and it was lovely. Tap, snap, attached. Then I asked the question that safeguarding software lives or dies on - "what happens when this child doesn't have photo consent?" - and realised the happy path I'd just been admiring would have cheerfully photographed a child whose parents had explicitly said no. That isn't a small bug in a forest school. That's the whole reason a parent might never trust you again.
The camera that refuses to open
The rule I landed on is blunt, and the bluntness is the point. A child without photo consent cannot be photographed. Not "a warning appears first". Not "it gets flagged in the report afterwards". The camera does not open. If a leader tries to take a picture and there's a child in the group without consent, the app stops them at the door rather than tidying up after the fact.
This feels heavy-handed until you picture the actual moment. It's a wet Tuesday, there are a dozen children and a fire, the leader has about four seconds of attention to spare, and they're holding a phone. "Remember to check the consent list" is not a safeguard in that moment. It's a hope. A rule that depends on a busy human remembering it under pressure isn't a rule, it's a wish. The only version that actually protects a child is the one where the wrong action is impossible, not merely discouraged.
The AI's instinct is to make it work
Here's the thing though. Everything about how an AI is trained pushes in exactly the wrong direction for this. Ask for a camera and you get a camera that opens. Ask it to handle an edge case and it'll handle it gracefully - which usually means letting the action through with a polite note attached. The model is relentlessly optimised to make the thing function, and "function" reads as "the user gets to do what they were trying to do".
So the unsafe action was the easy, default, frictionless one, and every instinct in the model leaned towards it. I had to keep saying, in effect, no - the default is no. Not "open the camera and check consent as you go". Closed unless proven open. The AI wasn't being careless; it was being helpful, and helpful is the wrong setting when the helpful thing is the thing you're trying to prevent. I'm the product manager here, and the product decision was that the software should be deliberately unhelpful in precisely one direction.
Deny by default, and prove it
Once you frame it that way, the testing flips too, and this is the bit I keep thinking about. The test that matters isn't "can a consented child be photographed". Of course they can; that's the easy path and it more or less writes itself. The test that matters is "does the camera stay shut for a child who isn't consented". The assertion is about the thing not happening.
That's a strange muscle to build, because most of our testing is about confirming features work. Here the most important test confirms a feature deliberately doesn't. Deny by default, and then write the test that proves the denial holds even when someone changes the code around it six months from now. If that test ever goes green when it should be red, you don't have a failing test - you have a child who could be photographed without consent, which is a different category of problem entirely.
The gate has to be on both ends
The first version of the gate lived entirely in the app. Camera won't open, job done. Except it isn't, because a gate that only lives on the device is a suggestion. Anything that can talk to the server directly can stroll straight around it.
So the consent check runs in two places: on the device, where the camera won't open, and on the server, where an upload tied to a non-consented child is refused outright. The device gate is for the honest user in the moment; the server gate is the one that actually holds. If a rule matters, it has to be enforced where it can't be bypassed, which is never only the client. This isn't paranoia about my own leaders - it's just an honest acknowledgement of where the real boundary sits. The AI, left to itself, will happily build the client-side gate and call it done, because on the screen it looks finished. Looking finished and being safe are not the same thing.
Where this goes beyond cameras
Once you start seeing software this way, the pattern is everywhere in anything that has to keep people safe or keep you compliant. The camera is just the clearest example.
Ratios are the same shape. You can't quietly start a session that's under-ratio and sort it out later - the app stops you, and going ahead anyway is a deliberate, logged override with a reason attached, not a silent default. The risk-benefit assessment is the same shape again: it's captured before the activity, as a gate you pass through, not a form you reconstruct afterwards to satisfy an inspector. In each case the constraint isn't bolted onto the feature. The constraint is the feature. The whole value of the thing is that it makes the unsafe or non-compliant path harder than the safe one.
What I'd tell you
In ordinary software, "make it work" is the goal and constraints are the annoying bits you handle grudgingly. In safeguarding or compliance software, that's inverted: the constraint is the product, and "just make it work" is precisely the instinct you're guarding against. AI is genuinely brilliant at the first kind of software and quietly, confidently wrong for the second, because it will smooth the path you actually need to block.
So when you're building the kind of software where being wrong has a real-world cost, don't ask the AI to make it work and then sprinkle the safety on top. Decide the gates yourself - that's the human's job, the product manager's job - and state them as the unbreakable rules they are. Then write the tests that prove the gate holds, especially the tests that assert the thing must not happen. Let the AI build the rest at the speed it's so good at. But the cage around the dangerous bit? You design that, and you check it yourself. A camera that refuses to open isn't a limitation of the Forest School App. It's the most important feature in it.
