There is a particular flavour of terror that only happens after everything has gone right. The build was clean. Staging was green. The sandbox payment went through end to end, the little confirmation screen appeared, and I sat back feeling like a man who had earned his cup of tea. Then we flipped it live, a real customer tried to pay, and the whole thing fell over with an error message that, at first glance, made absolutely no sense.
The error was this: '<' is an invalid start of a value. A JSON parser, choking on a less-than sign. I'll be honest with you, the first time you see that one it looks like gibberish. It is not gibberish. It is one of the most useful error messages in the business, once you know what it's actually telling you. And it taught me something about the gap between "works in the test environment" and "works in the real world" that no amount of green ticks had prepared me for.
The job: rip out an ageing payment gateway
The task itself was the unglamorous sort that keeps real software alive. The setup was an ageing card-payment integration bolted onto an old gateway, and the job was moving it to a new card processor. Nothing exciting on paper. Swap the plumbing, keep the money flowing, try not to break anything that takes people's card details.
This is exactly the kind of work I now do with AI sitting alongside me, and it's the kind of work where being the product manager actually matters. The AI doesn't decide we're migrating. It doesn't decide which path to keep open if things go wrong. It executes; I steer. And on this one, the steering started with a problem before I'd written a single line of integration code.
When the AI can't read the docs
I wanted the AI to read the new processor's API documentation and build against it. Sensible. Except their docs are a JavaScript single-page application. The whole thing renders client-side, which means a fetch tool that grabs the raw HTML gets back a near-empty shell and a polite suggestion to enable JavaScript. The AI could see the page existed. It could not see a single word of what was on it.
That sounds a lot like a dead end. It is not. If you can't get the AI the shape of the API up front, build something that doesn't need to know the shape yet. So that's what we did. The first pass was deliberately API-shape-agnostic: an integration layer that handled the flow, the states, the error paths, without hard-coding the exact field names and response structure of a processor whose docs we couldn't read.
Once I'd manually worked out the actual request and response shapes, we did a second pass and wrote a properly typed client on top of that agnostic foundation. Typed, because I like the compiler shouting at me before a customer does. The order of operations mattered here. Build the thing that survives not-knowing first, then tighten it up once you know. It's a pattern I'll be reaching for again whenever a third party hides their docs behind a wall of JavaScript.
Green lights all the way down
And it worked. The sandbox gave us a payment link, we walked a test payment all the way through, and it completed. On staging, every path I could think to exercise came back green. This is the bit where, if I'm not careful, I start believing the job is done.
Here's the thing though. A sandbox is a controlled, friendly little world that has been built specifically to say yes to you. It exists to prove your code talks to their code. What it cannot do, by its very nature, is reproduce the environment your production server actually lives in. And that's where the trouble was waiting.
The less-than sign that gives the game away
Live. Real customer. '<' is an invalid start of a value.
Once the initial confusion passed, the penny dropped, and this is the lesson I want you to take away even if you read nothing else. When a JSON parser complains about a stray "<", it is almost never a JSON problem. It is an HTML page wearing a JSON costume. The very first character of an HTML document is a less-than sign, the opening of a <!DOCTYPE> or an <html> tag. Your parser expected { and got <. Something handed your code a web page where an API response should have been.
So what was the web page? It was a block page. The new processor's web application firewall — a WAF — had looked at the request coming from our production server, decided it didn't like it, and served back an HTML "computer says no" page instead of letting the request through to the API. The give-away that you are being blocked rather than served is exactly this: a human-readable HTML page turning up where machine-readable JSON belongs.
The firewall you can't see from the sandbox
The cause was an IP-address block. The WAF was refusing requests coming from our production server's IP. And the sandbox never showed this because the sandbox doesn't sit behind the same firewall, doesn't care about your server's IP, and is positively eager to let you in. The block only exists in production. It cannot, by definition, be caught in a test environment that doesn't have the firewall in front of it.
It got better, in the way these things do. The hosting plan used dynamic outbound IP addresses. So there wasn't a single fixed IP I could hand the processor and ask them to whitelist — the address could change. The one stable thing in the whole mess was the support ID the WAF stamped into its block response. That ID identifies the blocked request to the processor regardless of which of our shifting IPs it happened to come from, which turns out to be the thing their support team actually needs to find you in their logs.
This is an entire class of failure that simply does not exist until you go live. WAFs, IP allowlists, dynamic infrastructure, the dozen invisible bits of network plumbing between your code and theirs. Your tests can be perfect and your code can be correct and you can still get a less-than sign where your JSON should be, because the failure isn't in the code at all. Nobody writes about this part. We all write about the clever build and the typed client. Almost nobody writes about the firewall nobody could see until a real customer hit it.
The senior move is the boring one
Now, the heroic instinct here is to stay up late firefighting. Hammer the processor's support line, redeploy, fiddle with infrastructure, try to force the IP block open while customers can't pay. Resist it.
We reverted to the old gateway. And the reason we could do that calmly, as a config swap rather than a 2am emergency, is the bit I'm proudest of. The migration had been built defensively: the old code path was left fully intact alongside the new one. We hadn't ripped the old gateway out the moment the new one looked ready. So reverting wasn't a panicked redeploy of yesterday's build — it was flipping a configuration switch back. Customers kept paying on the old gateway, the money kept flowing, and the processor got on with sorting out the IP block on their side without anyone holding their breath.
That's the genuinely senior call, and it's the un-heroic one. Revert to the thing that works, keep the business running, and finish the migration when the third party has unblocked you. No medals for it. No war story about the all-nighter. Just a quiet config swap and a client who never knew there was a problem.
What I'm taking from this one
A few things I want to hold onto, because I know I'll need them again.
First, "works in sandbox" is a much smaller claim than it feels like at the time. It proves your code is correct. It proves nothing about the production network it has to live in. Treat the first live transaction as its own test, because it is testing things the sandbox literally cannot.
Second, learn the tells. A JSON parser choking on a < means HTML, and HTML where you expected an API response usually means you're being blocked, not served. That five-second diagnosis would have saved me a good deal of staring.
Third, and this is the one I'd tattoo on the inside of my eyelids: build the migration so the old path stays alive until the new one has survived production, not just the sandbox. Defensive design is what turns a reversion into a shrug instead of a crisis. The AI is brilliant at building the new thing quickly. It's still my job, as the human in the chair, to insist we keep the old thing breathing until we're truly sure. The migration finished eventually, on a quiet afternoon, once the firewall stopped saying no. Which is exactly how a migration ought to finish.
