There are some documents you simply do not paste into a cloud API. Not because you don't trust the provider exactly, but because the moment the text leaves your laptop, you've made a decision on behalf of people who never agreed to it. I had a set of documents like that recently. Genuinely sensitive. The kind where you read the terms of service of every AI tool you own and then quietly close all of them.
And yet I wanted help. The work was the sort of thing AI is brilliant at: read a pile of related documents, spot the inconsistencies, reason about who said what and when. I just couldn't let any of it off the machine. So I set myself a small, stubborn goal. I wanted a local Claude. A model living entirely on my own hardware that could read these files, never phone home, and still be useful.
I'll be honest with you, I expected this to be a weekend. It was not a weekend.
Where local AI actually is, with the marketing stripped off
The first thing that happens when you go looking for "run AI locally" is you get told it's solved. Just install Ollama. And to be fair, Ollama is lovely. But Ollama is a model runner. It pulls a model down and gives you something to talk to. What it does not give you is any sense of your documents. There's no layer that reads a folder of files and feeds the right bits into the model. That's a whole separate problem called retrieval, and Ollama doesn't pretend to solve it.
So you go up a level to something like AnythingLLM, which does have the retrieval layer. The RAG part, where it chunks your documents up and pulls relevant pieces into the prompt. Promising. Except it wanted me to upload documents into its own workspace, and pointing its agent at a normal Windows folder full of files simply didn't work. Pointing @agent at a folder won't work. It's not built to wander around your filesystem the way Claude Code does on my machine every day.
Fine, I thought, I'll go the other way. I'll take a small model, drop it into an agent loop, and let it call tools to read the folder itself. And a small 4B model dropped into agent mode just hangs. Sits there. Thinks about it. Produces nothing. Nobody writes about this part, by the way. The blog posts all stop at "and then it summarised my PDFs." Mine stopped at a spinner.
The number that actually mattered wasn't the version number
Here's the thing though. I kept reasoning about this like a software person, comparing model names and version numbers, when the real constraint was sitting in my laptop the whole time. I've got an RTX 5070 Laptop GPU. Sounds fancy. It has 8 GB of VRAM. The machine has 63 GB of system RAM, but for AI that almost doesn't matter, because the only question that counts is: does the model fit in 8 GB of VRAM?
If it fits in VRAM, the GPU runs it and it's quick. If it doesn't, it spills onto the CPU and system RAM and it crawls. I looked at a bigger, cleverer variant and it was 23 GB. Even the smallest cut of it was 17 GB. Both of those would have run on my CPU at the speed of a tired sloth, and I'd have spent the whole time wondering whether it had crashed.
What actually fit was qwen3:8b — roughly 5 GB at Q4 quantisation, and crucially it supports tool-calling, which you need if the model is going to read files itself. So it became the daily driver. Not because it was the best model on any leaderboard. Because it fit in the box I actually own. VRAM fit beats version number every single time, and I wish someone had told me that on day one instead of letting me drool over benchmark charts for models I could never run.
The small-model ceiling, which I hit roughly every four minutes
Now, an 8B model is small. And small models do small-model things. I watched qwen3:8b confidently invent a subfolder that did not exist and then try to read from it. I watched it reach for entirely the wrong tool for the job. I watched it loop happily past its 10-round tool-call budget, doing the AI equivalent of pacing around a room. And my personal favourite: it would open a tiny file, see not very much, and decide the file was "empty" and move on. The files were not empty. They were just small.
Ouch. Also fair. I was asking a small model to navigate a filesystem like a junior developer, and it doesn't have the spare intelligence to do that and the actual task.
The fixes were the boring, classic ones, and they worked:
- Stop making it navigate. Instead of letting the model wander the folder, I preloaded every document straight into the context up front. No filesystem to get lost in. The text is just there.
- Drop the temperature to 0.3. A creative model is the last thing you want when the job is "read exactly what's on the page." Lower temperature, fewer flights of fancy.
- Make the directory-listing tool forgiving. When the model asked for a folder slightly wrong, the tool met it halfway instead of throwing a tantrum.
Better. But it laid bare a harder truth that no amount of prompt-tuning was going to fix.
The pivot: stop asking the small model to be clever
I kept trying to get qwen3:8b to do the nuanced bit. The actual reasoning. Read these documents, weigh them up, tell me where they don't line up. And it just couldn't. Local models at this size are, candidly, shallow and generic next to a frontier model. Where Claude gives you a considered, specific read, the local model gives you something that sounds like it skimmed a Wikipedia article on the topic. The harder I pushed, the more obvious it became that chasing a bigger and bigger local model was the wrong fight entirely.
So I stopped. And I asked a different question: what is the actual part that has to stay on my laptop?
It wasn't the reasoning. The reasoning could happen anywhere, in principle. The thing that could never leave was the identity of the people in the documents. The names. The initials. The detail that ties this text to real human beings.
Which meant I'd been building the wrong tool. I didn't need a local genius. I needed a local redactor. A small, dumb, fast model whose entire job is to strip out identifiers and nothing else. And stripping identifiers is the one thing a small model can genuinely do well, because it's discovery, not judgement. Find the names, swap them out, write down what you swapped. No nuance required.
How the redactor actually works
The design got simple very quickly, which is usually the sign you've finally found the right shape. The local model reads the documents and pulls out every identifier. A person referred to by their initials — say "FM" — becomes [PERSON 1]. If the same person is referred to in three different ways across the documents, all of those aliases fold into the one pseudonym, so the anonymised version stays internally consistent. And the mapping from real identifier to pseudonym gets written to a small file, .redaction/map.json, that never leaves the machine.
Once that's done, the documents are anonymised. "FM" is now "[PERSON 1]" everywhere, the map is locked in a local file, and the text that comes out the other side describes events involving people who can no longer be identified from it. That text is safe to send to the frontier model. So the local model does the private bit, Claude does the clever bit, and at no point does a real person's identity touch the cloud. If I need to read the result back in real terms, the map lives on my laptop and I can reverse it myself.
The test that made me trust it
There's a trap in this kind of find-and-replace work and it's an obvious one once you've been bitten. If you redact "FM" by blindly swapping that string everywhere it appears, you don't just catch the name. You catch the "fm" buried inside other words. The word CONFIRM contains an "FM". So does "affirm." A naive redactor turns a perfectly innocent "CONFIRM" into "CON[PERSON 1]" and now your anonymised document is gibberish, and worse, you can't trust any of it.
So the make-or-break test was straightforward: redact "FM" everywhere it appears as a name, and leave the word "CONFIRM" completely untouched. The fix is word-boundary matching, so "FM" only matches when it stands alone as a token, not when it's wedged inside a longer word. I ran it. "FM" became "[PERSON 1]" everywhere it was a name. "CONFIRM" stayed "CONFIRM." That was the moment I actually trusted the thing.
What I'd tell myself at the start
The lesson I keep coming back to is that "local model versus frontier model" is a false binary, and I wasted a fair bit of time treating it like a choice I had to make. It isn't. The two are good at completely different jobs, and the trick is to give each one the job it's actually suited to.
Use the local model for the work that must stay private — reading the documents and stripping the identifiers. That's discovery, and small models are fine at discovery. Then use the frontier model for the nuanced reasoning, on content where the real people have already been anonymised away. The privacy and the intelligence stop fighting each other the moment you split the job along that line.
And the other lesson, the cheaper one I'd have paid good money to learn on day one: when you're picking a local model, stop reading the leaderboards. VRAM fit beats version number every single time. The cleverest model in the world is no use to you if it crawls along on your CPU. Find out how much VRAM you've got, find the best model that fits inside it, and build around that. I went looking for a tiny Claude and ended up building something better — a small tool that knows its one job, hands the thinking off to something cleverer, and keeps the people in those documents exactly where they belong. On my laptop.
