RAG is not a moat

Almost every founder pitching me an AI product tells me the same thing about their moat, and it’s almost never true. They point at the model they picked, or the RAG pipeline they built, like it’s some kind of secret sauce. I get it, you spent three weeks wiring together an embeddings store, a retriever, and a prompt template, and it feels like an achievement. It is an achievement. It is also something any competent engineer can rebuild over a weekend with a laptop and a credit card.

I’ve built this stack more times than I can count now across CoolHax Labs projects, and the wiring itself stopped being interesting to me a while back. Chunk the documents, embed them, store the vectors, retrieve the top few, stuff them into a prompt. That’s maybe two days of real work if you already know what you’re doing. The model underneath is a commodity too. Whatever edge one model has this month gets erased by whichever one ships next month. None of that is where the actual value sits.

The stack was never the point

If your defensibility story starts with “we use RAG” or “we fine-tuned a model,” I already know how the rest of the meeting goes. That’s table stakes at this point, the kind of thing a competent team ships in their first sprint. Proprietary data nobody else has access to matters more. So does an eval harness that tells you, honestly, which version of your product actually works better, so you improve faster than whoever copied your UI last month. Distribution you’ve already earned counts too, and trust built up over months of good answers counts most of all, mostly because it’s the slowest thing to build and the easiest thing to lose.

Anyways, this isn’t a popular thing to say to a founder who just spent their runway on infrastructure. But it’s the honest read, and the dishonest one doesn’t help anyone ship something that survives contact with a real competitor.

What Holt actually protects

I open sourced Holt, my personal AI agent OS, and people sometimes ask why I’d give away the code if I actually cared about defensibility. Holt’s whole premise is that your memory, the conversations, the context, the years of accumulated state about how you work and think, stays yours and persists on your own machine instead of living inside someone else’s platform. A competitor can fork the repo tomorrow. They still don’t have your memory. They still don’t have mine. The retrieval logic was replaceable in an afternoon. The thing being retrieved, and who controls it, wasn’t.

That’s basically the same lesson from the fintech work, just inverted. We shipped production AI to 23 million users there, and the model choice changed more than once during that stretch without users noticing. What didn’t change was the data pipeline feeding it and the verification layer catching it when it was wrong. Rip out the model, the product survives. Rip out the data or the eval discipline, it doesn’t.

The uncomfortable part

I’ll admit I used to believe the stack was the hard part too, back when RAG still felt novel. It took building the same pipeline for the fourth or fifth client before I noticed I was solving the identical problem every time, which is usually a sign you’re looking at commodity work, not IP. The actual hard conversations with clients now are about data rights, about whether they even have data clean enough to be worth retrieving from, about whether anyone on their team will look closely enough at outputs to catch the model being confidently wrong.

None of that shows up in a demo. A demo just needs the pipeline to answer one good question in front of the right person. It shows up six months in, when the competitor who copied your architecture over a weekend still can’t catch up, and they can’t quite figure out why, because the code was never what they were missing.

I still get asked, at least once a week, what stack I’d recommend for a new AI product. I answer it, because it’s a fair question and the answer is genuinely simple now. Then I ask what data they have that nobody else has, and the room usually goes quiet for a bit.