We're Gonna Need a Bigger Moat

Written from inside the ten weeks it describes: the LLaMA weights had leaked onto Discord in early March, Vicuna had just hit 90% of ChatGPT's benchmark performance for around $300 of fine-tuning, and Steve is racing to explain, with a lot of swearing, why that sequence should terrify anyone whose business plan amounted to "GPT plus a wrapper." The mode is manic news explainer, stuffed with jokes about Zuck losing the "Clown Jewels" and what Microsoft and OpenAI were allegedly stocking in the executive washroom. Steve was building Cody at Sourcegraph when he wrote it, and the piece doesn't hide the conflict of interest; it closes by turning the moat question on his own employer, and readers can judge that answer for themselves.

As prophecy it split down the middle. Open-weights models really did commoditize the base layer, the LLaMA family really did become a standard, and "fine-tune something small on your own data" grew into an actual enterprise market. But the essay also swallows the leaked memo's most quotable number whole — training costs falling from a billion dollars to a hundred — which conflated fine-tuning a leaked model with training one from scratch, and the frontier labs, whose moat was supposedly gone "even by the end of this year," spent the next two years widening it instead. Steve hedges exactly once, with "We do not know when the lines will cross," and that sentence has aged better than most of the confident ones around it.