For about a month, I tried to match our proprietary models with open-weight ones.
Not across the company — just in the engine room. OwlMeans is an AI development pipeline: you describe what you want as user stories, and a team of specialized AI roles turns them into a full-stack app you own. Under the hood, every one of those roles calls a language model. Today that’s a combination of Anthropic and OpenAI models — Claude doing most of the heavy lifting, with OpenAI in the mix for specific roles. It’s a setup I trust. It also has a price.
So I did the obvious experiment. The open-weight models had a very loud year — by mid-2025 the best open-weight LLMs were coming out of China, with GLM, Kimi, Qwen, and DeepSeek trading blows with the proprietary frontier on coding. They are a fraction of the price. If I could run my pipeline on open weights without the output getting worse, the math was irresistible.
I spent the month wiring them in, benchmarking, and tuning. Here is what actually happened — including the part I didn’t expect, which turned out to be the whole point.
The bet: the pipeline should make the model interchangeable
Here is the thing that made me think this would even work.
OwlMeans doesn’t hand a model a vague prompt and pray. It shatters a project into very small, tightly-scoped tasks, each running with very little context — a single user story, a single fix, a single file’s worth of change. A business-analyst role frames the work, an architect designs it, a developer role implements one slice, a fixer cleans up. Nobody asks one model to “build the app.” They ask it to do one small, well-specified thing.
That design has a quiet side effect: the smaller and more specified the task, the less the choice of model matters. When the context is tiny and the instruction is exact, almost any competent model lands in roughly the same place. There’s less room to be brilliant, and less room to wander off. This is also why our token costs stay sane — the research on agent loops is blunt that token cost grows with the square of the context you keep re-billing each step, so keeping each task small is the difference between a workable bill and a runaway one.
So my hypothesis was simple: if the pipeline already commoditizes the task, it should commoditize the model. Swap a proprietary model for GLM, and the app that comes out the other end should be basically the same app.
I was mostly right about that. And it didn’t matter the way I hoped.
What broke first: the models that think too much
The open-weight path did not go gently.
The first wall was the “thinking” models. GLM, Kimi, Qwen — they reason by default. That sounds great until you put a hard cap on the output and the model spends its entire budget reasoning silently, then returns nothing. Empty content. The coder role would fire, the model would think itself right up to the token ceiling, and hand back an empty string. The pipeline would choke on a non-answer.
I set effort: none. It was silently ignored. The model kept thinking.
This is not a quirk unique to me — it’s the dirty secret of reasoning models in production. The reasoning effort is, as one good write-up puts it, a cost dial, not a quality dial: turn it up and you can triple your thinking tokens for a few points of benchmark, and you get billing surprises and empty outputs when the reasoning eats the room your answer was supposed to live in. The fix was unglamorous: a hard, absolute cap on reasoning tokens per role, tuned model by model, so the thinking can’t starve the output. Coders get a tight reasoning leash; the planning roles get more rope. There is no universal setting. Each model wanted its own.
The second wall was structured output. My pipeline leans on models returning clean, schema-valid JSON so the next stage can act on it. Proprietary APIs are very good at this now — OpenAI’s strict structured-output mode drives format failures below 0.1%. A lot of open-weight models, on a lot of providers, are not there. They produce malformed JSON in a meaningful slice of responses, especially on long outputs and nested schemas. So I stopped forcing JSON on the shaky models and let them use tool calls instead, and I rewrote the source-handling so the agent extracts just the relevant snippets and applies patch-diffs rather than rewriting whole files. Smaller context, fewer chances to break the format. That one change did more for open-weight reliability than any model swap.
The third wall was the providers themselves. The same open model behaves differently depending on who’s hosting it — speed, reliability, even whether structured output works at all. I benchmarked a pile of them. Some were great. Some failed schema validation on every call. Some truncated. A few simply couldn’t serve the models I needed. I ended up routing through OpenRouter, which spreads each model across sixteen-to-twenty-plus underlying providers and fails over between them — and I leaned on its :nitro variant, which sorts providers by raw throughput to claw back speed. By the end I could flip the entire pipeline between an open-weight preset and a proprietary one with a single flag, and run the exact same project through both.
That switch is what finally let me compare them honestly. And the comparison is where it got interesting.
The surprise: the output was basically the same
I expected the open models to produce visibly worse code. They didn’t.
Because the pipeline breaks everything into tiny tasks, the end result came out nearly the same across models. Same structure, same behavior, same app. The differences were small and almost aesthetic — CSS quality, mostly, and the occasional embellishment. Here’s the part I find genuinely funny: the more advanced models were the worse offenders. Give a frontier model a small, exact task and it gets creative — adds a flourish to the UI nobody asked for, decides your form needed an animation. Haiku just does the thing. The smarter the model, the more likely it is to improvise when improvising is exactly what you don’t want.
So the quality question — the one everybody fights about on benchmark leaderboards — mostly dissolved. Which left me staring at two axes and only two: price and speed.
And I’d argue that’s the honest place for this whole industry to land. The neutral data shops already separate models along exactly these lines — intelligence, speed, and price as three different columns — because once intelligence is “good enough” for the task, speed and price are what you actually live with.
Where the cheap models stop being cheap
On paper, open weights win price in a landslide. They’re a fraction of what the Anthropic-and-OpenAI preset costs per token. That’s the entire reason to do this.
But per-token price is a trap, and I walked straight into it.
On the proprietary side, the Anthropic models carry most of the load — and Sonnet and Haiku make fewer mistakes, and they infer faster than what I get through OpenRouter, even with Nitro on. Per request the speed difference isn’t dramatic. It adds up. And the mistakes are the real tax: with the open-weight preset, the agent has to stop and fix things more often. A bad JSON return here, a confused diff there, a retry, a re-plan. Every one of those is more tokens and more wall-clock.
This is the math nobody puts on the pricing page. A model that’s five times cheaper per token but needs to retry isn’t five times cheaper in practice — a 20% per-step failure rate roughly doubles your token bill, and in an agentic loop those failures compound across steps. The whole industry is quietly relearning that cheaper tokens have produced bigger bills, because the retries eat the savings alive.
That’s exactly what I watched happen. The price advantage on the meter got chewed up by the stop-and-fix loop. And when I stacked everything together — the slower inference, the extra retries, the re-planning — the real-world gap between my proprietary preset and my open-weight preset came out around 5x. Not on a benchmark. On the actual job, end to end.
To be clear about what I’m not claiming: this isn’t open weights being bad. It’s open weights, in my pipeline, today, on the providers I tested, with the tuning I’ve done so far. Which brings me to the part I’m least certain about.
Why I don’t trust my own numbers yet
Here’s the uncomfortable admission. I have a 5x number. I don’t fully trust it, and I don’t think you should either — not because it’s wrong, but because of how these things are usually measured.
The benchmark era is breaking down in plain view. The standard coding benchmarks are saturated and contaminated; the field spent 2025 watching the scorecard stop reflecting real-world ability. And the lazy shortcut everyone reaches for — have a strong model grade the output — is itself unreliable: LLM-as-judge shows large, order-sensitive variance on coding tasks. Asking one frontier model to tell me whether another model’s code is good is not evidence. It’s a vibe with a confidence interval.
So my next move isn’t another benchmark. It’s to look at the actual code these models vibe-coded, with my own eyes and real metrics — does it build, does it pass, is it maintainable, would I want to own it. It’s entirely possible that GLM and Kimi turn out to produce better overall results when I judge the real artifact instead of a synthetic score. It’s also possible the open models just aren’t tuned right yet — the reasoning budgets and the dozen small knobs almost certainly need per-model adjustment, and I’ve only scratched that. My honest expectation: the moment you move off synthetic benchmarks and start comparing real outputs, the whole picture gets murkier, not cleaner.
I’ll keep digging. But the month already taught me the thing that actually matters, and it isn’t a model.
The lesson worth more than the result
Step back from the leaderboard for a second.
Everything that made open weights usable in my pipeline — the per-model reasoning caps so thinking doesn’t starve the output, the tool-calling fallback for shaky JSON, the snippet-extraction and patch-diffs that shrink the context, the provider failover and throughput routing, the per-role model assignments — none of that is the model. It’s the structure around the model. It’s a month of unglamorous engineering that sits between “cheap tokens exist” and “you shipped working software.”
That’s the whole product, really. A coding agent is a coding agent — not a software development team. The model is a commodity; you can swap it tomorrow, and after this month I can tell you exactly how much swapping it actually buys you. The value was never the model. The value is the pipeline that makes any model produce code you can own and maintain — and that absorbs the retries, the format failures, the per-model tuning, so you don’t pay for them in your weekends.
This is the same promise OwlMeans makes to customers, and I just spent a month proving it to myself from the inside: OwlMeans is the team around the agent. You describe your product as user stories. The pipeline breaks it down, runs it, fixes it, and hands you a typed monorepo that’s yours — code you keep building with any agent you like. The cost is predictable per story, because the pipeline is what makes it predictable.
You can skip my month
So here’s the honest pitch, and I’ll make it the way I’d want it made to me.
If you’re vibe-coding a real product, you are going to face this exact question — which model, which provider, how do I keep the bill down without the output falling apart. You can do what I did: spend weeks wiring up presets, discovering that effort: none is a lie, watching your cheap tokens evaporate into retries, and learning per-model what’s actually true. It was worth it for me, because building this engine is my job.
It is not your job.
You can let the pipeline that already fought this fight do it for you — pick the model economics, eat the retries, enforce the structure, and give you back ownable, maintainable code with a cost you can predict per story. That’s the offer. I burned a month and a lot of tokens to find the 5x. You can just have the pipeline.
If saving time and tokens on vibe-coding is the problem, this is the cheapest month of R&D you’ll never have to run.
OwlMeans is an AI development pipeline: describe what you want as user stories and get full-stack apps, chatbots, AI agents, and data pipelines — typed, SSO-ready, and yours to keep building with any agent. See what it can do →