I spent about a month trying to replace the proprietary models in our development pipeline with open-weight ones.
That was a month I could have spent on the product itself. I made the bet because every specialized role in OwlMeans calls a language model. A change there reaches the whole development flow.
At the time, our preset combined Anthropic and OpenAI models. I tried GLM, Kimi, DeepSeek and Qwen through hosted providers. I wanted to know whether a different preset could do the same work reliably.
This June experiment is not a benchmark. It does not establish a cost advantage for OwlMeans.
Our pipeline splits a product into small pieces. A role works on a user story, a design decision or a bounded code change. It receives the context for that job rather than a request to invent the whole application.
I thought that structure might make changing models straightforward. If the task was narrow enough, perhaps the model would have fewer opportunities to misunderstand it.
That was an engineering hypothesis I could test in my own system. It was also a tempting detour for a solo founder. Improving the machinery felt useful to every future project, which made it easy to justify another day of tuning.
I needed to look at the whole run as well as the app it produced.
Empty answers
The first problem was reasoning.
Some of the models used the available output budget on internal reasoning and returned no usable content. The coder role finished its request, but the next stage had nothing to apply.
I tried disabling reasoning with a setting that the provider did not consistently honor. Then I put explicit limits around the affected roles and adjusted them model by model.
That took more attention than the preset switch suggested. A planning role and a coder did not need the same treatment. A setting that worked for one model was not a rule I could safely copy to the others.
The next problem was the response format. Our stages pass structured results to one another. An answer that looks sensible to me can still fail the schema the next step expects.
I added a tool-call path where a model could not reliably return the expected structure. I also changed how code moved through the pipeline. The agent worked with relevant snippets and patches instead of repeatedly handling whole files.
Those changes became part of the system I had to maintain. The experiment was already larger than choosing a model from a list.
The provider in the middle
A model name did not identify the complete execution environment.
The host affected which features were available and how requests behaved. I tested providers and eventually used OpenRouter routing and failover in the experimental preset.
OpenRouter’s provider-routing documentation explains the controls available for selecting and filtering hosts. My pipeline depended on a particular response shape, so those controls mattered. I still had to test the complete route I was using.
I could finally switch between presets and run the same project through them. By then, the switch carried reasoning limits, response handling, code extraction and provider choices behind it.
The app at the end
In those runs, the resulting applications were broadly similar. I noticed differences in CSS and occasional additions to the interface that I had not requested.
I also saw capable models embellish a small task when I wanted a literal implementation. That was an observation about my workload and configuration. It would be a poor basis for ranking the models generally.
As the person running the system, I had to deal with less visible differences. Failed structured results needed another attempt, and I had to attend to patches that did not apply. Even an ordinary step could stop because of a provider mismatch.
A final screenshot hides that work.
Substituting models meant taking on more work inside the pipeline.
I had an informal comparison from my own metering. I did not trust it enough to turn it into a general performance claim.
The models had different tuning. Provider behavior was part of the result. Retries and repair steps made the path through a run matter as much as the final artifact.
A proper comparison would need a defined task, the same acceptance checks, consistent metering and reported failures. My runs had given me engineering notes.
So this revision leaves out the old numerical headline. The public address remains unchanged so existing links keep working.
I still had to decide where to spend my time. I could keep tuning the experimental preset, or return my attention to the rest of the product. More measurements would only help if they answered a decision I actually needed to make.
A month of tuning
I kept a working path for trying different models. I also found several assumptions that had been hidden inside the pipeline: how a role reasons, how it returns a result, how much source it receives and what happens when its provider fails.
That is useful knowledge for the engine I am building. It does not prove that another founder should repeat the same experiment.
If I did it again, I would define the stopping conditions earlier. I would decide which failure patterns I needed to understand before spending another week changing settings.
The month bought me a better understanding of my own system. It also took a month away from the rest of OwlMeans. Both belong in the account.
Revised 15 October 2026. This preserves the June experiment. Model and provider behavior has continued to change; the observations above describe those runs.