Our hardest end-to-end test could spend forty minutes waiting for an agent. The first version gave it an eighty-minute timeout. When the run stopped doing useful work, I often waited most of that time before finding out.
I kept thinking about Dujiangyan. UNESCO describes an irrigation system begun in the third century BC which controls river flow without a dam. It is still in use, with maintenance.
The comparison was useful for one decision in my test. I needed a signal from the running process, rather than another guess at how long the process ought to take.
The test drives OwlMeans through the browser. It registers a user, submits a prompt, waits for a specification and confirms it. A new Kubernetes sandbox starts for the project. Then an agent develops the first user story.
That last step took between ten and forty minutes in the runs behind this June article.
Mocking the model would answer a different question. Here I wanted to know whether the browser, pipeline and project environment could complete the flow together.
The browser harness uses @owlmeans/test-ui around Playwright. Its test login uses our supervisor mechanism, so this particular test does not establish that a real Google sign-in works.
The sandbox comes from @owlmeans/kephemeral. It provisions the temporary namespace, storage, workload and network policy. It also watches the workload and can recover a crashed pod.
That recovery helps the environment stay available. It does not tell the test whether the agent’s answer is correct.
The waiting loop
The agent consumes tokens while it works. I used changes in the recorded balance as a sign that the run was still active.
The test sampled that balance every twenty seconds. If it kept changing, the detector allowed the wait to continue. If it stayed flat for ten minutes, the detector failed the run as potentially stalled.
This was a liveness heuristic for our pipeline. A changing balance can mean unproductive work. A flat balance can also mean a legitimate wait, delayed accounting or a problem outside the model.
I still needed assertions for the expected story state and browser behavior. Spending tokens cannot satisfy those assertions.
The thresholds were choices for the June flow, rather than defaults I would prescribe to another team. Record what your long-running operation actually emits, then decide how long a silence deserves investigation.
Playwright’s testing guidance also stresses test isolation. Our slow wait was only one source of unreliable runs.
The abandoned sandbox
Interrupt a test after provisioning and it can leave a project behind. The next run then meets an environment the test did not intend to create.
We reused one test profile. Before and after each run, the harness destroyed that profile’s test projects. Cleanup before the run mattered because the previous process might never have reached its teardown.
The deletion scope must stay narrow. A cleanup routine which reaches unrelated projects would be a much worse problem than a flaky test.
Report cleanup failures. If resources remain, starting the next test as though cleanup succeeded hides useful evidence. Check the actual starting state before proceeding.
That removed a recurring source of interference in this setup. Later failures still needed investigation.
The route home
The cluster ran on a server at home, behind an ISP connection without a usable inbound public address.
Our route used a cloud VM with a reserved public IP, an Nginx proxy and WireGuard. Traffic went from Cloudflare to the relay, through the encrypted tunnel to the home server, and then to the Kubernetes service.
This was our relay arrangement. It should not be confused with using Cloudflare Tunnel’s connector as the whole transport.
Terraform, Ansible and shell scripts configured the route. A watchdog checked firewall rules every sixty seconds and repaired the drift it recognised. It could recover particular rules. The network could still fail.
The reserved address was intended to remain stable across the teardowns described in the setup. Provider state, credentials and configuration still needed maintaining.
A network failure should now give me a network failure to inspect. Waiting longer for an agent cannot repair a broken route to its environment.
This work made the test easier to diagnose. I could distinguish a quiet agent, an abandoned sandbox and a broken path into the cluster instead of giving all three another hour.
The login shortcut, provisioning library, cleanup permissions, token accounting and network configuration still need maintenance. Changes to any of them can invalidate an assumption in the test.
OwlMeans Services can include this kind of delivery and test infrastructure. The work has to fit the actual stack and failure modes; the June setup is not a promise to deploy any system in a few days.
I traded time spent waiting for time spent maintaining the harness. For this flow, that was worth it.
Originally published 23 June 2026. Revised 15 October 2026 to separate the liveness detector from correctness checks and qualify the infrastructure claims.