Acceptance-testing the open-source release on fresh machines: 22 issues

OpenDsec's automated tests always ran, and passed, on our development machine and our production cluster. But the development machine has everything installed, and the production cluster was tuned step by step. A new user starts from a fresh machine and a README. So before the release we took a new user's place: fresh Ubuntu 24.04 VMs, the documentation, and every feature a user might touch.

What we tested

Area	How
Single machine	Install per the README; run the examples, the SDK feature test and 54 admin command checks; reboot; uninstall
No KVM	Containers only
E2B	E2B's Python SDK (sync and async), JS SDK, code interpreter, snapshots and network options
Offline	Install from the offline bundle on a VM without internet; use the E2B SDK offline
Cluster	3 control and 3 compute nodes with NFS and HAProxy: deploy per the docs, hand over to fleet, add a node, rolling upgrade, config change, maintenance, backup, monitoring
Failures	Hard-reset a control node and kill the IAM leader under load
Kubernetes	Single-node k3s

Two serious issues

On a current Ubuntu kernel, every microVM resume failed. Ubuntu 24.04's 6.8.0-147 kernel kills a process placed with CLONE_INTO_CGROUP into a cgroup that has ever been through cgroup.kill. OpenDsec ends the VM process on pause and starts a new one in the same cgroup on resume, so every resume failed. Our production machines run an older kernel, which is why we never saw it; a kernel upgrade would have exposed it. Killing now freezes the cgroup and sends SIGKILL to each process instead of using cgroup.kill, and a regression test fails on the affected kernel.

Restarting a control node failed a few calls with unknown edge. After a restart the watcher only learned about compute nodes when they re-registered, up to 10 s later, and meanwhile the apiserver treated them as gone. A hard reset of one control node failed 3 of 3,911 calls in a load test. The watcher now starts from the nodes its peers know, and the apiserver keeps a node it briefly cannot see for 2 minutes. The same hard reset now fails none.

The rest

Most of the other 20 are things a new user hits on day one:

• Install: fresh machines lack make, and the script only failed halfway through the build; fresh Ubuntu lacks python3-venv; fresh cloud images have no package lists until apt update.
• CLI: opendsecctl exec dropped argument quoting, breaking python3 -c '...'; --help did not work; batch create printed no IDs; importing a missing local file went to Docker Hub.
• E2B: a single-machine install had no E2B gateway, and on clusters that deny internet by default E2B sandboxes had none, while E2B allows it by default.
• Operations: opendsecops restart skipped the E2B gateway; the disk alert used a fixed threshold and fired on every machine under 200 GB; on Kubernetes the apiserver could not write the image store.
• Performance: the NFS 4.1 we recommended cannot find holes in sparse files, so creating from a memory snapshot on another node read the whole memory file and took 141 s; with 4.2 it takes 62 s. The docs now say 4.2, and store-check warns about 4.1.

Each is listed in the CHANGELOG.

Not covered

Real GPUs, ARM machines, multi-node Kubernetes, and distributions other than Ubuntu 24.04. If you run OpenDsec on one of these and something breaks, please open an issue.

Takeaway

A full test run on a development machine shows the code is right on our machines. It does not show a user can get it working from the docs. Both serious issues came from the environment: a newer kernel, and restart timing that production never produced. Walking through the docs on a fresh machine finds problems a development machine never will.