# OpenDsec documentation (English) > Open-source sandboxes on your own machines for RL rollouts and code-executing AI agents. Firecracker microVMs in ~0.1 s, fork with memory, and an E2B-compatible API. # Docs ## Find it by task | I want to… | Read | |---|---| | Try it on one machine | [Get started](/en/docs/quickstart); details in [Single-node install](/en/docs/install) | | Deploy a production cluster from scratch | [Cluster deployment](/en/docs/cluster) | | Install in a data center without internet access | [Deployment options: air-gapped](/en/docs/deploy#air-gapped) | | Understand sandboxes, backends, timeouts, sessions | [Concepts](/en/docs/concepts) | | Call OpenDsec from a program | [Caller guide](/en/docs/user-guide), [Python SDK reference](/en/docs/python-sdk) | | Work from the command line | [CLI reference](/en/docs/cli) | | Import images, build environments, warm up with snapshots | [Images and snapshots](/en/docs/images) | | Control what sandboxes can reach | [Network policy](/en/docs/network) | | Onboard teams, assign quotas and permissions, see usage | [Projects and permissions](/en/docs/iam) | | Move code written for the E2B SDK | [Migrating from E2B](/en/docs/e2b) | | Change the cluster config | [Configuration reference](/en/docs/configuration) | | Upgrade, add machines, maintain, renew certificates, handle failures | [Operations manual](/en/docs/operations) | | Set up monitoring, understand alerts | [Monitoring and alerts](/en/docs/monitoring) | | Something went wrong | [Troubleshooting](/en/docs/troubleshooting) | | Assess security, harden for production | [Security model](/en/docs/security) | | Know what stays compatible between versions | [Versioning](/en/docs/versioning) | | Understand the internals | [How the components work](/en/docs/components); design and measurements in [SPEC](/docs/spec) (Chinese) | | Licensing and the enterprise edition | [License and commercial use](/en/docs/license) | ## About these docs Apart from Get started, Deployment options, FAQ and License, these pages come from the `docs/` directory of the [opendsec repository](https://github.com/Kernflux/opendsec/tree/main/docs) (English in `docs/en/`) and are synced daily, maintained together with the code. "Edit this page on GitHub" at the bottom of each page opens its source file; changes are welcome. --- # Get started Install every OpenDsec component on one Linux machine, then create sandboxes with the Python SDK, the E2B SDK or the command line. On a fresh 8-core VM the whole run takes about 10 minutes, most of it building. ## Requirements - Linux x86_64 with systemd, cgroup v2 and kernel 5.15 or newer. Fully verified on Ubuntu 24.04. - With `/dev/kvm` you get microVMs too; without it, containers only. For microVMs, use bare metal or a cloud VM with nested virtualization. - Run every step below as root: the installer sets up system services and writes its files to `/root/opendsec-ops`. ## 1. Install the prerequisites ```sh sudo -i apt update && apt install -y make git python3-venv curl -fsSL https://go.dev/dl/go1.27.3.linux-amd64.tar.gz | tar -C /usr/local -xz ``` ## 2. Install OpenDsec ```sh git clone https://github.com/kernflux/opendsec cd opendsec deploy/quickstart.sh ``` The script: 1. checks the machine, builds the binaries, and downloads Firecracker and the guest kernel, checking each against `deploy/deps.lock`; 2. installs IAM, watcher, apiserver, edge and the E2B gateway as systemd services; 3. imports `python:3.12` and creates the project `root/demo` with a caller; 4. runs one container and one microVM sandbox as a check. ::: tip Other options - `--no-microvm`: containers only. - `--kernel FILE`: use a guest kernel you already have instead of downloading it. - `--build-kernel`: build the guest kernel yourself. - No internet access on the machine? Use the [offline bundle](/en/docs/deploy#air-gapped). ::: ## 3. Create a sandbox Load the caller's environment, then install the SDK: ```sh . /root/opendsec-ops/demo.env python3 -m venv .venv .venv/bin/pip install ./sdk/python .venv/bin/python sdk/python/examples/quickstart.py ``` `demo.env` holds the caller's address, token and CA, and also sets the three variables the E2B SDK reads. Pick any of these: ::: code-group ```python [Python SDK] import asyncio from opendsec import DSecClient, DSecRunArgs async def main(): # reads OPENDSEC_ADDR, OPENDSEC_TOKEN and friends async with DSecClient() as client: sb = await client.run_sandbox(DSecRunArgs( backend="microvm", image="root/py:3.12", cpu_cores_limit=1, memory_limit_mb=1024)) try: r = await sb.run_shell("python3 -c 'print(6*7)'") print(r.stdout) # four copies, memory included forks = await sb.fork(count=4) for f in forks: await f.stop() finally: await sb.stop() asyncio.run(main()) ``` ```python [E2B SDK] # pip install e2b # demo.env already sets E2B_API_URL, E2B_SANDBOX_URL, E2B_API_KEY from e2b import Sandbox sbx = Sandbox.create() print(sbx.commands.run("uname -r").stdout) # on microVMs the snapshot includes memory snap = sbx.create_snapshot() copy = Sandbox.create(snap.snapshot_id) copy.kill() sbx.kill() ``` ```sh [CLI] . /root/opendsec-ops/env # the admin environment opendsecctl status ID=$(opendsecctl sandbox create -backend microvm \ -image root/py:3.12 | grep -m1 '"id"' | cut -d'"' -f4) opendsecctl exec "$ID" python3 -c 'print(6*7)' opendsecctl sandbox stop "$ID" ``` ::: ## 4. Uninstall ```sh deploy/quickstart.sh --uninstall ``` ## Next - A multi-node cluster or an offline install: see [Deployment](/en/docs/deploy). - Moving from E2B: see [docs/E2B.md](/en/docs/e2b). - Every command: `opendsecctl --help`. --- # Concepts This page introduces every concept you meet when using OpenDsec: sandboxes and their backends, lifecycle and timeouts, sessions, tasks, images and snapshots, networking, projects and permissions, and errors. The [Python SDK reference](/en/docs/python-sdk) and the [CLI reference](/en/docs/cli) are easier to follow after it. ## 1. Architecture ``` Python SDK / opendsecctl / E2B SDK | gRPC + TLS, bearer token v +-----------------+ auth, quota +-----------------+ | apiserver x N | ------------> | IAM x 3 | | (stateless) | <------------ | (Raft) | +-----------------+ +-----------------+ | ^ snapshot +-----------------+ | +------------------- | watcher | | create, exec, files +-----------------+ v +----------+ +----------+ +----------+ | edge | | edge | | edge | one per compute node | sandbox | | sandbox | | sandbox | +----------+ +----------+ +----------+ \ | / shared image store (NFS / CephFS / JuiceFS) ``` - The **apiserver** is the only way in. It keeps no state, so you can run as many as you like behind a load balancer. - **IAM** holds projects, principals, permissions, quotas and tokens, replicated over three replicas with Raft. - The **watcher** probes every edge each second and aggregates each node's capacity and load into a snapshot the apiservers schedule from. - An **edge** runs on every compute node and creates and manages that node's sandboxes. After the apiserver picks a node, its edge makes the final call based on the node's actual memory, disk and CPU pressure. - The **shared image store** is mounted on every apiserver and edge node and holds image layers and snapshots. How each component works inside is in [How the components work](/en/docs/components). ## 2. Sandboxes A sandbox is an isolated environment with its own file system, process space, network and resource limits. You run commands in it, read and write its files, reach services running inside it, and stop it when you are done. Each sandbox has: | Field | Meaning | |---|---| | `id` | `.`, e.g. `node-a1.k3m9x...`. The first part is the node that created it | | `project` | The project it belongs to; its resources count against that project's quota | | `principal` | Who created it | | `edge_id` | The node it is on now. It changes when a paused sandbox moves to another node; the `id` does not | | `spec` | What was requested: backend, image, resources, timeouts, network rules, task, labels | | `state` | Current state; see below | | `created_at`, `last_active_at`, `ended_at` | Unix timestamps (seconds) | | `message` | Why it became `FAILED` or `STOPPED`, e.g. `idle timeout (ttl_running_stop) reached` | ### 2.1 Backends A creation must name its backend (`backend`); the SDK does not choose one for you. | Backend | Built on | Start | Isolation | When paused | Fork and snapshots | Use it for | |---|---|---|---|---|---|---| | `microvm` | Firecracker VM + jailer | cold start about 0.1 s; from a memory snapshot about 55 ms | strongest: its own kernel, a uid per VM | snapshotted and the process ends, all memory freed; resumes in about 50 ms | yes, with memory | the default; untrusted code | | `container` | Linux container: overlayfs, cgroup v2, namespaces, seccomp | about 0.1 s | shares the host kernel | processes frozen; with `pause_reclaim`, memory goes to swap | fork copies files only; `pack_diff` builds images | building images; trusted, performance-sensitive code | | `fncall` | a pool of pre-started containers | about 2 ms | as container | as container | as container | many short function calls | | `fullvm` | QEMU full VM | about 0.23 s | VM | vCPUs stopped, memory kept | no | a full system (the image's own init) or PCI passthrough (GPUs) | | `sim` | simulated lifecycle; runs nothing | — | — | — | — | tests and scheduling benchmarks | Which backends a cluster offers depends on what the administrator enabled on each node ([configuration reference](/en/docs/configuration#edges-backends)). A machine without `/dev/kvm` offers only `container` and `fncall`. ### 2.2 Resources | Parameter | Meaning | Default | |---|---|---| | `cpu_cores_limit` | CPU cores; may be fractional | 1 | | `memory_limit_mb` | Memory (MB). For a microVM this is the guest's memory | 2048 | | `disk_mb` | Writable disk size (MB); 0 means the backend's default | the backend's `default_disk_mb`: microvm 2048 and container 1024 in the quickstart and example configs; 10240 when the config sets none | | `gpus`, `gpu_mode` | Number of GPUs; `exclusive` or `shared` | 0 | | `qos` | `LS` (latency-sensitive) or `BE` (best effort). BE sandboxes run at the lowest CPU priority (cgroup `cpu.idle`) and are reclaimed from first when the node is short of memory | `LS` | Nodes admit sandboxes against their capacity **after overcommit**: CPU is overcommitted 8x by default (`cpu_overcommit`), memory not at all. Placement ranks nodes by whichever is higher, what is allocated or what is actually in use. ### 2.3 States ``` create | v CREATING ---- error ----> FAILED | v +------> RUNNING -----------------+ | | | RESUMING PAUSING STOPPING ^ | | | v v +-------- PAUSED -------------> STOPPED ``` | State | Meaning | |---|---| | `CREATING` | Being created: file system and network prepared, process or VM started | | `RUNNING` | Running; commands and file calls work | | `PAUSING`, `PAUSED` | Pausing, paused. **A runtime call to a paused sandbox (a command, a file call, HTTP) resumes it**; callers do not need to call `resume` first | | `RESUMING` | Resuming | | `STOPPING`, `STOPPED` | Stopping, stopped. Its file system and processes are cleaned up and its quota is returned at once | | `FAILED` | Creation or the sandbox failed; the reason is in `message` | Records of ended sandboxes (`STOPPED`, `FAILED`) stay on the node for 10 minutes (`stopped_retention`) and can still be looked up meanwhile; `list_sandboxes(include_ended=True)` lists them. ### 2.4 Timeouts Each sandbox has two timeouts, set at creation, in seconds: | Timeout | Meaning | Default | Limit | |---|---|---|---| | `ttl_running_stop` (idle timeout) | A running sandbox with no calls and no command running for this long is stopped | 3600 (1 hour) | none | | `ttl_max` (lifetime) | Counted from **creation**, paused time included; the sandbox stops when it is reached | 86400 (1 day) | 604800 (7 days) | - 0 means the cluster default. Defaults and the limit are set by the administrator under `admission` ([configuration reference](/en/docs/configuration#admission)). A request whose `ttl_max` exceeds the limit is rejected (`invalid`). - **Idle** means no runtime call is in progress and the last call is older than the idle timeout. A command running for two hours keeps the sandbox from being idle. - Paused time does not count toward the idle timeout: it starts from zero again on resume. - `touch()` resets the idle timer and does nothing else. - **Pause instead of stop on timeout**: with `ttl_paused_s` set (used through the API and E2B's `autoPause` for now), a sandbox reaching its idle timeout or lifetime is paused and kept `ttl_paused_s` more seconds. Any call during that time resumes it. - E2B's `set_timeout` maps to the API's `SetSandboxTimeout`: the lifetime becomes the sandbox's current age plus the timeout. **`stop()` sandboxes when you are done**: a sandbox holds its project's quota until it stops. ## 3. Running commands: sessions Commands run in **sessions**. A session is a long-lived shell in the sandbox: - in one session, `cd` and `export` carry over from one command to the next; - the default session is `default`; any other name opens a new session, independent of the others; - a session's user is fixed by its first command (the `user` parameter: a name or `uid[:gid]`; default `init_user`, i.e. root) and cannot change afterwards; - `close_session()` ends a session and kills every process it started. **Keep the agent and the grader apart**: run the agent's commands in a session of an unprivileged user (e.g. `session="agent", user="1000"`) and grade in another, root session. An unprivileged session cannot reach other sessions, read their output records or change system files. Write the grader's secret files with strict permissions (e.g. `umask 077`). **Output limits**: - the first 64 MB of each command's stdout and stderr are recorded; the rest is read and dropped (the command keeps running), and the dropped byte count is in `ShellResult.dropped_bytes`; - each session keeps the last 512 MB of output records; older ones are deleted and `get_output` can no longer fetch them. **Timeouts**: when a command's `timeout` expires, the processes it started are killed; the session stays. **Lost connections**: if the edge or apiserver restarts while a command runs, the SDK reconnects and fetches the result without running the command twice; streamed output continues where it broke off, with nothing repeated or lost. Besides sessions, a sandbox can run standalone **processes** (the API's `StartProcess` and related calls; E2B's `commands.run` is built on them): each has its own pid, stdin and signals, optionally a PTY. A process keeps running when the caller disconnects, and the caller can attach to it again later. ## 4. Tasks A sandbox can be created with a `task_id`, such as the name of an RL training job. All sandboxes of a task can then be acted on together: | Action | Effect | |---|---| | `pause_task` | Pause every sandbox of the task, e.g. when training is preempted: microVMs free all their memory; containers with `pause_reclaim` push it to swap | | `resume_task` | Resume every paused sandbox. Optional: the next call to a sandbox resumes it anyway | | `stop_task` | Stop every sandbox, when the job ends | The result counts the sandboxes that succeeded and failed, with the failures' reasons. ## 5. Batch creation and fork - **Batch creation**: `run_sandboxes(args, count)` creates up to 32768 sandboxes in one call. The server queues them at the rate limit, so rate limiting never fails part of a batch. Results come back in index order; failed items are `DSecError`s. - **Fork**: `sb.fork(count=N)` copies a running sandbox into N running sandboxes (up to 256). - The source pauses briefly (about 10 ms for a microVM) and carries on with its `id` and timeouts unchanged. - Copies inherit the source's spec; they can get extra labels, another `task_id` and timeouts of their own. - microVM copies include memory: processes continue from the moment of the fork. Each copy gets its own hostname, machine-id and random seed. Copies on the same node share memory pages. - Container copies get the file system only, not the processes. - Copies go to the source's node first; when it is full, to other nodes, which pull the snapshot straight from the source node. - Each copy counts against quota on its own. Batch creation and fork carry idempotency keys: when a server restart breaks the connection, the SDK's retry creates only what is still missing, never extra sandboxes. ## 6. Images and snapshots Sandboxes start from **images**, kept in the shared image store and named `/:`, e.g. `root/py:3.12` or `root/rl/swe-4821:v1`. The tag defaults to `latest`. | Source | How | Notes | |---|---|---| | Imported from a registry | `opendsecctl image import` | Any OCI registry, Docker Hub and private registries included; `docker save` files too | | Composed | `compose_image` | Several images stacked in order (base environment + repository + tools), without copying data | | Made from a sandbox | `pack_diff` (container) | Saves only the sandbox's changes against the image it started from, minus build residue (`/tmp`, caches, shell history) | | Snapshot | `create_snapshot` | Saved as is. microVMs must include memory (`memory=True`); sandboxes created from it continue from the moment of the snapshot | Image permissions follow the project the image belongs to: publishing (import, compose, `pack_diff`, snapshot, delete) needs `layer.publish`, using (creating sandboxes, viewing) needs `layer.use`. See [Images and snapshots](/en/docs/images). ## 7. Network Sandbox networking **denies by default**: nothing is reachable unless allowed. Allow destinations with `network_rules`, whose keys can be: - the name of a rule set the administrator defined (e.g. `pypi` = `pypi.org` + `files.pythonhosted.org`); - a domain, subdomains included, e.g. `github.com`; - an IPv4 address or network, e.g. `1.2.3.4`, `10.20.0.0/16`; - `*`: every public address. Any of them can take `:port[/protocol]`, e.g. `pypi.org:443`, `10.0.0.0/8:5432/tcp`. `true` allows; `false` takes back something the cluster's default rules allow. `update_network()` changes the rules of a running sandbox. Private, link-local and multicast addresses are reachable only through an address or network rule that names them; private addresses a domain resolves to are always blocked. See [Network policy](/en/docs/network). ## 8. Projects, principals and permissions - **Projects** form a tree rooted at `root`, e.g. `root/team-a/rl`. Every sandbox and image belongs to a project. - A **principal** is a caller, with one or more **tokens** (`dsk..`). Its kind is `user`, `service` or `agent`, a label only: permissions do not depend on it. - **Permissions** are granted on a project and inherited by its subprojects: | Permission | Allows | |---|---| | `sandbox.create` | Creating sandboxes in the project (forks included) | | `sandbox.operate` | Operating the project's sandboxes: view, list, run commands, files, pause, resume, stop, network changes | | `layer.use` | Using the project's images: create sandboxes from them, view them | | `layer.publish` | Publishing the project's images: import, compose, `pack_diff`, snapshot, delete | | `project.admin` | Managing the project: subprojects, principals, grants, tokens, usage; on root, cluster operations too | | `fleet.view`, `fleet.operate` | On root: viewing and operating the cluster management service (upgrades, config changes) | - **Quotas** are set on projects and checked up the tree: a subproject's usage also counts in every ancestor. Quota keys are `sandboxes`, `cpu_cores`, `memory_mb`, `gpus`. - **Bounded delegation**: a project admin can grant only permissions they hold, and a subproject's quota cannot exceed any ancestor's. See [Projects, principals and permissions](/en/docs/iam). ## 9. Errors Every error carries a stable reason string (`DSecError.reason` in the Python SDK): | reason | Meaning | What to do | |---|---|---| | `unauthenticated` | Token invalid, expired or revoked | Use another token | | `forbidden` | A permission is missing | The message names it; ask the project admin to grant it | | `not_found` | No such sandbox, project or image, or the sandbox was cleaned up | — | | `quota_exceeded` | Over the quota of the project or an ancestor | Stop sandboxes you no longer use, or ask for more quota | | `rate_limited` | Over the rate limit | Back off and retry | | `no_capacity` | The cluster cannot place it now (every candidate node full or under pressure) | Retry later, or lower concurrency | | `invalid` | Bad arguments, e.g. a timeout over the limit, a malformed rule, an unknown backend | Read the message | | `unavailable` | A server is restarting or failing over | The SDK already retried, for up to 60 s by default | | `timeout` | The call timed out | Raise `timeout`, or check the cluster's load | | `internal` | Server-side error | Note the time and sandbox id and check the server logs | **Which calls are safe to retry**: creation, batch creation and fork carry idempotency keys, so retries create nothing extra; commands carry a request id, so a retry never runs one twice; queries can always be retried. The SDK retries `unavailable` by itself; other reasons are the caller's decision. **Node failure**: when the machine a sandbox runs on fails, the sandbox is lost and later calls return `not_found` or `unavailable`. Take out results you need to keep. --- # FAQ ## What is OpenDsec? Open-source sandbox infrastructure for reinforcement-learning rollouts and code-executing AI agents. It runs isolated sandboxes (Firecracker microVMs, containers and QEMU VMs) on your own Linux machines, from one machine to a cluster, and serves its own Python SDK as well as an E2B-compatible API. It implements the DSec paper (arXiv:2609.22978) and is AGPL-3.0. ## Does it work without KVM? Yes. Without `/dev/kvm` the installer sets up the container backend only, and the SDK and E2B API work as usual. For microVMs, use bare metal or a cloud VM with nested virtualization. ## Which Linux distributions are supported? Fully verified on fresh Ubuntu 24.04. It needs x86_64, kernel 5.15 or newer, cgroup v2 and systemd. Other distributions should work but have not each been verified; please [open an issue](https://github.com/Kernflux/opendsec/issues) if one does not. ## How does it relate to E2B? OpenDsec serves an E2B-compatible API. Code written for the E2B SDK runs once `E2B_API_URL`, `E2B_SANDBOX_URL` and `E2B_API_KEY` point at your gateway, and the sandboxes run on your own machines. E2B's Python SDK (sync and async) and JS/TS SDK acceptance checks all pass. Directory watching, signed file URLs and template builds are not supported yet. The differences and how to migrate are in [docs/E2B.md](/en/docs/e2b). ## Is there an open-source, self-hosted alternative to E2B? Yes: OpenDsec runs on your own machines and speaks E2B's API, so E2B SDK code moves over by changing three environment variables. E2B's own runtime is also open source and can be self-hosted on one machine. How OpenDsec, E2B, Daytona and Modal differ is in [the comparison](/en/docs/compare). ## How does it relate to DeepSeek? It doesn't. OpenDsec is an independent implementation of the DSec paper that DeepSeek published. It is not endorsed by DeepSeek and contains none of its code. ## microVM, container or gVisor: which isolation? OpenDsec offers microVMs (Firecracker, each with its own kernel and a jailer with its own uid) for untrusted code, which is the default, and containers (namespaces, cgroups, seccomp; they share the host kernel) for trusted, performance-sensitive code and image builds. It does not use gVisor. Both start in about 0.1 s; only microVMs fork and pause with their memory. See [Concepts](/en/docs/concepts#_2-1-backends). ## How many sandboxes can a cluster run? On 2 × AMD EPYC 9Y24 bare metal (384 threads, 1.5 TB RAM per node), one node created 5120 microVMs in a single 37-second burst, and two nodes created 8192 containers with no failures. Memory is usually the limit: with the memory optimizations on (virtio-pmem images, free page reporting, zram), 16 idle 512 MB VMs used about 31% of the host memory they used without them, and 5120 idle 512 MB VMs used about 91 MB each. ## Is it suited to RL training? That is what it is built for. Batch creation makes up to 32768 sandboxes in one call; sandboxes carry a `task_id`, so a preempted training job can pause all of its sandboxes at once and free their memory; `fork(count=N)` copies a running microVM with its memory for tree search; retries never create duplicate sandboxes. See [Concepts](/en/docs/concepts). ## Does it run on Kubernetes? Experimentally: the manifests in `deploy/k8s/` run container sandboxes (no microVMs), and multi-node is not fully verified. The verified ways are a single machine and a systemd-managed cluster; see [Deployment](/en/docs/deploy). ## Is it free for commercial use? Yes. The servers are AGPL-3.0 and the SDK and clients Apache-2.0, so programs that call OpenDsec are not bound by the AGPL. If you modify the servers and offer them to others over a network, the AGPL asks you to publish your changes. A commercial license, support and an enterprise edition are available; see [License](/en/docs/license). ## How do I install without internet access? Use the offline bundle; see [Deployment](/en/docs/deploy#air-gapped). ## Is it ready for production? It is at 0.x, and interfaces may change before 1.0; see [VERSIONING.md](/en/docs/versioning). Rolling upgrades, a hard reset of a control node and an IAM leader change have each been run under load with no failed calls. For support or a commercial license, see [License](/en/docs/license). ## Does it support GPUs? GPUs can be given to container sandboxes exclusively or shared, and the full VM backend supports PCI passthrough. Neither has been tested on real GPUs yet. ## Something else? See [Troubleshooting](/en/docs/troubleshooting), which lists the problems we have met by symptom, with causes and fixes. If yours is not there, please [open an issue](https://github.com/Kernflux/opendsec/issues). --- # OpenDsec vs E2B, Daytona and Modal OpenDsec is sandbox infrastructure you run on your own Linux machines: Firecracker microVMs, containers and QEMU VMs, with fork and pause that keep memory, and an API that E2B's SDKs talk to. E2B, Daytona and Modal are mainly hosted services. This page compares the four on what matters for RL rollouts and code-executing agents, and says when each fits. Facts about the other products come from their own documentation and repositories, checked on 2026-10-25 (links under [Sources](#sources)). Hosted products change often: check their docs before deciding. Where their docs did not say, the table says so rather than guess. If something here is wrong or out of date, please [open an issue](https://github.com/Kernflux/opendsec/issues). ## At a glance | | OpenDsec | E2B | Daytona | Modal Sandboxes | |---|---|---|---|---| | Where it runs | Your machines only: one Linux machine or a cluster, also air-gapped. No hosted service | Hosted cloud. Open-source runtime for self-hosting on one machine ("Embed"); BYOC for enterprise | Hosted cloud; "bring your own compute" deployments | Modal's cloud only | | License | AGPL-3.0 servers; Apache-2.0 SDK and clients | Apache-2.0 (SDKs and runtime) | Open-source repository archived in 2026 (last release v0.190.0, AGPL-3.0); development continues in a private codebase | Proprietary | | Isolation | Firecracker microVM with jailer (default); also containers and QEMU VMs | Firecracker microVM | "Dedicated kernel"; container and VM runtimes (the docs do not name the technology) | gVisor by default; a VM runtime is alpha | | Fork a running sandbox with memory | Yes: `fork(count=N)`, up to 256 copies, source pauses ~10 ms | Yes: snapshots keep memory by default; many sandboxes from one snapshot | Not described in the docs we reviewed | Memory snapshots are alpha | | Pause and resume with memory | Yes; resume ~50 ms; a whole task at once | Yes; pause ~4 s per GiB of RAM, resume ~1 s | Not described in the docs we reviewed | Through memory snapshots (alpha) | | Measured start | ~0.1 s cold microVM; ~55 ms from a memory snapshot | Not stated in the docs we reviewed | "Under 90 ms" | Not stated in the docs we reviewed | | Longest sandbox lifetime | Set by the administrator (default limit 7 days) | 1 hour (Hobby), 24 hours (Pro) | Not stated in the docs we reviewed | 24 hours | | GPUs | Containers (exclusive or shared) and QEMU PCI passthrough; not yet tested on real GPUs | Not offered | NVIDIA and AMD GPU sandboxes | Yes (gVisor runtime) | | E2B SDK | Supported through a gateway ([what differs](/en/docs/e2b)) | Native | Own SDK | Own SDK | | Cost | Free software on your hardware; commercial license optional | Per-second vCPU and RAM, plus plans | Hosted pricing | Hosted pricing | ## When to choose which **OpenDsec** fits when: - sandboxes must run on your own machines: a private data center, an air-gapped network, data that may not leave, or hardware you already own; - you run RL rollouts at scale: one burst of 5120 microVMs on a single 384-thread node took 37 s, and 8192 containers on two nodes all succeeded; - you need to branch running state: `fork(count=N)` copies a microVM with its memory, for tree search or many rollouts from one prepared environment; - your code already uses the E2B SDK and you want to keep it while moving onto your own infrastructure. **E2B** fits when you want a managed service and do not want to run infrastructure, or a single self-hosted machine is enough. **Modal** fits when sandboxes need GPUs today, or your workloads already run on Modal. **Daytona** fits when you want its hosted sandboxes, including GPU ones. Its open-source repository is archived, so self-hosting means running the last open release without updates. ## What OpenDsec does not do (yet) - **No hosted service:** you install and operate it; [Get started](/en/docs/quickstart) takes about 10 minutes on one machine, a cluster is described in [Cluster deployment](/en/docs/cluster). - **0.x:** interfaces may change before 1.0 ([versioning](/en/docs/versioning)). - **GPUs untested:** the code is there, but nobody has run it on real GPUs yet. - **E2B gaps:** directory watching, signed file URLs and template builds are not supported yet ([details](/en/docs/e2b)). - **Linux x86_64 only;** microVMs need `/dev/kvm` (bare metal, or a cloud VM with nested virtualization). ## Sources Checked on 2026-10-25. - OpenDsec: [Concepts](/en/docs/concepts) (backends, timeouts, fork), [measurements](/en/) on 2 × AMD EPYC 9Y24 bare metal, [E2B compatibility](/en/docs/e2b), [License](/en/docs/license). - E2B: [open source](https://e2b.dev/open-source.md) and [e2b-dev/runtime](https://github.com/e2b-dev/runtime) (Apache-2.0, Firecracker, Embed), [persistence](https://docs.e2b.dev/sandbox/persistence) (pause and resume), [snapshots](https://docs.e2b.dev/sandbox/snapshots), [pricing](https://e2b.dev/pricing) (plans, session length, no GPUs). - Daytona: [daytonaio/daytona](https://github.com/daytonaio/daytona) (archived; last release v0.190.0 on 2026-06-23), [docs](https://www.daytona.io/docs) (isolation, start time, GPU sandboxes, bring your own compute). - Modal: [Sandboxes guide](https://modal.com/docs/guide/sandboxes) (gVisor, GPUs, 24-hour timeout), [Sandbox snapshots](https://modal.com/docs/guide/sandbox-snapshots) (memory snapshots alpha), [VM Sandboxes](https://modal.com/docs/guide/vm-sandboxes.md) (alpha). --- # Deployment | Option | Status | For | |---|---|---| | [Single machine](#single-machine) | Verified | trying it out, development, small workloads | | [Cluster](#cluster) | Verified | production, any number of compute nodes | | [Air-gapped](#air-gapped) | Verified | data centers without internet access | | [Kubernetes](#kubernetes) | Experimental | existing Kubernetes clusters, container sandboxes only | ## Single machine One command installs everything; see [Get started](/en/docs/quickstart), and [Single-node install](/en/docs/install) for the options, what gets installed, file locations and common problems. ## Cluster - Three control nodes (IAM, watcher, apiserver, fleet) plus any number of compute nodes (edge). - Every apiserver and edge mounts the same shared image store: NFS 4.2, CephFS or JuiceFS. - An L4 load balancer in front, health-checking `/readyz`. - Once installed, fleet takes over: staged rollouts, config changes, new nodes and maintenance, with failed nodes rolled back automatically. Step by step from scratch: [Cluster deployment](/en/docs/cluster). Day-to-day operations afterwards: [Operations manual](/en/docs/operations). ::: warning NFS version Mount NFS with `vers=4.2`. Version 4.1 cannot find the holes in sparse files, which makes creating from a memory snapshot on another node more than twice as slow. `opendsecctl image store-check` checks for this. ::: ## Air-gapped Build a bundle on a machine with internet access and install it on the target. The bundle carries the guest kernel, a base image and the Python wheels (the E2B SDK included), for one node or a cluster. ```sh deploy/offline/build-bundle.sh # on a connected machine # copy it over, unpack, then: ./install.sh ``` See [deploy/offline/README.md](https://github.com/Kernflux/opendsec/blob/main/deploy/offline/README.md). ## Kubernetes Manifests in `deploy/k8s/` run container sandboxes. They need at least three nodes and a ReadWriteMany storage class. Multi-node is not fully verified yet; see [deploy/README.md](https://github.com/Kernflux/opendsec/blob/main/deploy/README.md). --- # Single-node install in detail `deploy/quickstart.sh` installs every OpenDsec component on one Linux machine. It uses the same tooling as a multi-node cluster (`opendsecops`), so the single-node cluster can later be upgraded and grow more machines. This page covers its requirements and options, what it does, where things end up, and common problems. The short version is [Get started](https://kernflux.github.io/en/docs/quickstart). ## 1. Requirements | Item | Requirement | Check | |---|---|---| | Architecture | x86_64 | `uname -m` | | Kernel | 5.15 or newer (fully verified on Ubuntu 24.04) | `uname -r` | | Init | systemd | `ls /run/systemd/system` | | cgroups | v2 (unified) | `stat -fc %T /sys/fs/cgroup` prints `cgroup2fs` | | File systems | overlay | `grep overlay /proc/filesystems` | | Commands | `curl`, `tar`, `sha256sum`, `mkfs.ext4`, `ip`, `awk` (in Ubuntu by default) | — | | Build | Go 1.27 or newer and `make` (not needed with prebuilt binaries or `--release`) | `go version` | | Privileges | root | — | | Ports | 7001, 7002, 7011, 7100, 7201, 7400, 9461–9464 and 9467 free | `ss -ltn` | | microVMs | `/dev/kvm`: bare metal, or a cloud VM with nested virtualization. Without it, containers only | `ls /dev/kvm` | | Resources | 4 cores, 8 GB of RAM and 40 GB of disk or more recommended | — | The script first checks each item, printing `[ok]` or `[x]`, and stops on any `[x]` before changing anything. ## 2. Installing ```sh sudo -i # everything below as root apt update && apt install -y make git python3-venv curl -fsSL https://go.dev/dl/go1.27.3.linux-amd64.tar.gz | tar -C /usr/local -xz git clone https://github.com/kernflux/opendsec && cd opendsec deploy/quickstart.sh ``` On a fresh 8-core VM this takes 8 to 11 minutes, mostly building. ### Options | Option | Meaning | |---|---| | (none) | container and microVM with `/dev/kvm`, container only without. The guest kernel is downloaded from the GitHub release and checked against the sha256 in `deploy/deps.lock` | | `--no-microvm` | Containers only | | `--kernel FILE` | Use this guest kernel instead of downloading one | | `--build-kernel` | Build the guest kernel (a few more minutes; needs `build-essential flex bison bc libelf-dev libssl-dev`) | | `--release DIR` | Install a packaged release directory instead of building (offline bundles use this) | | `--uninstall` | Remove everything; see §6 | | `--help` | Usage | ### Environment variables | Variable | Default | Meaning | |---|---|---| | `OPENDSEC_IMAGE` | `docker.io/library/python:3.12` | Base image to import; may be an image file | | `OPENDSEC_ADDR_IP` | the default route's source address | Address the components listen on. Set it on a machine with several interfaces, or for access from other machines | | `OPENDSEC_OPS` | `/root/opendsec-ops` | Ops directory | | `OPENDSEC_DEBS` | — | An offline bundle's package repository (`debs/`), used for system packages instead of the apt sources | For access from this machine only: `OPENDSEC_ADDR_IP=127.0.0.1 deploy/quickstart.sh`. ## 3. What the script does 1. **Checks the machine** (the items in §1) and settles the node name (the host name), address, sandbox state directory (`/data/opendsec-state` if `/data` is a mount point, else `/var/lib/opendsec/edge-state`) and whether to install microVMs. 2. **Builds and packages**: `make build`, then `opendsecops release` packs the binaries, systemd units, Firecracker, jailer and the guest kernel into a release under `/root/opendsec-ops/releases/`. 3. **Writes the config**: an inventory (this machine, ssh address `local`), a cluster config `opendsec.json` and an admin token in the ops directory. The config: - a single-replica IAM (Raft, so it can grow to three later), a root quota of 10000 sandboxes, tokens valid for at most 90 days; - the image store at `/mnt/opendsec-images` (a local directory); - a `pypi` rule set and **no default network rules: sandboxes reach no network by default**; - the container backend (at most 2000 sandboxes, 1 GB default disk); with KVM, the microvm backend too (at most 1000, 2 GB default disk, EROFS image disks, pmem and memory reclaim on); - an E2B gateway for project `root/demo`, with template `base` = `root/py:3.12`. 4. **Issues certificates and installs**: `opendsecops certs` creates the CA and component certificates; `opendsecops install all` creates the user and directories, delivers the release, config, certificates and systemd units, and applies node settings (sysctl, modules, NFS client identity and so on). 5. **Starts** everything with `opendsecops start all`, in the order IAM → watcher → apiserver → E2B gateway → edge, waiting for each to be ready. 6. **Imports an image**: `OPENDSEC_IMAGE` as `root/py:3.12`. 7. **Creates a project and a caller**: project `root/demo` (quota 100 sandboxes), principal `demo`, granted `sandbox.create`, `sandbox.operate` and `layer.publish` on `root/demo` and `layer.use` on `root`. 8. **Checks**: as `demo`, creates one container and one microVM sandbox, runs `python3 -c 'print(6*7)'`, expects 42, and stops them. Finally it prints how the administrator and the caller use the cluster. ## 4. Afterwards ### Environments ```sh . /root/opendsec-ops/env # administrator: admin token, project root . /root/opendsec-ops/demo.env # caller: a token for root/demo; also sets E2B_API_URL, E2B_SANDBOX_URL, E2B_API_KEY ``` ### Files and directories | Path | Contents | |---|---| | `/root/opendsec-ops/` | Ops directory: inventory, `opendsec.json`, `admin.token`, `pki/` (the CA private key included), `releases/`, `env`, `demo.env`. **It holds secrets: keep it private and back it up** | | `/opt/opendsec/current` | The current release (pointing at `/opt/opendsec/releases/`): `bin/`, `vm/` (guest kernel), `agents/`, `units/` | | `/usr/local/bin/opendsec-*`, `opendsecctl` | Links to the current release's binaries | | `/etc/opendsec/` | This machine's config, certificates and `admin.token` | | `/var/lib/opendsec/iam` | IAM data (Raft log and snapshots) | | `/data/opendsec-state` or `/var/lib/opendsec/edge-state` | Sandbox disks, snapshots and records | | `/mnt/opendsec-images` | Image store | | `/var/log/opendsec/audit` | Audit log | | `/var/backups/opendsec` | Daily IAM backups | ### Services | Unit | Port (metrics, `/readyz`) | |---|---| | `opendsec-iam@0` | 7001, Raft 7011 (9461) | | `opendsec-watcher@0` | 7002 (9462) | | `opendsec-apiserver@0` | 7100 (9463) | | `opendsec-edge` | 7201 (9464) | | `opendsec-e2b@0` | 7400 (9467) | | `opendsec-iam-backup.timer` | backs up IAM daily at 03:00 | ```sh systemctl status 'opendsec-*' journalctl -u opendsec-edge -f deploy/prod/opendsecops status # is every component ready opendsecctl status # cluster snapshot (after . /root/opendsec-ops/env) ``` ### Access from other machines - Install with `OPENDSEC_ADDR_IP` set to an address other machines reach (by default, the default route's address). - Give callers `/root/opendsec-ops/pki/ca.crt`, a token and the address; they set `OPENDSEC_ADDR=
:7100` and `OPENDSEC_CA=`. - Open only 7100 in the firewall (and 7400 for the E2B SDK). ### Rebooting Every service starts at boot. A reboot loses all sandboxes (paused ones included); images, projects, principals and tokens are kept. ## 5. Upgrading A single-node cluster has no fleet; upgrade it with `opendsecops`: ```sh cd opendsec && git pull make build && deploy/prod/opendsecops release bin/ deploy/prod/opendsecops upgrade # the version is in the previous command's output deploy/prod/opendsecops status ``` Components restart one at a time; running sandboxes are unaffected. To go back: `deploy/prod/opendsecops rollback `. Read the "Breaking" section of `CHANGELOG.md` before upgrading. ## 6. Uninstalling ```sh deploy/quickstart.sh --uninstall ``` Stops and disables every service, kills leftover sandboxes, removes the nftables tables, network interfaces and namespaces and mounts, and deletes `/etc/opendsec`, `/opt/opendsec`, `/var/lib/opendsec`, `/var/log/opendsec`, `/var/backups/opendsec`, the ops directory, the image store and the sandbox state directory. Kept: the node settings (`/etc/sysctl.d`, udev rules, `/etc/modules-load.d`, networkd files) and software you installed yourself, such as Go and make. ## 7. Common problems | Symptom | Cause and fix | |---|---| | `[x] command make` | `apt install -y make` | | `[x] Go 1.27+ to build` | Install Go as in §2, or build elsewhere and copy `bin/` here | | `[x] cgroup v2` | The system still uses cgroup v1: add `systemd.unified_cgroup_hierarchy=1` to the kernel command line and reboot | | `[x] port 7100 is in use` | Another program holds the port, or an earlier install: `--uninstall` first | | `[x] /root/opendsec-ops already holds a cluster` | An earlier install: `--uninstall` first, or manage the existing cluster with `opendsecops` | | `warning: no /dev/kvm: containers only` | No KVM on this machine. Enable nested virtualization on the cloud VM, or use bare metal | | `could not fetch the guest kernel pinned in deploy/deps.lock` | GitHub is unreachable: `--kernel FILE`, `--build-kernel` or `--no-microvm` | | Importing `python:3.12` is slow or fails | Docker Hub is slow from here: point `OPENDSEC_IMAGE` at a mirror, or `docker save` the image to a file and pass its path | | `python3 -m venv` says ensurepip is unavailable | `apt install -y python3-venv` | | The final check says `... sandbox ... python printed '...', want 42` | The sandbox was created but the command did not print the expected output: look at `journalctl -u opendsec-edge` and `opendsecctl status`, then run `opendsecctl exec python3 -c 'print(6*7)'` yourself to see the error | | microVM creation fails | `journalctl -u opendsec-edge` has the reason; check `ls -l /dev/kvm`; on a cloud VM, check nested virtualization is on | More in [Troubleshooting](/en/docs/troubleshooting). --- # Cluster deployment This page deploys a production cluster from scratch: planning, infrastructure, installation, handing over to fleet, the first caller, and acceptance. Day-to-day operations afterwards (upgrades, config changes, new machines, certificates, failures) are in the [operations manual](/en/docs/operations). The procedure was verified as written on a fresh cluster of 3 control-plane nodes, 3 compute nodes, NFS and HAProxy; installation took about 8 minutes (preparing the machines not included). ## 1. Planning ### Machines | Role | Count | Suggested size | Runs | |---|---|---|---| | Control-plane node | 3 (one per failure domain) | 4 cores, 8 GB to start, local SSD | one replica each of IAM, watcher, apiserver and fleet; one of them also runs Prometheus and Alertmanager | | Compute node | as needed | bare metal or a cloud VM with nested virtualization; local NVMe for sandbox state; memory decides how many sandboxes fit | edge, node agent | | Ops machine | 1 (may be a control-plane node) | any | `opendsecops`; holds all of the cluster's secrets | - The cluster keeps creating and serving sandboxes with any one control-plane node down. IAM is a 3-replica Raft group and tolerates one failure. - When a compute node fails, its sandboxes are lost, and new ones go to other nodes. - In a small cluster, control-plane services can share machines with edges (several roles on one inventory line). ### Components and ports | Component | Where | Instances | Port (metrics) | State | |---|---|---|---|---| | `opendsec-iam` | control-plane nodes | 3 (or 5), Raft | 7001, Raft 7011 (9461) | `/var/lib/opendsec/iam`, local SSD | | `opendsec-watcher` | control-plane nodes | 2–3 | 7002 (9462) | none | | `opendsec-apiserver` | control-plane nodes, behind the load balancer | 2+ | 7100 (9463) | none; audit log in `/var/log/opendsec/audit` | | `opendsec-fleet` | control-plane nodes | 3 | 7300 (9465) | releases in `/var/lib/opendsec/fleet`; desired state in IAM | | Prometheus, Alertmanager | one control-plane node | 1 | 9090, 9093 | local | | `opendsec-edge` | every compute node, as root | 1 per node | 7201 (9464) | `state_dir`, local NVMe | | `opendsec-nodeagent` | every machine, as root | 1 per node | (9466) | none | | `opendsec-e2b` (optional) | any node | as needed | 7400 (9467) | none | | node_exporter | every machine | 1 per node | 9100 | none | **Network**: open only the load balancer's 7100 to callers (and 7400 for the E2B SDK). Keep every other port inside the private network: 7001, 7011, 7002, 7201 and 7300 carry mTLS traffic between components, 9461–9467 and 9100 are metrics, 9090 and 9093 are for operators. ## 2. Infrastructure ### 2.1 Every node - Linux x86_64, systemd, cgroup v2, kernel 5.15 or newer (verified on Ubuntu 24.04 with kernel 6.8). - A private network between the nodes. - Root ssh from the ops machine to every node, by default with the key `/root/.ssh/opendsec-ops` (`OPENDSEC_SSH_KEY` changes it): ```sh ssh-keygen -t ed25519 -N '' -f /root/.ssh/opendsec-ops # add /root/.ssh/opendsec-ops.pub to root's authorized_keys on every node ``` - Compute nodes: `/dev/kvm` present (`ls -l /dev/kvm`); local NVMe mounted at `/data` (the sandbox state directory defaults to `/data/opendsec-state`; XFS allows faster reflink copies). The packages the edge needs (erofs-utils, e2fsprogs, nftables, iproute2) are installed for you. ### 2.2 Shared storage Mounted at `/mnt/opendsec-images` on every apiserver and edge node. A NAS (NFS), CephFS or JuiceFS all work: - **Root squash must be off** (on a NAS, set the permission group to "no root restriction"). - NFS mount options, in `/etc/fstab`: ``` :/ /mnt/opendsec-images nfs4 vers=4.2,nconnect=8,hard,timeo=600,lookupcache=positive,_netdev 0 0 ``` An NFS server on a Linux machine (for tests or small clusters): ```sh apt install -y nfs-kernel-server mkdir -p /srv/opendsec-images echo '/srv/opendsec-images 10.0.0.0/16(rw,sync,no_subtree_check,no_root_squash)' >> /etc/exports exportfs -ra ``` Why these options: [Images and snapshots §5](/en/docs/images). ### 2.3 Load balancer TCP forwarding of port 7100 to every apiserver: - plain TCP, not HTTP or gRPC: the apiservers terminate TLS themselves; - HTTP health check on port 9463, path `/readyz`, every 2 s, unhealthy after 2 failures; - an idle timeout of at least 30 s: the apiservers ping idle connections every 20 s; - keep the client's source address if the load balancer can (otherwise failed authentications are all charged to the load balancer's address). HAProxy (appended to `/etc/haproxy/haproxy.cfg`): ``` frontend opendsec_api bind 10.0.0.10:7100 mode tcp timeout client 120s default_backend opendsec_apiservers backend opendsec_apiservers mode tcp balance roundrobin timeout server 120s option httpchk GET /readyz default-server check port 9463 inter 2s fall 2 rise 2 server ctl-1 10.0.0.11:7100 server ctl-2 10.0.0.12:7100 server ctl-3 10.0.0.13:7100 ``` In a cloud, use the provider's layer-4 load balancer (TCP listener) with the health check above. A load balancer is optional: callers can put every apiserver's address in `OPENDSEC_ADDR`, comma separated, and the SDK round-robins and avoids one that is down. ### 2.4 Ops machine ```sh apt update && apt install -y make git curl -fsSL https://go.dev/dl/go1.27.3.linux-amd64.tar.gz | tar -C /usr/local -xz export PATH=$PATH:/usr/local/go/bin git clone https://github.com/kernflux/opendsec && cd opendsec export OPENDSEC_OPS=/root/opendsec-ops mkdir -p -m 0700 $OPENDSEC_OPS ``` `$OPENDSEC_OPS` will hold the inventory, cluster config, admin token and the CA's private key. **It is the key to the whole cluster**: give only the people who need it access, and back it up somewhere safe. ## 3. Installation ### 3.1 Package a release ```sh make build && deploy/prod/opendsecops release bin/ ``` Packs the binaries, systemd units, Firecracker, jailer and the guest kernel (downloaded and checked per `deploy/deps.lock`) into a release under `$OPENDSEC_OPS/releases/`. - If the guest kernel cannot be downloaded: `OPENDSEC_KERNEL= deploy/prod/opendsecops release bin/`, or put the file into `~/.cache/opendsec-deps/vmlinux-6.1.167-opendsec.2` (its sha256 must match deps.lock). - For a containers-only cluster: `OPENDSEC_DEPS_SKIP=guest-kernel`. ### 3.2 Describe the cluster ```sh cp deploy/prod/inventory.example $OPENDSEC_OPS/inventory cp deploy/prod/cluster.example.json $OPENDSEC_OPS/opendsec.json ``` **inventory**: one machine per line: name, ssh address, private address, roles. ``` ctl-1 203.0.113.11 10.0.0.11 iam=0 watcher=0 apiserver=0 fleet=0 ctl-2 203.0.113.12 10.0.0.12 iam=1 watcher=1 apiserver=1 fleet=1 ctl-3 203.0.113.13 10.0.0.13 iam=2 watcher=2 apiserver=2 fleet=2 monitor node-a1 203.0.113.21 10.0.1.21 edge=node-a1 node-b1 203.0.113.22 10.0.1.22 edge=node-b1 node-c1 203.0.113.23 10.0.1.23 edge=node-c1 ``` - Roles: `iam=`, `watcher=`, `apiserver=`, `fleet=`, `edge=`, `e2b=`, `monitor` (Prometheus and Alertmanager go on this machine). - Every IAM replica, watcher, apiserver and edge in the cluster config must appear on exactly one inventory line. - The ssh address `local` runs commands on this machine without ssh. **opendsec.json**: replace the example's addresses with your private addresses (the IAM, watcher, apiserver and fleet sections), then adjust: - `iam.root_quota`: the whole cluster's quota; - `admission`: rate limits (the example raises them to 20000 creations and 100000 calls per second); - `network.rule_sets`, `network.default_rules`: rule sets and default rules (the example has no default rules, so sandboxes reach no network by default); - `edge_template.backends`: which backends compute nodes offer, how many sandboxes each, default disk sizes; - `edges`: nodes that need their own settings. New machines need no entry; `edge_template` covers them. Every field: [Configuration reference](/en/docs/configuration). Validate the result: ```sh bin/opendsecctl pki init -config $OPENDSEC_OPS/opendsec.json -out $(mktemp -d) ``` **Admin token**: ```sh printf 'dsk.admin.%s\n' "$(openssl rand -hex 16)" > $OPENDSEC_OPS/admin.token chmod 0600 $OPENDSEC_OPS/admin.token ``` ### 3.3 Certificates ```sh deploy/prod/opendsecops certs api.opendsec.internal,10.0.0.10 ``` Creates the cluster CA (its private key stays in `$OPENDSEC_OPS/pki`) and every component's certificate. The argument is the load balancer's address or DNS name, which goes into the apiserver certificates; every apiserver certificate also carries `api.opendsec.internal` (the name the SDK checks when given several addresses). ### 3.4 Install on every node ```sh deploy/prod/opendsecops install all ``` On each node: creates the user and directories, delivers the release, config, certificates and systemd units, installs the packages needed, and applies node settings (sysctl, kernel modules, NFS client identity, log filtering). **Nothing starts yet.** Each node gets only the CA certificate and its own components' certificates and keys; the admin token goes only to the IAM nodes. ### 3.5 Mount the shared storage On every apiserver and edge node, add the fstab line from §2.2 and mount it, then check on one of them: ```sh mount /mnt/opendsec-images /opt/opendsec/current/bin/opendsecctl image store-check -store /mnt/opendsec-images ``` Every required item must be `ok`. The NFS client identity is set by `install` and takes effect on a fresh mount: if a node mounted the share before `install`, reboot it once. ### 3.6 Start ```sh deploy/prod/opendsecops start all deploy/prod/opendsecops status ``` Starts IAM → watcher → apiserver → E2B gateway → fleet → edge in that order, waiting for each `/readyz` to return 200. IAM bootstraps with the admin token on its first start. `status` shows whether each component is ready and the current IAM leader, and exits 1 if any is not ready. ### 3.7 Monitoring ```sh deploy/prod/opendsecops monitor https://alerts.example.com/hook ``` Installs node_exporter everywhere, and Prometheus and Alertmanager on the `monitor` node, loads the alert rules (`deploy/monitoring/alerts.yml`), and sends alerts to the webhook. Without a webhook, alerts go nowhere. Metrics and alerts: [Monitoring and alerts](/en/docs/monitoring). ## 4. Hand over to fleet From here on, upgrades, config changes and new machines go through fleet in staged rollouts with automatic rollback. ```sh deploy/prod/opendsecops install-agent all # start a node agent on every machine export PATH=$PATH:$OPENDSEC_OPS/current/bin export OPENDSEC_ADDR=10.0.0.10:7100 OPENDSEC_TOKEN=$(cat $OPENDSEC_OPS/admin.token) OPENDSEC_CA=$OPENDSEC_OPS/pki/ca.crt V=$(basename $(readlink $OPENDSEC_OPS/current)) opendsecctl fleet release push $OPENDSEC_OPS/releases/$V opendsecctl fleet config put $OPENDSEC_OPS/opendsec.json -note "initial" opendsecctl fleet import -inventory $OPENDSEC_OPS/inventory -release $V -config-gen 1 opendsecctl fleet node set node-a1 -domain a # failure domains: rollouts spread over them, at most half a batch per domain opendsecctl fleet node set node-b1 -domain b opendsecctl fleet node set node-c1 -domain c opendsecctl fleet nodes ``` In `fleet nodes`, every machine should be reporting (`SEEN` a few seconds ago), with `RUNNING` equal to `RELEASE` and every `UNITS` ready. After the handover, `opendsecops upgrade` and `restart` refuse to run; upgrades go through `opendsecctl fleet rollout` ([operations manual §6](/en/docs/operations)). `opendsecops` stays for certificates, new machines and emergencies. ## 5. First image and first caller As root on a node that mounts the image store: ```sh export OPENDSEC_ADDR=10.0.0.10:7100 OPENDSEC_TOKEN= OPENDSEC_CA=/etc/opendsec/pki/ca.crt opendsecctl image import docker.io/library/python:3.12 -as root/py:3.12 -store /mnt/opendsec-images ``` Onboard a team (on the ops machine): ```sh opendsecctl project create root team-a -quota sandboxes=1000 opendsecctl principal create root/team-a trainer -kind agent # the token is shown only in this output opendsecctl grant root/team-a trainer sandbox.create,sandbox.operate opendsecctl grant root trainer layer.use # may use the images under root/ ``` Give the caller the load balancer's address, the token (separately, over a secure channel), `$OPENDSEC_OPS/pki/ca.crt`, the SDK (`sdk/python` or a packaged wheel) and the [caller guide](/en/docs/user-guide). ## 6. Acceptance ```sh # smoke test: one sandbox per backend, running python3 -c 'print(6*7)' OPENDSEC_TOKEN= OPENDSEC_PROJECT=root/team-a \ .claude/skills/opendsec-deploy/scripts/smoke.sh root/py:3.12 container,microvm # features: creation, commands, files, network, pause, fork, batches, images, one by one python3 -m venv .venv && .venv/bin/pip install ./sdk/python OPENDSEC_TOKEN= OPENDSEC_PROJECT=root/team-a .venv/bin/python sdk/python/examples/test_features.py # load test: 600 s at concurrency 8; prints successes, failures and latency percentiles at the end OPENDSEC_TOKEN= OPENDSEC_PROJECT=root/team-a .venv/bin/python sdk/python/examples/load_test.py 600 8 ``` While the load test runs, try a failure drill: hard-reset a control-plane node, or stop the IAM leader's process, and check that the load test reports 0 failures. ## 7. Data centers without internet access Use the offline bundle: build it on a connected machine with `deploy/offline/build-bundle.sh` and unpack it on the ops machine. The steps are the same as above, except: - system packages come from the bundle's `debs/`: set `OPENDSEC_DEBS=/debs`; - the release is the bundle's own: `/deploy/prod/opendsecops release-add /release/`; - images are imported from the bundle's `images/`, e.g. `opendsecctl image import /images/python-3.12.tar -as root/py:3.12 -store /mnt/opendsec-images`; - the SDK and the E2B SDK come from the bundle's `wheels/`: `pip install --no-index --find-links wheels opendsec`. Details in `deploy/offline/README.md`. ## 8. Next - Upgrades, config changes, new machines, maintenance, certificates, failures: [operations manual](/en/docs/operations) - [Monitoring and alerts](/en/docs/monitoring) - Service levels: [SLO](/en/docs/slo) - Hardening: [Security model](/en/docs/security) - E2B SDK access: [operations manual §4, "E2B gateway"](/en/docs/operations) --- # OpenDsec caller guide > For administrators: this guide is written for the people who use the cluster. Fill in the items marked "administrator fills in" and hand it to them. OpenDsec provides isolated environments for running code, called sandboxes. You create sandboxes with the Python SDK, run commands and read and write files in them, and stop them when you are done. ## 1. What you get | File or item | Meaning | |---|---| | `opendsec--py3-none-any.whl` | The Python SDK; needs Python 3.10 or newer | | `ca.crt` | The cluster CA, to verify the server's certificate | | token | Your credential, `dsk..`. **Treat it like a password** and keep it out of code repositories | | project | E.g. `root/prod`. Every sandbox you create counts against this project's quota | | address | See below | **Address** (several addresses, comma separated, spread calls over every reachable apiserver and avoid one that is down): - with a load balancer, a single address, e.g. `api.opendsec.internal:7100`; - without one, every apiserver's address, e.g. `10.0.0.11:7100,10.0.0.12:7100,10.0.0.13:7100`. - Your cluster administrator gives you the address. ## 2. Install and a first sandbox ```sh pip install opendsec--py3-none-any.whl export OPENDSEC_ADDR=10.0.0.11:7100,10.0.0.12:7100,10.0.0.13:7100 export OPENDSEC_TOKEN=dsk.prod-caller.xxxxxxxx export OPENDSEC_PROJECT=root/prod export OPENDSEC_CA=/path/to/ca.crt python quickstart.py ``` ```python import asyncio from opendsec import DSecClient, DSecRunArgs async def main(): async with DSecClient() as client: # reads the variables above sb = await client.run_sandbox(DSecRunArgs( backend="microvm", image="root/py:3.12", cpu_cores_limit=1, memory_limit_mb=1024, ttl_running_stop=600)) # stops after 10 idle minutes try: r = await sb.run_shell("python3 -c 'print(6*7)'") print(r.stdout) # 42 finally: await sb.stop() asyncio.run(main()) ``` ## 3. Common operations | Operation | SDK | |---|---| | Create | `client.run_sandbox(DSecRunArgs(...))`; batches with `client.run_sandboxes(args, count)`, up to 32768 per call | | Run a command | `sb.run_shell(cmd, timeout=...)` returns stdout, stderr and the exit code; environment variables and the current directory carry over within one `session` | | Background commands / streaming | `sb.run_shell_background(...)` + `sb.get_output(...)`; `sb.stream_shell(...)` | | Files | `sb.write_file(path, data)`, `sb.read_file(path)`, `sb.list_dir(path)`, `sb.remove(path)` | | HTTP services in the sandbox | `sb.http(url)` | | Pause / resume | `sb.pause()` / `sb.resume()`: a paused sandbox keeps its memory state and continues where it left off | | Copy | `sb.fork(count=N)`: N copies of a running sandbox, same memory and files (microvm) | | Network rules | `sb.update_network({...})` | | Make an image | container: `sb.pack_diff("root/prod/:")` saves files only, without build residue in /tmp, caches and so on; the image works on container and microvm. microvm: `sb.create_snapshot(image, memory=True)` saves memory too, and sandboxes created from it continue from the moment it was saved; microvm cannot save files only, so build files-only images on container | | Stop | `sb.stop()`. **Always stop sandboxes when you are done**: they hold quota until they time out | | Separate the agent from grading | `run_shell(cmd, session="agent", user="1000")` runs the agent's commands as an unprivileged user, with grading in another session (root by default). A session's user is fixed by its first command. An unprivileged session cannot reach other sessions, read their output records or change system files; write files the agent must not see with strict permissions (e.g. `umask 077`) | ## 4. Specs and limits - **backend**: - `microvm`: Firecracker virtual machines, the strongest isolation, starting in about 0.2 s, with pause, resume and fork. **Recommended.** - `container`: containers, starting in about 0.1 s. - **Images**: images the administrator imported, such as `root/py:3.12` (python:3.12) from the quickstart; `client.list_images()` lists them. Ask your administrator for other images. - **Network**: - Whether sandboxes reach the internet by default depends on the cluster config (`network.default_rules`; administrator fills in). Clusters installed with the quickstart reach nothing by default; allow what you need in `network_rules`, e.g. `{"*": True}` or a rule set `{"pypi": True}`. - Private addresses (the VPC, cluster nodes) and cloud metadata are unreachable by default: `*` excludes them, and private addresses a domain resolves to are always blocked. To reach one internal service, name its address or network explicitly, e.g. `{"10.30.0.10:5432/tcp": True}` ([Network policy](/en/docs/network)). - Narrowing rules: keys of `network_rules` can be rule set names, domains (with subdomains), IPv4 addresses or networks, or `"*"` (every public address), optionally followed by `:port[/protocol]` to allow only those ports, e.g. `{"*": False, "pypi.org:443": True, "files.pythonhosted.org:443/tcp": True}`. Ports may be ranges such as `8000-8080`; the protocol is `tcp` or `udp`, both when left out. - IPv4 only. - **Lifecycle**: - The idle timeout `ttl_running_stop` defaults to 1 hour. A sandbox is idle while no command runs and no other call arrives; it stops when the timeout is reached. - The lifetime `ttl_max` defaults to 1 day and can be at most 7 days. - **Quota**: each project's number of sandboxes and resources at a time are limited (administrator fills in). Creations beyond it get `quota_exceeded`. - **Disk**: the writable disk defaults to 2 GB for microvm and 1 GB for container; set `disk_mb` to change it. - **Command output**: the first 64 MB of each command's stdout and stderr are recorded and the rest is dropped (`ShellResult.dropped_bytes`) while the command keeps running; each session keeps its last 512 MB of output records, and older output can no longer be fetched with `get_output`. - **Rate limits**: the creations and calls per second each principal may make on each apiserver are set in the cluster config (administrator fills in; the defaults are 2000 creations and 10000 calls, the example production config 20000 and 100000). Batch creations are queued by the server and never fail on the rate. ## 5. Errors and retries | `DSecError.reason` | Meaning | What to do | |---|---|---| | `unavailable` | A server is restarting or failing over | The SDK already retried, for up to 60 s by default (`retry_timeout`); you only see this after that | | `no_capacity` | The cluster cannot place it right now | Retry later, or lower concurrency | | `quota_exceeded` | Over the project's quota | Stop sandboxes you no longer use, or ask for more quota | | `rate_limited` | Too many requests | Back off and retry | | `unauthenticated` / `forbidden` | Token invalid or expired, or a permission missing | Check the token and the project | | `not_found` | The sandbox does not exist or was cleaned up | — | | `invalid` | Bad arguments | Read the message | - Creations are safe to retry: the SDK gives each creation an idempotency key, so even if an answer is lost in a server restart, a retry returns the same sandbox. - When the machine a sandbox runs on fails, the sandbox is lost and calls return `not_found` or `unavailable`. Take out results you need to keep in time. ## 6. Tokens - Tokens have a lifetime (the cluster's `max_token_ttl_s`, 90 days with the quickstart; when yours expires: administrator fills in; `opendsecctl token list` shows it). Ask your administrator for a new token before then (`opendsecctl token create `). The old token is revoked once the new one works. - If a token leaks, tell your administrator at once to revoke it (`opendsecctl token revoke`); revocation takes effect on every apiserver within 1 s. ## 7. More - [Concepts](/en/docs/concepts): backends, states, timeouts, sessions, tasks, images, network, permissions, errors - [Python SDK reference](/en/docs/python-sdk): every class and method - [CLI reference](/en/docs/cli): every `opendsecctl` command - [Images and snapshots](/en/docs/images), [Network policy](/en/docs/network), [Projects, principals and permissions](/en/docs/iam) - [Troubleshooting](/en/docs/troubleshooting) --- # Python SDK reference `opendsec` is OpenDsec's asynchronous Python SDK, built on gRPC and `asyncio`. This page lists all of its classes and methods. The concepts behind them (backends, states, timeouts, sessions, permissions) are in [Concepts](/en/docs/concepts). - Python 3.10 or newer; depends on `grpcio>=1.73.1`, `protobuf>=6.31,<8` and `googleapis-common-protos>=1.60`. - Licensed Apache-2.0: programs that call OpenDsec are not bound by the server's AGPL. - Every method is a coroutine: `await` it inside an `async` function. ## Install ```sh # from the source repository python3 -m venv .venv && .venv/bin/pip install ./sdk/python # from a wheel your administrator gave you pip install opendsec-0.11.0-py3-none-any.whl ``` Offline: the offline bundle's `wheels/` directory holds the SDK and all its dependencies; install with `pip install --no-index --find-links wheels opendsec`. ## Quick example ```python import asyncio from opendsec import DSecClient, DSecRunArgs, DSecError async def main(): async with DSecClient() as client: # reads the OPENDSEC_* variables sb = await client.run_sandbox(DSecRunArgs( backend="microvm", image="root/py:3.12", cpu_cores_limit=1, memory_limit_mb=1024, ttl_running_stop=600, # stop after 10 idle minutes network_rules={"pypi": True})) # PyPI only try: r = await sb.run_shell("pip install requests && python3 -c 'import requests; print(requests.__version__)'") print(r.exit_code, r.stdout, r.stderr) finally: await sb.stop() asyncio.run(main()) ``` ## Connecting: `DSecClient` ```python DSecClient(endpoint=None, token=None, project=None, ca_file=None, server_name=None, retry_timeout=60) ``` | Parameter | Environment variable | Default | Meaning | |---|---|---|---| | `endpoint` | `OPENDSEC_ADDR` | `127.0.0.1:7100` | Apiserver address `host:port`. Several addresses, comma separated (`10.0.0.11:7100,10.0.0.12:7100`), spread calls over every reachable apiserver and move off one that goes down | | `token` | `OPENDSEC_TOKEN` | empty | The caller's credential `dsk..`, sent with every call | | `project` | `OPENDSEC_PROJECT` | `root` | Default project for creations, listings, task actions and image listings | | `ca_file` | `OPENDSEC_CA` | none | The cluster's CA certificate. Set, the client uses TLS; unset, plaintext, for development clusters only | | `server_name` | — | none; `api.opendsec.internal` with several addresses | The name the server certificate is checked against. Every apiserver certificate carries `api.opendsec.internal` | | `retry_timeout` | — | 60 | How long (seconds) calls keep retrying while a server is unreachable (`unavailable`: a restart, a failover). 0 disables retries | ```python async with DSecClient() as client: # recommended: connects on entry, closes on exit ... client = DSecClient(endpoint="10.0.0.10:7100", token=tok, project="root/team-a", ca_file="ca.crt") await client.open() # or open and close it yourself ... await client.close() ``` Connection settings (automatic): reconnects start after 100 ms and back off to at most 2 s; a keepalive every 10 s detects a silently lost connection within about 15 s; messages may be up to 80 MB. ## Creation arguments: `DSecRunArgs` ```python @dataclass class DSecRunArgs: backend: str = "sim" image: str = "" cpu_cores_limit: float = 1.0 memory_limit_mb: int = 2048 gpus: int = 0 gpu_mode: str = "" disk_mb: int = 0 ttl_running_stop: int = 0 ttl_max: int = 0 network_rules: dict[str, bool] = {} init_user: str = "root" task_id: str = "" qos: str = "LS" labels: dict[str, str] = {} project: str | None = None ``` | Field | Meaning | |---|---| | `backend` | `microvm`, `container`, `fncall`, `fullvm` or `sim`. **The default, `sim`, only simulates and runs nothing: always set it** | | `image` | Image name `/:`, e.g. `root/py:3.12`. Needs `layer.use` on the image's project | | `cpu_cores_limit` | CPU cores; may be fractional | | `memory_limit_mb` | Memory (MB). Creating from a snapshot with memory, it must equal the snapshot's | | `gpus`, `gpu_mode` | Number of GPUs; `gpu_mode` is `exclusive` or `shared` (default: the node decides) | | `disk_mb` | Writable disk size (MB); 0 means the backend's default | | `ttl_running_stop` | Idle timeout (seconds); 0 means the cluster default (usually 1 hour) | | `ttl_max` | Lifetime (seconds) from creation, paused time included; 0 means the cluster default (usually 1 day); usually at most 7 days | | `network_rules` | Network rules, e.g. `{"pypi": True}`, `{"*": True}`, `{"github.com:443": True}`. See [Network policy](/en/docs/network) | | `init_user` | The sandbox's default user, also the sessions' default user | | `task_id` | Task name, for task actions such as `pause_task` | | `qos` | `LS` (latency-sensitive, default) or `BE` (best effort) | | `labels` | Arbitrary key-value labels, kept in the sandbox's spec | | `project` | The project to create in; default: the client's `project` | `DSecContainerRunArgs` is a subclass with `backend="container"` by default and a `container_image` field (as the paper writes it) that replaces `image` when set: ```python DSecContainerRunArgs(container_image="root/py:3.12", memory_limit_mb=4096, cpu_cores_limit=4) ``` ## Creating and querying ### `run_sandbox(args, timeout=120) -> Sandbox` Creates a sandbox and returns once it is `RUNNING`. - `timeout`: limit for the whole creation (seconds); `None` for none. - While the server is unreachable, the call retries within `retry_timeout`. Each creation carries a random idempotency key: even if the first answer was lost, a retry returns the sandbox the first attempt created, never a second one. - Needs `sandbox.create` on the project and `layer.use` on the image's project. - Common errors: `quota_exceeded`, `no_capacity`, `invalid` (no such image or backend, a timeout over the limit), `forbidden`. ### `run_container(args, timeout=120) -> Sandbox` The same as `run_sandbox`, usually with `DSecContainerRunArgs`. ### `run_sandboxes(args, count, timeout=600) -> list[Sandbox | DSecError]` Creates `count` sandboxes of one spec, up to 32768 per call. - The list is in index order: `Sandbox` for each success, `DSecError` for each failure (not raised). - The server queues creations at the rate limit, so rate limiting never fails part of a large batch. - A lost connection is retried; only what is still missing gets created. - If the batch request itself fails (no permission, say), `DSecError` is raised. ```python res = await client.run_sandboxes(DSecRunArgs(backend="microvm", image="root/py:3.12", task_id="job-7"), 500) ok = [x for x in res if not isinstance(x, DSecError)] failed = [x for x in res if isinstance(x, DSecError)] ``` ### `run_containers(args, count, timeout=600)` The same as `run_sandboxes`. ### `get_sandbox(sandbox_id) -> Sandbox` Looks a sandbox up by id. Needs `sandbox.operate` on its project. Ended sandboxes stay on their node for 10 minutes; after that, `not_found`. ### `list_sandboxes(task_id="", include_ended=False, project=None) -> list[Sandbox]` Lists the sandboxes of a project and all its subprojects. `task_id` narrows to one task; `include_ended=True` includes recently ended ones. Needs `sandbox.operate`. ### `cluster_status() -> Snapshot` The cluster snapshot: each edge's address, health, draining state, backends, capacity, allocated resources, sandboxes per backend, free memory and disk, and pressure (protobuf `Snapshot`; fields in `proto/opendsec/v1/cluster.proto` and `EdgeState` in `types.proto`). ## Sandbox handle: `Sandbox` `run_sandbox`, `get_sandbox` and the others return `Sandbox` objects. | Attribute | Meaning | |---|---| | `id` | Sandbox id | | `state` | State name, e.g. `"RUNNING"`. The **last seen** state; `refresh()` updates it | | `edge_id` | The node it runs on | | `info` | The full record (protobuf `Sandbox`): `project`, `principal`, `spec`, `created_at`, `last_active_at`, `ended_at`, `message` | ### Lifecycle Every lifecycle method returns the `Sandbox` itself (`info` updated) and retries while the server is unreachable. They need `sandbox.operate`. | Method | Meaning | |---|---| | `refresh()` | Fetch the record again | | `stop()` | Stop and clean up. The quota comes back at once | | `pause()` | Pause. A microVM is snapshotted and its process ends, freeing all its memory; a container's processes are frozen | | `resume()` | Resume. Optional: a runtime call to a paused sandbox resumes it | | `touch()` | Reset the idle timer; resumes a paused sandbox | | `update_network(rules, replace=False)` | Change network rules: merged into the current ones by default, replaced with `replace=True`. Taking an allowance back applies to new connections at once and cuts existing ones too | ### Running commands ```python run_shell(cmd, timeout=None, session="default", env=None, cwd=None, user=None) -> ShellResult ``` Runs a shell command in a session and returns its result when it ends. | Parameter | Meaning | |---|---| | `cmd` | A whole shell command, interpreted by the session's shell: pipes, `&&` and so on work | | `timeout` | Seconds. When it expires, the processes the command started are killed, the session stays, and `ShellResult.timed_out` is `True` | | `session` | Session name. Within a session, `cd` and `export` carry over between commands; sessions are independent of each other | | `env`, `cwd` | Environment variables and working directory for this command only | | `user` | The session's user, fixed by its **first** command: a name or `uid[:gid]` (e.g. `"1000"`). Later values for the same session have no effect | A lost connection (edge or apiserver restart) is retried without running the command twice. ```python r = await sb.run_shell("cd /srv && export X=1") r = await sb.run_shell("pwd; echo $X") # /srv 1 r = await sb.run_shell("pytest -q", timeout=600, cwd="/srv/repo") if not r.ok: print(r.exit_code, r.stderr) ``` `ShellResult` fields: | Field | Meaning | |---|---| | `cmd_id` | The command's number in its session | | `exit_code` | Exit code | | `stdout`, `stderr` | Output decoded as UTF-8 (undecodable bytes become `�`); the raw bytes are `stdout_bytes`, `stderr_bytes` | | `done` | Whether the command has ended (background commands may not have) | | `timed_out` | Whether it was killed by its timeout | | `duration` | Run time (seconds) | | `truncated` | The output exceeds what one call returns (16 MB per stream) and was not all returned; the full output (up to 64 MB per stream) is still recorded: fetch the rest with `get_output(cmd_id, stdout_offset=...)` | | `dropped_bytes` | Bytes dropped past the 64 MB per-stream record limit | | `ok` | `done and exit_code == 0 and not timed_out` | ### Background commands ```python cmd_id = await sb.run_shell_background("python3 train.py", session="train") ... r = await sb.get_output(cmd_id, session="train", wait=30) # wait up to 30 s print(r.done, r.stdout) ``` - `run_shell_background(cmd, timeout=None, session="default", env=None, cwd=None, user=None) -> int` starts the command and returns its number at once. - `get_output(cmd_id, session="default", wait=0, stdout_offset=0, stderr_offset=0) -> ShellResult` returns the recorded output, waiting up to `wait` seconds for the command to end; `stdout_offset` and `stderr_offset` start from a byte position, for reading incrementally. ### Streaming output ```python async for kind, data in sb.stream_shell("make test", timeout=1800): if kind == "stdout": sys.stdout.buffer.write(data) elif kind == "stderr": sys.stderr.buffer.write(data) elif kind == "exit": print("exit code", data.exit_code) # data is a ShellResult without output ``` `stream_shell(cmd, timeout=None, session="default", env=None, cwd=None, user=None)` yields `("stdout", bytes)` and `("stderr", bytes)` chunks as they come, then `("exit", ShellResult)`. A broken connection resumes where it broke: the command keeps running in the sandbox and nothing is repeated or lost. ### Sessions - `list_sessions() -> list[str]`: the sandbox's sessions. - `close_session(session)`: end a session, killing every process it started. ### Files Paths are absolute paths in the sandbox; file calls run as the session's user (`session` parameter, default `default`). | Method | Meaning | |---|---| | `read_file(path, offset=0, length=0) -> bytes` | Read a file; `length=0` reads to the end | | `write_file(path, data, mode=0, append=False) -> FileInfo` | Write a file; `data` is `bytes` or `str` (encoded as UTF-8). `mode` sets the permissions (e.g. `0o755`; 0 for the default); `append=True` appends | | `list_dir(path) -> list[FileInfo]` | List a directory | | `stat(path) -> FileInfo` | File information | | `remove(path, recursive=False)` | Remove a file; `recursive=True` removes a directory and its contents | `FileInfo` fields: `name`, `type` (`file`, `dir`, `symlink`, `other`), `size`, `mode`, `mtime`. Size limits: - one `read_file` returns at most 64 MB; read larger files in pieces with `offset` and `length`; - keep one `write_file` under 64 MB (a message may be up to 80 MB); write larger files in pieces with `append=True`; - in older code, a message through the apiserver was limited to 4 MB, and going over it reported `quota_exceeded` (really a message that was too large). The first entry under "Fixed" in CHANGELOG 0.11.0 raised the limit to 80 MB on the servers, the Go client and the Python SDK; such errors now report `invalid`. ```python # Read a large file in pieces. Past the end, read_file returns b"". chunks, off = [], 0 while True: part = await sb.read_file("/data/big.bin", offset=off, length=32 << 20) if not part: break chunks.append(part) off += len(part) ``` ### HTTP services in the sandbox ```python r = await sb.http("http://127.0.0.1:8000/health", method="GET", headers=None, body=b"", timeout=30) print(r.status, r.headers, r.body) ``` Makes an HTTP request from inside the sandbox's network namespace, to reach services running in it. Returns `HttpResult(status, headers, body)`. For a direct TCP connection (WebSocket, say), use the API's `ConnectPort`, or reach the port through the E2B gateway. ### Fork, snapshots and images ```python fork(count=1, ttl_running_stop=0, ttl_max=0, labels=None, task_id="", timeout=600) -> list[Sandbox | DSecError] ``` Copies a running sandbox into `count` (1 to 256) running sandboxes. - The source pauses briefly and carries on, its `id` and timeouts unchanged. - Copies inherit the source's spec; `labels` are added to theirs, `task_id` replaces the task (default: the source's), and `ttl_running_stop` and `ttl_max` are the copies' timeouts (0 for the cluster default). - microVM copies include memory, so processes continue from the fork; container copies get the files only. - Results are in index order, failures as `DSecError`. Each copy counts against quota. A retry after a lost connection creates only the copies still missing. - Needs `sandbox.create` on the project and `layer.use` on the source image's project. ```python create_snapshot(image, memory=False, overwrite=False) -> ImageInfo ``` Saves the sandbox as image `image` (`/:`) exactly as it is; nothing is cleaned up. - `memory=True` (microVM only) includes memory: sandboxes created from the image resume from the snapshot, with its processes running. Their `memory_limit_mb` must equal the snapshot's. - microVMs require `memory=True`. For a files-only image, build on container and use `pack_diff`. - `overwrite=True` replaces an existing image of that name. - Needs `layer.publish` on the image's project. ```python pack_diff(image, scrub=(), overwrite=False) -> ImageInfo ``` Saves the sandbox's changes against the image it started from as a new image `image`, stacked on that image (container-type backends only). - Removed by default: `/tmp/*`, `/var/tmp/*`, `/root/.cache`, `/root/.bash_history`, `/home/*/.bash_history`, `/root/.python_history`. `scrub` adds more (globs relative to `/`, e.g. `["srv/repo/.git"]`). - The image works for both container and microVM. - Needs `layer.publish` on the image's project. ```python sb = await client.run_container(DSecContainerRunArgs(container_image="root/rl/python:3.12", network_rules={"pypi": True, "github": True})) await sb.run_shell("git clone https://github.com/org/repo /srv/repo && pip install -e /srv/repo") img = await sb.pack_diff("root/rl/repo-env:v1") await sb.stop() # from now on, create from root/rl/repo-env:v1 instead of installing again ``` `ImageInfo` fields: `name`, `layers` (bottom first), `env`, `workdir`, `user`, `parent`, `source`, `created_at`, `created_by`, `snapshot`, `memory`. ## Task actions | Method | Meaning | |---|---| | `pause_task(task_id, project=None)` | Pause every sandbox of the task | | `resume_task(task_id, project=None)` | Resume every sandbox of the task | | `stop_task(task_id, project=None)` | Stop every sandbox of the task | They return a `TaskActionResult`: `affected` (succeeded), `failed`, `errors` (the failures' reasons). They need `sandbox.operate`. ## Images | Method | Meaning | Permission | |---|---|---| | `list_images(project=None) -> list[ImageInfo]` | Images of the project and its subprojects | `layer.use` | | `get_image(name) -> ImageInfo` | One image | `layer.use` | | `compose_image(name, parts, env=None, workdir="", user="", overwrite=False) -> ImageInfo` | Stack the `parts` bottom first into a new image without copying data. `env` adds environment variables; `workdir` and `user` set the defaults | `layer.publish` on the new image's project, `layer.use` on each part's | | `delete_image(name)` | Delete the image's registration. `opendsecctl image gc` reclaims its layers | `layer.publish` | ```python await client.compose_image("root/rl/swe-4821:v1", ["root/rl/python:3.12", "root/rl/django-ws:a1b2", "root/tools/agent-kit:3"]) ``` Importing from a registry is CLI-only (`opendsecctl image import`): it writes into the store on a machine that mounts it. ## Projects, principals and tokens These methods use the same API as `opendsecctl`, for people and agents alike, under the same permission rules ([Projects, principals and permissions](/en/docs/iam)). | Method | Meaning | Needs | |---|---|---| | `whoami() -> str` | The principal of the current token | — | | `create_project(parent, name, quota=None) -> Project` | Create a subproject of `parent`. `quota` e.g. `{"sandboxes": 100, "cpu_cores": 400}`; keys left out take the parent's limit | `project.admin` on `parent` | | `get_project(project=None) -> Project` | `path`, `quota`, `usage` (current), `policies` (each principal's permissions), `children` | any permission on the project | | `set_quota(project, quota) -> Project` | Set the quota (replaced as a whole); may not exceed any ancestor's | `project.admin` on the parent project | | `delete_project(project)` | Delete an empty project (no subprojects, no usage, no principals of its own) | `project.admin` on the parent project | | `create_principal(project, principal_id, kind="agent", ttl=0) -> str` | Create a principal in the project and return its first token (**returned only this once**). `ttl`: the token's lifetime in seconds, 0 for the server default | `project.admin` | | `delete_principal(principal)` | Delete a principal and all its tokens | `project.admin` on the principal's project | | `grant(project, principal, perms) -> Project` | Grant permissions, e.g. `["sandbox.create", "sandbox.operate"]`. Only permissions you hold yourself | `project.admin` and each permission granted | | `revoke(project, principal, perms=()) -> Project` | Revoke permissions; all of them when `perms` is empty | `project.admin` | | `create_token(principal=None, ttl=0, description="") -> (str, TokenInfo)` | Issue a token, by default for yourself | yourself: nothing; others: `project.admin` on their project | | `list_tokens(principal=None) -> list[TokenInfo]` | Tokens without secrets: `id`, `created_at`, `expires_at` (0: never), `description`, `created_by` | as above | | `revoke_token(principal, token_id)` | Revoke a token; it stops working on every apiserver within about 1 s | as above | | `get_usage(project=None, start="", end="") -> UsageReport` | Usage of the project and its subprojects per UTC day; `start` and `end` are `YYYY-MM-DD`, inclusive | `project.admin` | Each row of `UsageReport.rows`: `project`, `day`, `sandboxes_started`, `sandbox_hours`, `cpu_core_hours`, `memory_gb_hours`, `gpu_hours`. ## Errors: `DSecError` Every failure raises `DSecError`, except single items of a batch creation or fork, which go in the result list. ```python from opendsec import DSecError, QUOTA_EXCEEDED, NO_CAPACITY try: sb = await client.run_sandbox(args) except DSecError as e: if e.reason in (QUOTA_EXCEEDED, NO_CAPACITY): await asyncio.sleep(5) # try again later else: raise ``` - `e.reason`: a stable reason string, with a constant of the same name in the module: `UNAUTHENTICATED`, `FORBIDDEN`, `NOT_FOUND`, `QUOTA_EXCEEDED`, `RATE_LIMITED`, `NO_CAPACITY`, `INVALID`, `UNAVAILABLE`, `TIMEOUT`, `INTERNAL`. What each means and what to do: [Concepts §9](/en/docs/concepts#_9-errors). - `e.message`: the server's explanation. **What is retried automatically**: only `unavailable` (server unreachable), within `retry_timeout`, for creation, batch creation, fork, lifecycle calls, commands, file reads, directory listings, sessions and streams. `write_file`, `remove`, `http`, `update_network`, `pack_diff`, `create_snapshot` and the management methods are not retried; on failure the caller decides. ## Patterns ### RL rollouts ```python async def rollout(client, step_image, episode): sb = await client.run_sandbox(DSecRunArgs(backend="microvm", image=step_image, cpu_cores_limit=2, memory_limit_mb=2048, task_id="ppo-run-42", ttl_max=3600)) try: for action in episode: r = await sb.run_shell(action, session="agent", user="1000", timeout=120) ... score = await sb.run_shell("python3 /grader/grade.py", session="grader") # a root session return float(score.stdout) finally: await sb.stop() # training is preempted await client.pause_task("ppo-run-42") # afterwards, the next call to each sandbox resumes it; or explicitly: await client.resume_task("ppo-run-42") ``` ### Branching from one state ```python base = await client.run_sandbox(DSecRunArgs(backend="microvm", image="root/rl/swe-4821:v1")) await base.run_shell("cd /srv/repo && git apply /tmp/partial.patch") branches = await base.fork(count=8, task_id="search-17") results = await asyncio.gather(*(b.run_shell(f"python3 try.py --variant {i}") for i, b in enumerate(branches) if not isinstance(b, DSecError))) ``` ### A warmed-up environment ```python sb = await client.run_sandbox(DSecRunArgs(backend="microvm", image="root/rl/model-server:v1", memory_limit_mb=4096)) # start a long-running service that loads a model (slow), then waits for requests on port 8000 await sb.run_shell("(nohup python3 /srv/serve.py > /tmp/serve.log 2>&1 &)") await sb.run_shell("until curl -sf http://127.0.0.1:8000/ready; do sleep 1; done", timeout=600) await sb.create_snapshot("root/rl/model-server:warm", memory=True) await sb.stop() # created from the snapshot, the service is already running with the model loaded: ready in about 55 ms fast = await client.run_sandbox(DSecRunArgs(backend="microvm", image="root/rl/model-server:warm", memory_limit_mb=4096)) ``` More examples in the repository's `sdk/python/examples/`: `quickstart.py`, `test_features.py` (checks each feature in turn), `load_test.py` (load test), `decoupled_rollout.py` (decoupled rollouts). --- # CLI reference: opendsecctl `opendsecctl` is OpenDsec's command-line tool, for administrators and callers alike: create and operate sandboxes, manage images, projects, principals and tokens, inspect the cluster, issue certificates, and manage cluster releases (fleet). ``` opendsecctl [arguments] [options] opendsecctl help # every command opendsecctl fleet # help for the fleet subcommands ``` - It is installed as `/usr/local/bin/opendsecctl` (single-node install) or `/opt/opendsec/current/bin/opendsecctl` (cluster nodes); on the ops machine it is `$OPENDSEC_OPS/current/bin/opendsecctl`. - Options may come before or after positional arguments: `opendsecctl sandbox stop -addr x` and `opendsecctl sandbox stop -addr x ` are the same. - Unless noted, output is JSON (protobuf's JSON form; fields at their default value are left out). - Exit code 0 on success; on failure it prints `error: : ` and exits non-zero. `exec` exits with the command's own exit code. ## Connection and environment | Option | Environment variable | Default | Meaning | |---|---|---|---| | `-addr` | `OPENDSEC_ADDR` | — | Apiserver address; several, comma separated | | `-token` | `OPENDSEC_TOKEN` | — | Token. Prefer the variable, so it stays out of shell history and process lists | | `-project` | `OPENDSEC_PROJECT` | `root` | Default project | | `-ca` | `OPENDSEC_CA` | — | Cluster CA certificate; set, TLS is used | | `-store` | `OPENDSEC_IMAGE_STORE` | — | Mount point of the image store (`image import`, `image gc`, `image store-check`) | | `-deadline` | — | 5m | Time limit for the whole command; none for `image import` | After a single-node install, the administrator's and the caller's environments are: ```sh . /root/opendsec-ops/env # administrator: admin token, project root . /root/opendsec-ops/demo.env # caller: a token for project root/demo, plus the E2B SDK's variables ``` ## Sandboxes ### `sandbox create` ``` opendsecctl sandbox create -backend [-image I] [-cpu N] [-mem MB] [-disk MB] [-gpus N] [-task T] [-ttl-idle S] [-ttl-max S] [-net rule,...] [-count N] ``` | Option | Default | Meaning | |---|---|---| | `-backend` | `sim` | `microvm`, `container`, `fncall`, `fullvm`, `sim`. **The default, `sim`, only simulates: always set it** | | `-image` | — | Image, e.g. `root/py:3.12` | | `-cpu` | 1 | CPU cores | | `-mem` | 1024 | Memory, MB | | `-disk` | 0 | Writable disk, MB; 0 for the backend's default | | `-gpus` | 0 | GPUs | | `-task` | — | Task name | | `-ttl-idle` | 0 | Idle timeout (seconds); 0 for the cluster default | | `-ttl-max` | 0 | Lifetime (seconds); 0 for the cluster default | | `-net` | — | Network rules, comma separated; `!rule` takes back one of the cluster defaults, e.g. `-net 'pypi,github.com:443,!npm'` | | `-count` | 1 | Number of sandboxes | With `-count 1` the output is the sandbox's JSON. With a larger `-count`, each result is a line (`indexid`, or `indexerrorreason: message` for a failure), followed by a summary: ``` $ opendsecctl sandbox create -backend microvm -image root/py:3.12 -count 3 -task job-1 0 node-a1.4k2m… 1 node-b1.9xq1… 2 node-a1.p0zz… created 3/3 in 412ms; failures: map[] ``` The id in a script: ```sh ID=$(opendsecctl sandbox create -backend microvm -image root/py:3.12 | grep -m1 '"id"' | cut -d'"' -f4) ``` ### `sandbox get | stop | pause | resume | touch ` View, stop, pause, resume, or reset the idle timer. Prints the updated sandbox. ### `sandbox list [-task T] [-all]` Sandboxes of `-project` and its subprojects; `-task` narrows to a task, `-all` includes recently ended ones. ### `sandbox fork [-count N] [-task T] [-ttl-idle S] [-ttl-max S]` Copies a running sandbox into N (1 to 256); output as for batch creation. ### `sandbox network ... [-replace]` Changes network rules: `pypi` allows, `!pypi` takes back. Merged into the current rules by default; `-replace` replaces them. ```sh opendsecctl sandbox network $ID pypi github.com:443 opendsecctl sandbox network $ID '!github.com:443' opendsecctl sandbox network $ID '*' -replace # only the whole public internet ``` ### `exec [-session S] [-user U] [-timeout S] [--] ` Runs a command in a sandbox, streaming its stdout and stderr, and exits with its exit code. - **One argument** is a whole shell command, interpreted by the sandbox's shell: `opendsecctl exec $ID 'cd /srv && make test | tail'`. - **Several arguments** are each quoted and joined, as `ssh` and `docker exec` do: `opendsecctl exec $ID python3 -c 'print(6*7)'` runs exactly `python3 -c 'print(6*7)'` in the sandbox. - If an argument starts with `-` (as in `ls -la`), put `--` before the command so opendsecctl does not take it as its own option: `opendsecctl exec $ID -- ls -la /`. - `-session`: session name, default `default`; `-user`: the session's user when this command starts it; `-timeout`: seconds; on expiry it prints `opendsecctl: command timed out`. > Before 0.11, several arguments were joined with spaces and no quoting. Scripts that added an extra layer of quotes to work around that (e.g. `"'print(6*7)'"`) must drop it. ### `cp : ` / `cp :` Copies one file between this machine and a sandbox, up to 64 MB; use the SDK's chunked reads and writes for larger files. ## Tasks ``` opendsecctl task pause|resume|stop ``` Acts on every sandbox of the task in `-project` and prints how many succeeded and failed. ## Images ### `image import -as /: -store ` Imports an image into the shared store and registers it. Run it **as root on a machine that mounts the image store**; it needs `layer.publish` on the target project. The source can be: | Source | Example | |---|---| | Public registry | `docker.io/library/python:3.12`, `python:3.12`, `ghcr.io/org/image:tag` | | Private registry | `harbor.example.com/team/app:1.0` | | Registry without TLS | add `-plain-http` | | OCI layout, directory or tar | `./python-3.12.oci.tar` | | `docker save` tar | `./app.tar`; with several images in the file, `file#name` picks one | Local paths must start with `/` or `.`; a missing file is an error (it is never looked up in a registry). Private registry credentials are taken from, in order: 1. `-username U -password-stdin`: the password or access token is read from stdin, e.g. `echo "$TOKEN" | opendsecctl image import ... -username 'robot$ci' -password-stdin`; 2. `-registry-auth FILE`: a docker-format credentials file; 3. `$REGISTRY_AUTH_FILE`, `$DOCKER_CONFIG/config.json`, `~/.docker/config.json` (what `docker login` saved). An import has no overall time limit (a large image can take long); progress goes to stderr. ### `image save -out ` Saves a registry image as an OCI tar, for machines without internet access to `image import ./file`. Needs no cluster. Takes `-plain-http`, `-username`, `-password-stdin`. ### `image compose ... [-overwrite]` Stacks images bottom first into one, without copying data: ```sh opendsecctl image compose root/rl/swe-4821:v1 root/rl/python:3.12 root/rl/django-ws:a1b2 root/tools/agent-kit:3 ``` ### `image list [-project P]`, `image get `, `image delete ` List the images of a project and its subprojects; show one image; delete an image's registration (`image gc` reclaims the layers). ### `image pack [-scrub a,b] [-overwrite]` `pack_diff`: saves a container sandbox's changes against the image it started from as a new image. `/tmp`, caches and shell history are removed by default; `-scrub` adds paths (globs relative to `/`). ### `image snapshot [-memory] [-overwrite]` Saves a sandbox as an image as it is. microVMs need `-memory` (memory included). ### `image gc -store [-grace 24h] [-dry-run]` Removes layers and snapshots no image refers to. Anything younger than `-grace` (default 24 hours) stays, so imports in progress and fresh snapshots are safe. Try `-dry-run` first to see what would go. It needs the connection settings but works on the store only. ### `image store-check -store ` Checks whether a file system (a newly mounted NAS, say) can hold the image store: permissions, hard links, renames, xattrs, sparse files and more. Every required item must be `ok`. Needs no cluster. ``` $ opendsecctl image store-check -store /mnt/opendsec-images ok write, sync and read back a file ok files a root process creates are root's (no root squash) ok chown to another user (image files keep their owners) ok exclusive create (one edge builds a VM disk at a time) ok rename a file over another (manifests, records) ok rename a directory (layers are published so) ok hard links (snapshots link their disks) ok symbolic links ok finds the holes of sparse files (snapshots move only their data; NFS needs vers=4.2) ok 0:0 character devices (layer whiteouts) ok overlay over layers kept here, whiteouts honored ok trusted.* xattrs ok user.* xattrs /mnt/opendsec-images can hold the image store ``` Items that are not required (sparse-file holes, `trusted.*` xattrs) only warn when they fail, and the store still works; a failed required item rules the file system out. ## Projects, principals, permissions and tokens | Command | Meaning | |---|---| | `project create [-quota k=v,...]` | Create a subproject, e.g. `project create root team-a -quota sandboxes=1000,cpu_cores=4000,memory_mb=4096000` | | `project get ` | Quota, current usage, each principal's permissions, subprojects | | `project delete ` | Delete an empty project | | `principal create [-kind agent\|user\|service] [-ttl S]` | Create a principal; **the token in the output is shown only this once** | | `principal delete ` | Delete a principal and all its tokens | | `grant [,...]` | Grant permissions, e.g. `grant root/team-a trainer sandbox.create,sandbox.operate` | | `revoke [,...]` | Revoke permissions; all of them when none are named | | `token create [] [-ttl S] [-desc D]` | Issue a token, by default for yourself | | `token list []` | List tokens (no secrets) | | `token revoke ` | Revoke; takes effect cluster-wide within about 1 s. `primary` is the legacy primary token | | `whoami` | The principal of the current token | | `usage [] [-from YYYY-MM-DD] [-to YYYY-MM-DD] [-daily] [-csv]` | Usage: by default each project's total over the period; `-daily` per day and project; `-csv` CSV output | Quota keys: `sandboxes`, `cpu_cores`, `memory_mb`, `gpus`. What the permissions mean: [Projects, principals and permissions](/en/docs/iam). ## Cluster | Command | Meaning | Needs | |---|---|---| | `status` | Cluster snapshot: each edge's health, draining state, capacity, allocations, sandbox counts, free memory and disk, pressure | — | | `edge drain [-undo]` | Stop (`-undo`: resume) placing new sandboxes on an edge; its sandboxes keep running | `project.admin` on root | | `iam backup -out FILE` | Back up IAM (gzipped JSON, mode 0600): principals, projects, grants, quotas, token hashes | as above | | `iam restore FILE` | Replace IAM's contents with a backup (quota leases of running sandboxes are kept) | as above | | `iam members` | List the IAM Raft replicas | as above | | `iam member add -raft-addr H:P -iam-addr H:P` | Add a replica started with `opendsec-iam -join` | as above | | `iam member remove ` | Remove a replica (its machine is gone, say) | as above | | `version` | This build's version | — | ## Certificates These need no cluster. | Command | Meaning | |---|---| | `pki init -config opendsec.json -out DIR [-days 365] [-ca-days 3650] [-hosts h1,h2]` | Create a cluster CA and a certificate for every component in the config. `-hosts` adds host names or IPs to every certificate | | `pki ca -out DIR [-days N]` | A new CA only (CA rotation) | | `pki issue -ca DIR -role -id [-hosts ip1,ip2] -out DIR [-days 365]` | A certificate for one component. Roles: `edge`, `apiserver`, `watcher`, `iam`, `fleet`, `node`, `ops` | Production clusters usually use `opendsecops certs`, which calls these; see [Cluster deployment](/en/docs/cluster). ## Local clusters and benchmarks | Command | Meaning | |---|---| | `cluster up` | Run a whole cluster in this process, for development and tests | | `bench` | Scheduling benchmark on an in-process simulated cluster; `-compare` compares random placement with power-of-2 | `cluster up` options: | Option | Default | Meaning | |---|---|---| | `-config` | a generated simulated cluster | Cluster config file | | `-sim-edges` | 4 | Simulated edges without `-config` | | `-apiservers` | 1 | Apiservers without `-config` | | `-runtime` | `sim` | Runtime of the container backend in generated clusters: `sim` or `linux` (real containers, needs root) | | `-microvm-kernel` | — | Also offer the `microvm` backend, with this guest kernel | | `-image-store` | — | Shared image store directory (enables the store and `pack_diff`) | | `-state-dir` | `/var/lib/opendsec` | Sandbox state directory (linux runtime) | | `-host-image` | true | Offer the development image `host` (the host's `/` as a read-only lower layer: it exposes host files, development only) | | `-disk-reserve-mb` | 512 | Reject creations while the state directory has less free space; -1 disables | | `-tls` | false | Create a CA and certificates and run with mTLS | | `-audit-dir` | — | Apiserver audit log directory | | `-metrics` | — | Prometheus metrics address, e.g. `:9464` | | `-write-config` | — | Write the effective config (with the bound addresses) to this file | | `-log` | `info` | Log level | ```sh ./bin/opendsecctl cluster up -sim-edges 1 -runtime linux -image-store /mnt/opendsec-images \ -microvm-kernel /opt/opendsec/vm/vmlinux-6.1.167-opendsec # then set OPENDSEC_ADDR, OPENDSEC_TOKEN and OPENDSEC_PROJECT as it prints ``` ## fleet: cluster releases and configuration These go through an apiserver and need `fleet.view` (read) or `fleet.operate` (change) on root; `project.admin` includes both. Every action is audited. | Command | Meaning | |---|---| | `fleet nodes [-state S] [-role R] [-domain D]` | Machines: state, failure domain, roles, desired and running release, units ready, last heartbeat | | `fleet node get ` | One machine | | `fleet node accept -roles edge=[,...] [-domain D] [-addr-ip IP] [-labels k=v,...]` | Accept a newly reporting machine: deliver the current release and config and start it | | `fleet node set [-roles ...] [-domain D] [-addr-ip IP] [-labels k=v,k=]` | Change roles, failure domain, labels (`k=` removes a label). Changes the record only; deploys nothing | | `fleet node sync ` | Bring a machine to its desired release, config and roles | | `fleet node cordon \| uncordon ` | Stop or resume placing sandboxes on it | | `fleet node evacuate [-deadline 1h]` | Cordon, then wait for its sandboxes to end | | `fleet node retire ` | Take a machine out: stop and disable its components | | `fleet node pin [-reason R]`, `unpin ` | Pin or unpin: pinned machines are left out of rollouts | | `fleet import -inventory FILE -release V [-config-gen G] [-domain D]` | Register machines already running release V (when moving from opendsecops) | | `fleet releases` | Uploaded releases | | `fleet release pack -bin DIR -units DIR [-deps DIR] -out DIR` | Package a release directory (usually done by `opendsecops release`) | | `fleet release push [-notes N]` | Upload a release to every fleet replica | | `fleet config put [-note N]` | Upload a new cluster config as a new config generation | | `fleet config get [-gen N] [-out FILE]` | Fetch a config generation | | `fleet rollout create [-release V] [-config-gen G] [...]` | Start an upgrade or config change; options below | | `fleet rollout status [] [-steps]` | Progress, of the newest rollout by default; `-steps` lists every machine | | `fleet rollout pause \| resume \| abort \| rollback ` | Pause, continue, abort, roll back | | `fleet rollouts` | Past rollouts | `fleet rollout create` options: | Option | Meaning | |---|---| | `-release V`, `-config-gen G` | Target release, target config generation (at least one) | | `-roles`, `-domains`, `-nodes`, `-labels` | Only machines with these roles, domains, names, labels | | `-canary N` | Machines upgraded alone first; default 1, -1 for none | | `-bake S` | Observation time (seconds) after the canary and after each batch; default 300, -1 for none | | `-batch N` | Machines per batch; default 5% of the non-control-plane machines (at least 1, at most 100) | | `-max-per-domain N` | At most this many machines of one failure domain per batch; default half the batch (at least 1) | | `-pause S` | Seconds between batches; default 60, -1 for none | | `-max-failures N` | Failed machines tolerated before pausing; default 0, i.e. pause at the first failure | | `-step-timeout S` | Timeout per machine (seconds), default 600 | | `-note N` | A note | With no apiserver available, connect to fleet directly from the ops machine with the ops certificate: ```sh opendsecctl fleet nodes -direct -fleet 10.0.0.1:7300,10.0.0.2:7300 -pki $OPENDSEC_OPS/pki -ops-id ops ``` The full release procedure is in the [operations manual §6](/en/docs/operations). --- # Images and snapshots Sandboxes start from images. OpenDsec keeps images in an **image store** shared by every node, each image a stack of read-only layers. This page covers image names, importing, making images from sandboxes, using snapshots, and the store's requirements and cleanup. ## 1. Names An image name is `/:`: | Part | Rule | Example | |---|---|---| | Project | An existing project path; each segment `[A-Za-z0-9][A-Za-z0-9._-]`, up to 64 characters | `root`, `root/rl` | | Name | `[a-z0-9][a-z0-9._-]`, up to 128 characters, lower case only | `py`, `swe-4821` | | Tag | `[A-Za-z0-9_][A-Za-z0-9._-]`, up to 128 characters; `latest` when left out | `3.12`, `v1`, `step3` | An image belongs to the project in its name, and permissions follow that project: | Action | Permission | |---|---| | Create sandboxes from it, view, list | `layer.use` on the image's project | | Import, compose, `pack_diff`, snapshot, delete | `layer.publish` on the image's project | A name without `/` (such as `host`) is a node-local image (`edges[].images`), for development and tests only. ## 2. Importing Import from a registry or a file with `opendsecctl image import`. Run it **as root on a machine that mounts the image store** (any compute node, or the single-node machine): it writes the layers into the store directly, then registers the image through the API. ```sh # public registry (docker.io/library/ can be left out for Docker Hub) opendsecctl image import python:3.12 -as root/py:3.12 -store /mnt/opendsec-images # private registry: docker login's saved credentials, or pass them echo "$HARBOR_TOKEN" | opendsecctl image import harbor.example.com/team/app:1.0 -as root/app:1.0 \ -store /mnt/opendsec-images -username 'robot$ci' -password-stdin # an internal registry without TLS opendsecctl image import 10.0.0.5:5000/base:1 -as root/base:1 -store /mnt/opendsec-images -plain-http # files: a docker save tar, an OCI layout directory or tar docker save myimage:dev -o myimage.tar opendsecctl image import ./myimage.tar -as root/rl/myimage:dev -store /mnt/opendsec-images opendsecctl image import ./bundle.tar#app:1 -as root/app:1 -store /mnt/opendsec-images # a file holding several images ``` - The image's environment variables, working directory and default user come from its own config (a Dockerfile's `ENV`, `WORKDIR`, `USER`). `ENTRYPOINT` and `CMD` do not run by themselves: you run commands once the sandbox is up. - Identical layers are stored once: a second image on the same base reuses its layers. - `-overwrite` replaces an image of the same name; without it, an existing name is an error. - On a single-node install, `. /root/opendsec-ops/env` already sets `OPENDSEC_IMAGE_STORE`, so `-store` can be left out. **Data centers without internet access**: save the image to a file on a connected machine, copy it over and import it: ```sh opendsecctl image save python:3.12 -out python-3.12.oci.tar # on a connected machine; no cluster needed opendsecctl image import ./python-3.12.oci.tar -as root/py:3.12 -store /mnt/opendsec-images ``` ## 3. Composing Stack several images in order into a new one, **copying no data**: ```sh opendsecctl image compose root/rl/swe-4821:v1 \ root/rl/python:3.12 \ # bottom: the base environment root/rl/django-ws:a1b2 \ # middle: a repository and its dependencies root/tools/agent-kit:3 # top: the agent's tools ``` ```python await client.compose_image("root/rl/swe-4821:v1", ["root/rl/python:3.12", "root/rl/django-ws:a1b2", "root/tools/agent-kit:3"], env={"TASK": "4821"}, workdir="/srv/repo") ``` - Later images lie on top of earlier ones: a file in an upper layer wins, and a file an upper layer deleted is gone from the result. - Environment variables are merged in order, with `env` applied last; `workdir` and `user` default to the parts'. - It suits RL's "base environment × repository × tools" combinations: each part is built once, and any combination costs no extra space. ## 4. Making images from sandboxes Two ways: | | `pack_diff` | `create_snapshot` | |---|---|---| | Backends | container, fncall | container, microvm | | Saves | The sandbox's changes against its image, as a new layer on top of that image | The sandbox's whole state; with memory on microVMs | | Cleanup | Removes `/tmp`, `/var/tmp`, `/root/.cache`, shell and Python history, plus what you add | None; saved as is | | A sandbox created from it | Starts fresh, with the files as saved | microVM: resumes from the snapshot with its processes running | | Works for | container and microVM | the same backend (a container snapshot holds files only) | | Typical use | Reusable environments: dependencies installed, code built | Warm starts: models loaded, services running; the state at one step | ### pack_diff ```python sb = await client.run_container(DSecContainerRunArgs( container_image="root/rl/python:3.12", network_rules={"pypi": True, "github": True})) await sb.run_shell("git clone https://github.com/org/repo /srv/repo && pip install -e /srv/repo") img = await sb.pack_diff("root/rl/repo-env:v1", scrub=["srv/repo/.git"]) await sb.stop() ``` ```sh opendsecctl image pack $ID root/rl/repo-env:v1 -scrub srv/repo/.git ``` - `scrub` takes globs relative to `/`. - The new image records its parent image (`parent`) and has one layer more. - microVMs do not support `pack_diff`. For a files-only image, build on container: the image works on microVMs too. ### Snapshots ```python sb = await client.run_sandbox(DSecRunArgs(backend="microvm", image="root/py:3.12", memory_limit_mb=2048, network_rules={"pypi": True})) await sb.run_shell("pip install -q numpy") await sb.run_shell("(nohup python3 -c 'import numpy, http.server as h; h.test(h.SimpleHTTPRequestHandler, port=8000)' > /tmp/srv.log 2>&1 &)") img = await sb.create_snapshot("root/rl/warm:v1", memory=True) await sb.stop() # created from the snapshot in about 55 ms: the HTTP server, numpy loaded, is already running on port 8000 new = await client.run_sandbox(DSecRunArgs(backend="microvm", image="root/rl/warm:v1", memory_limit_mb=2048)) ``` ```sh opendsecctl image snapshot $ID root/rl/warm:v1 -memory ``` - microVMs must include memory (`memory=True` / `-memory`). - Sandboxes created from a snapshot with memory must use the snapshot's `memory_limit_mb`. - A snapshot is placed only on nodes running the Firecracker version it was made with; after a cluster upgrades Firecracker, older snapshots work only on nodes not yet upgraded, so make them again when needed. - Every sandbox created from a snapshot gets its own hostname, machine-id and random seed. - A snapshot can be used at once on the node of its sandbox; meanwhile it is published to the image store in the background, and other nodes can use it once that finishes (seconds to tens of seconds, depending on the store). Creations on other nodes wait for it until then. ### Fork and snapshots `fork` also takes a snapshot and creates the copies from it, but the snapshot is not registered as an image and is not needed once the copies exist. To create from one state many times over a long period, use `create_snapshot`; for a few copies now, use `fork`. ## 5. The image store ### Requirements The store is mounted at the same path on every apiserver and edge node (default `/mnt/opendsec-images`), the config's `image_store.root`. NFS, CephFS, JuiceFS and other shared file systems work: - **Root squash must be off**: edges write snapshots as root, and apiservers delete them. - With NFS, use version 4.2 and these mount options: `vers=4.2,nconnect=8,hard,timeo=600,lookupcache=positive,_netdev`. - 4.1 works too, but cannot find the holes of sparse files: using a memory snapshot on another node then reads it at its full size (more than twice as slow in our tests). - `lookupcache=positive` stops the client from caching "file not found". Without it, a node that looked for an image before it was registered could keep missing it for up to a minute. - Every node needs a distinct NFS client identity (`deploy/node/install.sh` sets one from `/etc/machine-id`); otherwise the server takes nodes with the same host name for one client. Check the mount before use: ```sh opendsecctl image store-check -store /mnt/opendsec-images ``` Every required item must be `ok`. The items are listed in the [CLI reference](/en/docs/cli#image-store-check-store-dir). ### Layout ``` /layers//rootfs/... layer contents (overlay format) /layers//meta.json /images///.json image manifests: the layers and defaults /snapshots//... snapshots (disks, memory) /vmdisks/... microVM image disks (EROFS with erofs_images) ``` Do not change these files by hand; use the `opendsecctl image` commands. ### How nodes use images - **Containers**: each node keeps local copies of image layers (`layer_cache`, on by default), copied from the store on first use; later containers stack those local layers. Running containers are unaffected when the store is unreachable. - **microVMs**: each image becomes a read-only image disk on the node. With `erofs_images`, the disk is built once in the store as EROFS and copied to each node; otherwise every node flattens it to ext4 itself. With `image_pmem`, VMs of one image share the host's page cache. - While the store is unreachable, new creations, `pack_diff`, snapshots and cross-node forks wait until it is back or their calls time out; running microVMs are unaffected. ## 6. Deletion and cleanup ```sh opendsecctl image delete root/rl/old:v1 # delete the registration opendsecctl image gc -store /mnt/opendsec-images -dry-run # see what would go opendsecctl image gc -store /mnt/opendsec-images # remove layers and snapshots no image refers to ``` - `image delete` removes the image's registration only; its layers may still serve other images and are not deleted right away. - `image gc` removes layers and snapshots no image refers to. Anything younger than `-grace` (default 24 hours) stays, so imports in progress and fresh snapshots are safe. - Running sandboxes are unaffected: containers use the node's local copies, microVMs the node's image disks. - Edges reclaim their own local copies and disk caches. Run `image gc` regularly (daily, say) on a machine that mounts the store. --- # Network policy Sandbox networking denies everything by default: a sandbox can neither connect out nor be connected to. Callers allow the destinations they need; administrators define rule sets and cluster-wide default rules. This page covers how to write rules, how they are enforced, and the edge cases. It assumes the node has sandbox networking enabled (`edges[].network.enabled`, on in the single-node install and the example configs). On nodes without it, sandboxes only have a loopback interface and `network_rules` is ignored. ## 1. Writing rules `network_rules` is a dict: each key is a rule, `true` allows it, `false` takes it back. | Rule | Example | Meaning | |---|---|---| | Rule set name | `pypi` | A group of destinations the administrator defined in `network.rule_sets` | | Domain | `github.com` | The domain and all its subdomains (`api.github.com`, `raw.github.com`, …). `*.github.com` means the same | | IPv4 address | `1.2.3.4` | That address | | IPv4 network | `203.0.113.0/24`, `10.20.0.0/16` | The whole network | | `*` | `*` | Every **public** address (private ones excluded; see §4) | Any rule can be limited to ports with `:port[/protocol]`: | Rule | Meaning | |---|---| | `pypi.org:443` | Port 443 only, TCP or UDP | | `10.0.0.0/8:5432/tcp` | TCP 5432 only | | `*:443` | Port 443 of every public address | | `example.com:8000-8080` | A port range | | `1.1.1.1:53/udp` | UDP 53 only | Entries of rule sets can carry ports too. IPv4 only: sandboxes have IPv4 addresses only, and AAAA queries get empty answers. ### Examples ```python # Python packages only DSecRunArgs(..., network_rules={"pypi": True}) # the internet, HTTPS only DSecRunArgs(..., network_rules={"*:443": True}) # the internet, plus one database on the internal network DSecRunArgs(..., network_rules={"*": True, "10.20.0.15:5432/tcp": True}) # the cluster allows the internet by default; not for this sandbox DSecRunArgs(..., network_rules={"*": False}) # the cluster allows the internet by default; this sandbox gets PyPI on 443 only DSecRunArgs(..., network_rules={"*": False, "pypi.org:443": True, "files.pythonhosted.org:443": True}) ``` On the command line, `-net` takes rules comma separated, with `!` for `false`: ```sh opendsecctl sandbox create -backend microvm -image root/py:3.12 -net 'pypi,github.com:443' opendsecctl sandbox create -backend microvm -image root/py:3.12 -net '!*' ``` ## 2. Rule sets and default rules (administrators) In the cluster config's `network` section ([configuration reference](/en/docs/configuration#network)): ```json "network": { "rule_sets": { "pypi": ["pypi.org", "files.pythonhosted.org"], "npm": ["registry.npmjs.org"], "github": ["github.com", "codeload.github.com", "objects.githubusercontent.com"], "mirror": ["10.30.0.10:443/tcp"] }, "default_rules": {"mirror": true}, "upstream_dns": ["10.0.0.2"] } ``` - **Rule sets** name common destinations, so callers write `{"pypi": true}` without knowing the domains. A rule set applies only when `network_rules` names it. - **Default rules** apply to every sandbox, under its own `network_rules`: for the same key, the sandbox's value wins, so callers can take a default back with `false`. - For internet access by default: `"default_rules": {"*": true}`. - The single-node install (quickstart) has no default rules: sandboxes reach nothing by default. - Sandboxes created through the E2B gateway get internet access by default, as on E2B, unless the request turns it off ([Migrating from E2B](/en/docs/e2b)). - **Upstream DNS**: the resolvers the edges' DNS proxies query; default: the host's `/etc/resolv.conf`. Set them when the internal network has its own DNS. ## 3. Enforcement Each edge enforces the rules with nftables and a DNS proxy: - **Outbound**: every packet a sandbox sends goes through its own chain, in this order: 1. accept the addresses and networks its rules name (with their port limits); 2. drop private, link-local and multicast destinations; 3. accept addresses the DNS proxy resolved for it from allowed domains; 4. drop the rest. - **DNS**: the sandbox's DNS queries are redirected to the edge's DNS proxy. - Allowed domains: the query goes upstream, and the IPv4 addresses in the answer are added to the sandbox's allowed set before it gets the answer, so a connection can always use an address it just resolved. - Other domains: NXDOMAIN (no such domain). - With `*` allowed, every name resolves. - Addresses learned this way expire after an hour unused; resolving the name again adds them back. - **Inbound**: every new connection to a sandbox is dropped, whether from another sandbox, the host or outside. Replies to connections the sandbox opened pass. - **Spoofing**: packets with a forged source address match no rule and are dropped. - **Host services**: apart from DNS, a sandbox reaches no service on its host. - **Egress**: sandbox traffic leaves through the host with masquerading; the outside sees the node's address. To reach a service running in a sandbox, do not go over the network: use the SDK's `sb.http()`, the API's `ConnectPort`, or the E2B gateway's port forwarding. These enter the sandbox's network namespace from the edge and do not pass the rules above. ## 4. Private addresses These ranges are reachable only through an **address or network rule naming them**: `0.0.0.0/8`, `10.0.0.0/8`, `100.64.0.0/10`, `127.0.0.0/8`, `169.254.0.0/16` (cloud metadata `169.254.169.254` included), `172.16.0.0/12`, `192.0.0.0/24`, `192.168.0.0/16`, `198.18.0.0/15`, `224.0.0.0/4`, `240.0.0.0/4` - `*` does not include them. - Private addresses a domain resolves to are always blocked, even for an allowed domain. This stops DNS from being used to get around the rules. - To reach one internal service (a database, a package mirror, a model server), name its address and port in a network rule, e.g. `10.30.0.10:443/tcp`. Allow only what is needed: allowing whole private ranges lets code in the sandbox reach cluster nodes and other internal services. ## 5. Changing rules at run time ```python await sb.update_network({"github": True}) # merge: add a rule await sb.update_network({"github": False}) # take it back await sb.update_network({"pypi": True}, replace=True) # replace: only this rule (the defaults still apply) ``` ```sh opendsecctl sandbox network $ID github opendsecctl sandbox network $ID '!github' opendsecctl sandbox network $ID pypi -replace ``` - Needs `sandbox.operate`. - New allowances apply at once. - **Taking an allowance back applies at once and cuts existing connections**: their later packets go through the chain too. - The new rules are written into the sandbox's spec; `get_sandbox` shows the current ones. ## 6. Troubleshooting | Symptom | Likely cause | Check | |---|---|---| | `Could not resolve host` / no such domain | The domain is not allowed (the DNS proxy answers NXDOMAIN) | `sb.info.spec.network_rules` and the cluster's `default_rules`; subdomains are covered, parent domains are not | | Resolves, but the connection times out | A port limit does not match; or the name resolves to a private address | Try a rule without ports; `getent hosts ` shows what it resolves to | | An internal service times out | Private addresses need an explicit network rule | Add `
:/tcp` | | A rule is `invalid` | Not a valid domain, IPv4 address or network; a port outside 1–65535; a protocol other than tcp or udp | The message names the offending entry | | No rule works, and the sandbox only has `lo` | The node has no sandbox networking | The administrator checks `edges[].network.enabled` | | pip through a proxy or mirror fails | The mirror's domain is not in the rules | Add the mirror to a rule set, or use a network rule | Checking from inside the sandbox: ```sh cat /etc/resolv.conf # DNS points at the sandbox's gateway getent hosts pypi.org # allowed domains resolve python3 -c 'import urllib.request as u; print(u.urlopen("https://pypi.org/simple/", timeout=5).status)' ``` --- # Projects, principals and permissions OpenDsec organizes resources in a **project tree**, authenticates callers as **principals** holding **tokens**, controls who may do what with **permissions** granted on projects, and limits each project's resources with **quotas**. This page describes those rules and a few common setups. ## 1. Projects - Projects form a tree rooted at `root`, with paths separated by `/`, e.g. `root/team-a/rl`. - Project names cannot contain `/` or spaces. - Every sandbox and image belongs to a project: a sandbox to the project it was created in, an image to the project in its name (`root/team-a/py:3.12` belongs to `root/team-a`). - Listings of sandboxes and images include all subprojects. ```sh opendsecctl project create root team-a -quota sandboxes=1000,cpu_cores=4000,memory_mb=4096000 opendsecctl project create root/team-a rl opendsecctl project get root/team-a ``` `project get` shows the quota (`quota`), current usage (`usage`), the permissions each principal holds directly on the project (`policies`) and its subprojects (`children`). It needs any permission on the project. `opendsecctl project delete root/team-a/rl` deletes a project. Only empty projects can be deleted: no subprojects, no usage (no running sandboxes) and no principals of their own. Root cannot be deleted. ## 2. Principals and tokens A **principal** is whatever calls the API: a person, a service or an agent. ```sh opendsecctl principal create root/team-a trainer -kind agent ``` - Principal ids are unique across the cluster and cannot contain `.`, `/` or spaces. - A principal belongs to the project it was created in, whose admins manage it (issue and revoke its tokens, delete it). - `kind` is `user`, `service` or `agent` (default `agent`): a label only, with no effect on permissions. - Creation issues the first token, **shown only in that output**. - A new principal has no permissions; grant them separately (§3). A **token** is a principal's credential, sent as `authorization: Bearer ` (the SDK and opendsecctl do this): - Format `dsk...`. The older format `dsk..` is the principal's primary token with id `primary`; the administrator's bootstrap token has this form. - The server keeps only a hash of the secret: a lost token cannot be recovered, only replaced. - A principal can hold many tokens, e.g. one per training machine, so a leaked one can be revoked alone. ```sh opendsecctl token create trainer -ttl 2592000 -desc "gpu-node-07" # 30 days opendsecctl token list trainer opendsecctl token revoke trainer ``` | Rule | Detail | |---|---| | Who manages tokens | Every principal manages its own; `project.admin` on a principal's project manages that principal's | | Lifetime | `-ttl` seconds. With the cluster's `max_token_ttl_s` set: left out, the limit applies; above the limit, `invalid`. Without it: left out means never expiring | | Revocation | Takes effect on every apiserver within about 1 s: each revocation makes every apiserver drop its authentication cache | | After expiry | Calls get `unauthenticated` | **Replacing a token**: issue a new one → give it to the caller → the caller switches → confirm the old one is no longer used → revoke the old one. **Failed authentication limit**: once one client address fails more than `auth_fail_rate` authentications per second (default 5), new tokens from it are not checked for a while, against brute force. ## 3. Permissions Permissions are granted on a project and apply to it and **all its subprojects**: ```sh opendsecctl grant root/team-a trainer sandbox.create,sandbox.operate opendsecctl revoke root/team-a trainer sandbox.operate opendsecctl revoke root/team-a trainer # all of them ``` | Permission | Allows | |---|---| | `sandbox.create` | Creating sandboxes, batch creation, fork (copies are created in this project) | | `sandbox.operate` | Getting and listing sandboxes; commands, files, HTTP, processes, ports; pause, resume, stop, touch; network rule changes; task actions | | `layer.use` | Creating sandboxes from the project's images; viewing and listing them; using them as parts of `compose` | | `layer.publish` | Publishing the project's images: import, compose, `pack_diff`, snapshot, delete | | `project.admin` | Creating, setting quotas on and deleting subprojects; creating and deleting principals; granting and revoking permissions; managing its principals' tokens; usage | | `fleet.view` | On root: viewing the cluster management service (machines, releases, rollouts) | | `fleet.operate` | On root: releases, config changes, starting and controlling rollouts, managing machines | Notes: - Permissions **do not include each other**: `project.admin` alone does not let you create sandboxes; grant each permission needed. The exception is cluster management: `project.admin` on root works as `fleet.view` and `fleet.operate` too. - **Cluster-wide operations** need `project.admin` on root: draining an edge (`edge drain`), IAM backup and restore, IAM replica management. - Operating a sandbox checks `sandbox.operate` on **the sandbox's project**, whoever created it. Principals with `sandbox.operate` on the same project can operate each other's sandboxes; put callers who must be isolated from each other in different projects. - Creating a sandbox needs two permissions: `sandbox.create` on the target project and `layer.use` on the image's project. - The bootstrap administrator (the config's `admin_id`, default `admin`) holds every permission on root and cannot be deleted. ### Bounded delegation Project admins can hand management down, but never more than they have: - **Granting** needs `project.admin` on the project and every permission granted, held on that project. An admin without `layer.publish` cannot grant `layer.publish`. - **Quotas**: a subproject's quota cannot exceed any ancestor's. Setting it needs `project.admin` on the **parent** project; a project's own admins cannot raise their own quota. - So a whole subproject can safely be handed to a team or an agent: they create projects, principals, quotas and grants inside it, but cannot break the bounds set above them. ## 4. Quotas | Key | Meaning | |---|---| | `sandboxes` | Sandboxes existing at once (running or paused) | | `cpu_cores` | Sum of CPU cores | | `memory_mb` | Sum of memory (MB) | | `gpus` | Sum of GPUs | - A creation must fit the quota of its project **and of every ancestor**: a subproject's usage counts in each of them. Going over any level is `quota_exceeded`, and the message names the level and key. - Keys left out when a project is created or its quota set take the tightest ancestor limit at that moment, stored in the project. Root's quota is set by operators (`iam.root_quota`); unset, unlimited. - Lowering an ancestor's quota later takes effect at once: every ancestor is checked again at each creation, so the effective limit is always the smallest along the path. - Setting a quota replaces it as a whole: keys left out of `set_quota` take the tightest ancestor limit again. - Paused sandboxes keep their quota: the capacity they need to resume is still theirs. - Stopping a sandbox returns its quota at once. When a node fails, the watcher releases its sandboxes' quota after `lost_edge_grace` (default 120 s). - Quotas bound what is held **at once**, not cumulative use; for that, see §5. ```sh opendsecctl project create root/team-a eval -quota sandboxes=200,cpu_cores=400 # SDK: await client.set_quota("root/team-a/eval", {"sandboxes": 300, "cpu_cores": 600}) ``` ## 5. Usage ```sh opendsecctl usage root/team-a -from 2026-10-01 -to 2026-10-31 # each project's total over the period opendsecctl usage root -from 2026-10-01 -daily -csv > usage.csv # per day and project ``` ```python report = await client.get_usage("root/team-a", start="2026-10-01", end="2026-10-31") for row in report.rows: print(row.day, row.project, row.sandboxes_started, row.sandbox_hours, row.cpu_core_hours, row.memory_gb_hours, row.gpu_hours) ``` - Covers the project and all its subprojects: sandboxes started, sandbox-hours, CPU core-hours, memory GB-hours, GPU-hours. - Per UTC day. Each row counts the project's own sandboxes only; subprojects get rows of their own. Without `-daily`, the last row is the total. - What is counted is the time quota is held: running and paused alike. Sandboxes still running count up to the time of the query. - Needs `project.admin` on the project. - Kept for 400 days in IAM and replicated with Raft, so node restarts and failovers lose nothing. ## 6. Common setups ### Onboarding a team ```sh export OPENDSEC_ADDR=:7100 OPENDSEC_TOKEN=$(cat $OPENDSEC_OPS/admin.token) OPENDSEC_CA=$OPENDSEC_OPS/pki/ca.crt opendsecctl project create root team-a -quota sandboxes=1000,cpu_cores=4000,memory_mb=4096000 opendsecctl principal create root/team-a team-a-admin -kind user # the team's admin opendsecctl grant root/team-a team-a-admin project.admin,sandbox.create,sandbox.operate,layer.use,layer.publish opendsecctl grant root team-a-admin layer.use # may use the shared images under root/ ``` The team admin then creates subprojects and principals and hands out permissions up to their own. ### A training job's caller ```sh opendsecctl principal create root/team-a trainer -kind agent opendsecctl grant root/team-a trainer sandbox.create,sandbox.operate opendsecctl grant root trainer layer.use ``` ### Separate build and run accounts ```sh # the build account may publish images under root/team-a opendsecctl principal create root/team-a builder -kind service opendsecctl grant root/team-a builder sandbox.create,sandbox.operate,layer.use,layer.publish # the run account may only use them opendsecctl grant root/team-a trainer layer.use ``` Even if the code running jobs is compromised, it cannot tamper with the images training uses. ### Agents managing their own subproject Agents use the same API (the SDK's `create_project`, `create_principal`, `grant`, `create_token`) to give subtasks principals and quotas in their own subproject: ```python async with DSecClient(token=agent_admin_token, project="root/team-a/agent-7") as c: await c.create_project("root/team-a/agent-7", "sub-1", quota={"sandboxes": 10}) tok = await c.create_principal("root/team-a/agent-7/sub-1", "agent-7-sub-1", ttl=3600) await c.grant("root/team-a/agent-7/sub-1", "agent-7-sub-1", ["sandbox.create", "sandbox.operate"]) ``` Bounded delegation keeps the subtasks within the agent's own quota and permissions. ### E2B gateway callers The E2B gateway calls OpenDsec with the caller's own token, which needs `sandbox.create` and `sandbox.operate` on `e2b.project` and `layer.use` on the template images' project; snapshots (`create_snapshot()`) also need `layer.publish` on `e2b.project`. ## 7. Backup and restore IAM's whole content (principals, projects, permissions, quotas, token hashes, usage) can be backed up: ```sh opendsecctl iam backup -out iam-$(date +%F).json.gz # gzipped JSON, mode 0600 opendsecctl iam restore iam-2026-10-24.json.gz ``` - Needs `project.admin` on root. - On a deployed cluster, every IAM node backs up daily at 03:00 to `/var/backups/opendsec` (`opendsec-iam-backup.timer`); `opendsecops backup` takes one onto the ops machine right away. - Restoring replaces IAM's content. Quota leases of running sandboxes are not in backups; the next reconciliation (default 30 s) registers them again from the edges. - Principals, tokens and projects created after the backup do not exist after a restore and must be recreated. Full disaster recovery steps: [operations manual §5](/en/docs/operations). --- # Moving from E2B to OpenDsec OpenDsec's E2B gateway (`opendsec-e2b`) serves the API that E2B's SDKs talk to. Code written for E2B, and agent frameworks built on its SDKs, run on your own OpenDsec cluster without changes: you point the SDK at the gateway and use an OpenDsec token as the API key. The design and its tests are in SPEC §37. ## For callers: three environment variables ```sh export E2B_API_URL=http://:7400 # https://... when TLS is on export E2B_SANDBOX_URL=http://:7400 export E2B_API_KEY= # dsk.. ``` ```python from e2b import Sandbox sbx = Sandbox.create() # template "base" print(sbx.commands.run("python3 -V").stdout) sbx.files.write("/tmp/a.txt", "hello") sbx.kill() ``` E2B's JS SDK needs Node.js 22.19 or newer (Ubuntu 24.04's own `nodejs` is 18: take Node from nodejs.org): ```js import { Sandbox } from 'e2b' const sbx = await Sandbox.create() console.log((await sbx.commands.run('python3 -V')).stdout) await sbx.kill() ``` The code interpreter works the same way, on a template your administrator built for it (see below): ```python from e2b_code_interpreter import Sandbox sbx = Sandbox.create("code-interpreter-v1") print(sbx.run_code("6 * 7").text) # 42 ``` Your token's permissions, quota and audit trail apply as for any OpenDsec call: the gateway keeps no state of its own and makes every call with your token. ## What works Checked with E2B's Python SDK (sync and async), JS/TS SDK and code interpreter SDKs (`test/e2e/testdata/e2b_*`): | Area | Supported | |---|---| | Sandboxes | `create`, `connect`, `get_info`, `list`, `kill`, `set_timeout`, `is_running`, metadata, environment variables | | Commands | output and streaming, exit codes, `envs`, `cwd`, background processes, stdin, `kill`, `list`, reconnecting | | Terminals | `pty.create`, input, resize | | Files | `read`, `write` (any size, 2 MB chunks), `list`, `make_dir`, `rename`, `remove`, `exists`, `get_info` | | Ports | `get_host(port)`: HTTP, streamed responses, WebSocket to a server in the sandbox | | Code interpreter | `run_code` with results, logs, errors, charts, pandas tables, contexts, `envs` | | Fork and snapshots | `fork(count)`, `create_snapshot(name)`, `list_snapshots`, `delete_snapshot`, `Sandbox.create(snapshot_id)`; on microVMs memory is kept, so running processes carry over | | Lifecycle | `pause`, resume through `connect`, `lifecycle={"on_timeout": "pause"}` (autoPause), the refresh keepalive | | Network | `allow_internet_access`, `network={"allow_out": [...], "deny_out": ["0.0.0.0/0"]}`, `update_network` | ## What differs | | E2B | OpenDsec gateway | |---|---|---| | API key | E2B key | an OpenDsec token | | Templates | built by E2B's template build | names an administrator maps to an OpenDsec image, backend and size in the cluster config; an OpenDsec image name (`root/py:3.12`) also works. No template build API | | Sandbox ports | public access can be enabled | always need the token (the SDKs send it); no plain browser access | | Paused sandboxes | resumed by traffic only with autoResume | any call resumes them | | After an auto-resume | the original timeout restarts | the idle timeout applies | | Filesystem-only pauses | supported | refused (400): pauses keep memory | | Filesystem-only snapshots | supported | refused (400) on microVMs; container snapshots are filesystem-only (no memory to keep) | | Egress denials | any CIDR | only all traffic (`0.0.0.0/0`); everything else is allowed by rule | | Domain rules | `*.example.com` covers subdomains | also covers `example.com` itself | | `envVars` given at creation, in `run_code` | applied | not applied (pass `envs` to `run_code`) | | Code interpreter kernels | Python, JavaScript, R, Java, Bash | Python | | Not available | | watching directories, signed file URLs, MCP, volumes, metrics (an empty list), request transform rules, egress proxy | An option the gateway cannot honour is refused with 400 rather than ignored, so a program never runs with settings it did not get. ## Speed Measured through the gateway on microVMs (Firecracker), on the project's bare-metal test hosts (AMD EPYC 9Y24), 2026-10-23: | | Time | |---|---| | `Sandbox.create()`, template `base` (2 vCPU, 1 GB, cold boot) | p50 166 ms | | First command after creation | p50 12 ms | | `Sandbox.create("code-interpreter-v1")` (2 GB memory snapshot) | about 250 ms | | First `run_code` | about 40 ms | | `fork(count=2)` | about 100–150 ms | | `create_snapshot()` | about 40–70 ms | | `Sandbox.create(snapshot_id)` | about 65–85 ms | ## For administrators 1. Add an `e2b` section to the cluster config and roll it out (docs/OPERATIONS.md §4, "E2B 网关"): ```json "e2b": { "gateways": [{"addr": "10.0.0.11:7400"}, {"addr": "10.0.0.12:7400"}], "project": "root/e2b", "templates": {"base": {"image": "root/py:3.12", "backend": "microvm", "cpu": 2, "memory_mb": 1024}}, "max_timeout_s": 86400 } ``` 2. Give nodes the fleet role `e2b=`, one at a time (`opendsecctl fleet node set` then `fleet node sync`). 3. Put a load balancer in front: TCP 7400, health check HTTP 9467 `/readyz`, idle timeout at least 60 s. Terminate TLS there, or set `tls_cert` and `tls_key` on the gateways. 4. Give callers tokens with `sandbox.create` and `sandbox.operate` on the project and `layer.use` on the images' projects. Snapshots also need `layer.publish` on the project, since a snapshot is an image there. 5. The code interpreter needs its template: `deploy/e2b/build-code-interpreter.py` builds `root/code-interpreter:v1`, a memory snapshot of a microVM running E2B's code-interpreter server, and the template entry maps a name to it. 6. As on E2B, sandboxes made through the gateway can reach the internet unless the caller turns it off (`allow_internet_access=False`, or `deny_out` of all traffic), whatever the cluster's `network.default_rules`. --- # Configuration reference The whole cluster shares one JSON configuration file, deployed to `/etc/opendsec/opendsec.json` on every node. Every component reads it and uses the parts that concern it. This page describes each field and its default. - A complete production example: `deploy/prod/cluster.example.json`; the config a single-node install generates is `/root/opendsec-ops/opendsec.json`. - Durations are in **seconds** (fractions allowed). - For most numeric fields, 0 or leaving the field out means the default; some fields take -1 to turn something off, as noted. - Changing the config: on a cluster run by fleet, `opendsecctl fleet config put` and `fleet rollout create -config-gen ` ([operations manual §6.2](/en/docs/operations)); before fleet, `opendsecops push-config all` and `opendsecops restart`. - Validating a config: `opendsecctl pki init -config -out $(mktemp -d)` succeeds only if its format and constraints are right. ## Top level ```json { "tls": { ... }, // mTLS between components, TLS on the public API "service_token": "...", // development only: component auth without TLS "iam": { ... }, "watcher": { ... }, "apiserver": [ ... ], "placement": { ... }, "admission": { ... }, "image_store": { ... }, "network": { ... }, "edges": [ ... ], // edges listed one by one "edge_template": { ... }, // every other edge (new machines, Kubernetes) "fleet": { ... }, "e2b": { ... }, "extensions": { ... } // extensions in the build (none in the open-source build) } ``` At least one of `tls.dir` and `service_token` must be set; production clusters use `tls`. ## tls | Field | Meaning | |---|---| | `dir` | Certificate directory, e.g. `/etc/opendsec/pki`, holding `ca.crt` and each component's `-.crt` and `.key` (from `opendsecctl pki init` or `opendsecops certs`). Set, components use mTLS among themselves and the public API uses TLS | ## service_token For development clusters: a shared secret components authenticate with when there is no TLS. Do not set it in production; a config delivered through fleet may not contain it. ## iam | Field | Default | Meaning | |---|---|---| | `replicas` | — | Raft replicas, each `{"id", "addr", "raft_addr", "data_dir"}`. With replicas, `addr` and `state` are not used, and `admin_token` or `admin_token_file` is required | | `replicas[].id` | — | Replica id: unique, without `/`, `.` or spaces | | `replicas[].addr` | — | gRPC address, e.g. `10.0.0.1:7001` | | `replicas[].raft_addr` | — | Raft address, e.g. `10.0.0.1:7011` | | `replicas[].data_dir` | — | Raft log and snapshots, e.g. `/var/lib/opendsec/iam`, on a local SSD | | `addr` | — | Address of a single unreplicated IAM | | `state` | in memory | State file of a single IAM; unset, state lives in memory only and is lost on restart | | `admin_id` | `admin` | The administrator created at bootstrap | | `admin_token` | — | The administrator's token, `dsk..`. Placeholders containing `change-me` are refused | | `admin_token_file` | — | Read `admin_token` from this file instead, so the config holds no secret. Only IAM nodes need the file | | `root_quota` | unlimited | The root project's quota, e.g. `{"sandboxes": 10000}` | | `max_token_ttl_s` | 0 (none) | Longest lifetime of issued tokens. With a limit, tokens that never expire cannot be issued | The admin token is used only when IAM first starts (bootstraps): it creates the `admin_id` principal with every permission on root. ## watcher | Field | Default | Meaning | |---|---|---| | `replicas` | — | Several watchers, each `{"id", "addr"}`. All probe edges and serve snapshots; the one holding the IAM lock reconciles | | `addr` | — | Address of a single watcher | | `probe_interval` | 1 | How often each edge is probed | | `probe_timeout` | 0.5 | Timeout of one probe | | `unhealthy_after` | 3 | Consecutive failures before an edge counts as unhealthy | | `reconcile_interval` | 30 | How often quotas are reconciled with IAM | | `reconcile_min_age` | 60 | Leases younger than this are never released by reconciliation | | `lost_edge_grace` | 120 | After an edge is unhealthy this long, its sandboxes' quota is released | | `unbound_max_age` | 600 | Leases no edge claimed are released after this long | | `lock_ttl` | 10 | Lease time of the reconciliation lock | ## apiserver An array with one entry per apiserver instance; `opendsec-apiserver -index N` runs entry N. | Field | Default | Meaning | |---|---|---| | `addr` | — | Listen address, e.g. `10.0.0.1:7100` | | `audit_dir` | off | Audit log directory, e.g. `/var/log/opendsec/audit` | | `audit_max_mb` | 256 | Rotate an audit log file at this size | | `audit_keep` | 8 | Files kept, the current one included; the oldest go first | | `audit_runtime` | false | Also audit runtime calls (commands, files, HTTP, processes, ports) and read-only queries (getting and listing sandboxes, cluster status, projects, usage, token lists). By default every other call is audited: creation, fork, stop, pause, resume, network changes, image publishing and deletion, projects, principals, grants, tokens, cluster operations. Commands are recorded as their length and SHA-256 only, never their text | | `snapshot_refresh` | 1 | How often the cluster snapshot is refreshed from the watchers | ## placement Scheduling parameters, applied by each apiserver. | Field | Default | Meaning | |---|---|---| | `k` | 2 | power-of-k: pick k random candidate nodes and use the least loaded | | `max_attempts` | 3 | Tries on other nodes after an edge rejects a creation | | `reject_penalty` | 2 | A node that rejected is ranked down for this long | | `overlay_grace` | 2 | A sandbox just placed counts toward its node's load for this long, until the snapshot shows it | | `usage_floor` | 0.2 | A node's actual CPU and memory use below this share does not count as load (the host's own baseline); -1 counts it always | | `burst_threshold` | 0.8 | Only for `burst_eligible_images` on the `container` backend: once local (`onprem`) nodes are used beyond this share, sandboxes go to `zone: cloud` nodes first, with a probability growing with the excess; below it, cloud nodes are only a fallback when no local node fits. ≤0 disables | | `burst_eligible_images` | — | Images that may run on cloud nodes (their layers are mirrored there). Other images and backends never go to cloud nodes | ## admission Creation policy each apiserver enforces. Timeouts: 0 means the built-in default, -1 means none. Rates are per principal and per apiserver instance (N apiservers admit up to N times the rate); -1 disables a limit. | Field | Default | Meaning | |---|---|---| | `default_ttl_running_stop_s` | 3600 | Idle timeout when a request sets none | | `default_ttl_max_s` | 86400 | Lifetime when a request sets none | | `max_ttl_max_s` | 604800 (7 days) | Longest lifetime a request may ask for; more is `invalid` | | `request_rate` | 10000 | Calls per second per principal (commands and file calls included) | | `request_burst` | 2 × `request_rate` | Burst | | `create_rate` | 2000 | Sandbox creations per second per principal | | `create_burst` | 32768 | Burst: a whole batch may start at once | | `auth_fail_rate` | 5 | Failed authentications per second per client address; beyond it, new tokens from that address are not checked for a while | | `max_inflight_creates` | 2048 | Concurrent creations per apiserver | A single creation over the rate gets `rate_limited`; batch creations are queued at the rate by the server and never partly fail on it. With the apiservers behind a load balancer, have it keep the client's source address, or every failed authentication counts against the load balancer's address. ## image_store | Field | Meaning | |---|---| | `root` | Mount point of the shared image store, e.g. `/mnt/opendsec-images`, the same path on every apiserver and edge node. Without it, only edge-local images (`edges[].images`) exist | File system requirements: [Images and snapshots §5](/en/docs/images). ## network The cluster-wide network policy. | Field | Meaning | |---|---| | `rule_sets` | Rule sets: name → destinations (domains, IPv4 addresses or networks, optionally with `:port[/protocol]`). E.g. `{"pypi": ["pypi.org", "files.pythonhosted.org"], "github": ["github.com", "codeload.github.com", "objects.githubusercontent.com"]}` | | `default_rules` | Rules every sandbox has, under its own `network_rules`. `{"*": true}` lets every sandbox reach the internet by default; `{"pypi": true}` only PyPI | | `upstream_dns` | Resolvers the edges' DNS proxies use; default: the host's `/etc/resolv.conf` | Rule syntax and behaviour: [Network policy](/en/docs/network). ## edges and edge_template `edges` lists edges one by one; `edge_template` applies to any edge not listed (`opendsec-edge -id ` uses the template when no entry has that id). New machines usually need no entry in `edges`. | Field | Default | Meaning | |---|---|---| | `id` | — | Edge id, `[a-z0-9-]`, 1 to 32 characters; required in `edges` | | `addr` | — | Listen address, e.g. `0.0.0.0:7201` | | `advertise` | the bound address | Address registered with the watchers | | `zone` | `onprem` | `onprem` or `cloud`, nothing else. `cloud` nodes are used only past `burst_threshold` | | `capacity` | automatic | `{"cpu_cores", "memory_mb", "gpus", "disk_mb"}`. With a real backend: with neither `cpu_cores` nor `memory_mb` set, they are the host's CPU count and total memory (setting only one leaves the other unset); without `disk_mb`, it is the size of the file system holding `state_dir`. Overcommit applies on top | | `cpu_overcommit` | 8 | CPU overcommit factor | | `mem_overcommit` | 1 | Memory overcommit factor | | `disk_overcommit` | 1 | Writable-disk overcommit factor | | `mem_reserve_mb` | automatic | Reject creations while the host's MemAvailable is below this. 0 means automatic (10% of RAM, at least 128 MB); -1 disables | | `disk_reserve_mb` | automatic | Reject creations while the state directory's file system has less free space. 0 means automatic (5% of the file system, at least 1024 MB); -1 disables | | `cpu_pressure_limit` | 0 (off) | Reject creations while CPU pressure (`/proc/pressure/cpu` some avg10, percent) is above this. A burst of creations raises it too, so set it from the node's own measurements, e.g. 50 | | `paused_mem_charge` | 1 | Share of their memory that paused sandboxes whose pause frees memory (microVMs; containers with `pause_reclaim`) still count for admission, 0 to 1 | | `persist_state` | on with a real backend | Write sandbox records to disk and re-adopt the sandboxes after an edge restart | | `keep_sandboxes_on_exit` | false | Leave sandboxes running when the edge exits. Needed for rolling upgrades | | `stopped_retention` | 600 | How long records of ended sandboxes are kept | | `state_dir` | `/var/lib/opendsec` | Sandbox disks, snapshots and records; should be local NVMe. XFS allows reflink copies | | `agent_dir` | next to the edge binary | Directory of `opendsec-aether` and `opendsec-chronus` | | `images` | — | Local static images: name → overlay lower directories (topmost first). Not needed with an image store | | `network` | off | Sandbox networking; below | | `gpu` | — | GPU devices; below | | `backends` | `{"sim": {...}}` | The backends this edge offers; below | ### edges[].network | Field | Default | Meaning | |---|---|---| | `enabled` | false | Enable sandbox networking. Off, sandboxes only have a loopback interface | | `subnet` | `10.88.0.0/16` | Sandbox address pool, a /30 per sandbox | | `dns_port` | 15353 | The edge's DNS proxy port on the host | Edges sharing a machine need distinct `subnet`s and `dns_port`s. ### edges[].gpu | Field | Meaning | |---|---| | `devices` | GPU device nodes, e.g. `["/dev/nvidia0", "/dev/nvidia1"]`. `capacity.gpus` defaults to their number | | `extra` | Other devices every GPU sandbox needs, e.g. `/dev/nvidiactl`, `/dev/nvidia-uvm` | | `mounts` | Host files or directories bound read-only into the sandbox (driver libraries, `nvidia-smi`) | | `shared_slots` | Sandboxes per device with `gpu_mode: shared`, default 4 | | `pci` | PCI devices passed through to `fullvm` sandboxes with VFIO, e.g. `["0000:3b:00.0"]` | ### edges[].backends Backend name → backend config. The name is what creations put in `backend`, e.g.: ```json "backends": { "microvm": {"runtime": "firecracker", "max_sandboxes": 4000, "default_disk_mb": 2048, "kernel": "/opt/opendsec/current/vm/vmlinux-6.1.167-opendsec", "firecracker": "/opt/opendsec/current/bin/firecracker", "jailer": "/opt/opendsec/current/bin/jailer", "image_pmem": true, "erofs_images": true, "free_page_reporting": true, "cold_age_s": 120}, "container": {"runtime": "linux", "max_sandboxes": 8000, "default_disk_mb": 1024}, "fncall": {"runtime": "linux", "max_sandboxes": 4000, "default_disk_mb": 1024, "warm_pool": {"root/py:3.12": 64}} } ``` Common fields: | Field | Default | Meaning | |---|---|---| | `runtime` | `sim` | `linux` (containers), `firecracker` (microVMs), `qemu` (full VMs), `sim` (simulated) | | `max_sandboxes` | — | Most sandboxes of this backend running at once on the node | | `default_disk_mb` | 10240 | Writable disk size when a request sets no `disk_mb` | | `disk_io` | unlimited | Per-sandbox disk I/O limits: `{"read_mbps", "write_mbps", "read_iops", "write_iops"}`, 0 for unlimited | | `core_sched` | on if the kernel supports it | A core scheduling cookie per sandbox: two sandboxes never run on the two hardware threads of one core at the same time, which closes SMT side channels and keeps best-effort load off a latency-sensitive sandbox's core. `true` requires kernel support; `false` turns it off | `linux` runtime (container, fncall): | Field | Default | Meaning | |---|---|---| | `pids_max` | 4096 | Process limit per sandbox | | `pause_reclaim` | false | A pause pushes the container's memory to swap, read back ahead of use on resume. Needs host swap and `vm.swappiness` above 0 | | `warm_pool` | — | Image → containers kept started ahead of requests (the fncall backend). A request without snapshot, user, working directory, path restrictions or GPUs, and with the default disk, takes one and is ready in milliseconds | | `layer_cache` | true | Containers run on node-local copies of their image layers, copied on first use, so running containers are unaffected when the image store (NFS) is unreachable | `firecracker` runtime (microvm): | Field | Default | Meaning | |---|---|---| | `kernel` | required | Guest kernel (vmlinux) path | | `firecracker`, `jailer` | the programs of that name on PATH | Paths of Firecracker and jailer | | `uid_base` | 1000000000 | Each VM runs as its own uid from `[uid_base, uid_base+65536)` | | `vm_overhead_mb` | 64 | Added to the guest memory for the VM's cgroup limit (the VMM itself, page tables, device buffers) | | `erofs_images` | false | Images become EROFS disks built once in the image store and copied to each node, instead of flattened to ext4 on every node. Needs a guest kernel with EROFS and `mkfs.erofs` (erofs-utils) on the node | | `image_pmem` | false | Attach the image as virtio-pmem with DAX: VMs of one image share the host page cache | | `free_page_reporting` | false | Memory the guest frees goes back to the host (virtio-balloon) | | `cold_age_s` | 0 (off) | Above 0, DAMON in the guest pages out memory idle this long to compressed zram | | `zram_pct` | 100 | zram size, percent of the VM's memory (0 to 400) | | `cold_boot_dirty_tracking` | true | Cold-booted VMs track dirty pages, so a pause writes only pages they touched. `false`: faster memory writes (host huge pages), but the first pause or snapshot writes all memory | `image_pmem`, `free_page_reporting` and `cold_age_s` need OpenDsec's guest kernel (`deploy/guest-kernel`, published as a GitHub release). `qemu` runtime (fullvm): | Field | Default | Meaning | |---|---|---| | `qemu` | `qemu-system-x86_64` on PATH | QEMU path | | `kernel` | — | Guest kernel | | `full_init` | false | Hand PID 1 to the image's own `/sbin/init` (aether runs beside it) | `sim` runtime: `create_latency_ms` (median creation time, lognormal), `latency_sigma`, `failure_rate`, for scheduling benchmarks. ## fleet The cluster management service; optional. | Field | Default | Meaning | |---|---|---| | `replicas` | — | Each `{"id", "addr", "data_dir"}`; `addr` e.g. `10.0.0.1:7300`, `data_dir` default `/var/lib/opendsec/fleet` | | `heartbeat_s` | 10 | Node agent report interval | | `lock_ttl` | 10 | Lease time of the leader lock | | `max_downloads` | 64 | Concurrent release downloads served per replica | | `keep_releases` | 10 | Releases kept besides those in use | | `edge_port` | 7201 | Port edges listen on at the node's private address | | `ready_ports` | iam 9461, watcher 9462, apiserver 9463, edge 9464, fleet 9465, e2b 9467 | Each role's `/readyz` port, which node agents check after restarting a unit | | `prometheus_url` | — | Prometheus for rollout gates | | `gates` | — | Rollout gates: `[{"name", "query", "max"}]`. While a batch bakes, each PromQL query must stay at or below `max`, or the rollout pauses | ```json "fleet": { "replicas": [{"id": "0", "addr": "10.0.0.1:7300"}, {"id": "1", "addr": "10.0.0.2:7300"}, {"id": "2", "addr": "10.0.0.3:7300"}], "heartbeat_s": 10, "prometheus_url": "http://10.0.0.3:9090", "gates": [{"name": "create errors", "query": "sum(rate(opendsec_apiserver_create_seconds_count{reason!=\"ok\"}[5m]))", "max": 1}] } ``` ## e2b The E2B-compatible gateway; optional. | Field | Default | Meaning | |---|---|---| | `gateways` | — | Gateway instances: `[{"addr", "tls_cert", "tls_key"}]`. `addr` e.g. `10.0.0.1:7400`; set both `tls_cert` and `tls_key` for the gateway to serve HTTPS itself, or neither for plain HTTP behind a TLS-terminating load balancer | | `project` | required with gateways | Sandboxes are created in this project. A request can name another with the `X-OpenDsec-Project` header | | `domain` | — | The sandboxes' domain returned to SDKs, which then reach `-.` (needs wildcard DNS and a certificate pointing at the gateways). Unset, clients set `E2B_SANDBOX_URL` | | `templates` | — | E2B template id → sandbox. SDKs default to `base`. A template id containing `/` is taken as an image name with `default_template`'s other settings | | `default_template` | — | As above, the defaults | | `max_timeout_s` | 0 (admission cap only) | Cap on a sandbox's timeout | Template fields: | Field | Default | Meaning | |---|---|---| | `image` | required | Image | | `backend` | `microvm` | Backend | | `cpu` | 2 | CPU cores | | `memory_mb` | 512 | Memory. Must equal the snapshot's when the image is a memory snapshot | | `disk_mb` | the backend's | Writable disk | | `start_cmd` | — | Started with `bash -l -c` in every new sandbox (an E2B template's start command) | | `ready_port` | — | Wait for this port to listen before handing the sandbox out | | `ready_timeout_s` | 60 | Longest wait | More in [Migrating from E2B](/en/docs/e2b). ## Command-line flags of the components Usually passed by the systemd units (`deploy/systemd/`); you rarely set them yourself. | Program | Flags | |---|---| | `opendsec-iam` | `-config`, `-replica ` (which replica to run, replicated mode), `-join` (a new replica: do not bootstrap; wait for `opendsecctl iam member add`), `-metrics`, `-log` | | `opendsec-watcher` | `-config`, `-replica `, `-metrics`, `-log` | | `opendsec-apiserver` | `-config`, `-index ` (run entry N of `apiserver`), `-drain-delay` (on SIGTERM, fail `/readyz` and keep serving this long, default 6s), `-grace` (then give calls in flight this long, default 30s), `-metrics`, `-pprof`, `-log` | | `opendsec-edge` | `-config`, `-id [,...]` (`all` for every configured edge; unlisted ids use `edge_template`), `-addr`, `-advertise`, `-metrics`, `-pprof`, `-log` | | `opendsec-fleet` | `-config`, `-replica `, `-metrics`, `-log` | | `opendsec-e2b` | `-config`, `-index `, `-ca` (default `tls.dir/ca.crt`), `-drain-delay` (default 5s), `-metrics`, `-log` | | `opendsec-nodeagent` | `-config` (default `/etc/opendsec/opendsec.json`), `-name`, `-addr`, `-fleet`, `-root` (default `/opt/opendsec`), `-check`, `-metrics`, `-log` | `-metrics
` serves `/metrics`, `/healthz` and `/readyz`. Default ports per role: [Monitoring and alerts](/en/docs/monitoring). --- # OpenDsec operations manual This manual covers installing a multi-node production cluster, day-to-day operations and handling failures. The design background is in SPEC §27 (Chinese), the components in `deploy/README.md`. To try OpenDsec on one machine, see `deploy/quickstart.sh` ([Single-node install](/en/docs/install)). The step-by-step first deployment is in [Cluster deployment](/en/docs/cluster). ## 1. Topology Using the example inventory `deploy/prod/inventory.example`: | Node | Components | Notes | |---|---|---| | ctl-1, ctl-2, ctl-3 | one replica each of iam, watcher, apiserver and fleet; ctl-3 also runs Prometheus and Alertmanager | control-plane nodes, one per failure domain, 4 cores and 8 GB to start, no sandboxes | | node-a1, node-b1, node-c1 … | edge, node agent | compute nodes: bare metal or cloud VMs with nested virtualization, running sandboxes | - The cluster keeps creating and serving sandboxes with any one control-plane node down. When a compute node fails, its sandboxes are lost and new ones go to other nodes. - In a small cluster, control-plane components and edges can share machines (several roles on one inventory line). - Shared storage (a NAS) is mounted at `/mnt/opendsec-images` on every apiserver and edge node, holding images and snapshots. - An internal load balancer (LB below) listens on port 7100 (layer-4 TCP) in front of every apiserver. Clients use only the LB's address. ### Ports (open only inside the VPC) | Port | Use | Who connects | |---|---|---| | 7100 | apiserver public API (TLS) | LB, ops machine | | 7001 / 7011 | IAM gRPC / Raft replication | every component / IAM replicas | | 7002 | watcher | apiserver, edge | | 7201 | edge | watcher, apiserver, other edges, fleet | | 7300 | fleet (cluster management, §6) | node agents, apiserver, other fleet replicas | | 7400 | E2B gateway (optional, §4) | callers using the E2B SDK, LB | | 9461–9467 | metrics, `/healthz`, `/readyz` (iam, watcher, apiserver, edge, fleet, node agent, E2B gateway) | Prometheus, LB health checks (9463; E2B gateway 9467), node agents (local) | | 9100 | node_exporter | Prometheus | | 9090 / 9093 | Prometheus / Alertmanager | operators | ### Infrastructure requirements - **NAS**: NFS 4.1 or newer, with root squash turned off (on the NAS, "no root restriction" in the permission group). Edges write snapshots as root and apiservers delete them. - Once mounted, and before moving anything onto it, run `opendsecctl image store-check -store /mnt/` (no cluster needed); every required item must be ok. Missing `trusted.*` xattr support is fine (SPEC §29.5). - Mount options: `vers=4.2,nconnect=8,hard,timeo=600,lookupcache=positive,_netdev`. 4.1 works too, but cannot find the holes of sparse files: creating a sandbox from a memory snapshot on another node then reads the disk and memory at full size (more than twice as slow in our tests; `store-check` warns about it). `lookupcache=positive` stops the client from caching "file not found": without it, a node that looked for an image before it was registered could keep missing it for up to a minute (SPEC §30). - fstab line: `:/ /mnt/opendsec-images nfs4 vers=4.2,nconnect=8,hard,timeo=600,lookupcache=positive,_netdev 0 0`. - Each node needs a distinct NFS client identity. Without one, an NFSv4 client identifies itself by host name, and nodes with the same host name (cloned from one image, say) are taken by the server for one client and keep invalidating each other's open files and locks (SPEC §33.4). `deploy/node/install.sh` sets the identity from `/etc/machine-id` (`/etc/modprobe.d/opendsec-nfs.conf`); a new identity takes effect only on a fresh mount with no users of the old one, most simply after a reboot. To check: on the NFS server, the `name` in `/proc/fs/nfsd/clients/*/info` should differ for every node; or watch the server's syslog for `rpc.mountd` printing `client attached/detached` over and over. - **LB** (a cloud layer-4 load balancer, or HAProxy / LVS of your own): - Listen on TCP 7100; no layer-7 HTTP/gRPC listener: the apiservers terminate TLS themselves. - HTTP health check on port 9463, path `/readyz`, every 2 seconds, unhealthy after 2 failures. - Idle timeout of at least 30 seconds: the apiservers ping idle connections every 20 seconds, so quiet streams (batch creations, long commands) are not cut by the LB. - Keep the client's source address if the LB can; otherwise rate-limited authentication failures all count against the LB (see `deploy/README.md`, "Admission and rate limits"). - The apiserver certificates must contain the LB's address: run `opendsecops certs `, then `opendsecops push-config all` and `opendsecops restart`. **Data centers without internet access**: use the offline bundle (`deploy/offline/README.md`, SPEC §38). The steps are the same as below, except that packages come from `OPENDSEC_DEBS`, the release from `opendsecops release-add`, and images from files. ## 2. The ops machine and opendsecops The ops machine can be any machine that reaches every node over ssh (a control-plane node too). Its directory is `/root/opendsec-ops` by default (`OPENDSEC_OPS` changes it); the ssh key defaults to `/root/.ssh/opendsec-ops` (`OPENDSEC_SSH_KEY`). Contents: ``` inventory nodes and roles (see deploy/prod/inventory.example) opendsec.json cluster config (see deploy/prod/cluster.example.json) admin.token admin token, mode 0600 pki/ the cluster CA (ca.key lives only here) releases/ packaged releases backups/ IAM backups current -> releases/ ``` `deploy/prod/opendsecops help` lists every command. Common ones: | Command | Effect | |---|---| | `opendsecops release bin/` | Package what `make build` produced into a release (version from `opendsecctl version`) | | `opendsecops certs [LB address]` | Create the CA and certificates; with an LB address, reissue the apiserver certificates | | `opendsecops install all` | First install: users and directories, release, config, certificates, systemd units, node preparation. Starts nothing | | `opendsecops start all` | Start in the order IAM → watcher → apiserver → edge and wait until all are ready | | `opendsecops status` | Check every component's `/readyz` and show the current IAM leader | | `opendsecops versions` | The version each component runs | | `opendsecops upgrade ` | Rolling upgrade | | `opendsecops rollback ` | Roll back: the same procedure onto an older release | | `opendsecops push-config all` + `opendsecops restart` | After changing the config or certificates: deliver, then rolling restart | | `opendsecops backup` | One IAM backup onto the ops machine, now | | `opendsecops rotate-ca 1\|2\|3` | Rotate the cluster CA without downtime (§4, "Certificates") | | `opendsecops monitor [webhook]` | Install node_exporter, Prometheus and Alertmanager, with alerts to the webhook | Each node receives only `ca.crt` and its own components' certificates and keys. `admin.token` goes only to IAM nodes, which need it at startup; the cluster config names `admin_token_file`, never the token itself. ## 3. First install ```sh make build && deploy/prod/opendsecops release bin/ # edit $OPENDSEC_OPS/inventory and opendsec.json, write admin.token (dsk.admin.<32 random hex digits>) deploy/prod/opendsecops certs deploy/prod/opendsecops install all # mount the NAS at /mnt/opendsec-images on each node (fstab) deploy/prod/opendsecops start all deploy/prod/opendsecops monitor deploy/prod/opendsecops status ``` Moving from a single IAM (an `iam.state` file) to three replicas: 1. On the old cluster: `opendsecctl iam backup -out iam.json.gz`. 2. Stop the old services and `install` and `start` with the new config. The new IAM bootstraps with the same admin token. 3. `opendsecctl iam restore iam.json.gz`. 4. Quota leases are not in backups. Sandboxes already on the edges are registered with IAM again at the next reconciliation (`reconcile_interval`, default 30 s). ## 4. Day-to-day operations ### Upgrades ```sh make build && deploy/prod/opendsecops release bin/ deploy/prod/opendsecops upgrade ``` - The sequence: 1. deliver the new release to every node and check its sha256; 2. restart components one by one, each only after the previous one's `/readyz` returns 200; 3. IAM followers first, the leader last; 4. then watchers, apiservers and edges. After each edge restarts, wait until it is healthy again in the cluster snapshot. - If any step times out (`OPENDSEC_READY_TIMEOUT`, default 180 s), it stops there. Components already upgraded keep the new version; to go back, `opendsecops rollback `. - On SIGTERM, an apiserver first fails `/readyz` and keeps serving 6 more seconds while the LB takes it out, then gives calls in flight up to 30 seconds. The SDK retries broken calls and resumes streams (SPEC §26). - Edge restarts do not touch sandboxes, provided `keep_sandboxes_on_exit` and `persist_state` are set and the unit uses `KillMode=process`. The new edge process re-adopts them. - Each node keeps the last 3 releases (`OPENDSEC_KEEP_RELEASES`). Running sandboxes keep the agents of the release they started with. ### Maintaining a machine ```sh opendsecctl edge drain node-a1 # stop placing new sandboxes on node-a1; existing ones keep running # wait for its sandboxes to end; paused ones can move (SPEC §26.2) systemctl stop opendsec-edge # on node-a1; on a control-plane node, opendsec-{apiserver,watcher,fleet,iam}@ # maintain..., then start the services again and run: opendsecctl edge drain node-a1 -undo ``` Maintain only one IAM node at a time, or the majority is lost. Rebooting a machine follows the same steps (drain → `systemctl reboot` → checks → `drain -undo`). After the reboot, check: `/data`, `/mnt/opendsec-images` and swap are mounted, `sysctl vm.swappiness` is 10, all four `/readyz` return 200, and the edge is healthy again in `opendsecctl status`. Notes: - A machine reboot loses all its sandboxes, paused microVMs included: when the edge re-adopts one and finds its network namespace gone, it removes it with its snapshot, and callers then get not_found. - During the reboot, Prometheus fires `OpenDsecComponentDown`, `OpenDsecEdgeUnhealthy` and the like; Alertmanager needs a receiver (`opendsecops monitor `), or the alerts reach nobody. - A measured reboot (bare metal, about 107 s of firmware self-test) is in SPEC §27. ### Certificates - Component certificates are valid 825 days, but never past the CA's expiry; the CA is valid 10 years. `OpenDsecCertExpiringSoon` fires 30 days before expiry. - **Renewing component certificates**: on the ops machine, delete the `.crt` and `.key` concerned, run `opendsecops certs` to reissue them (the LB addresses are kept), then `opendsecops push-config all` and `opendsecops restart`. - **Rotating the CA without downtime**: three steps, each with a config push and a rolling restart. 1. `opendsecops rotate-ca 1`: create a new CA; every node trusts both the old and the new CA, component certificates unchanged for now. 2. `opendsecops rotate-ca 2`: reissue every component certificate from the new CA. 3. `opendsecops rotate-ca 3`: drop the old CA. After step 1, give every client `$OPENDSEC_OPS/pki/ca.crt` (both CAs); after step 3, again (the new CA only). - Why three steps: until restarted, components use their old certificates and trust lists. Reissuing straight from a new CA would make restarted and not-yet-restarted components reject each other. ### Adding edges With fleet (§6), add machines with `opendsecops add-node` and `opendsecctl fleet node accept` (§6.3). Before fleet: 1. Add an entry to `edges` in `opendsec.json` and a line to the inventory. 2. `opendsecops certs` (issues the new edge's certificate). 3. `opendsecops install `, `opendsecops push-config all`, `opendsecops restart`, `opendsecops start `, in that order. ### Onboarding a tenant ```sh export OPENDSEC_ADDR=:7100 OPENDSEC_TOKEN=$(cat $OPENDSEC_OPS/admin.token) OPENDSEC_CA=$OPENDSEC_OPS/pki/ca.crt opendsecctl project create root -quota sandboxes= opendsecctl principal create root/ -kind agent # the token is shown only in this output opendsecctl grant root/ sandbox.create,sandbox.operate opendsecctl grant root layer.use # may use the images under root/ ``` - A caller's onboarding package: the [caller guide](/en/docs/user-guide) (with the address and project filled in), the SDK wheel, `ca.crt`, `sdk/python/examples/quickstart.py` and the token. Send the token separately over a secure channel. - Tokens last at most 90 days (`max_token_ttl_s`). Before expiry, issue a new one with `opendsecctl token create `, and once the caller has switched, revoke the old one with `opendsecctl token revoke`. ### Usage (SPEC §41) ```sh opendsecctl usage root/ -from 2026-10-01 -to 2026-10-31 # each project's total over the period opendsecctl usage root -from 2026-10-01 -daily -csv > usage.csv # per day and project, CSV ``` - Covers the project and all its subprojects: sandboxes started, sandbox-hours, CPU core-hours, memory GB-hours, GPU-hours. Per UTC day; each row counts the project's own sandboxes only, subprojects get rows of their own, and the last row is the total. - What is counted is quota leases: all the time a sandbox holds resources, paused time included (the capacity is still its own). Running sandboxes count up to the time of the query. - Needs `project.admin` on the project. Kept 400 days, in IAM, replicated with Raft, so node restarts and failovers lose nothing. - Python SDK: `await client.get_usage("root/", start="2026-10-01", end="2026-10-31")`. ### Entry points - **With an LB**: the SDK uses `OPENDSEC_ADDR=:7100`. - **Without an LB**: the SDK uses every apiserver's address, e.g. `OPENDSEC_ADDR=10.0.0.11:7100,10.0.0.12:7100,10.0.0.13:7100`, round-robins and fails over by itself (SPEC §27.8). - **Open to the internet**: allow only TCP 7100, and only from the callers' source addresses; the apiserver certificates must contain the public address or domain (`opendsecops certs`). Never expose any other port: IAM, watcher, edge and fleet ports, 9461–9466, 9100, 9090, 9093. - **Moving from direct addresses to an LB**: 1. `opendsecops certs api.opendsec.internal,,...`; 2. `opendsecops push-config all` and `opendsecops restart`; 3. tell callers to switch `OPENDSEC_ADDR` to the LB; 4. once the old addresses see no traffic, tighten the security group. ### E2B gateway (optional, SPEC §37) Lets callers using the E2B SDK connect directly. 1. Add an `e2b` section to the cluster config and deliver it through fleet (§6.2): ```json "e2b": { "gateways": [{"addr": "10.0.0.11:7400"}, {"addr": "10.0.0.12:7400"}], "project": "root/e2b", "templates": {"base": {"image": "root/py:3.12", "backend": "microvm", "cpu": 2, "memory_mb": 1024}}, "max_timeout_s": 86400 } ``` 2. Give machines the role, one at a time: `opendsecctl fleet node set -roles ,e2b=` (the index is the position in `gateways`), then `opendsecctl fleet node sync `. The node agent brings the machine to its desired release and config and enables and starts `opendsec-e2b@`; once it is ready, do the next machine. 3. Put an LB in front: forward TCP 7400, health check HTTP 9467 `/readyz`, idle timeout at least 60 s (the SDK's streams send keepalives every 50 s). For HTTPS from outside, terminate TLS on the LB, or set `tls_cert` and `tls_key` in `gateways`. 4. Callers set `E2B_API_URL` and `E2B_SANDBOX_URL` to the gateway and `E2B_API_KEY` to their OpenDsec token. The token needs `sandbox.create` and `sandbox.operate` on `project` and `layer.use` on the images' project; for snapshots (`create_snapshot()`), also `layer.publish` on `project`, where snapshots are stored as images. The gateway is stateless and calls the apiserver with each caller's own token, so permissions, quotas and audit are exactly as with the OpenDsec SDK. **E2B code interpreter** (the `e2b_code_interpreter` package) needs a template of its own: 1. Download E2B's code-interpreter repository and build the image with `deploy/e2b/build-code-interpreter.py`. It creates a microVM, installs Jupyter and E2B's services, and takes a snapshot with memory, `root/code-interpreter:v1`. It needs internet access (pip, apt) and `layer.publish` on `root`. 2. Add `"code-interpreter-v1": {"image": "root/code-interpreter:v1", "backend": "microvm", "cpu": 2, "memory_mb": 2048}` to `e2b.templates`. `cpu` and `memory_mb` must match the snapshot. 3. Deliver the config as in §6.2. Check it with `test/e2e/testdata/e2b_code_interpreter_check.py`. ## 5. Failures ### One IAM node down - The other two replicas keep serving. `OpenDsecIAMQuorumAtRisk` fires: **a second node must not fail now**. - If the machine comes back: start `opendsec-iam@`, and it catches up by itself. - If it does not (a new machine with a new address), replacing replica 2 with a new replica id 3: ```sh opendsecctl iam member remove 2 # opendsec.json: change the replicas entry with id 2 to id 3 at the new address; change the inventory to match opendsecops certs && opendsecops install echo 'OPENDSEC_IAM_FLAGS=-join' > /etc/opendsec/iam.env # on the new node opendsecops start # does not bootstrap; waits to be added. /readyz fails until then, so this step times out, as expected opendsecctl iam member add 3 -raft-addr :7011 -iam-addr :7001 rm /etc/opendsec/iam.env # after joining, the data directory holds Raft state and -join no longer matters opendsecops push-config all && opendsecops restart # so the other components use the new address list ``` - If the machine keeps its address but got a new disk: start it with `-join`. It is still a member, and the leader sends it a snapshot. ### All of IAM lost (disaster recovery) 1. Empty `/var/lib/opendsec/iam` on every replica and start them. The new cluster bootstraps with the admin token. 2. `opendsecctl iam restore `. Backups are in `/var/backups/opendsec` on every IAM node, and in `backups/` on the ops machine. 3. After one reconciliation, sandboxes alive on the edges are registered in the quotas again. 4. Principals, tokens and projects created after the backup are lost and must be recreated. ### One compute node down - The other nodes keep serving. - Sandboxes on the failed machine are lost. After `lost_edge_grace` (120 s), the watcher releases their quota. - When the machine comes back, the edge re-adopts sandboxes still running (unless the whole machine rebooted). After a reboot nothing can be re-adopted, and paused microVMs are removed too (see "Maintaining a machine"). ### NAS unavailable - Running microVMs are unaffected: they run from local copies of their images and the snapshot cache. - Running containers are unaffected too: they use node-local copies of their layers (`layer_cache`, on by default). With `layer_cache` off, a container's read-only layers come straight from the store, and reading files not in the page cache hangs until the NAS is back (SPEC §33: a 60-second NFS outage held a command for 88 seconds). - Operations that need the store wait until the NAS returns or their calls time out: new creations (creations waited up to 94 seconds during an outage in our tests, and all succeeded afterwards), pack, snapshot, cross-node fork. - Alert `NodeImageStoreMissing`. NFS is mounted `hard`, so I/O that was stuck completes once the NAS is back. ### System disk - Audit logs are in `audit_dir` (default `/var/log/opendsec/audit`), at most `audit_keep` files per apiserver (default 8 of 256 MB each). Ship them elsewhere regularly if they must be kept long. - Every sandbox produces some kernel and networkd messages. `deploy/node/install.sh` keeps them in the journal only (which by default uses at most 10% of its file system) and turns off networkd-dispatcher, which has no scripts. Without that, `/var/log/syslog` grows by more than 1 GB a day under load (SPEC §33.2). ### Alerts and what to check | Alert | Check first | |---|---| | OpenDsecComponentDown / NotReady | `opendsecops status`; on the node, `journalctl -u `. `curl :/readyz` returns why it is not ready | | OpenDsecIAMNoLeader | Are fewer than 2 replicas up; is port 7011 open between them | | OpenDsecEdgeUnhealthy | Is the machine up, is `opendsec-edge` running | | OpenDsecCreateErrors / CreateSlow | `opendsec_apiserver_create_seconds` by `reason` for the error mix; the edges' `opendsec_edge_create_phase_seconds` | | OpenDsecCapacityLow / EdgeUnderPressure | Add edges, or lower project quotas | | OpenDsecCertExpiringSoon | Renew as in §4, "Certificates" | | OpenDsecIAMBackupStale | `systemctl status opendsec-iam-backup`; look at `/var/backups/opendsec` | | OpenDsecSLOCreateBurnFast / BurnSlow, SLORuntimeBurnFast / BurnSlow | The error budget burns too fast ([SLO](/en/docs/slo)): as for CreateErrors, break down by `reason` and `method` to see which errors and since when; was there a release, a config change or a node failure at the time | Every alert: [Monitoring and alerts](/en/docs/monitoring). ## 6. The fleet service (SPEC §28) With many machines, upgrades and config changes are no longer pushed over ssh from the ops machine one by one: the fleet service advances them in batches, and a node agent on each machine carries them out. `opendsecops` stays for the first install, certificates, adding machines and emergencies. ### 6.1 Parts - **`opendsec-fleet@`**: three replicas on the control-plane nodes next to IAM, port 7300 (metrics 9465). The replica holding the IAM lock `fleet-controller` leads; the others forward requests to it. Desired state lives in IAM's key-value area and is backed up with IAM. - **`opendsec-nodeagent`**: one per machine, running as root (metrics 9466). It reports to fleet every 10 seconds and carries out what fleet asks: download a release, switch releases, write `/etc/opendsec/{opendsec.json,edge.env}`, restart `opendsec-*` services. It accepts these actions only. - The agent runs from `/opt/opendsec/agent/opendsec-nodeagent`, which does not switch with releases; a new release's agent replaces it only after passing a `-check` self-test. - With all of fleet down, node agents change nothing, and components and sandboxes keep running. - **CLI**: `opendsecctl fleet ...`, through an apiserver, needs `fleet.view` (read) or `fleet.operate` (change) on root; `project.admin` includes both. Every action is audited. With every apiserver down, connect to fleet directly from the ops machine with `-direct -fleet -pki $OPENDSEC_OPS/pki -ops-id ops`. ### 6.2 Releases and upgrades ```sh make build && deploy/prod/opendsecops release bin/ # package, with the firecracker, jailer and kernel pinned in deploy/deps.lock opendsecctl fleet release push $OPENDSEC_OPS/releases/ opendsecctl fleet rollout create -release # default: 1 canary, 5 minutes of bake, 5% per batch, 60 s between batches opendsecctl fleet rollout status [-steps] ``` - Order: every target machine downloads the release first; control-plane nodes upgrade one by one, the IAM leader's node last; then one edge as the canary, and after its bake, the edges in batches, with at most half of a batch from any one failure domain. - Each edge is drained before its upgrade; after it, its own `/readyz` and the watcher snapshot must be fine before scheduling resumes. Edge restarts leave existing sandboxes alone. - Before each batch: the machines already upgraded in this rollout are still fine, and every control-plane component is ready. With `fleet.prometheus_url` and `fleet.gates` set, those PromQL queries are checked during the bake too. - **When something fails**: the failed machine goes back to its old release by itself and is pinned, and the rollout pauses. Then: - `opendsecctl fleet rollout resume `: accept the failures and upgrade the remaining machines; - `opendsecctl fleet rollout rollback `: take the upgraded machines back to the old release in reverse order, in batches; - `opendsecctl fleet rollout abort `: stop where it is. Once the cause is found, unpin with `opendsecctl fleet node unpin `. - **Config only**: `opendsecctl fleet config put opendsec.json -note ...` creates a new config generation; then `opendsecctl fleet rollout create -config-gen `. The config may not contain `service_token` or `iam.admin_token`. - With fleet replicas, `opendsecops upgrade` and `restart` refuse to run; in an emergency, add `OPENDSEC_FORCE=1`. ### 6.3 Adding, maintaining and retiring machines ```sh deploy/prod/opendsecops add-node n17 edge=n17 # certificates, install, node agent opendsecctl fleet nodes -state pending # it shows up here once its agent reports opendsecctl fleet node accept n17 -roles edge=n17 -domain az2 # deliver the current release and config and start it ``` - New edges need no entry in `edges`: `edge_template` in the cluster config covers every unlisted edge. - Adding a role to an existing machine (e.g. `e2b`): `node set -roles ...`, then `node sync `, one machine at a time. Changing roles alone starts nothing; the machine is deployed at its next rollout or sync. fleet built before 2026-10-23 deadlocked rollouts when several machines got roles at once, each waiting for the other's new units (SPEC §37.7); use sync in that case. - Maintenance: `opendsecctl fleet node cordon ` stops scheduling there (fleet keeps it so), and `node uncordon` after maintenance; `node evacuate ` cordons and then waits for every sandbox on the machine to end. - Retiring: first `node evacuate `, and once its sandboxes are gone, `opendsecctl fleet node retire `: the node agent stops and disables the machine's components. fleet refuses to retire a machine not evacuated (not cordoned, or still with sandboxes); machines no longer sending heartbeats can be retired directly. Move control-plane roles off a machine first. ### 6.4 Moving from opendsecops to fleet (once, without downtime) 1. In the inventory, add `fleet=` to the control-plane nodes; in `opendsec.json`, add a `fleet` section (see `deploy/prod/cluster.example.json`), and point microvm's `firecracker`, `jailer` and `kernel` at `/opt/opendsec/current/...`. 2. `opendsecops certs`: issues the fleet, node agent and ops certificates. 3. One last rolling upgrade with opendsecops (to a release with fleet and the node agent): `opendsecops release bin/`, `opendsecops push-config all`, `OPENDSEC_FORCE=1 opendsecops upgrade `, then `opendsecops start all` to start the fleet replicas. 4. `opendsecops install-agent all`: start a node agent on every machine. 5. Register: `opendsecctl fleet release push $OPENDSEC_OPS/releases/`; `opendsecctl fleet config put $OPENDSEC_OPS/opendsec.json`; `opendsecctl fleet import -inventory $OPENDSEC_OPS/inventory -release -config-gen 1 -domain `. 6. `opendsecctl fleet nodes`: every machine reports, `RUNNING` matches `RELEASE`, and every `UNITS` is ready. --- # Monitoring and alerts Every OpenDsec component serves Prometheus metrics and health checks. `opendsecops monitor` sets up node_exporter, Prometheus, Alertmanager and the alert rules. This page lists every metric and every alert, and what to do about each alert. Service level objectives and how they are computed are in [SLO](/en/docs/slo). ## 1. Endpoints Every component is started with `-metrics
` (the systemd units do this) and serves there: | Path | Meaning | |---|---| | `/metrics` | Prometheus metrics | | `/healthz` | 200 while the process is alive | | `/readyz` | 200 while it can serve, else 503 with the reason (IAM without a leader, an apiserver without a cluster snapshot yet, shutting down, …) | | Role | Metrics port | Prometheus job | |---|---|---| | iam | 9461 | `opendsec-iam` | | watcher | 9462 | `opendsec-watcher` | | apiserver | 9463 | `opendsec-apiserver` | | edge | 9464 | `opendsec-edge` | | fleet | 9465 | `opendsec-fleet` | | node agent | 9466 | `opendsec-nodeagent` | | E2B gateway | 9467 | `opendsec-e2b` | | node_exporter | 9100 | `node` | Load balancer health checks use the apiserver's `9463/readyz` (the E2B gateway's `9467/readyz`). On SIGTERM, an apiserver fails `/readyz` first and keeps serving 6 more seconds, so the load balancer takes it out before it stops. ## 2. Setup ```sh deploy/prod/opendsecops monitor https://alerts.example.com/hook ``` - Installs node_exporter on every machine, and Prometheus (9090) and Alertmanager (9093) on the `monitor` node. - Scrape targets come from the inventory; the alert rules are `deploy/monitoring/alerts.yml`. - Alerts go out as webhooks (Alertmanager's webhook format). **Without a webhook, the receiver is reset to "send nothing"**: before running it again, look up the existing receiver in `/etc/prometheus/alertmanager.yml` and pass it along. - Run it again after adding machines or changing roles, to scrape the new ones. With a Prometheus of your own, add scrape targets on the ports above and load `deploy/monitoring/alerts.yml`. ## 3. Metrics ### Every component | Metric | Type | Labels | Meaning | |---|---|---|---| | `opendsec_build_info` | gauge | `version` | Always 1; the running version | | `opendsec_ready` | gauge | — | 1 while `/readyz` returns 200 | | `opendsec_rpc_server_handled_total` | counter | `service`, `method`, `reason` | RPCs served; `reason` is `ok` or the error reason (`quota_exceeded`, …) | | `opendsec_rpc_server_handling_seconds` | histogram | `service`, `method` | Time to serve an RPC (streams: until they end) | | `opendsec_tls_cert_expiry_timestamp_seconds` | gauge | `identity` | When the certificates the process loaded expire (Unix time); `ca` is the CA | | `opendsec_host_mounts` | gauge | — | Entries in the process's mount table | ### apiserver | Metric | Type | Labels | Meaning | |---|---|---|---| | `opendsec_apiserver_create_seconds` | histogram | `backend`, `reason` | Sandbox creation through the apiserver (placement, admission, edge), by outcome | | `opendsec_apiserver_placement_retries_total` | counter | `reason` | Creations retried on another node after an edge rejected them, by the edge's reason | | `opendsec_apiserver_fork_seconds` | histogram | `held` | Fork time from the snapshot to the last copy; `held`: whether the snapshot stayed on the source node | | `opendsec_apiserver_move_seconds` | histogram | `reason` | Paused sandboxes moved to another node | ### edge | Metric | Type | Labels | Meaning | |---|---|---|---| | `opendsec_edge_sandboxes` | gauge | `edge`, `backend`, `state` | Sandboxes per state | | `opendsec_edge_capacity` | gauge | `edge`, `resource` | Schedulable capacity (overcommit applied): `cpu_cores`, `memory_mb`, `gpus`, `disk_mb` | | `opendsec_edge_allocated` | gauge | `edge`, `resource` | Resources allocated to sandboxes | | `opendsec_edge_mem_available_mb` | gauge | `edge` | The node's MemAvailable | | `opendsec_edge_disk_available_mb` | gauge | `edge` | Free space on the state directory's file system | | `opendsec_edge_under_pressure` | gauge | `edge`, `pressure` | 1 while creations are rejected under memory, disk or CPU pressure | | `opendsec_edge_draining` | gauge | `edge` | 1 while draining (no new sandboxes) | | `opendsec_edge_moved_in` | gauge | `edge` | Sandboxes that moved here from another node | | `opendsec_edge_created_total` | counter | `edge` | Sandboxes created | | `opendsec_edge_rejected_total` | counter | `edge` | Creations rejected (no capacity, draining, pressure) | | `opendsec_edge_create_seconds` | histogram | `backend`, `reason` | Creation on the node (admission to running) | | `opendsec_edge_create_phase_seconds` | histogram | `backend`, `phase` | Creation phases (prepare, network, config, start, …), to find what is slow | | `opendsec_edge_op_seconds` | histogram | `backend`, `op`, `reason` | Pause, resume, stop, snapshot | | `opendsec_edge_runtime_info` | gauge | `edge`, `component`, `version` | Versions of the runtime programs the node uses (e.g. firecracker) | | `opendsec_snapshot_transfer_seconds` | histogram | `kind` | Snapshots moved between nodes: fetched from another edge, pulled from or published to the store | | `opendsec_snapshot_transfer_bytes_total` | counter | `kind` | Snapshot data received from other edges (holes and zero runs excluded) | ### IAM and watcher | Metric | Meaning | |---|---| | `opendsec_iam_leader` | 1 on the IAM replica that leads | | `opendsec_iam_leases` | Quota leases held (one per live sandbox) | | `opendsec_iam_backup_last_success_timestamp_seconds` | Last successful IAM backup (from the backup script, through node_exporter's textfile collector) | | `opendsec_watcher_edges_healthy` | Edges answering | | `opendsec_watcher_edges_unhealthy` | Edges not answering | | `opendsec_watcher_edges_draining` | Edges draining | ### fleet (on the leader only) | Metric | Labels | Meaning | |---|---|---| | `opendsec_fleet_leader` | `replica` | 1 on the leading replica | | `opendsec_fleet_nodes` | `state`, `domain` | Machines per state and failure domain | | `opendsec_fleet_node_release` | `release` | Machines per running release | | `opendsec_fleet_heartbeat_age_seconds` | — | Age of the oldest heartbeat among active machines | | `opendsec_fleet_rollout_active` | `state` | 1 while a rollout is in this state | | `opendsec_fleet_rollout_progress` | — | Share of the active rollout's steps done | ### Useful queries ```promql # creations per second, by outcome sum by (reason) (rate(opendsec_apiserver_create_seconds_count[5m])) # p99 creation time, by backend histogram_quantile(0.99, sum by (le, backend) (rate(opendsec_apiserver_create_seconds_bucket[5m]))) # server-side error ratio (caller errors such as quota and bad arguments excluded) sum(rate(opendsec_rpc_server_handled_total{reason=~"internal|unavailable|timeout"}[5m])) / sum(rate(opendsec_rpc_server_handled_total[5m])) # running sandboxes per node and backend sum by (edge, backend) (opendsec_edge_sandboxes{state="RUNNING"}) # share of memory allocated sum(opendsec_edge_allocated{resource="memory_mb"}) / sum(opendsec_edge_capacity{resource="memory_mb"}) # which creation phase is slow histogram_quantile(0.99, sum by (le, phase) (rate(opendsec_edge_create_phase_seconds_bucket{backend="microvm"}[10m]))) # versions running count by (version) (opendsec_build_info) ``` ## 4. Alerts The rules are in `deploy/monitoring/alerts.yml`, 25 of them. Severities: `critical` needs action now, `warning` the same day, `info` is worth a look. ### Control plane | Alert | Severity | For | Meaning | What to do | |---|---|---|---|---| | `OpenDsecComponentDown` | critical | 1m | Prometheus cannot scrape a component | `opendsecops status`; on that machine, `systemctl status ` and `journalctl -u ` | | `OpenDsecComponentNotReady` | critical | 2m | A component reports not ready | `curl :/readyz` returns the reason | | `OpenDsecIAMNoLeader` | critical | 30s | IAM has no leader: creations and authentication fail | Are fewer than 2 IAM replicas up? Is port 7011 open between them? | | `OpenDsecIAMQuorumAtRisk` | warning | 2m | Fewer than 3 IAM replicas up; one more failure loses the quorum | Bring the replica back soon; do not maintain another IAM node meanwhile | | `OpenDsecWatcherDegraded` | warning | 2m | Fewer than 2 watchers up | Bring the watcher back | | `OpenDsecAPIServerDegraded` | warning | 2m | Fewer than 2 apiservers ready | Bring the apiserver back; check the load balancer's health checks | | `OpenDsecCertExpiringSoon` | warning | 1h | A certificate expires within 30 days | Renew it: [operations manual §4, "Certificates"](/en/docs/operations) | | `OpenDsecIAMBackupStale` | warning | — | No successful IAM backup for 36 hours | `systemctl status opendsec-iam-backup`; check `/var/backups/opendsec` and disk space | | `OpenDsecVersionSkew` | info | 2h | More than one version running for over 2 hours | An upgrade that stopped halfway? `opendsecctl fleet rollout status` | ### Sandboxes | Alert | Severity | For | Meaning | What to do | |---|---|---|---|---| | `OpenDsecEdgeUnhealthy` | critical | 1m | Edges not answering the watchers; their quota is released after `lost_edge_grace` | Is the machine up and `opendsec-edge` running? | | `OpenDsecCreateErrors` | critical | 5m | Too many creations fail for server-side reasons | Break `opendsec_apiserver_create_seconds` down by `reason`; check edge logs | | `OpenDsecCreateSlow` | warning | 10m | p99 creation time of a backend is too high | Find the slow phase with `opendsec_edge_create_phase_seconds`; check the image store and node disks | | `OpenDsecEdgeUnderPressure` | warning | 5m | An edge rejects creations under memory, disk or CPU pressure | See the `pressure` label; free resources, add capacity or lower quotas | | `OpenDsecCapacityLow` | warning | 15m | Most of the cluster's sandbox memory is allocated | Add compute nodes or lower quotas | | `OpenDsecEdgeDiskLow` | warning | 10m | An edge's state file system has under 10% free | Clean up; look for many snapshots or large sandbox disks | | `NodeFilesystemAlmostFull` | warning | 10m | A file system on a node is almost full | Which mount point: logs, audit logs, state directory? | | `NodeImageStoreMissing` | critical | 5m | An edge node does not have the image store mounted | `mount /mnt/opendsec-images` on it; check the NAS | ### fleet | Alert | Severity | For | Meaning | What to do | |---|---|---|---|---| | `OpenDsecFleetNoLeader` | warning | 1m | No fleet replica leads: no heartbeats are taken and rollouts stand still (components keep running) | Check the fleet replicas and IAM | | `OpenDsecFleetNodeSilent` | warning | 2m | A node agent has not reported for a while | `opendsecctl fleet nodes` shows which; `systemctl status opendsec-nodeagent` | | `OpenDsecFleetRolloutPaused` | warning | 30m | A rollout has been paused for 30 minutes | `opendsecctl fleet rollout status -steps` for the reason, then resume, roll back or abort | | `OpenDsecFleetDrift` | info | 10m | Machines run different releases while no rollout is active | Pinned machines, or one out of sync: `fleet node sync ` | ### SLO | Alert | Severity | For | Meaning | |---|---|---|---| | `OpenDsecSLOCreateBurnFast` | critical | 2m | Creation errors burn the error budget 14.4x too fast: 2 days of this spend the 30-day budget | | `OpenDsecSLOCreateBurnSlow` | warning | 15m | 6x: 5 days spend it | | `OpenDsecSLORuntimeBurnFast` | critical | 2m | Runtime calls (commands, files, stop, pause, resume, fork), 14.4x | | `OpenDsecSLORuntimeBurnSlow` | warning | 15m | As above, 6x | What to do: break the errors down by `reason` and `method` and find when they started; was there a release, a config change or a node failure at that time? How the SLOs are computed: [SLO](/en/docs/slo). ## 5. Logs | What | Where | |---|---| | Component logs | journald: `journalctl -u opendsec-edge`, `journalctl -u 'opendsec-apiserver@*'`, … | | Warnings and errors only | `journalctl -u 'opendsec-*' -p warning --since -1h` | | Audit log | `audit_dir` on apiserver nodes (default `/var/log/opendsec/audit`), JSON lines, one per call: time, principal, method, project, sandbox, outcome, latency, source address. Commands are recorded as their length and SHA-256 only | | IAM backups | `/var/backups/opendsec` on IAM nodes | Log levels are set with each component's `-log` flag (`debug`, `info`, `warn`, `error`); the default is `info`. Under systemd each log line starts with its syslog priority (such as `<4>`), which journald records as the entry's priority, so `-p warning` finds the warnings and errors. Builds from before 2026-10-25 lack the prefix and log everything at the normal priority: use `journalctl -u 'opendsec-*' --since -1h | grep -E 'level=(WARN|ERROR)'`. --- # Troubleshooting Find the symptom, then the cause and the fix. Every entry comes from a problem we actually hit. If yours is not here, collect the information in §1 and [open an issue](https://github.com/Kernflux/opendsec/issues). ## 1. Collect this first ```sh opendsecctl version # client version deploy/prod/opendsecops status # is every component ready (ops machine) deploy/prod/opendsecops versions # versions each component runs opendsecctl status # cluster snapshot: each edge's health, capacity, pressure opendsecctl fleet nodes # with fleet: each machine's state journalctl -u 'opendsec-*' -p warning --since -1h # on the nodes: warnings and errors of the last hour curl -s :9463/readyz # the reason when not ready (ports per role: Monitoring and alerts) ``` When a call fails, note the error's `reason` and full message, the time and the sandbox id. The first part of a sandbox id is its node: look at `journalctl -u opendsec-edge` there. ## 2. Installation | Symptom | Cause | Fix | |---|---|---| | quickstart reports `[x] command make`, `[x] Go 1.27+` | A fresh machine without build tools | See [Single-node install §2](/en/docs/install) | | `apt install` says `has no installation candidate` | Fresh cloud images have no package lists | `apt update` first | | `python3 -m venv` says ensurepip is unavailable | `python3-venv` is missing | `apt install -y python3-venv` | | `pip install` says externally-managed-environment | Ubuntu 24.04 keeps pip out of the system Python | Use a virtual environment: `python3 -m venv .venv && .venv/bin/pip install ...` | | `could not fetch the guest kernel pinned in deploy/deps.lock` | GitHub releases are unreachable | `--kernel FILE`, `--build-kernel` or `--no-microvm`; for clusters, `OPENDSEC_KERNEL=` | | `[x] port 7100 is in use`, `already holds a cluster` | An earlier install | `deploy/quickstart.sh --uninstall`, then install again | | IAM fails to start; its log calls the admin token a placeholder | The config still has the example's `change-me` | Generate a real token (`dsk.admin.$(openssl rand -hex 16)`) | | Config validation says `zone ... must be "onprem" or "cloud"` | `edges` or `edge_template` has another `zone` | Only these two values are allowed | | edge fails to start: `erofs_images needs mkfs.erofs` | The node lacks erofs-utils | `apt install erofs-utils`; newer `opendsecops install` installs it | | `opendsecops release` cannot download the kernel | As above, GitHub unreachable | Set `OPENDSEC_KERNEL`, or put the file into `~/.cache/opendsec-deps/` | | ssh never returns (the ops machine running remote commands) | A background process on the node is still tied to the session | `ssh -f host 'setsid nohup cmd > log 2>&1 < /dev/null &'` | ## 3. Creating sandboxes | Error | Cause | Fix | |---|---|---| | `quota_exceeded: root/team-a: sandboxes 100+1 > 100` | The project's (or an ancestor's) quota is used up | Stop sandboxes you no longer need (find them with `list_sandboxes`), or ask for more quota. The message names the level and key | | `no_capacity` | Every candidate node is full, under memory, disk or CPU pressure, or draining | `opendsecctl status`: look at each node's `alloc`, `under_pressure`, `draining`; add capacity or retry later | | Every creation is `no_capacity` though the edges are healthy | An edge's `zone` is not `onprem` (older versions did not check) | Set it to `onprem` | | `invalid: ... ttl_max ... exceeds` | The requested lifetime is over `max_ttl_max_s` | Ask for less, or have the administrator raise the limit | | `invalid: backend "microvm" ...` / unknown backend | The cluster lacks that backend (nodes without KVM, say) | `opendsecctl status` shows each node's `backends` | | `not_found: image ...` | A typo in the image name, or the image was never imported | `opendsecctl image list -project root` | | An edge reports `image ... not found` for about a minute after the image was registered | The NFS client cached "file not found" | Add `lookupcache=positive` to the NFS mount options | | `forbidden: ... lacks sandbox.create on ...` | No permission to create in the project | The project admin grants it | | `forbidden: ... lacks layer.use on root` | No permission to use the image's project | `grant root layer.use` | | `invalid: memory ... snapshot` / creating from a snapshot fails | `memory_limit_mb` differs from the snapshot's | Use the snapshot's memory size | | Creating from a snapshot keeps waiting | The snapshot is still being published to the store (made moments ago, used on another node) | Wait for the publish; a slow store makes it take longer | | Only some nodes can use a snapshot | Snapshots go only to nodes with the same Firecracker version | Make snapshots again after upgrading Firecracker | | Batch creations through a load balancer fail with `Stream removed`; the load balancer logs `cD` | The load balancer's idle timeout cut a quiet stream | Set the idle timeout to 30 s or more; newer servers ping every 20 s | | `rate_limited` | Over the per-principal rate | Back off and retry; for batches use `run_sandboxes` (the server queues them and they never fail on the rate) | ## 4. Commands and files | Symptom | Cause | Fix | |---|---|---| | `invalid: sandbox ... is STOPPED` | The sandbox stopped: stopped by a caller, or it reached its idle timeout or lifetime | `get_sandbox` shows the `message` (e.g. `idle timeout (ttl_running_stop) reached`); raise the timeouts if you need longer | | `not_found` | The sandbox was cleaned up (10 minutes after it ended), or its node failed or rebooted | A node reboot loses its sandboxes; create new ones | | `ShellResult.timed_out` is True | The command ran past its `timeout` | Raise `timeout`, or use a background command and `get_output` | | Output incomplete, `truncated` is True | One call returns at most 16 MB per stream | Fetch the rest with `get_output(cmd_id, stdout_offset=...)` | | `dropped_bytes` above 0 | The output exceeded the 64 MB per-stream record limit | Write the output to a file and read it in pieces | | Writing or reading files over 4 MB fails with `quota_exceeded: grpc: received message larger than max` | In older code a message was limited to 4 MB (the first entry under "Fixed" in CHANGELOG 0.11.0 fixed it) | Upgrade; meanwhile read and write in pieces under 3 MB | | A command printing more than 4 MB fails as a whole | As above | As above; meanwhile redirect the output to a file and read it in pieces | | `opendsecctl exec $ID python3 -c "'print(1)'"` prints nothing | 0.11 changed how `exec` handles arguments, and the old workaround's extra quotes now reach Python | Write `opendsecctl exec $ID python3 -c 'print(1)'` | | Arguments of `exec` are taken as opendsecctl's options | The command has arguments starting with `-` | Put `--` before it: `opendsecctl exec $ID -- ls -la` | | A session of an unprivileged user cannot write system directories | The session runs as its `user` | Expected; run root work in another session | | Another `user` for the same session had no effect | The session's user was fixed by its first command | Use a new session name | ## 5. Network See [Network policy §6](/en/docs/network). The two most common: - A name does not resolve: the domain is not allowed (the DNS proxy answers "no such domain" when no rule matches). - An internal service times out: private addresses need an explicit address or network rule, and private addresses a domain resolves to are always blocked. ## 6. microVMs | Symptom | Cause | Fix | |---|---|---| | Nodes do not offer the microvm backend | No `/dev/kvm` | Bare metal, or nested virtualization | | Every resume after a pause fails (Ubuntu 24.04, kernel 6.8.0-1xx) | This kernel kills processes started later in a cgroup that has been through `cgroup.kill`. Fixed in the code of 2026-10-24 (CHANGELOG 0.11.0, "Fixed") | Upgrade OpenDsec | | VMs never come up; the log has `connect: invalid argument` | The state directory's path is too long: vsock's unix socket path exceeds 108 bytes | Use a shorter `state_dir` | | The node runs short of memory | VMs do not hand memory back to the host | Turn on `free_page_reporting`, `cold_age_s` and `image_pmem` (needs OpenDsec's guest kernel); lower `paused_mem_charge` so paused VMs count for less | | Creating from a memory snapshot on another node is slow | NFS 4.1 cannot find the holes of sparse files and reads the whole memory file | Mount NFS with `vers=4.2` (`umount -l`, then `mount`) | ## 7. Images | Symptom | Cause | Fix | |---|---|---| | `image import` fails halfway with `context deadline exceeded` | Older CLIs had a 5-minute overall limit | Upgrade; imports now have no overall limit | | `image import ./x.tar` tries to download from `registry-1.docker.io` | Older CLIs took a missing local file for an image name | Check the path; newer CLIs report the missing file | | `image import` fails on permissions | Not root, or the store has root squash on | Run as root on a node that mounts the store; turn root squash off on the NAS | | `store-check` fails a required item | The file system does not meet the requirements | Adjust the NAS (root squash, hard links, xattrs, …) for the failing item | | The image store keeps growing | Deleting an image only removes its registration | Run `opendsecctl image gc` regularly | | The NFS server logs `client attached/detached` over and over | Nodes share an NFS client identity (same host name) | `deploy/node/install.sh` sets the identity from machine-id; reboot the nodes for it to take effect | ## 8. Cluster and upgrades | Symptom | Cause | Fix | |---|---|---| | `too many colons in address` | opendsecctl before 0.11 does not take comma-separated addresses | Upgrade, or give a single address for now | | TLS says `not api.opendsec.internal` with several addresses | Older certificates lack that name | `opendsecops certs ` to reissue → `push-config all` → rolling restart | | A few calls fail with `unknown edge` during an upgrade | A restarted watcher does not yet know some edges. Fixed in the code of 2026-10-24; it can still happen a few times while upgrading from an older version | Retry; it stops once the upgrade is done | | `opendsecops upgrade` says `this cluster has fleet replicas` | The cluster is run by fleet | Use `opendsecctl fleet rollout create` | | A rollout paused with a machine pinned | The machine did not become ready after upgrading and was rolled back | `fleet rollout status -steps` and the machine's logs for the cause; after fixing it, `fleet node unpin`, then resume or roll back | | A rollout hangs after roles were added to two machines at once | fleet before 2026-10-23 made them wait for each other | Add roles one machine at a time: `fleet node set`, then `fleet node sync` | | A config change did not reach the E2B gateway | Older `opendsecops restart` skipped the gateway | Upgrade; or `systemctl restart opendsec-e2b@` on that machine | | The disk alert keeps firing on machines with small disks | Older alert rules used a fixed threshold | Use the new `alerts.yml` (alerts below 10% free); `opendsecops monitor` reloads it | | `OpenDsecEdgeUnhealthy` | The edge stopped, the machine is down, or the network is cut | Once back, the edge re-adopts sandboxes still running (unless the machine rebooted) | | All sandboxes are gone after a machine reboot | Sandboxes (paused ones too) do not survive a machine reboot | Expected; snapshot state you need to keep | ## 9. E2B SDK | Symptom | Cause | Fix | |---|---|---| | `401: ... API key is malformed: expected the "e2b_" prefix` | `E2B_API_URL` is not set, so the SDK talks to E2B's own service | Point `E2B_API_URL` and `E2B_SANDBOX_URL` at the gateway | | The JS SDK fails with a syntax error or does not start | Node is too old | E2B's JS SDK needs Node 22.19 or newer | | `pause(mode="filesystem")` does a full pause instead of failing | E2B's Python SDK 2.46 does not know `mode` and drops it silently | Use E2B SDK 2.59 or newer; OpenDsec refuses filesystem-only pauses | | E2B sandboxes cannot reach the internet | Gateways before 0.11 used the cluster's default network rules | Upgrade; gateways now allow the internet by default, as E2B does | | Building the code interpreter template fails | The build needs internet access and `layer.publish` | The newer build script allows internet for its build sandbox; check the permission | | `Sandbox.create(snapshot_id)` times out | Using a fresh snapshot on another node waits for it to reach the image store | Raise `timeout`, especially with a slow store | --- # Security model OpenDsec runs untrusted code: commands a model generates during RL, scripts an agent runs. This page explains what isolates sandboxes from each other, from their host and callers from each other, how traffic and secrets inside the cluster are protected, and a hardening checklist for production. To report a vulnerability, see `SECURITY.md` in the repository root; do not open a public issue. ## 1. Threat model | Assumption | Meaning | |---|---| | Code in sandboxes is fully untrusted | It may run as root, try to escape, scan networks, exhaust resources | | Callers do not trust each other | Callers of different projects cannot see or operate each other's sandboxes and images | | Cluster nodes and the private network are trusted | Components still use mTLS, but a compromised node is out of scope | | The ops machine is the most sensitive | It holds the CA's private key and the admin token | ## 2. Sandbox isolation ### microVM (recommended for untrusted code) - Each sandbox is a Firecracker virtual machine **with its own kernel**: root in the sandbox is root of the guest kernel only, and the attack surface is Firecracker's few virtual devices instead of every system call of the host kernel. - Firecracker is started by jailer: chrooted into its own directory and running as **a uid of its own per VM** (from `uid_base`, default one billion), so even a compromised VMM cannot touch other VMs' files. - Each VM is in its own cgroup, limited to the guest memory plus `vm_overhead_mb`. - Core scheduling (`core_sched`) is on by default: two sandboxes never run on the two hardware threads of one core at the same time, which closes SMT side channels. ### container - Linux namespaces (mount, pid, net, uts, ipc, cgroup) + overlayfs + cgroup v2, sharing the host kernel. - **Capabilities**: Docker's default set only, without `CAP_MKNOD` (without a device cgroup, a device node could reach the host's disks). - **seccomp**: these system calls return EPERM: mounting and namespace switching (`mount`, `umount2`, `pivot_root`, `setns`, the new mount API), kernel modules and kexec, `bpf`, keyrings, file handles (`open_by_handle_at`, the "shocker" escape), `userfaultfd`, `perf_event_open`, raw I/O ports, swap, quotas, reboot. `clone` and `unshare` with `CLONE_NEWUSER` are refused (a user namespace would give the dropped capabilities back); `clone3` returns ENOSYS so libc falls back to `clone`. - A process limit `pids_max` (default 4096); CPU, memory and disk I/O limited by cgroups. - Core scheduling on by default. **Containers share the host kernel**, so a kernel vulnerability can lead to an escape. Prefer microVMs for untrusted code. ### Both backends - **Read-only agents**: the agents in the sandbox (aether, chronus) are mounted read-only; code in the sandbox cannot change them. - **Session users**: commands can run in sessions of an unprivileged user (`user="1000"`). An unprivileged session cannot reach other sessions' channels, read their output records or change system files. With the agent's commands and the grader in different sessions, the agent cannot tamper with grading. - **Path restrictions** (API fields `readonly_paths`, `masked_paths`): make paths in the sandbox read-only, or hide them (an empty directory or file in their place). In containers root cannot undo them; in a microVM the guest's root can, so pair them with a non-root `init_user`. - **Resource limits**: CPU, memory, disk size and disk I/O (`disk_io`) are all bounded; a node short of memory or disk rejects new sandboxes to protect the running ones. - **Output limits**: at most 64 MB of each command's output is recorded, at most 512 MB per session, so output cannot fill a node's disk. ## 3. Network isolation See [Network policy](/en/docs/network). In short: - Outbound traffic is denied by default and allowed by rules, enforced in the host's nftables, which root in the sandbox cannot change. - Inbound connections are always refused: sandboxes cannot reach each other or services on the host (except DNS). - Packets with forged source addresses are dropped. - Private, link-local (cloud metadata `169.254.169.254` included) and multicast addresses are reachable only through network rules that name them; private addresses a domain resolves to are always blocked, so DNS cannot be used to get around the rules. - Taking an allowance back cuts existing connections at once. ## 4. Isolation between callers - Every call carries a token; IAM stores only token hashes. - Permissions are granted per project and inherited downwards; bounded delegation keeps admins from handing out more permissions or quota than they have ([Projects, principals and permissions](/en/docs/iam)). - Operating a sandbox needs `sandbox.operate` on the sandbox's project, so callers of one project can operate each other's sandboxes: **put callers who must be isolated from each other in different projects**. - Images are authorized by project: the principal running jobs can have `layer.use` only, so it cannot change the images training relies on. - Tokens can expire (the cluster can enforce a limit with `max_token_ttl_s`), and a revocation takes effect cluster-wide within about 1 s. - After too many failed authentications from one address, new tokens from it are not checked for a while, against brute force. ## 5. Inside the cluster - **mTLS**: all traffic between components is mutual TLS. Each component's certificate carries its role and id (e.g. `edge-node-a1`), and each internal service accepts only certain roles: an edge's sandbox operations only from apiservers, its status queries from watchers, apiservers and fleet, and snapshot fetches from other edges; IAM's management calls only from apiservers. - **Public API**: apiservers serve TLS, and clients verify it against the cluster CA. Apiserver certificates carry the load balancer's address and `api.opendsec.internal`. - **Where secrets live**: | Secret | Where | Not where | |---|---|---| | CA private key | only on the ops machine, `$OPENDSEC_OPS/pki` | any node | | Admin token | the ops machine; `/etc/opendsec/admin.token` on IAM nodes (IAM needs it to bootstrap) | the cluster config holds only `admin_token_file`; other nodes do not have it | | Component certificates and keys | only on the nodes running those components | other nodes | | Caller tokens | the callers | the server stores only hashes | - **Audit logs**: each apiserver records every call that changes something (creations, stops, pauses, forks, network changes, image publishing, projects, principals, permissions, tokens, cluster operations): time, principal, method, project, object, outcome, source address. Commands are recorded as their length and SHA-256, never their text. With `audit_runtime`, runtime calls and read-only queries are recorded too. - **fleet's node agent** runs as root. It connects to fleet itself (with its node certificate, checking fleet's certificate) and performs only a fixed set of actions: download and verify a release, switch releases, write the config, restart OpenDsec's services. A new node agent replaces the old one only after passing a self-check. ## 6. Hardening checklist Check each item when deploying a production cluster: - [ ] Only the load balancer's 7100 (and 7400 with the E2B SDK) is reachable by callers; every other port only from the private network. - [ ] Untrusted code runs on the `microvm` backend. - [ ] Default network rules (`network.default_rules`) are as narrow as possible; internal services are allowed by exact address and port only. - [ ] `iam.max_token_ttl_s` is set (e.g. 90 days) and tokens are rotated. - [ ] Callers are split into projects by team or purpose, with only the permissions they need; principals running jobs do not get `layer.publish`. - [ ] Audit logging is on (`audit_dir`) and the logs are shipped elsewhere regularly. - [ ] `$OPENDSEC_OPS` is accessible to operators only and has an offline backup (it holds the CA's private key). - [ ] `opendsecops monitor ` has an alert receiver; watch `OpenDsecCertExpiringSoon` and `OpenDsecIAMBackupStale`. - [ ] The development `host` image (which exposes the host's `/` to sandboxes) is not used in production, and `service_token` is not set. - [ ] Agents' commands run in unprivileged sessions, grading in another session. - [ ] The NAS holding the image store is reachable from cluster nodes only. --- # OpenDsec service level objectives (SLO) This document defines what an OpenDsec cluster promises its callers, how that is measured, and what to do when it is missed. The targets are suggested starting points; operators can adjust them to their needs, but must then change the numbers in the `opendsec-slo` group of `deploy/monitoring/alerts.yml` to match. Measurements are recorded in SPEC §33 (Chinese): fault injection and a 48-hour stability test. ## 1. Indicators (SLIs) | SLI | Good events | All events | Source | |---|---|---|---| | Creation availability | Successful creations | All creations, minus the callers' own errors | apiserver `opendsec_apiserver_create_seconds_count` | | Creation latency | Successful creations finished within the threshold | Successful creations | apiserver `opendsec_apiserver_create_seconds_bucket` | | Runtime availability | Successful Exec, ExecStream, ReadFile, WriteFile and lifecycle calls (Stop, Pause, Resume, Fork) | The same, minus the callers' own errors | apiserver `opendsec_rpc_server_handled_total` | | End-to-end success | A complete caller cycle succeeds | All cycles | the caller: `sdk/python/examples/soak.py` | **How errors are classified**: - **Callers' errors**: `invalid`, `unauthenticated`, `forbidden`, `not_found`, `quota_exceeded`, `rate_limited`, `conflict`. These are the caller's problem and are left out of the SLIs. - **Server errors** count as bad events: - in both kinds of SLI: `unavailable`, `internal`; - in creation only: `timeout` (waiting for a creation slot or for the rate limiter until timing out) and `no_capacity` (insufficient capacity planning, also the server side's responsibility); - `timeout` in runtime calls does not count: it is mostly the caller's own timeout, or a command that has not finished. **Server-side versus caller-side indicators**: the server counts every attempt, and the SDK retries `unavailable`, so one fault may be several failures on the server side while the caller only sees a slowdown. The caller's end-to-end success rate is what finally counts; the server-side indicators are for alerting and diagnosis. ## 2. Objectives Over a rolling 30-day window. | SLI | Objective | 30-day error budget | |---|---|---| | Creation availability | 99.9% | 0.1%: 1000 failures per million creations | | Creation latency (container) | 99% within 5 s | 1% | | Creation latency (microvm) | 99% within 3 s | 1% | | Runtime availability | 99.95% | 0.05% | | End-to-end success (`soak.py`) | 99.9% | 0.1% | **Assumptions**: - no more than 70% of the cluster's memory is allocated; - fewer than 3000 containers per edge. Beyond these, creations slow down noticeably, because the kernel serializes `pivot_root` (SPEC §32); the latency objectives no longer apply, and the cluster should grow first. **Behaviour under single failures** (verified by fault injection, SPEC §33): | Failure | Promise | |---|---| | Any component process crashes or restarts (IAM, apiserver, watcher, edge, fleet, nodeagent) | Callers see no failures; calls may slow down, for at most 60 s | | One control-plane node is cut off from the others | Callers see no failures | | An edge process hangs or crashes | Its sandboxes keep running and can be operated again once the edge is back | | 2 of 3 IAM replicas unavailable | Failures allowed: creation and login need an IAM majority; back to normal within 60 s of recovery | | An edge node loses its network or fails entirely | Failures allowed: operations on its sandboxes fail, and on a machine failure those sandboxes are lost, as designed; new creations are unaffected and go to other nodes | | The image store (NFS) is unreachable | Running sandboxes are unaffected (containers use local copies of their layers, `layer_cache`). New creations may fail: they wait until the store is back and fail if the outage outlasts their call timeout | | An edge process hangs | Creations sent to it move to another edge after about 15 s; operations on its sandboxes wait for it to recover | **Not covered by the SLOs**: - failures of the callers' own programs in their sandboxes; - sandboxes lost to a node reboot (by design; see the operations manual §5); - image import times. ## 3. Measuring **Server side**: the Prometheus rules are in the `opendsec-slo` group of `deploy/monitoring/alerts.yml`: - recording rules such as `opendsec:create_availability:ratio_rate5m`; - two burn-rate alerts (multi-window, multi-burn-rate, as in the Google SRE workbook): | Alert | Condition | Meaning | |---|---|---| | `OpenDsecSLOCreateBurnFast` | Error rates over 1 hour and over 5 minutes both above 14.4 times the budget | At this rate, 2 days spend the 30-day budget | | `OpenDsecSLOCreateBurnSlow` | Error rates over 6 hours and over 30 minutes both above 6 times the budget | At this rate, 5 days spend the budget | Runtime availability has the same two. Availability over the past 30 days in Prometheus: ```promql 1 - sum(increase(opendsec_apiserver_create_seconds_count{reason=~"unavailable|internal|timeout|no_capacity"}[30d])) / sum(increase(opendsec_apiserver_create_seconds_count{reason!~"invalid|unauthenticated|forbidden|not_found|quota_exceeded|rate_limited|conflict"}[30d])) ``` **Caller side**: run `soak.py` for a long time with a caller's onboarding bundle, then summarize with `soak_report.py`, per hour or per period: ```sh python soak.py 172800 8 soak.csv # 48 hours, concurrency 8 python soak_report.py soak.csv ``` Run `load_test.py` or `soak.py` with every release and config change and check the failure count (operations manual §4). ## 4. When the error budget runs out - **Less than half the budget left**: hold non-urgent releases and config changes, and find where the errors come from first. - **Budget spent**: - only changes that improve stability, each with a load test; - write the cause and the improvements into an incident record. - **A burn-rate alert fires**: follow the operations manual §5. Start with `opendsec_apiserver_create_seconds_count` grouped by `reason`, then the edges' `opendsec_edge_rejected_total`. --- # Versioning and compatibility This page says what stays compatible between OpenDsec releases, and what an upgrade or a rollback needs. Changes are listed per release in [CHANGELOG.md](https://github.com/Kernflux/opendsec/blob/main/CHANGELOG.md). ## Version numbers - A release is `MAJOR.MINOR.PATCH`, from the `VERSION` file, tagged `v`. - Builds outside a release carry a timestamp: `make build` gives `0.11.0-202610180528`. `opendsecctl version` and the `opendsec_build_info` metric show it. - **Before 1.0** (now), a minor release (0.11 → 0.12) may break compatibility. The changelog then lists every break under "Breaking" and says what to do. A patch release (0.11.0 → 0.11.1) never breaks anything. - **From 1.0**, breaking changes need a new major version. Minor releases add features compatibly, and patch releases fix bugs. - Security fixes go into the latest minor release only (see [SECURITY.md](https://github.com/Kernflux/opendsec/blob/main/SECURITY.md)). ## What is covered | Surface | Compatibility | |---|---| | Public gRPC API (`opendsec.v1.DSec`, `proto/opendsec/v1/opendsec.proto`) and the Python SDK | Changes only add things: new RPCs, new fields, new enum values. Fields are not renumbered or reused. Error reasons (`no_capacity`, `quota_exceeded`, ...) are stable strings. An SDK works with clusters of its own minor version and the next one. Example: the 0.10 SDK runs against 0.11 clusters, and was tested under load during the 0.10 → 0.11 production upgrade. | | Internal gRPC APIs: IAM, watcher and edge (`cluster.proto`), in-sandbox agents (`runtime.proto`), fleet | Compatible between consecutive minor versions, so that a rolling upgrade can run old and new components side by side (see below). They are not an API for other programs. | | Cluster config (`opendsec.json`) | New fields are optional and have defaults. A field to be removed or renamed is deprecated for one minor release first. Validation may become stricter (for example, 0.11 rejects an `edge_template` zone that placement cannot use); the changelog lists such changes. | | CLI (`opendsecctl`) | Commands and flags are kept; changes in how arguments are parsed are listed in the changelog. JSON output follows the protobuf JSON mapping of the API. | | Metrics and alert names | May change in minor releases before 1.0; the changelog lists renames. | | Go packages (`pkg/client`, `internal/...`) | Not covered. `internal/` cannot be imported from outside the module anyway. | ## Upgrades **Supported path:** from one minor version to the next (0.10 → 0.11), and between patch versions of one minor version. To go further, upgrade one minor version at a time. **Rolling upgrades keep running sandboxes.** The order is the same with the fleet service (`opendsecctl fleet rollout create`) and with `opendsecops upgrade`: 1. IAM followers, then the IAM leader. The followers can apply any command the old leader proposes. New kinds of commands are proposed only once the leader runs the new version, which by then all replicas do. 2. watchers, then apiservers, then fleet replicas. 3. edges, one at a time (or batch by batch with fleet), each drained first. What carries over: - **Edges**: an edge re-adopts the sandboxes the previous version left running. This needs `persist_state`, `keep_sandboxes_on_exit` and `KillMode=process` (the default units). The SDK resumes interrupted calls and output streams. - **In-sandbox agents**: every release ships aether and chronus under its own `agents/` directory. A running sandbox keeps the agents it started with until it ends, and edges talk to agents from the previous minor version. - **IAM state**: Raft logs and snapshots move forward. Backups restore across versions: 0.11 reads the formats `opendsec-iam-backup/2` and `/1`, and `dsec-iam-backup/1` from before the rename. - **Guest kernel**: a new kernel in a release is used by new microVMs only. Running and paused VMs keep theirs. **Runtime pins:** - **Firecracker version**: a memory snapshot (a pause snapshot, a fork, an image created with `memory=true`) restores only under the Firecracker version that took it. Placement enforces this through the edges' reported runtime versions. A release that changes Firecracker therefore leaves older snapshots runnable only on edges still on the old version. The changelog says so whenever `deploy/deps.lock` moves Firecracker. - **Image store formats**: layers, manifests and snapshots are read by later releases. Formats of derived files carry a version in their name (for example the EROFS disks under `vmdisks/`); a new format is built alongside the old one, and image GC removes the old one. ## Rollbacks - **To the previous release**: run the same rolling procedure. With fleet, use `opendsecctl fleet rollout rollback `, or roll out the old release; otherwise `opendsecops rollback `. - **State written by the newer release** may not be understood by the older one. In particular, a backup in a newer format does not restore into an older IAM. Take an IAM backup (`opendsecops backup`) before every upgrade, and keep the previous release's packages (fleet keeps them, and so does each node: the last 3, see `OPENDSEC_KEEP_RELEASES`). - **A changelog entry marked "no rollback"** means the release changes stored state in a way the previous release cannot read. Such a change only comes in a minor release before 1.0, and never in a patch release. ## Deprecation Something to be removed (an RPC, a field, a config key, a flag) is first marked deprecated in the changelog, the proto comments or the config reference. It keeps working for at least one more minor release. --- # How the components work This document goes through how each OpenDsec component is built: what it does, where its code is, how it works inside, where its state lives, and what happens when it fails. Deployment is in the README's "Deploy a cluster" and the [operations manual](/en/docs/operations); the reasons for each design decision and the measurements are in [SPEC.md](/docs/spec) (Chinese; "§x" below refers to its sections). ## 0. Overview ``` callers (Python SDK / opendsecctl) | gRPC + TLS, bearer token v apiserver xN (stateless, placement built in) ---> IAM x3 (Raft: identities, quotas, key-value area) | ^ snapshot every second ^ reconciliation | watcher xN (probes edges, aggregates scheduling state) | mTLS v edge (one per compute node, root) ---> backends: container / fncall / microvm / fullvm | unix socket or vsock v aether (PID 1 in the sandbox) ---> chronus (one per shell session) fleet x3 + nodeagent (every machine): releases, config, staged upgrades shared image store (NFS / CephFS / JuiceFS): layers, manifests, snapshots, microVM image disks ``` Source layout: | Directory | Contents | |---|---| | `cmd/opendsec-*` | Each component's main: flag parsing and startup only; the logic is in `internal/` | | `internal/cluster` | Assembles components from the config and starts their gRPC services; `opendsecctl cluster up` uses it too | | `internal/config` | The cluster config (one JSON file): structure, defaults, validation | | `proto/opendsec/v1` | Every gRPC interface: `opendsec.proto` (public API), `iam.proto`, `cluster.proto` (watcher and edge), `runtime.proto` (aether), `fleet.proto` | | `gen/go`, `sdk/python/opendsec/_pb` | Code generated from the protos | ### The path of one creation Taking `client.run_sandbox(...)`: 1. The **SDK** calls `CreateSandbox` with its token and project. With several apiserver addresses it round-robins, and on `unavailable` retries on another. 2. The **apiserver** (`place` in `internal/apiserver/apiserver.go`): 1. rate-limits per principal and checks the number of creations in flight; 2. fills in default TTLs, and checks that the image exists and the caller has `layer.use`; 3. makes a lease id; the sandbox id is `.`; 4. **places first**: the built-in placement picks an edge; 5. **then admits**: one IAM `Admit` call authenticates, authorizes `sandbox.create` and reserves quota, binding the reservation to the chosen edge. 3. **IAM** commits the reservation as one command through Raft: the project's and all its ancestors' usage grow by the sandbox's resources, and if any level would go over, nothing changes. 4. The **edge** receives `CreateSandbox` and makes the final admission decision: capacity, and memory, disk and CPU pressure. If they do not allow it, it returns `no_capacity`. Otherwise it: 1. records the sandbox on disk; 2. allocates its network; 3. has the backend create it; 4. sets it RUNNING and records that on disk. 5. When an edge returns `no_capacity` or `unavailable`, the apiserver gives that edge a short penalty and tries another, up to `max_attempts` times (default 3). In IAM the same lease is simply rebound to the new edge. If every attempt fails, the lease is released. 6. Later calls for this sandbox are forwarded straight to the edge named in its id; no table lookup is needed. ## 1. apiserver **Role**: the cluster's only entry point: authentication, authorization, admission, placement, and forwarding to edges. Code in `internal/apiserver`. - **Stateless**: a sandbox id encodes its edge, so any apiserver can serve any request, and a restart loses nothing. The one exception is sandboxes that moved (§26.2): their new location is recovered from the watcher snapshot (edges report `moved_in`) and from the original edge's answer (`move.go`). - **Authentication cache**: - token-to-principal results and authorization results are each cached for 30 s, never past the token's own expiry; - in the background, IAM's security epoch is checked every second. Revoking a token or a permission changes the epoch, which clears the caches, so a revocation takes effect on every apiserver within about 1 s (§14.3). - **Scheduling state**: every `snapshot_refresh`, the cluster snapshot is fetched from the watchers for the built-in placement. If none can be fetched, the last one is kept. - **Readiness**: with no IAM answer or watcher snapshot for 15 s, `/readyz` fails, and the load balancer takes the apiserver out. - **Admission** (`admission.go`, §18): - default TTLs and the lifetime cap; - per-principal token buckets; batch creations queue on the server and never partly fail; - a limit on creations in flight (default 2048); - authentication is paused for a source address with too many failures. - **Batch creation** `CreateSandboxes`: up to 32768 per call, results streamed one by one. - **Fork** (`fork.go`): - one snapshot of the source sandbox, then count copies, up to 256 per call; - each copy is an ordinary creation (authorized and charged on its own), placed on the source's node first; - copies placed on other nodes fetch the snapshot files straight from the source edge. - **Image API** (`images.go`): - registers, composes and deletes image manifests, writing to the shared store directly; - writes need `layer.publish`, use needs `layer.use`, so the account that builds environments and the account that runs jobs can be separate. - **Runtime calls** (`runtime.go`): exec, files and HTTP are authorized for `sandbox.operate` and forwarded to the edge, which passes them to aether in the sandbox. A paused sandbox is resumed by the edge; if its node has no room, it is moved to another node and resumed there. - **Audit** (`audit.go`): one JSON line per public call, rotated by size. - **Extension points** (`ext`, §34): - an extension's authenticator runs before IAM: a token it recognizes (an SSO token, say) maps straight to a principal, and IAM never sees that token; authorization and quotas stay with IAM, and Admit carries the principal instead. - audit extensions receive each line after it is written to the local file. - The open-source build links no extensions, so none of this applies there. - **fleet proxy** (`fleet.go`): serves `FleetAdmin` on the public port; after authorizing `fleet.view` or `fleet.operate`, it forwards the call to fleet with the caller's identity. ## 2. placement (inside the apiserver) **Role**: picks an edge for each new sandbox. Code in `internal/placement/placement.go` (§3.4). - **Two steps**: 1. **filter**: healthy, not draining, offers the backend, matching hardware (GPUs), enough allocatable capacity, has the image; for creations from snapshots, the same runtime versions; 2. **power-of-k**: draw k candidates uniformly at random (default 2) and take the least loaded. - **Load** is the highest of three: - allocated resources as a share of capacity; - actual CPU use (smoothed over 10 s by the edge); - actual memory use. Actual use below `usage_floor` (default 0.2) does not count, so different idle baselines of hosts do not skew placement. - **Concurrency without coordination**: each apiserver overlays the sandboxes it just placed, not yet in the snapshot, onto the snapshot (in-flight overlay), so several apiservers placing at once do not all pick the same node. - **Penalties**: a node that rejected a creation becomes the last choice for a short while. Not excluded outright: when every node is restarting, trying one beats failing. - **Locality**: - fork copies prefer the source's node; - creations from snapshots prefer nodes that already cache the snapshot. - **Cloud burst**: `zone` is `onprem` or `cloud`. When local utilization exceeds `burst_threshold`, some cloud-eligible images go to the cloud zone with a probability. - **Errors**: when no node fits, it tells "the image does not exist" (`not_found`) from "every node with the image is full" (`no_capacity`), with the reasons nodes were skipped (memory, disk, CPU pressure, runtime version mismatch). ## 3. IAM **Role**: identities, projects, permissions and quotas; also distributed locks, and a key-value area where other components keep small durable state. Code in `internal/iam` (§3.2, §15). - **Data model**: - **projects**: nested, e.g. `root/team-a/rl`; unset quota keys take the parent's; - **principals**: belong to a project and may hold several tokens; - **tokens**: `dsk..`; only the secret's hash is stored; they can expire and be revoked; - **permissions**: granted on projects and valid for their subprojects; delegation is bounded, nobody grants what they do not hold. - **Deterministic commands** (`commands.go`): - every state change is a Command. Nondeterministic values (time, token ids, secret hashes) are made by the proposer and written into the command, so every replica ends up with the same result. - Standalone: commands apply directly and the state is written to a JSON file. - Replicated (`raft.go`, hashicorp/raft + BoltDB): commands are replicated through Raft. Followers forward requests to the leader as they are (`service.go`), so clients may connect to any replica. - **Quota leases**: - each sandbox has a lease recording its resources, counted in its project's and every ancestor's usage; - `Admit` authenticates, authorizes and reserves in one go; reserving the same lease again is idempotent (it only rebinds the edge); - leases are bound to edges, and when an edge is judged lost, its leases are released. - **Reconciliation** (`reconcile.go`, §15.2): - watchers regularly report the leases actually alive on each edge; - IAM commits only the differences: leases on an edge that IAM did not know are adopted, leases IAM holds that the edge does not are released; - leases younger than `reconcile_min_age` are never released; - normally releases come from edges: when sandboxes end, the edge queues their leases and releases them with `ReleaseLeases` after 1000 of them or 50 ms, one Raft commit per batch, so stopping a million sandboxes at once does not clog IAM (§31); - leases no edge claimed for a long time are cleared by `SweepUnbound`. - **Locks**: named locks with a TTL; watchers and fleet elect leaders with them. - **Key-value area** (`kv.go`, §28.3): - accepts fixed prefixes only (fleet's nodes, releases and rollout records, for instance); - each write gets a globally increasing revision, with compare-and-set; - IAM does not interpret the contents. It is replicated and backed up with the rest of the state. - **Backup and restore** (`ops.go`): gzipped JSON with principals, projects, token hashes, the security epoch and the key-value area. No leases (reconciliation rebuilds them after a restore). - **Membership changes**: `opendsecctl iam member add/remove`; a new replica started with `-join` waits to be added (§27.2). - **Bootstrap**: on first start, the root project and the administrator are created with the admin token from the config. A token still holding the example's placeholder (`change-me`) is refused. ## 4. watcher **Role**: probes edge health, aggregates scheduling state, drives quota reconciliation. Code in `internal/watcher/watcher.go` (§3.5). - **No durable state**: after a restart, its view is rebuilt from the config's edge list, the edges' periodic registrations, polling the edges, and the other watchers' snapshots. - **Probing**: - every `probe_interval` (default 1 s), each edge's `Status` is called concurrently; - 3 consecutive failures make an edge unhealthy; - an edge answering with an unexpected id counts as a failure. - **Snapshot**: after each round, a versioned snapshot with each edge's capacity, allocations, actual use, pressure, cached snapshots and runtime versions, for the apiservers to fetch. - **Several instances**: all probe and all serve snapshots. Only the one holding the IAM lock reconciles: it renews the lock every TTL/3 and considers itself leader until "request time + TTL − 2 s" by its own clock, so clock skew against IAM never lets two instances reconcile at once. - **Reconciliation**: every `reconcile_interval` (default 30 s). - healthy edges: their live leases go to IAM for reconciliation; - edges unhealthy for longer than `lost_edge_grace` (default 120 s): reconciled as empty, releasing their quota. An edge that comes back with sandboxes has them adopted in the next round; - each edge is reconciled on its own, so one unreachable edge only skips itself. - **Registration**: with mTLS, an edge can register only itself (its certificate identity must match). ## 5. edge (the node's control plane) **Role**: the control plane on each node: final admission, the sandbox state machine, TTL expiry, liveness checks, releasing resources, reporting to the watchers, and forwarding runtime calls to aether. Code in `internal/edge`, running as root. - **State machine**: CREATING → RUNNING ⇄ PAUSING/PAUSED/RESUMING → STOPPING → STOPPED, FAILED on failure. Each sandbox has an operation lock: one lifecycle operation at a time. - **Final admission** (`create`): capacity accounted per sandbox (CPU, memory, disk, GPUs, per-backend counts), plus three pressure checks: - memory: MemAvailable below the reserve (default 10% of RAM, at least 128 MB); - disk: free space on the file system holding `state_dir`; - CPU: PSI, optional. The edge has the last word: the snapshot placement used may be stale. - **Persistence and re-adoption** (`store.go`, `recover.go`, §13): - one record file per live sandbox, written as temp file → fsync → rename; - the record is written before the backend is called, so an edge that crashes during a creation still knows the sandbox after restarting; - when a creation finishes, the edge asks IAM where the lease is bound (§36): if the caller gave up and created it on another edge, or the lease was released, the sandbox is deleted right there, leaving no orphan; if IAM cannot be asked, it is kept; - at startup, before serving: RUNNING and PAUSED sandboxes are handed to their backends to adopt again, the rest are marked failed and released; the backends then clean up leftovers on the node that match no record (cgroups, processes, disks, networks); - with `keep_sandboxes_on_exit` and systemd's `KillMode=process`, edge restarts and upgrades do not touch running sandboxes. - **Background loop** (`Run`): - expires sandboxes by TTL: the idle timeout `ttl_running_stop` and the lifetime `ttl_max`; - probes backend liveness; - cleans up ended records; - when memory is short, inflates balloons of sandboxes that have them (`balloon.go`, §24): best-effort sandboxes first, then those idle longest, a quarter per round, at most half; deflates once the node has room again. - **Runtime calls** (`runtime.go`): - each call resumes a paused sandbox first (transparently to the caller), resets the idle TTL, and keeps idle expiry from interrupting while the call runs; - then it gets aether's gRPC connection from the backend and forwards the call. - **Reporting**: `Status` returns capacity, allocations, actual use and pressure (`host/usage.go` reads `/proc/stat`, `/proc/pressure/cpu` and `/proc/meminfo`; CPU smoothed over 10 s), plus live leases for reconciliation. Edges also register with the watchers periodically (`RegisterLoop`). - **Between nodes** (`move.go`, `peerfile`, §26): - snapshot files stay on the node that took them; other nodes fetch them from there when needed; - only non-zero data is sent: memory files and disks are mostly holes, and the receiver keeps them as holes; - a paused sandbox can move to another node: the new node resumes it under the same id, and the move completes only once the old node lets go, so it never runs on both. - **Metrics**: `internal/edge/metrics.go`, read live at scrape time. ### Backend interface `internal/edge/backend/backend.go` defines the `Backend` interface every backend implements: `Create`, `Stop`, `Pause`, `Resume`, `Probe`. Optional capabilities, implemented as needed: | Interface | Purpose | |---|---| | `Runtime` | Returns the gRPC connection to aether in the sandbox | | `LossReporter` | Tells the edge when it notices a sandbox died (its aether connection broke) | | `Snapshotter` / `Packer` | fork, snapshots, `pack_diff` | | `Networked` | Needs sandbox networking | | `ImageLister` | Can only run fixed local images (development) | | `Versioned` | Reports runtime versions, e.g. firecracker 1.17.0 | The `sim` backend only simulates lifecycles, for scheduling tests and `cluster up -sim-edges`. ## 6. Container backends (container, fncall) Code in `internal/edge/backend/container` (§11.2, §21). - **Root file system**: - lower layers: node-local copies of the image layers (`layercache.go`, §33.1): each layer is copied whole from the store to `//layers/` the first time it is used, shared by every container backend on the node. Running containers therefore no longer read the store and are unaffected when it (a NAS) is unreachable. - layers are immutable, so copies are keyed by layer id; - copies unused by any sandbox for 6 hours are removed by the edge; - `layer_cache: false` stacks layers straight from the store again (loaded on demand, file by file); - upper layer: a sparse ext4 image file per sandbox (`disk.go`) on a loop device, `disk_mb` in size. This disk is mounted in the sandbox's own mount namespace, not on the host. Otherwise each sandbox adds an entry to the host mount table, which every new namespace copies, and with thousands of sandboxes each start slows by about 0.1 s. - **Startup**: "re-exec yourself", as runc does. The edge starts its own binary again with a special argv[0]; that process is the sandbox's setup process (`init.go`), which in new mount/pid/uts/ipc/net/cgroup namespaces: 1. mounts the writable disk and overlay, then proc, sys read-only, and the sandbox's own cgroup (so Go or a JVM inside can size themselves to the limits); 2. pivot_roots into the sandbox's root and sets the hostname; 3. makes kernel interfaces read-only or masks them per `sandboxfs` (defaults as in Docker), plus the paths the caller asked for; 4. lowers the agent's OOM score, so that when memory runs out the sandbox's processes are killed, not the agent; 5. finally installs the seccomp filter, drops surplus capabilities (Docker's default set minus CAP_MKNOD), and execs aether as PID 1. - **Resources**: cgroup v2, in `linuxrt/cgroup.go`. - limits on CPU, memory and pids, optionally disk I/O; - BE sandboxes get `cpu.idle`; - each sandbox gets its own core scheduling cookie (`linuxrt/coresched.go`, §29.1), so two sandboxes never share a physical core's hardware threads at once, which prevents interference and blocks SMT side channels. - **Pause**: the cgroup freezer. With `pause_reclaim`, memory is also pushed to swap with `process_madvise(MADV_PAGEOUT)` and read back ahead of use on resume (`linuxrt/reclaim.go`, §22.1). - **GPUs** (`linuxrt/gpu.go`): - exclusive: a whole device; - shared: a slot on a device without an exclusive user. - **Snapshots and `pack_diff`**: file system only. A snapshot freezes the writable layer into a new image layer. `pack_diff` removes /tmp, caches and shell history. Container forks carry no memory. - **fncall**: the same implementation plus a warm pool (`pool.go`, §22.3). For the images listed in the config, sandboxes are started ahead of time; a request takes one and gets its limits, network and id set. About 2 ms through the API. ## 7. microVM backend (Firecracker) Code in `internal/edge/backend/microvm` (§19). This is the main backend. - **One Firecracker process per sandbox**, started by jailer: - its own chroot; - a uid per sandbox; - new network/IPC/UTS namespaces, inside the sandbox's own cgroup. - **Disks**: all VMs share one guest kernel; each VM gets four disks: 1. **agent disk**: read-only, with aether and chronus (`BuildAgentDisk`); 2. **image disk**: read-only, the image's layers flattened into one disk (`ImageCache` in `disks.go`). Each set of layers is built once, shared by every VM of the image on the node; virtio-pmem + DAX lets them share the host's page cache. With `erofs_images`, the image disk is EROFS and built only once, in the store: the first node that needs it builds it under the exclusive `vmdisks/.lock`, and other nodes just copy it (`images/vmdisk.go`, §29.4); 3. **writable disk**: the sandbox's own, sparse ext4; 4. **config disk**: tiny, unmounted once the guest has read it, so a VM restored from a snapshot can get a new one. - **In the guest**: aether runs as PID 1 with `--vm-init`, assembles the root file system, then serves the same Runtime API as in containers, over vsock (`internal/agent/aether/vminit.go`, `vsock.go`). zram, DAMON cold-page reclaim and free page reporting are turned on per config (`vmmem.go`). - **Network**: Firecracker's "network for clones" layout: each VM's netns has a tap device of the same name and the same guest address, translated by netpol inside the netns, so copies restored from snapshots need no change to the guest's network config. While a VM is paused there is no VMM process; a minimal placeholder process keeps the netns alive (`linuxrt/netnsholder.go`). - **Pause = snapshot and exit** (`Pause`): 1. pause the vCPUs; 2. the first time, a full snapshot; afterwards, dirty pages only (Diff), merged into the previous memory file (`diff.go`); 3. end the Firecracker process, so the memory really goes back to the host; 4. punch holes for zero pages in the memory file. Resume restores in a new process, in about 50 ms. - **Creating from snapshots, fork** (`createRestored`): the snapshot's files are hard-linked into the jail. A snapshot on another file system or node is first fetched to a node-local copy (`snapcache.go`), which every copy on the node then links to, sharing the page cache. After the restore, aether receives `PostRestore` and gives the copy a new clock, random seed, hostname and machine-id (`aether/restore.go`). - **Liveness**: the edge keeps a `Watch` stream open to aether; when it breaks, the sandbox counts as lost. After an edge restart, `Adopt` reconnects to running VMs. - **Metrics**: each creation's phases (prepare, network, config disk, boot or restore) are timed. ## 8. Full VM backend (fullvm, QEMU) Code in `internal/edge/backend/fullvm` (§23). - One QEMU process per sandbox (q35 machine, KVM), with the same disk layout as microVMs on virtio-pci, and aether over vhost-vsock. - Unlike microVMs: a full PCI bus, so GPUs can be passed through with VFIO (not yet tested on real devices); with `full_init`, the image's own `/sbin/init` (systemd) runs as PID 1. - Pause only stops the vCPUs and keeps the memory; no memory snapshots and no fork. Controlled through QMP (`qmp.go`). ## 9. Network policy (netpol) Code in `internal/edge/netpol` (§20, §29.2). The paper uses eBPF; this uses nftables. - **Topology**: each sandbox has its own network namespace, connected to the host by a veth pair, with a /30 from the edge's subnet (default 10.88.0.0/16). microVMs add a tap inside the netns. - **Default deny**: one nftables table per edge (`nft.go`). - New connections to a sandbox are always dropped, whether from other sandboxes, the host or outside; replies pass. - Every packet a sandbox sends enters its own chain, matched by veth and source address, so forged source addresses match nothing and are dropped. Established connections go through the chain too, so taking a rule back cuts its connections at once. - The chain: accept allowed networks; drop private, link-local and multicast addresses; accept addresses the DNS proxy resolved; drop the rest. - **Allowing by domain**: each edge runs a DNS proxy (`dns.go`). - Only allowed domains (subdomains included) are forwarded upstream, and the IPv4 addresses in the answer join the sandbox's allowed set before the sandbox gets the answer; - other names get NXDOMAIN; AAAA queries get empty answers (sandboxes have IPv4 only); - the proxy tells sandboxes apart by the query's source address. - **Rules** (`rules.go`): - a domain, an IPv4 network, a rule set named in the cluster config (such as `pypi`), or `*` (every public address); - each rule can be limited to a protocol and port range; each port combination gets an nftables set; - rules can change while the sandbox runs (`UpdateNetwork`). - **Performance**: per-sandbox rules go through netlink directly (`nl.go`). The `nft` and `ip` commands reread the whole ruleset or copy the mount table every time, 100–300 ms each with thousands of sandboxes; only the edge's base rules are set with `nft`, once, at startup. ## 10. Image store (images) Code in `internal/images` (§17, §24, §29). The image store is a shared directory every edge and apiserver mounts: ``` /layers//rootfs/... layers, overlay format (whiteouts are 0:0 character devices) /layers//meta.json /images///.json manifests: layer list, env, workdir, user /snapshots// snapshots (file system layer + a microVM's memory and state) /vmdisks/.erofs microVM EROFS image disks /inuse/ layers used by running sandboxes (for GC) ``` - **Layers are immutable**: written to a temporary directory, then renamed into place as a whole. Manifests likewise, temporary file plus rename. - **Import** (`oci.go`): `opendsecctl image import` pulls straight from OCI registries (Docker Hub by default); layers may be gzip-compressed, zstd-compressed or uncompressed. Whiteouts are restored as 0:0 character devices, which needs root. - **Private registries** (`auth.go`, §35): credentials from `-username` plus `-password-stdin`, or the config file docker login saved; token services (Basic auth exchanged for a token), identity tokens (OAuth2 refresh tokens) and plain Basic auth are supported. Redirects to another host or port (a CDN, object storage) carry no credentials. - **Opaque directories** (`opaque.go`): overlay marks opaque directories with the xattr `trusted.overlay.opaque`, which NFS does not carry. So at import, opaque directories are expanded: for each name visible in lower layers but absent from this one, a whiteout is written. - **Composition**: an image is a list of layers; `compose` concatenates several images' layers in order, copying no data. - **Merging** (`merge.go`): when a container fork chain gets too long, several layers are squashed into one; the result is still overlay format and stacks the same as the originals. - **GC** (`gc.go`): layers and snapshots still referenced by manifests, snapshot records or running sandboxes are kept; the rest past the grace period are removed, along with interrupted writes. - **Self-check** (`storecheck.go`): `opendsecctl image store-check` tries each capability the store needs in a directory: root not mapped to another user, chown, exclusive create, rename, hard links, 0:0 devices, overlay stacking and so on. ## 11. Agents in the sandbox: aether and chronus Code in `internal/agent`. Both are statically linked; each version lives in `/opt/opendsec/agents//` and is mounted read-only into sandboxes. Running sandboxes keep the version they were started with. - **aether** (`aether/aether.go`): one per sandbox, usually PID 1. - The edge talks only to it: over a unix socket in containers, vsock in VMs. - It routes calls by session id to the chronus serving that session, starting it on first use; closing a session kills chronus's whole process tree (`proctree`). - File calls, directory listings and HTTP requests made from inside the sandbox also go to the session's chronus, so they run as the session's user, with relative paths resolved against the session's current directory. - The `Watch` stream tells the edge it is alive. - **The process API** (`procs/`, §37) is aether's own, outside sessions: each command is a separate process, addressed by pid or tag, with stdin, signals and an optional PTY. A broken stream does not end its process, which can be reattached. User switching and environment reuse chronus's `Environ`. The E2B gateway builds envd's process API on it. - **chronus** (`chronus/`): one process per shell session, holding a long-lived shell, so `cd`, exported variables and functions carry over between commands. - Command output is written to record files: callers can disconnect and fetch it later by offset (`GetOutput`), or receive it as a stream (`ExecStream`). - Each command carries a request id: when the SDK retries, chronus finds the command already running instead of starting it twice. - It is a subreaper: processes commands put in the background stay in the session's process tree and are cleaned up when the session closes. - Output limits (`record.go`, §29.3): each stream records up to a set number of bytes, and the rest is read and dropped (`dropped_bytes` in the result); each session's records are bounded too, the records of the earliest finished commands going first. - Sessions can run as a given user (`user`), different sessions as different users (§29.3). ## 12. fleet and nodeagent (cluster management) Code in `internal/fleet`, `internal/nodeagent`, `internal/release` (§28). - **Releases** (`release.go`): a release is a directory with: - `bin/`: every component, plus firecracker and jailer; - `agents/`, `units/`, `vm/` (the guest kernel); - `deps.lock`: the pinned runtime dependencies; - `SHA256SUMS`: checksums of every other file. `opendsecctl fleet release pack` packages it. Once pushed to fleet, each replica keeps a local tgz. - **fleet**: three replicas; the one holding the IAM lock `fleet-controller` leads, and the others forward requests to it. - All desired state lives in IAM's key-value area, replicated and backed up with IAM: the node list (roles, failure domain, target release, config generation), releases, rollouts. - Node heartbeats are soft state, in the leader's memory only. - **Controller** (`controller.go`): the leader runs a `tick` periodically, sending actions to nodes that lag and advancing rollouts in progress: - one step per node, going pending → staging (download) → staged → applying (switch release, write config, restart in order) → verifying (wait for the node's `/readyz` and the watcher snapshot to be fine) → ok. - Control-plane nodes first, one at a time, the IAM leader's node last; then one edge as a canary, and after the bake time the edges in batches, with a cap per failure domain in each batch. - Each edge is drained before its upgrade and resumes scheduling after. Before each batch: machines already upgraded are still fine and every control-plane component is ready; PromQL gates (`gates`) are checked too when configured. - A failure while applying or verifying sends the machine back to its old release and pins it, and the whole rollout pauses until a person decides to continue, roll back or abort. - **nodeagent** (`agent.go`): one per machine, as root. - Reports to fleet every 10 s (release, unit states, actions in progress) and picks up actions from the answer. - Accepts five actions only: `stage`, `apply`, `restart`, `start`, `stop`. It changes only `/opt/opendsec`, `/etc/opendsec` (allow-listed files) and OpenDsec's own systemd units, and decides nothing by itself. - With all of fleet down, it changes nothing, and components and sandboxes keep running. - It runs from a separate copy in `/opt/opendsec/agent/` that does not switch with releases; a new release's nodeagent replaces it only after passing a `-check` self-test. - **opendsecops** (`deploy/prod/opendsecops`): a bash tool on the ops machine for the first install, certificates, adding machines and emergencies, reaching nodes over ssh (nodes whose inventory address is `local` run on this machine). ## 13. Security and RPC infrastructure Code in `internal/pki`, `internal/rpcutil` (§14). - **mTLS between components**: - one certificate per component instance, with its identity in the URI SAN `opendsec:///`; - each gRPC service has an ACL of which roles may call which methods (`security.go`): only apiservers can call IAM's `Admit`, for instance, and edges can register only themselves. - Development mode can use a shared service token instead, without per-caller authorization. - **Public port**: apiservers serve TLS, and callers send bearer tokens. Apiserver certificates carry the fixed name `api.opendsec.internal`, which the SDK checks when connecting to several apiservers by address. - **Error reasons**: every error carries a stable string in `ErrorInfo.reason`, such as `no_capacity`, `quota_exceeded`, `not_found`, `unavailable`. The SDK and retry logic go by it, never by parsing messages. - **Connections**: - servers ping idle connections every 20 s, so load balancers do not cut quiet streams on their idle timeout; - internal clients ping every 10 s, detecting a silently lost connection within about 15 s; reconnect backoff is capped at 2 s, so an edge is reachable again soon after a restart; - connections are pooled per address; - messages may be up to 80 MB each way. - **CA rotation**: three steps, without downtime (`opendsecops rotate-ca`; see the operations manual). ## 14. Clients - **Python SDK** (`sdk/python/opendsec/client.py`): an async API on grpc.aio. - `OPENDSEC_ADDR` can list several apiservers; the SDK round-robins over them and avoids broken ones. - On `unavailable`, it retries with backoff within `retry_timeout`. - `run_shell` carries a request id, so retries never run a command twice; `stream_shell` resumes from the offsets already received after a disconnect, losing and repeating nothing. - Callers choose the backend explicitly; the SDK does not abstract over backends. - **Go client** (`pkg/client`): used by `opendsecctl` and the end-to-end tests. - **opendsecctl** (`cmd/opendsecctl`): - the public API on the command line: sandboxes, exec, cp, images, IAM, fleet; - local tools: `cluster up` (an in-process cluster), `bench`, `pki`, `image store-check`, `fleet release pack`. ## 15. Metrics Code in `internal/metrics` (§26.3). Every binary serves, with `-metrics :`: - `/metrics`: Prometheus format; - `/healthz`: the process is alive; - `/readyz`: whether it can serve, with the reason when not. Main metrics: - calls and latency per RPC (by method and error reason); - apiserver creation latency and failure reasons; - placement retries; - per-backend sandbox counts on each edge, capacity and use, creation phase latencies; - fleet and nodeagent state; - `opendsec_build_info`: the running version. Every metric and the alert rules: [Monitoring and alerts](/en/docs/monitoring); the rules themselves are in `deploy/monitoring/alerts.yml`. ## 16. E2B gateway (opendsec-e2b) Code in `internal/e2b`, `cmd/opendsec-e2b` (§37). Lets E2B's SDKs run on OpenDsec without code changes. - **Two APIs on one port**: - E2B's management API (REST, `api.go`): create, get, list, delete, timeout, pause, connect; - network options and refresh (`network.go`): mapped to `network_rules` and `UpdateNetwork`; autoPause maps to `ttl_paused_s`; - fork and snapshots (`snapshots.go`): mapped to `ForkSandbox` and `CreateSnapshot`; a snapshot is an image in the gateway's project, and `snapshotID` is the image name; - E2B's in-sandbox envd API (`envd.go`): the process and file services over the Connect protocol, `/files` over HTTP. Requests with an `E2b-Sandbox-Id` header, or with host `-.`, are envd's. - **Stateless**: every request calls the apiserver with the caller's own token (their E2B API key), so permissions, quotas and audit work as usual. The only cache is the environment variables passed at sandbox creation, kept for one minute. - **Mapping**: - an E2B template is the image, backend and resources of `e2b.templates` in the cluster config; - metadata and envVars are stored as labels prefixed `e2b.`; - sandbox ids have `.` replaced by `-`; - timeouts use `SetSandboxTimeout`. - **Deployment**: fleet role `e2b=`, unit `opendsec-e2b@`, `/readyz` on port 9467. The unit runs as a `DynamicUser` and receives only the cluster config and the CA certificate, through `LoadCredential`. --- # License and commercial use ## Open source | Part | License | |---|---| | Servers and tools | AGPL-3.0 | | Python SDK (`sdk/python/`), Go client (`pkg/`), API definitions and generated code (`proto/`, `gen/`) | Apache-2.0 | Programs that call OpenDsec only use the client parts, so the AGPL does not apply to them. See [NOTICE](https://github.com/Kernflux/opendsec/blob/main/NOTICE). Contributions need a signed CLA; see [CONTRIBUTING.md](https://github.com/Kernflux/opendsec/blob/main/CONTRIBUTING.md). ## Commercial license You need a commercial license to: - keep a modified version of OpenDsec closed; - offer OpenDsec as a service without publishing your changes; - use the enterprise edition. The enterprise edition adds: - single sign-on with your identity provider (OIDC); - audit logs streamed to your SIEM or archive, hash-chained against tampering. For private deployments and support, contact **zhyb1528@aliyun.com**. --- # Acceptance-testing the open-source release on fresh machines: 22 issues OpenDsec's automated tests always ran, and passed, on our development machine and our production cluster. But the development machine has everything installed, and the production cluster was tuned step by step. A new user starts from a fresh machine and a README. So before the release we took a new user's place: fresh Ubuntu 24.04 VMs, the documentation, and every feature a user might touch. ## What we tested | Area | How | |---|---| | Single machine | Install per the README; run the examples, the SDK feature test and 54 admin command checks; reboot; uninstall | | No KVM | Containers only | | E2B | E2B's Python SDK (sync and async), JS SDK, code interpreter, snapshots and network options | | Offline | Install from the offline bundle on a VM without internet; use the E2B SDK offline | | Cluster | 3 control and 3 compute nodes with NFS and HAProxy: deploy per the docs, hand over to fleet, add a node, rolling upgrade, config change, maintenance, backup, monitoring | | Failures | Hard-reset a control node and kill the IAM leader under load | | Kubernetes | Single-node k3s | ## Two serious issues **On a current Ubuntu kernel, every microVM resume failed.** Ubuntu 24.04's 6.8.0-147 kernel kills a process placed with `CLONE_INTO_CGROUP` into a cgroup that has ever been through `cgroup.kill`. OpenDsec ends the VM process on pause and starts a new one in the same cgroup on resume, so every resume failed. Our production machines run an older kernel, which is why we never saw it; a kernel upgrade would have exposed it. Killing now freezes the cgroup and sends SIGKILL to each process instead of using `cgroup.kill`, and a regression test fails on the affected kernel. **Restarting a control node failed a few calls with `unknown edge`.** After a restart the watcher only learned about compute nodes when they re-registered, up to 10 s later, and meanwhile the apiserver treated them as gone. A hard reset of one control node failed 3 of 3,911 calls in a load test. The watcher now starts from the nodes its peers know, and the apiserver keeps a node it briefly cannot see for 2 minutes. The same hard reset now fails none. ## The rest Most of the other 20 are things a new user hits on day one: - **Install**: fresh machines lack `make`, and the script only failed halfway through the build; fresh Ubuntu lacks `python3-venv`; fresh cloud images have no package lists until `apt update`. - **CLI**: `opendsecctl exec` dropped argument quoting, breaking `python3 -c '...'`; `--help` did not work; batch create printed no IDs; importing a missing local file went to Docker Hub. - **E2B**: a single-machine install had no E2B gateway, and on clusters that deny internet by default E2B sandboxes had none, while E2B allows it by default. - **Operations**: `opendsecops restart` skipped the E2B gateway; the disk alert used a fixed threshold and fired on every machine under 200 GB; on Kubernetes the apiserver could not write the image store. - **Performance**: the NFS 4.1 we recommended cannot find holes in sparse files, so creating from a memory snapshot on another node read the whole memory file and took 141 s; with 4.2 it takes 62 s. The docs now say 4.2, and `store-check` warns about 4.1. Each is listed in the [CHANGELOG](https://github.com/Kernflux/opendsec/blob/main/CHANGELOG.md). ## Not covered Real GPUs, ARM machines, multi-node Kubernetes, and distributions other than Ubuntu 24.04. If you run OpenDsec on one of these and something breaks, please [open an issue](https://github.com/Kernflux/opendsec/issues). ## Takeaway A full test run on a development machine shows the code is right on our machines. It does not show a user can get it working from the docs. Both serious issues came from the environment: a newer kernel, and restart timing that production never produced. Walking through the docs on a fresh machine finds problems a development machine never will. --- # OpenDsec is open source Today we published the OpenDsec source on [GitHub](https://github.com/Kernflux/opendsec). OpenDsec is sandbox infrastructure for reinforcement-learning training and code-executing agents: it creates, pauses, copies and destroys isolated environments in bulk, on your own machines. ## Why Agentic RL training runs thousands of rollouts at once, and each needs an isolated environment that can install dependencies, run tests and reach the network. Training gets preempted, so environments must pause as a batch. The same state gets explored several ways, so environments must be copied together with their memory. DeepSeek described the system they built for this in [*DeepSeek Elastic Compute (DSec)*](https://arxiv.org/abs/2609.22978). OpenDsec follows that paper: a stateless apiserver, IAM with nested projects and quotas, a watcher that aggregates scheduling state, power-of-k placement with final admission on each node, layered images, a default-deny network policy and task-level pause. Where the paper is silent we made our own choices, and the [SPEC](/docs/spec) in the repository marks which design points come from the paper and which are ours. OpenDsec is an independent project. It is not affiliated with DeepSeek and contains none of its code. ## What it does - **Fast**: a Firecracker microVM cold-starts in about 0.1 s, is created from a memory snapshot in about 55 ms and resumes from a pause in about 50 ms. - **Fork**: `fork(count=N)` copies a running VM with its memory while pausing the source for about 10 ms. Copies on one node share memory pages; 64 of them restore concurrently in about 0.2 s. - **Scale**: one 384-thread node took a burst of 5120 microVMs in 37 s. - **E2B compatible**: code written for the E2B SDK runs on your machines after changing three environment variables. E2B's own Python and JS SDK acceptance checks all pass. - **Operable**: mTLS between components and a Raft-replicated IAM; staged fleet rollouts with automatic rollback; offline installs. Upgrading any component leaves running sandboxes alone. These numbers come from our own bare-metal machines; the SPEC says how each was measured. ## Try it One Linux machine with Ubuntu 24.04, about 10 minutes: ```sh sudo -i apt update && apt install -y make git python3-venv curl -fsSL https://go.dev/dl/go1.27.3.linux-amd64.tar.gz | tar -C /usr/local -xz git clone https://github.com/kernflux/opendsec && cd opendsec deploy/quickstart.sh ``` See [Get started](/en/docs/quickstart) for the details, and [Deployment](/en/docs/deploy) for clusters and offline installs. ## License The servers are AGPL-3.0. The Python SDK, the Go client and the API definitions are Apache-2.0, so programs that call OpenDsec are not bound by the AGPL. A [commercial license](/en/docs/license) covers closed modifications, offering OpenDsec as a service without publishing changes, and the enterprise edition's single sign-on and audit export. ## What's next OpenDsec is at 0.x, and interfaces may change before 1.0. Not yet verified: distributions other than Ubuntu 24.04, real GPUs, and multi-node Kubernetes. Issues and pull requests are welcome on GitHub.