<?xml version="1.0" encoding="utf-8"?><rss version="2.0" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Notebook</title><link>https://notebook.tines.com/</link><description>A shared notebook for the engineers, designers, and makers working at Tines.</description><language>en-US</language><atom:link href="https://notebook.tines.com/rss.xml" rel="self" type="application/rss+xml"/><item><title>The anatomy of a 3B step</title><link>https://notebook.tines.com/writing/anatomy-of-a-3b-step/</link><guid>https://notebook.tines.com/writing/anatomy-of-a-3b-step/</guid><pubDate>Mon, 16 Nov 2026 00:00:00 GMT</pubDate><dc:creator>Shayon Mukherjee</dc:creator><description>From building the image to tearing down the sandbox, how we give each 3B step a fresh Linux environment.</description><content:encoded><![CDATA[<div><p>At Tines we’re building 3B, where people work with an agent to automate enterprise software. Workflows, internal apps, and agents are all made of steps, each a directory with a Dockerfile and a program that reads stdin and writes stdout. The result becomes the input to whatever runs next.</p><p>Each run gets a fresh Linux sandbox, including a fresh gVisor application kernel, which we delete afterwards because we have to treat the program as untrusted even if it appears to have worked. That means its image and persistent files have to survive outside the environment we throw away.</p><h2 id="an-example-step">An example step</h2><p>A step runs the command defined by its image, so the same setup can start a compiled binary, a shell script, or a language runtime. For our example, we’ll follow a step written in Python that looks up IP addresses in an incoming security alert and keeps the answers in SQLite, so later alerts can reuse them. It needs <code>requests</code> in <code>requirements.txt</code>, a volume for its database, and a <a href="https://docs.3b.tines.com/en/articles/15885706-connect-to-your-tools-by-creating-connectors">connector</a> for the threat intelligence API. The API hostname below is illustrative.</p><figure class="code "><figcaption><span>enrich-alert/Dockerfile</span><span class="code-lang">dockerfile</span></figcaption><div class="code-body"><pre class="code-lines" aria-hidden="true">1
2
3
4
5
6
7
8</pre><pre><code><span class="tok-keyword">FROM</span> python:3.13-slim
<span class="tok-keyword">WORKDIR</span> /app
<span class="tok-keyword">COPY</span> requirements.txt ./
<span class="tok-keyword">RUN</span> --mount=type=cache,target=/root/.cache/pip \
    pip install -r requirements.txt
<span class="tok-keyword">COPY</span> --link . .
<span class="tok-keyword">VOLUME</span> [<span class="tok-string">&quot;alerts&quot;</span>]
<span class="tok-keyword">CMD</span> [<span class="tok-string">&quot;python&quot;</span>, <span class="tok-string">&quot;./script.py&quot;</span>]</code></pre></div></figure><figure class="code "><figcaption><span>enrich-alert/script.py</span><span class="code-lang">python</span></figcaption><div class="code-body"><pre class="code-lines" aria-hidden="true">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40</pre><pre><code><span class="tok-keyword">import</span> <span class="tok-name">json</span>
<span class="tok-keyword">import</span> <span class="tok-name">sqlite3</span>
<span class="tok-keyword">import</span> <span class="tok-name">sys</span>
<span class="tok-keyword">from</span> <span class="tok-name">urllib</span><span class="tok-punctuation">.</span><span class="tok-name">parse</span> <span class="tok-keyword">import</span> <span class="tok-name">quote</span>

<span class="tok-keyword">import</span> <span class="tok-name">requests</span>

<span class="tok-name">endpoint</span> <span class="tok-punctuation">=</span> <span class="tok-string">&quot;https://api.threatintel.example/v1/ip&quot;</span>
<span class="tok-name">alert</span> <span class="tok-punctuation">=</span> <span class="tok-name">json</span><span class="tok-punctuation">.</span><span class="tok-function">load</span><span class="tok-punctuation">(</span><span class="tok-name">sys</span><span class="tok-punctuation">.</span><span class="tok-name">stdin</span><span class="tok-punctuation">)</span>
<span class="tok-name">db</span> <span class="tok-punctuation">=</span> <span class="tok-name">sqlite3</span><span class="tok-punctuation">.</span><span class="tok-function">connect</span><span class="tok-punctuation">(</span><span class="tok-string">&quot;/storage/alerts/seen.db&quot;</span><span class="tok-punctuation">)</span>
<span class="tok-name">db</span><span class="tok-punctuation">.</span><span class="tok-function">execute</span><span class="tok-punctuation">(</span>
    <span class="tok-string">&quot;CREATE TABLE IF NOT EXISTS seen &quot;</span>
    <span class="tok-string">&quot;(address TEXT PRIMARY KEY, verdict TEXT)&quot;</span>
<span class="tok-punctuation">)</span>

<span class="tok-name">enrichment</span> <span class="tok-punctuation">=</span> <span class="tok-punctuation">{</span><span class="tok-punctuation">}</span>
<span class="tok-keyword">for</span> <span class="tok-name">address</span> <span class="tok-keyword">in</span> <span class="tok-name">alert</span><span class="tok-punctuation">[</span><span class="tok-string">&quot;addresses&quot;</span><span class="tok-punctuation">]</span>:
    <span class="tok-name">row</span> <span class="tok-punctuation">=</span> <span class="tok-name">db</span><span class="tok-punctuation">.</span><span class="tok-function">execute</span><span class="tok-punctuation">(</span>
        <span class="tok-string">&quot;SELECT verdict FROM seen WHERE address = ?&quot;</span><span class="tok-punctuation">,</span>
        <span class="tok-punctuation">(</span><span class="tok-name">address</span><span class="tok-punctuation">,</span><span class="tok-punctuation">)</span><span class="tok-punctuation">,</span>
    <span class="tok-punctuation">)</span><span class="tok-punctuation">.</span><span class="tok-function">fetchone</span><span class="tok-punctuation">(</span><span class="tok-punctuation">)</span>
    <span class="tok-keyword">if</span> <span class="tok-name">row</span> <span class="tok-keyword">is</span> <span class="tok-number">None</span>:
        <span class="tok-name">response</span> <span class="tok-punctuation">=</span> <span class="tok-name">requests</span><span class="tok-punctuation">.</span><span class="tok-function">get</span><span class="tok-punctuation">(</span>
            <span class="tok-string">f&quot;</span><span class="tok-punctuation">{</span><span class="tok-name">endpoint</span><span class="tok-punctuation">}</span><span class="tok-string">/</span><span class="tok-punctuation">{</span><span class="tok-function">quote</span><span class="tok-punctuation">(</span><span class="tok-name">address</span><span class="tok-punctuation">,</span> <span class="tok-name">safe</span><span class="tok-punctuation">=</span><span class="tok-string">&#x27;&#x27;</span><span class="tok-punctuation">)</span><span class="tok-punctuation">}</span><span class="tok-string">&quot;</span><span class="tok-punctuation">,</span>
            <span class="tok-name">timeout</span><span class="tok-punctuation">=</span><span class="tok-number">10</span><span class="tok-punctuation">,</span>
        <span class="tok-punctuation">)</span>
        <span class="tok-name">response</span><span class="tok-punctuation">.</span><span class="tok-function">raise_for_status</span><span class="tok-punctuation">(</span><span class="tok-punctuation">)</span>
        <span class="tok-name">verdict</span> <span class="tok-punctuation">=</span> <span class="tok-name">response</span><span class="tok-punctuation">.</span><span class="tok-function">json</span><span class="tok-punctuation">(</span><span class="tok-punctuation">)</span><span class="tok-punctuation">[</span><span class="tok-string">&quot;verdict&quot;</span><span class="tok-punctuation">]</span>
        <span class="tok-name">db</span><span class="tok-punctuation">.</span><span class="tok-function">execute</span><span class="tok-punctuation">(</span>
            <span class="tok-string">&quot;INSERT INTO seen VALUES (?, ?)&quot;</span><span class="tok-punctuation">,</span>
            <span class="tok-punctuation">(</span><span class="tok-name">address</span><span class="tok-punctuation">,</span> <span class="tok-name">verdict</span><span class="tok-punctuation">)</span><span class="tok-punctuation">,</span>
        <span class="tok-punctuation">)</span>
    <span class="tok-keyword">else</span>:
        <span class="tok-name">verdict</span> <span class="tok-punctuation">=</span> <span class="tok-name">row</span><span class="tok-punctuation">[</span><span class="tok-number">0</span><span class="tok-punctuation">]</span>
    <span class="tok-name">enrichment</span><span class="tok-punctuation">[</span><span class="tok-name">address</span><span class="tok-punctuation">]</span> <span class="tok-punctuation">=</span> <span class="tok-name">verdict</span>

<span class="tok-name">db</span><span class="tok-punctuation">.</span><span class="tok-function">commit</span><span class="tok-punctuation">(</span><span class="tok-punctuation">)</span>
<span class="tok-name">db</span><span class="tok-punctuation">.</span><span class="tok-function">close</span><span class="tok-punctuation">(</span><span class="tok-punctuation">)</span>
<span class="tok-name">alert</span><span class="tok-punctuation">[</span><span class="tok-string">&quot;enrichment&quot;</span><span class="tok-punctuation">]</span> <span class="tok-punctuation">=</span> <span class="tok-name">enrichment</span>
<span class="tok-function">print</span><span class="tok-punctuation">(</span><span class="tok-name">json</span><span class="tok-punctuation">.</span><span class="tok-function">dumps</span><span class="tok-punctuation">(</span><span class="tok-name">alert</span><span class="tok-punctuation">)</span><span class="tok-punctuation">)</span></code></pre></div></figure><p>In 3B, <code>VOLUME [&quot;alerts&quot;]</code> declares a persistent volume mounted at <code>/storage/alerts</code>. The example leaves out cache expiry, so a cached verdict stays there until it is removed. There’s a connector attached for the threat intelligence service as well. Its API key isn’t passed to Python, though, because the request gets authenticated after leaving the sandbox through a proxy that sits right outside the sandbox.</p><h2 id="running-linux-in-the-sentry">Running Linux in the sentry</h2><p>For a regular container, Linux handles the program’s syscalls in the host kernel, subject to namespace and seccomp restrictions. We use <a href="https://gvisor.dev/users/">gVisor</a> to handle them in a process called the <a href="https://gvisor.dev/docs/">sentry</a>. The sentry implements the Linux interface in Go, with its own task table, virtual memory system, and filesystems, and is itself an unprivileged process on the host.</p><p>So when Python asks for its PID, the sentry already has one in its task table, and if Python writes to a pipe, that just changes a buffer in the sentry. For networking, we use the host’s TCP/IP stack inside a separate network namespace for each sandbox, so connecting a socket does require a host syscall. The sentry asks the gofer, gVisor’s separate file server, for host files such as <code>/etc/resolv.conf</code> over the LisaFS protocol. We also use seccomp to restrict the host syscalls available to the sentry, which limits what it can do if it is compromised.</p><figure class="figure diagram wide"><div class="diagram-frame" role="img" aria-label="The sentry handles the step’s syscalls. It sends LisaFS requests to the gofer and makes its own host syscalls where needed."><svg viewBox="0 0 840 508" xmlns="http://www.w3.org/2000/svg">
  <defs>
    <marker id="diagram-1-sk-h" viewBox="0 0 8 8" refX="7" refY="4" markerWidth="7" markerHeight="7" orient="auto-start-reverse">
      <path d="M0 0 L8 4 L0 8 z" class="d-head"></path>
    </marker>
    <marker id="diagram-1-sk-hh" viewBox="0 0 8 8" refX="7" refY="4" markerWidth="7" markerHeight="7" orient="auto-start-reverse">
      <path d="M0 0 L8 4 L0 8 z" class="d-head-hot"></path>
    </marker>
  </defs>
  <text x="24" y="18" class="d-tag">GUEST</text>
  <rect x="24" y="26" width="168" height="48" rx="8" class="d-box"></rect>
  <text x="38" y="46" class="d-name">python script.py</text>
  <text x="38" y="62" class="d-note">image user, no caps</text>
  <text x="232" y="18" class="d-tag">GUEST KERNEL</text>
  <rect x="232" y="26" width="196" height="48" rx="8" class="d-box-hot"></rect>
  <text x="246" y="46" class="d-name">sentry</text>
  <text x="246" y="62" class="d-note">Linux, in Go, as a process</text>
  <text x="478" y="18" class="d-tag">HOST PROCESS</text>
  <rect x="478" y="26" width="144" height="48" rx="8" class="d-box"></rect>
  <text x="492" y="46" class="d-name">gofer</text>
  <text x="492" y="62" class="d-note">file server</text>
  <text x="664" y="18" class="d-tag">HOST KERNEL</text>
  <rect x="664" y="26" width="152" height="48" rx="8" class="d-box"></rect>
  <text x="678" y="46" class="d-name">Linux kernel</text>
  <text x="678" y="62" class="d-note">host syscalls</text>
  <path d="M96 74 V418" class="d-line d-dash"></path>
  <path d="M322 74 V418" class="d-line d-dash"></path>
  <path d="M550 74 V418" class="d-line d-dash"></path>
  <path d="M740 74 V418" class="d-line d-dash"></path>
  <rect x="368" y="92" width="22" height="316" rx="6" class="d-zone"></rect>
  <text x="379" y="430" text-anchor="middle" class="d-note">seccomp allowlist,</text>
  <text x="379" y="444" text-anchor="middle" class="d-note">namespaces, cgroup</text>
  <text x="108" y="111">getpid()</text>
  <text x="108" y="133" class="d-note">from the sentry’s task table</text>
  <path id="diagram-1-sk-a0" d="M96 118 H320" class="d-line-hot" marker-end="url(#diagram-1-sk-hh)"></path>
  <circle cx="96" cy="118" r="2.5" class="d-head-hot"></circle>
  <text x="108" y="155">futex(FUTEX_WAIT)</text>
  <text x="108" y="177" class="d-note">the thread parks in the sentry</text>
  <path id="diagram-1-sk-a1" d="M96 162 H320" class="d-line-hot" marker-end="url(#diagram-1-sk-hh)"></path>
  <circle cx="96" cy="162" r="2.5" class="d-head-hot"></circle>
  <text x="108" y="199">write(pipe)</text>
  <text x="108" y="221" class="d-note">into a sentry pipe buffer</text>
  <path id="diagram-1-sk-a2" d="M96 206 H320" class="d-line-hot" marker-end="url(#diagram-1-sk-hh)"></path>
  <circle cx="96" cy="206" r="2.5" class="d-head-hot"></circle>
  <text x="108" y="243">mmap(anonymous)</text>
  <text x="108" y="265" class="d-note">its own memory manager</text>
  <path id="diagram-1-sk-a3" d="M96 250 H320" class="d-line-hot" marker-end="url(#diagram-1-sk-hh)"></path>
  <circle cx="96" cy="250" r="2.5" class="d-head-hot"></circle>
  <path id="diagram-1-sk-b3" d="M322 250 H738" class="d-line d-dash" marker-end="url(#diagram-1-sk-h)"></path>
  <text x="732" y="243" text-anchor="end" class="d-note">backing pages, when needed</text>
  <text x="108" y="287">openat("/etc/resolv.conf")</text>
  <text x="108" y="309" class="d-note">a gofer file</text>
  <path id="diagram-1-sk-a4" d="M96 294 H320" class="d-line-hot" marker-end="url(#diagram-1-sk-hh)"></path>
  <circle cx="96" cy="294" r="2.5" class="d-head-hot"></circle>
  <path id="diagram-1-sk-b4" d="M322 294 H548" class="d-line d-dash" marker-end="url(#diagram-1-sk-h)"></path>
  <text x="542" y="287" text-anchor="end" class="d-note">LisaFS request</text>
  <text x="108" y="331">connect(api:443)</text>
  <text x="108" y="353" class="d-note">a host socket in its netns</text>
  <path id="diagram-1-sk-a5" d="M96 338 H320" class="d-line-hot" marker-end="url(#diagram-1-sk-hh)"></path>
  <circle cx="96" cy="338" r="2.5" class="d-head-hot"></circle>
  <path id="diagram-1-sk-b5" d="M322 338 H738" class="d-line d-dash" marker-end="url(#diagram-1-sk-h)"></path>
  <text x="732" y="331" text-anchor="end" class="d-note">its own connect()</text>
  <text x="108" y="375">unshare(CLONE_NEWNS)</text>
  <text x="108" y="397" class="d-note">the step’s filter says no</text>
  <path id="diagram-1-sk-a6" d="M96 382 H320" class="d-line-hot" marker-end="url(#diagram-1-sk-hh)"></path>
  <circle cx="96" cy="382" r="2.5" class="d-head-hot"></circle>
  <text x="334" y="386" class="d-hot">EPERM</text>
  <path d="M24 474 H52" class="d-line-hot"></path>
  <text x="60" y="478" class="d-note">the step’s syscall, handled by the sentry</text>
  <path d="M24 496 H52" class="d-line d-dash"></path>
  <text x="60" y="500" class="d-note">a host syscall or a request to the gofer</text>
  <g class="d-moving">
    <circle r="3.5" class="d-pulse"><animateMotion dur="11.0s" repeatCount="indefinite" calcMode="linear" keyPoints="0;0;1;1;0;0" keyTimes="0;0.0227;0.0636;0.0773;0.1182;1"><mpath href="#diagram-1-sk-a0"></mpath></animateMotion><animate attributeName="opacity" dur="11.0s" repeatCount="indefinite" values="0;0;1;1;0;0" keyTimes="0;0.0218;0.0227;0.1182;0.1191;1"></animate></circle>
    <circle r="3.5" class="d-pulse"><animateMotion dur="11.0s" repeatCount="indefinite" calcMode="linear" keyPoints="0;0;1;1;0;0" keyTimes="0;0.1591;0.2000;0.2136;0.2545;1"><mpath href="#diagram-1-sk-a1"></mpath></animateMotion><animate attributeName="opacity" dur="11.0s" repeatCount="indefinite" values="0;0;1;1;0;0" keyTimes="0;0.1582;0.1591;0.2545;0.2555;1"></animate></circle>
    <circle r="3.5" class="d-pulse"><animateMotion dur="11.0s" repeatCount="indefinite" calcMode="linear" keyPoints="0;0;1;1;0;0" keyTimes="0;0.2955;0.3364;0.3500;0.3909;1"><mpath href="#diagram-1-sk-a2"></mpath></animateMotion><animate attributeName="opacity" dur="11.0s" repeatCount="indefinite" values="0;0;1;1;0;0" keyTimes="0;0.2945;0.2955;0.3909;0.3918;1"></animate></circle>
    <circle r="3.5" class="d-pulse"><animateMotion dur="11.0s" repeatCount="indefinite" calcMode="linear" keyPoints="0;0;1;1;0;0" keyTimes="0;0.4318;0.4727;0.5591;0.6000;1"><mpath href="#diagram-1-sk-a3"></mpath></animateMotion><animate attributeName="opacity" dur="11.0s" repeatCount="indefinite" values="0;0;1;1;0;0" keyTimes="0;0.4309;0.4318;0.6000;0.6009;1"></animate></circle>
    <circle r="3" class="d-head"><animateMotion dur="11.0s" repeatCount="indefinite" calcMode="linear" keyPoints="0;0;1;0;0" keyTimes="0;0.4773;0.5136;0.5500;1"><mpath href="#diagram-1-sk-b3"></mpath></animateMotion><animate attributeName="opacity" dur="11.0s" repeatCount="indefinite" values="0;0;1;1;0;0" keyTimes="0;0.4764;0.4773;0.5500;0.5509;1"></animate></circle>
    <circle r="3.5" class="d-pulse"><animateMotion dur="11.0s" repeatCount="indefinite" calcMode="linear" keyPoints="0;0;1;1;0;0" keyTimes="0;0.5682;0.6091;0.6955;0.7364;1"><mpath href="#diagram-1-sk-a4"></mpath></animateMotion><animate attributeName="opacity" dur="11.0s" repeatCount="indefinite" values="0;0;1;1;0;0" keyTimes="0;0.5673;0.5682;0.7364;0.7373;1"></animate></circle>
    <circle r="3" class="d-head"><animateMotion dur="11.0s" repeatCount="indefinite" calcMode="linear" keyPoints="0;0;1;0;0" keyTimes="0;0.6136;0.6500;0.6864;1"><mpath href="#diagram-1-sk-b4"></mpath></animateMotion><animate attributeName="opacity" dur="11.0s" repeatCount="indefinite" values="0;0;1;1;0;0" keyTimes="0;0.6127;0.6136;0.6864;0.6873;1"></animate></circle>
    <circle r="3.5" class="d-pulse"><animateMotion dur="11.0s" repeatCount="indefinite" calcMode="linear" keyPoints="0;0;1;1;0;0" keyTimes="0;0.7045;0.7455;0.8318;0.8727;1"><mpath href="#diagram-1-sk-a5"></mpath></animateMotion><animate attributeName="opacity" dur="11.0s" repeatCount="indefinite" values="0;0;1;1;0;0" keyTimes="0;0.7036;0.7045;0.8727;0.8736;1"></animate></circle>
    <circle r="3" class="d-head"><animateMotion dur="11.0s" repeatCount="indefinite" calcMode="linear" keyPoints="0;0;1;0;0" keyTimes="0;0.7500;0.7864;0.8227;1"><mpath href="#diagram-1-sk-b5"></mpath></animateMotion><animate attributeName="opacity" dur="11.0s" repeatCount="indefinite" values="0;0;1;1;0;0" keyTimes="0;0.7491;0.7500;0.8227;0.8236;1"></animate></circle>
    <circle r="3.5" class="d-pulse"><animateMotion dur="11.0s" repeatCount="indefinite" calcMode="linear" keyPoints="0;0;1;1;0;0" keyTimes="0;0.8409;0.8818;0.8955;0.9364;1"><mpath href="#diagram-1-sk-a6"></mpath></animateMotion><animate attributeName="opacity" dur="11.0s" repeatCount="indefinite" values="0;0;1;1;0;0" keyTimes="0;0.8400;0.8409;0.9364;0.9373;1"></animate></circle>
  </g>
</svg></div><figcaption>The sentry handles the step’s syscalls. It sends LisaFS requests to the gofer and makes its own host syscalls where needed.</figcaption></figure><p>We use gVisor’s systrap platform to get into the sentry without hardware virtualization. This is useful for our self-hosted offering, since access to KVM isn’t a given in a customer’s environment.</p><p>After the program has run, we stop trusting its sentry and delete it, because a successful exit doesn’t establish that the program didn’t exploit its kernel. So the next execution gets another sentry, even when the same step runs again.</p><h2 id="starting-before-the-job-arrives">Starting before the job arrives</h2><p>We run millions of steps a day across what we call &quot;workers&quot;, each with a sandbox daemon called <code>sandboxd</code> (previously known as <code>gvisord</code>, with a little nod to systemd).</p><p>Before starting the pool, sandboxd creates its own user and network namespaces, so it can set up interfaces and routes there without needing those privileges over the host’s network. Each sandbox then gets a separate network namespace connected to the daemon’s network, where its TCP traffic is directed to our proxy, Pony.</p><p>With that in place, sandboxd prepares a pool of sandboxes ahead of time, starting the sentry and gofer and setting up the network so our runner can wait inside for a command. The runner starts the image’s configured command as the image’s user, with capabilities dropped, so in our example it runs <code>python ./script.py</code>. The slot remains occupied until the sandbox has been removed, including the resources it used outside the sentry.</p><figure class="figure diagram wide"><div class="diagram-frame" role="img" aria-label="Each slot goes through preparation, execution, and teardown before starting a replacement sandbox."><svg viewBox="0 0 720 330" xmlns="http://www.w3.org/2000/svg">
  <defs>
    <marker id="diagram-2-pool-head" viewBox="0 0 8 8" refX="7" refY="4" markerWidth="7" markerHeight="7" orient="auto-start-reverse">
      <path d="M0 0 L8 4 L0 8 z" class="d-head"></path>
    </marker>
    <marker id="diagram-2-pool-head-hot" viewBox="0 0 8 8" refX="7" refY="4" markerWidth="7" markerHeight="7" orient="auto-start-reverse">
      <path d="M0 0 L8 4 L0 8 z" class="d-head-hot"></path>
    </marker>
  </defs>

  <text x="24" y="32" class="d-tag">ONE WORKER, N SLOTS</text>
  <text x="696" y="32" text-anchor="end" class="d-note">each dot is one sandbox</text>

  
  <text x="288" y="58" text-anchor="middle" class="d-note">job arrives</text>
  <path d="M288 64 V84" class="d-line-hot" marker-end="url(#diagram-2-pool-head-hot)"></path>

  
  <rect x="68" y="90" width="140" height="176" rx="8" class="d-box"></rect>
  <rect x="218" y="90" width="140" height="176" rx="8" class="d-box-hot"></rect>
  <rect x="368" y="90" width="140" height="176" rx="8" class="d-box"></rect>
  <rect x="518" y="90" width="140" height="176" rx="8" class="d-box"></rect>

  <text x="80" y="111" class="d-name">preparing</text>
  <text x="80" y="127" class="d-note">boots a sandbox</text>
  <text x="230" y="111" class="d-name">ready</text>
  <text x="230" y="127" class="d-note">has run no step</text>
  <text x="380" y="111" class="d-name">running</text>
  <text x="380" y="127" class="d-note">exactly one step</text>
  <text x="530" y="111" class="d-name">retirement</text>
  <text x="530" y="127" class="d-note">kill and clean up</text>

  
  <g class="d-note">
    <text x="24" y="156">slot 0</text>
    <text x="24" y="184">slot 1</text>
    <text x="24" y="212">slot 2</text>
    <text x="24" y="240">slot 3</text>
  </g>
  <g>
    <path d="M78 152 H648 M78 180 H648 M78 208 H648 M78 236 H648" class="d-line d-dash"></path>
  </g>

  
  <g class="d-moving">
    <circle cy="152" r="5" class="d-pulse">
      <animate attributeName="cx" dur="9s" begin="0s" repeatCount="indefinite" values="138;138;288;288;438;438;588;588;138" keyTimes="0;0.16;0.24;0.42;0.5;0.76;0.84;0.99;1"></animate>
    </circle>
    <circle cy="180" r="5" class="d-pulse">
      <animate attributeName="cx" dur="9s" begin="-2.6s" repeatCount="indefinite" values="138;138;288;288;438;438;588;588;138" keyTimes="0;0.16;0.24;0.42;0.5;0.76;0.84;0.99;1"></animate>
    </circle>
    <circle cy="208" r="5" class="d-pulse">
      <animate attributeName="cx" dur="11s" begin="-6.1s" repeatCount="indefinite" values="138;138;288;288;438;438;588;588;138" keyTimes="0;0.12;0.2;0.3;0.38;0.8;0.88;0.99;1"></animate>
    </circle>
    <circle cy="236" r="5" class="d-pulse">
      <animate attributeName="cx" dur="9.5s" begin="-4.4s" repeatCount="indefinite" values="138;138;288;288;438;438;588;588;138" keyTimes="0;0.16;0.24;0.5;0.58;0.74;0.82;0.99;1"></animate>
    </circle>
  </g>

  
  <path d="M588 268 V290 Q588 300 578 300 H148 Q138 300 138 290 V272" class="d-line" marker-end="url(#diagram-2-pool-head)"></path>
  <text x="363" y="318" text-anchor="middle" class="d-note">a new sandbox in the same slot, only after the old one is gone</text>
</svg></div><figcaption>Each slot goes through preparation, execution, and teardown before starting a replacement sandbox.</figcaption></figure><p>We put many slots on a worker, sized for the CPU and memory it has available, with resources left for the worker’s own processes. A command can exit before everything belonging to it has gone away, which is why cleanup counts towards the slot’s lifetime too.</p><p>We check this with a test that sends 100 requests at once, each running <code>node -v</code> in a fresh sandbox. Those runs use sandboxes already waiting in the pool and can reuse image data cached on the worker. We measure image attachment and command execution separately from the full request and response, which also includes the work before a worker receives the job.</p><h2 id="who-answers-each-path">Who answers each path</h2><p>In our example, the Python installation is part of the base image. When Python reads one of its libraries, the bytes come from the image, but if the program changes that file, its writes go into the sandbox’s private tmpfs. The sentry implements the overlay and supplies <code>/tmp</code> and <code>/proc</code> too, with their temporary state discarded at the end of the run.</p><figure class="figure diagram wide"><div class="diagram-frame" role="img" aria-label="The image and temporary paths are served by the sentry; the gofer serves selected host files and persistent volumes."><svg viewBox="0 0 720 420" xmlns="http://www.w3.org/2000/svg">
  <defs>
    <marker id="diagram-3-mn-h" viewBox="0 0 8 8" refX="7" refY="4" markerWidth="7" markerHeight="7" orient="auto-start-reverse">
      <path d="M0 0 L8 4 L0 8 z" class="d-head"></path>
    </marker>
    <marker id="diagram-3-mn-hh" viewBox="0 0 8 8" refX="7" refY="4" markerWidth="7" markerHeight="7" orient="auto-start-reverse">
      <path d="M0 0 L8 4 L0 8 z" class="d-head-hot"></path>
    </marker>
  </defs>

  <text x="24" y="22" class="d-tag">THE STEP SEES</text>
  <text x="254" y="22" class="d-tag">SERVED BY</text>
  <text x="534" y="22" class="d-tag">BACKED BY</text>

  
  <g class="d-name">
    <text x="24" y="88">/</text>
    <text x="24" y="148">/tmp</text>
    <text x="24" y="178">/proc</text>
    <text x="24" y="278">/storage/alerts</text>
    <text x="24" y="308">/etc/resolv.conf</text>
  </g>
  <g>
    <path d="M150 84 H252" class="d-line" marker-end="url(#diagram-3-mn-h)"></path>
    <path d="M150 144 H252" class="d-line" marker-end="url(#diagram-3-mn-h)"></path>
    <path d="M150 174 H252" class="d-line" marker-end="url(#diagram-3-mn-h)"></path>
    <path d="M150 274 H252" class="d-line" marker-end="url(#diagram-3-mn-h)"></path>
    <path d="M150 304 H252" class="d-line" marker-end="url(#diagram-3-mn-h)"></path>
  </g>

  
  <rect x="240" y="40" width="220" height="162" rx="8" class="d-box-hot"></rect>
  <text x="254" y="61" class="d-name">sentry</text>
  <rect x="254" y="72" width="192" height="24" rx="5" class="d-cell"></rect>
  <text x="264" y="88">overlay: image + tmpfs</text>
  <rect x="254" y="102" width="192" height="24" rx="5" class="d-cell-hot"></rect>
  <text x="264" y="118" class="d-name">EROFS image reader</text>
  <rect x="254" y="132" width="192" height="24" rx="5" class="d-cell"></rect>
  <text x="264" y="148">tmpfs</text>
  <rect x="254" y="162" width="192" height="24" rx="5" class="d-cell"></rect>
  <text x="264" y="178">procfs</text>

  
  <path id="diagram-3-mn-lisa" d="M350 202 V226" class="d-line"></path>
  <text x="360" y="219" class="d-note">LisaFS</text>

  
  <rect x="240" y="226" width="220" height="104" rx="8" class="d-box"></rect>
  <text x="254" y="247" class="d-name">gofer</text>
  <text x="304" y="247" class="d-note">chroot, no AF_INET</text>
  <rect x="254" y="262" width="192" height="24" rx="5" class="d-cell"></rect>
  <text x="264" y="278">Nest: volumes</text>
  <rect x="254" y="292" width="192" height="24" rx="5" class="d-cell"></rect>
  <text x="264" y="308">fsgofer: host files</text>

  
  <rect x="520" y="84" width="180" height="60" rx="8" class="d-box"></rect>
  <text x="534" y="106" class="d-name">storage gateway</text>
  <text x="534" y="122" class="d-note">verify, cache, relay</text>

  <rect x="520" y="281" width="130" height="46" rx="8" class="d-box"></rect>
  <text x="534" y="301" class="d-name">worker disk</text>
  <text x="534" y="317" class="d-note">rootfs, /work</text>

  <rect x="240" y="364" width="460" height="40" rx="8" class="d-box"></rect>
  <text x="254" y="389" class="d-name">Spongeblob</text>

  <path id="diagram-3-mn-img" d="M446 114 H518" class="d-line-hot" marker-end="url(#diagram-3-mn-hh)"></path>
  <text x="462" y="107" class="d-note">ranges</text>
  <path id="diagram-3-mn-nest" d="M446 274 H490 V134 H518" class="d-line" marker-end="url(#diagram-3-mn-h)"></path>
  <text x="498" y="228" class="d-note">relay</text>
  <path d="M446 304 H518" class="d-line" marker-end="url(#diagram-3-mn-h)"></path>
  <path id="diagram-3-mn-blob" d="M680 144 V362" class="d-line" marker-end="url(#diagram-3-mn-h)"></path>

  <g class="d-moving">
    <circle r="3" class="d-pulse"><animateMotion dur="2.4s" repeatCount="indefinite" keyPoints="0;1;0" keyTimes="0;0.5;1" calcMode="linear"><mpath href="#diagram-3-mn-img"></mpath></animateMotion></circle>
    <circle r="3" class="d-pulse"><animateMotion dur="3.6s" begin="-0.8s" repeatCount="indefinite" keyPoints="0;1;0" keyTimes="0;0.5;1" calcMode="linear"><mpath href="#diagram-3-mn-nest"></mpath></animateMotion></circle>
    <circle r="3" class="d-pulse"><animateMotion dur="3s" begin="-0.3s" repeatCount="indefinite" keyPoints="0;1;0" keyTimes="0;0.5;1" calcMode="linear"><mpath href="#diagram-3-mn-blob"></mpath></animateMotion></circle>
    <circle r="3" class="d-pulse"><animateMotion dur="1.8s" begin="-0.5s" repeatCount="indefinite" keyPoints="0;1;0" keyTimes="0;0.5;1" calcMode="linear"><mpath href="#diagram-3-mn-lisa"></mpath></animateMotion></circle>
  </g>
</svg></div><figcaption>The image and temporary paths are served by the sentry; the gofer serves selected host files and persistent volumes.</figcaption></figure><h2 id="files-that-outlive-the-sandbox">Files that outlive the sandbox</h2><p>That’s different from, say, our SQLite file, which needs to survive the run so the next alert can find the answers this one saved. So when SQLite opens <code>/storage/alerts/seen.db</code>, the sentry sends a LisaFS request to our gofer’s Nest backend, which we wrote to serve persistent volumes. Nest reads from a snapshot and keeps this run’s writes separately until the run finishes. Concurrent runs each read their own snapshot, so SQLite’s locks only coordinate processes within a run. Stock gVisor code in the same gofer serves a few host files, such as <code>/etc/resolv.conf</code>.</p><p>Nest talks to Spongeblob through sandboxd’s storage gateway, which relays its requests. Spongeblob is our internal Rust database for filesystem data and snapshots. It powers a lot of other primitives we are able to offer at the filesystem level. Something we will cover in depth in another post.</p><p>Between Spongeblob, Nest, and Pony, we may have used up the sensible names on <code>runner</code> I think 😆.</p><p>A lookup like this may only need a few pages from the database. Our earlier implementation copied a volume file into a local backing file before returning from <code>open</code>, so gVisor could read and map it through a host file descriptor. But one production database had grown to 933 MB, which meant copying all of it before SQLite could look up a single row.</p><p>So we moved reads of committed files into the sentry’s page cache. Opening the file no longer copies its contents. Nest fetches the missing data from the snapshot when SQLite reads it, and if SQLite comes back to those pages, the sentry already has them. Writes stay separate, so Nest only has to publish the blocks the step changed.</p><p>That works well for a few pages, but if, say, the step scans the whole database, small fetches turn into one network wait after another. For larger reads, Nest now fetches 2 MiB windows, splitting each window across 4 concurrent requests, and reads ahead while the program keeps reading in order. We also changed the sentry to reclaim a quarter of its cache at a time under memory pressure, taking chunks read once before ones the program has returned to. So a scan can make progress without keeping the entire file in memory, and the cache gives memory back as the program needs it.</p><p>We measured those read-ahead and reclamation changes with the sentry page cache enabled in both versions. In our file-I/O benchmark, 200 cold lookups in a 900 MB database went from 8.42 seconds to 2.27 seconds. A SQL scan went from 13.0 seconds to 3.13 seconds, while a single point read stayed at 37 ms.</p><figure class="figure diagram wide"><div class="diagram-frame" role="img" aria-label="Selected cold-read benchmarks before and after the read-ahead changes, with the sentry page cache enabled in both. Median of 3 runs, 2 worker CPUs, cold storage packs, and 1 ms of added network latency."><svg viewBox="0 0 720 368" xmlns="http://www.w3.org/2000/svg">
  <text x="24" y="30" class="d-tag" text-anchor="start">NEST DATABASE READS</text>
  <text x="696" y="30" class="d-note" text-anchor="end">median · seconds</text>
  <rect x="78" y="53" width="10" height="10" rx="1" class="d-cell"></rect>
  <text x="98" y="62" class="d-note" text-anchor="start">before</text>
  <rect x="204" y="53" width="10" height="10" rx="1" class="d-cell-hot"></rect>
  <text x="224" y="62" class="d-note d-hot" text-anchor="start">after</text>
  <path d="M78 103 L78 174" class="d-line d-dash"></path>
  <path d="M78 223 L78 290" class="d-line d-dash"></path>
  <text x="78" y="316" class="d-note" text-anchor="middle">0</text>
  <path d="M266 103 L266 174" class="d-line d-dash"></path>
  <path d="M266 223 L266 290" class="d-line d-dash"></path>
  <text x="266" y="316" class="d-note" text-anchor="middle">5</text>
  <path d="M454 103 L454 174" class="d-line d-dash"></path>
  <path d="M454 223 L454 290" class="d-line d-dash"></path>
  <text x="454" y="316" class="d-note" text-anchor="middle">10</text>
  <path d="M642 103 L642 174" class="d-line d-dash"></path>
  <path d="M642 223 L642 290" class="d-line d-dash"></path>
  <text x="642" y="316" class="d-note" text-anchor="middle">15</text>
  <text x="78" y="88" class="d-name" text-anchor="start">200 lookups</text>
  <rect x="78" y="104" width="316.63" height="24" rx="3" class="d-cell"></rect>
  <text x="406.63" y="121" class="d-name" text-anchor="start">8.42</text>
  <rect x="78" y="139" width="85.3896" height="24" rx="3" class="d-cell-hot"></rect>
  <text x="175.39" y="156" class="d-name d-hot" text-anchor="start">2.27</text>
  <text x="78" y="208" class="d-name" text-anchor="start">SQL scan</text>
  <rect x="78" y="224" width="488.762" height="24" rx="3" class="d-cell"></rect>
  <text x="578.762" y="241" class="d-name" text-anchor="start">13.00</text>
  <rect x="78" y="259" width="117.65" height="24" rx="3" class="d-cell-hot"></rect>
  <text x="207.65" y="276" class="d-name d-hot" text-anchor="start">3.13</text>
  <path d="M78 290 L642 290" class="d-line"></path>
  <text x="360" y="348" class="d-note" text-anchor="middle">900 MB database · cold reads</text>
</svg></div><figcaption>Selected cold-read benchmarks before and after the read-ahead changes, with the sentry page cache enabled in both. Median of 3 runs, 2 worker CPUs, cold storage packs, and 1 ms of added network latency.</figcaption></figure><p>In a separate contention test, a hot reader sharing its step’s CPU with 2 scans took 30% longer because the faster scans used more of that CPU, even though the whole step finished sooner.</p><h2 id="building-the-image">Building the image</h2><p>When someone is working on a workflow and the agent edits a step’s code or Dockerfile, we build an updated image for that step as they iterate. That build runs in a sandbox as well, and BuildKit (with a couple of internal patches) turns the Dockerfile into image layers.</p><p>We use <a href="https://nydus.dev/docs/concepts/rafs/">Nydus</a> to export the image in its EROFS-compatible format. The file data in each layer is split into separately compressed, hashed chunks, while the filesystem metadata is written in <a href="https://docs.kernel.org/filesystems/erofs.html">EROFS</a>, a read-only format that records the directory tree and references to the file data. Nydus merges the layers’ metadata into one bootstrap file describing the finished image.</p><p>The builder pushes the bootstrap and layer blobs through the storage gateway’s registry, which checks the uploaded blobs against their digests before publishing them in Spongeblob.</p><figure class="figure diagram wide"><div class="diagram-frame" role="img" aria-label="The builder pushes through the gateway’s registry, where uploads are verified before publication."><svg viewBox="0 0 720 380" xmlns="http://www.w3.org/2000/svg">
  <defs>
    <marker id="diagram-5-bd-h" viewBox="0 0 8 8" refX="7" refY="4" markerWidth="7" markerHeight="7" orient="auto-start-reverse">
      <path d="M0 0 L8 4 L0 8 z" class="d-head"></path>
    </marker>
  </defs>

  
  <rect x="20" y="20" width="680" height="152" rx="12" class="d-zone"></rect>
  <text x="36" y="42" class="d-tag">BUILDER SANDBOX, THROWN AWAY AFTER</text>

  <rect x="40" y="70" width="140" height="70" rx="8" class="d-box"></rect>
  <text x="54" y="94" class="d-name">Dockerfile</text>
  <text x="54" y="111" class="d-note">FROM, RUN, COPY</text>

  <rect x="210" y="70" width="150" height="70" rx="8" class="d-box"></rect>
  <text x="224" y="94" class="d-name">builder</text>
  <text x="224" y="111" class="d-note">each RUN nested,</text>
  <text x="224" y="126" class="d-note">network via Pony</text>

  <rect x="390" y="70" width="140" height="70" rx="8" class="d-box"></rect>
  <text x="404" y="94" class="d-name">layers</text>
  <text x="404" y="111" class="d-note">timestamps clamped</text>

  <rect x="560" y="70" width="124" height="70" rx="8" class="d-box-hot"></rect>
  <text x="574" y="94" class="d-name">chunked</text>
  <text x="574" y="111" class="d-note">EROFS metadata</text>
  <text x="574" y="126" class="d-note">+ chunk blobs</text>

  <path id="diagram-5-bd-1" d="M180 105 H208" class="d-line" marker-end="url(#diagram-5-bd-h)"></path>
  <path id="diagram-5-bd-2" d="M360 105 H388" class="d-line" marker-end="url(#diagram-5-bd-h)"></path>
  <path id="diagram-5-bd-3" d="M530 105 H558" class="d-line" marker-end="url(#diagram-5-bd-h)"></path>
  <path id="diagram-5-bd-4" d="M622 140 V240" class="d-line" marker-end="url(#diagram-5-bd-h)"></path>
  <text x="630" y="200" class="d-note">push</text>

  
  <rect x="20" y="214" width="680" height="152" rx="12" class="d-ghost"></rect>
  <text x="36" y="356" class="d-tag">STORAGE GATEWAY AND SPONGEBLOB</text>

  <rect x="560" y="242" width="124" height="70" rx="8" class="d-box"></rect>
  <text x="574" y="266" class="d-name">registry</text>
  <text x="574" y="283" class="d-note">for this build</text>
  <text x="574" y="298" class="d-note">only</text>

  <rect x="390" y="242" width="140" height="70" rx="8" class="d-box"></rect>
  <text x="404" y="266" class="d-name">staging</text>
  <text x="404" y="283" class="d-note">private to this</text>
  <text x="404" y="298" class="d-note">attempt</text>

  <rect x="210" y="242" width="150" height="70" rx="8" class="d-box"></rect>
  <text x="224" y="266" class="d-name">tenant pool</text>
  <text x="224" y="283" class="d-note">one object per</text>
  <text x="224" y="298" class="d-note">digest</text>

  <rect x="40" y="242" width="140" height="70" rx="8" class="d-box"></rect>
  <text x="54" y="266" class="d-name">published</text>
  <text x="54" y="283" class="d-note">image, plus the</text>
  <text x="54" y="298" class="d-note">cache pointer</text>

  <path id="diagram-5-bd-5" d="M560 277 H532" class="d-line" marker-end="url(#diagram-5-bd-h)"></path>
  <path id="diagram-5-bd-6" d="M390 277 H362" class="d-line" marker-end="url(#diagram-5-bd-h)"></path>
  <path id="diagram-5-bd-7" d="M210 277 H182" class="d-line" marker-end="url(#diagram-5-bd-h)"></path>
  <text x="376" y="332" text-anchor="middle" class="d-note">copied only once the bytes match the digest</text>

  
  <path d="M285 242 V142" class="d-line d-dash" marker-end="url(#diagram-5-bd-h)"></path>
  <text x="295" y="197" class="d-note">this step’s previous cache, mounted lazily</text>

  
  <g class="d-moving">
    <circle r="4" class="d-pulse">
      <animateMotion dur="7s" repeatCount="indefinite" keyPoints="0;0;1;1" keyTimes="0;0.02;0.1;1" calcMode="linear"><mpath href="#diagram-5-bd-1"></mpath></animateMotion>
      <animate attributeName="opacity" dur="7s" repeatCount="indefinite" values="0;1;1;0;0" keyTimes="0;0.02;0.1;0.11;1"></animate>
    </circle>
    <circle r="4" class="d-pulse">
      <animateMotion dur="7s" repeatCount="indefinite" keyPoints="0;0;1;1" keyTimes="0;0.14;0.22;1" calcMode="linear"><mpath href="#diagram-5-bd-2"></mpath></animateMotion>
      <animate attributeName="opacity" dur="7s" repeatCount="indefinite" values="0;0;1;1;0;0" keyTimes="0;0.139;0.14;0.22;0.23;1"></animate>
    </circle>
    <circle r="4" class="d-pulse">
      <animateMotion dur="7s" repeatCount="indefinite" keyPoints="0;0;1;1" keyTimes="0;0.26;0.34;1" calcMode="linear"><mpath href="#diagram-5-bd-3"></mpath></animateMotion>
      <animate attributeName="opacity" dur="7s" repeatCount="indefinite" values="0;0;1;1;0;0" keyTimes="0;0.259;0.26;0.34;0.35;1"></animate>
    </circle>
    <circle r="4" class="d-pulse">
      <animateMotion dur="7s" repeatCount="indefinite" keyPoints="0;0;1;1" keyTimes="0;0.38;0.5;1" calcMode="linear"><mpath href="#diagram-5-bd-4"></mpath></animateMotion>
      <animate attributeName="opacity" dur="7s" repeatCount="indefinite" values="0;0;1;1;0;0" keyTimes="0;0.379;0.38;0.5;0.51;1"></animate>
    </circle>
    <circle r="4" class="d-pulse">
      <animateMotion dur="7s" repeatCount="indefinite" keyPoints="0;0;1;1" keyTimes="0;0.56;0.64;1" calcMode="linear"><mpath href="#diagram-5-bd-5"></mpath></animateMotion>
      <animate attributeName="opacity" dur="7s" repeatCount="indefinite" values="0;0;1;1;0;0" keyTimes="0;0.559;0.56;0.64;0.65;1"></animate>
    </circle>
    <circle r="4" class="d-pulse">
      <animateMotion dur="7s" repeatCount="indefinite" keyPoints="0;0;1;1" keyTimes="0;0.7;0.78;1" calcMode="linear"><mpath href="#diagram-5-bd-6"></mpath></animateMotion>
      <animate attributeName="opacity" dur="7s" repeatCount="indefinite" values="0;0;1;1;0;0" keyTimes="0;0.699;0.7;0.78;0.79;1"></animate>
    </circle>
    <circle r="4" class="d-pulse">
      <animateMotion dur="7s" repeatCount="indefinite" keyPoints="0;0;1;1" keyTimes="0;0.84;0.92;1" calcMode="linear"><mpath href="#diagram-5-bd-7"></mpath></animateMotion>
      <animate attributeName="opacity" dur="7s" repeatCount="indefinite" values="0;0;1;1;0;0" keyTimes="0;0.839;0.84;0.92;0.93;1"></animate>
    </circle>
  </g>
</svg></div><figcaption>The builder pushes through the gateway’s registry, where uploads are verified before publication.</figcaption></figure><p>Layer blobs are stored by digest within a tenant, so images can share a base layer or an identical <em>output layer</em> produced by another build. Spongeblob also stores a step’s build cache, including the pip directory in our example, so later builds can read cached layers and reuse saved cache directories as well.</p><figure class="figure diagram wide"><div class="diagram-frame" role="img" aria-label="The bootstrap describes the final tree and references file contents in shared layer blobs."><svg viewBox="0 0 720 430" xmlns="http://www.w3.org/2000/svg">
  <defs>
    <marker id="diagram-6-im-hh" viewBox="0 0 8 8" refX="7" refY="4" markerWidth="7" markerHeight="7" orient="auto-start-reverse">
      <path d="M0 0 L8 4 L0 8 z" class="d-head-hot"></path>
    </marker>
    <marker id="diagram-6-im-h" viewBox="0 0 8 8" refX="7" refY="4" markerWidth="7" markerHeight="7" orient="auto-start-reverse">
      <path d="M0 0 L8 4 L0 8 z" class="d-head"></path>
    </marker>
  </defs>

  
  <rect x="20" y="20" width="310" height="340" rx="10" class="d-box-hot"></rect>
  <text x="36" y="45" class="d-name">bootstrap</text>
  <text x="120" y="45" class="d-note">EROFS, no file contents</text>

  <text x="36" y="74" class="d-tag">DEVICES</text>
  <rect x="36" y="82" width="278" height="22" rx="4" class="d-cell"></rect>
  <text x="46" y="97">1  a base image layer</text>
  <rect x="36" y="108" width="278" height="22" rx="4" class="d-cell"></rect>
  <text x="46" y="123">2  pip install layer</text>
  <rect x="36" y="134" width="278" height="22" rx="4" class="d-cell"></rect>
  <text x="46" y="149">3  COPY . layer</text>

  <text x="36" y="182" class="d-tag">INODES</text>
  <rect x="36" y="190" width="278" height="40" rx="4" class="d-cell"></rect>
  <text x="46" y="206">…/python3.13/json/__init__.py</text>
  <text x="46" y="222" class="d-note">chunk: device 1, block 58</text>
  <rect x="36" y="236" width="278" height="40" rx="4" class="d-cell"></rect>
  <text x="46" y="252">…/site-packages/requests/api.py</text>
  <text x="46" y="268" class="d-note">chunk: device 2, block 3</text>
  <rect x="36" y="282" width="278" height="40" rx="4" class="d-cell"></rect>
  <text x="46" y="298">/app/script.py</text>
  <text x="46" y="314" class="d-note">chunk: device 3, block 0</text>
  <text x="36" y="344" class="d-note">and every other inode and directory</text>

  
  <text x="380" y="68" class="d-name">blob 1</text>
  <text x="440" y="68" class="d-note">python:3.13-slim</text>
  <g>
    <rect x="380" y="80" width="22" height="22" rx="3" class="d-cell"></rect>
    <rect x="404" y="80" width="22" height="22" rx="3" class="d-cell"></rect>
    <rect x="428" y="80" width="22" height="22" rx="3" class="d-cell"></rect>
    <rect x="452" y="80" width="22" height="22" rx="3" class="d-cell"></rect>
    <rect x="476" y="80" width="22" height="22" rx="3" class="d-cell"></rect>
    <rect x="500" y="80" width="22" height="22" rx="3" class="d-cell"></rect>
    <rect x="524" y="80" width="22" height="22" rx="3" class="d-cell"></rect>
    <rect x="548" y="80" width="22" height="22" rx="3" class="d-cell"></rect>
    <rect x="572" y="80" width="22" height="22" rx="3" class="d-cell"></rect>
    <rect x="596" y="80" width="22" height="22" rx="3" class="d-cell-hot"></rect>
    <rect x="620" y="80" width="22" height="22" rx="3" class="d-cell"></rect>
    <rect x="644" y="80" width="22" height="22" rx="3" class="d-cell"></rect>
    <rect x="668" y="80" width="22" height="22" rx="3" class="d-cell"></rect>
  </g>

  <text x="380" y="178" class="d-name">blob 2</text>
  <text x="440" y="178" class="d-note">pip install</text>
  <g>
    <rect x="380" y="190" width="22" height="22" rx="3" class="d-cell"></rect>
    <rect x="404" y="190" width="22" height="22" rx="3" class="d-cell"></rect>
    <rect x="428" y="190" width="22" height="22" rx="3" class="d-cell"></rect>
    <rect x="452" y="190" width="22" height="22" rx="3" class="d-cell-hot"></rect>
    <rect x="476" y="190" width="22" height="22" rx="3" class="d-cell"></rect>
    <rect x="500" y="190" width="22" height="22" rx="3" class="d-cell"></rect>
    <rect x="524" y="190" width="22" height="22" rx="3" class="d-cell"></rect>
    <rect x="548" y="190" width="22" height="22" rx="3" class="d-cell"></rect>
  </g>

  <text x="380" y="288" class="d-name">blob 3</text>
  <text x="440" y="288" class="d-note">COPY .</text>
  <g>
    <rect x="380" y="300" width="22" height="22" rx="3" class="d-cell-hot"></rect>
    <rect x="404" y="300" width="22" height="22" rx="3" class="d-cell"></rect>
  </g>

  <text x="380" y="140" class="d-note">each cell is one compressed chunk</text>
  <text x="380" y="155" class="d-note">with its own digest</text>

  
  <path d="M314 214 H346 V116 H607 V106" class="d-line-hot" marker-end="url(#diagram-6-im-hh)"></path>
  <path d="M314 260 H356 V226 H463 V216" class="d-line-hot" marker-end="url(#diagram-6-im-hh)"></path>
  <path d="M314 311 H376" class="d-line-hot" marker-end="url(#diagram-6-im-hh)"></path>

  
  <rect x="380" y="368" width="300" height="46" rx="8" class="d-ghost"></rect>
  <text x="394" y="388" class="d-name">another step’s image</text>
  <text x="394" y="404" class="d-note">same base, so the same blob 1</text>
  <path d="M680 391 H704 V91 H694" class="d-line d-dash" marker-end="url(#diagram-6-im-h)"></path>
</svg></div><figcaption>The bootstrap describes the final tree and references file contents in shared layer blobs.</figcaption></figure><p>Loading the bootstrap gives the sentry the directory tree up front, so a fresh run can get started with the pages its program touches during startup while the rest of the image stays in Spongeblob until needed.</p><p>We use this for the builder too. In one production trace, BuildKit found a cache hit in about a quarter of a second, then spent 23.5 seconds downloading and unpacking parent layers for a <code>COPY</code> that took 0.03 seconds. The builder was fresh, so it still had to reconstruct the files those cached instructions had produced. We changed it to mount the cached layers through the same lazy image path we use for executions.</p><p>That made the next edit faster, but a sequence of edits showed something else. A warm build could reuse those layers and then fail to export them into its own cache. The exporter dropped records it couldn’t load, along with the records depending on them, so one build saved only 3 of its 14 layers. The edit after that had to compile dependencies again.</p><p>Our benchmark had only checked the first edit after a cold build, so it missed this. We changed the exporter to try the other results for a cache record when one couldn’t load, and added a regression test that makes 2 successive edits. In the five-build comparison below, the third and fifth builds fell from 51 seconds each to 7.9 seconds each. The first build still took about the same time.</p><figure class="figure diagram wide"><div class="diagram-frame" role="img" aria-label="Five consecutive builds of the same Rust fixture, before and after the cache-export fix. Each point is an individual benchmark build duration, in seconds."><svg viewBox="0 0 720 355" xmlns="http://www.w3.org/2000/svg">
  <text x="24" y="30" class="d-tag" text-anchor="start">SUCCESSIVE BUILDS</text>
  <text x="696" y="30" class="d-note" text-anchor="end">individual builds · seconds</text>
  <path d="M78 58 L106 58" class="d-line d-dash"></path>
  <circle cx="92" cy="58" r="3.5" class="d-box"></circle>
  <text x="116" y="62" class="d-note" text-anchor="start">before</text>
  <path d="M204 58 L232 58" class="d-line-hot"></path>
  <rect x="214.5" y="54.5" width="7" height="7" rx="1" class="d-cell-hot"></rect>
  <text x="242" y="62" class="d-note d-hot" text-anchor="start">after</text>
  <path d="M78 290 L642 290" class="d-line"></path>
  <text x="64" y="294" class="d-note" text-anchor="end">0</text>
  <path d="M78 240 L642 240" class="d-line d-dash"></path>
  <text x="64" y="244" class="d-note" text-anchor="end">30</text>
  <path d="M78 190 L642 190" class="d-line d-dash"></path>
  <text x="64" y="194" class="d-note" text-anchor="end">60</text>
  <path d="M78 140 L642 140" class="d-line d-dash"></path>
  <text x="64" y="144" class="d-note" text-anchor="end">90</text>
  <path d="M78 90 L642 90" class="d-line d-dash"></path>
  <text x="64" y="94" class="d-note" text-anchor="end">120</text>
  <text x="78" y="311" class="d-note" text-anchor="middle">1</text>
  <text x="219" y="311" class="d-note" text-anchor="middle">2</text>
  <text x="360" y="311" class="d-note" text-anchor="middle">3</text>
  <text x="501" y="311" class="d-note" text-anchor="middle">4</text>
  <text x="642" y="311" class="d-note" text-anchor="middle">5</text>
  <path d="M78 101.667 L219 271.667 L360 205 L501 275.833 L642 205" class="d-line d-dash"></path>
  <circle cx="78" cy="101.667" r="4" class="d-box"></circle>
  <circle cx="219" cy="271.667" r="4" class="d-box"></circle>
  <circle cx="360" cy="205" r="4" class="d-box"></circle>
  <text x="370" y="195" class="d-name" text-anchor="start">51</text>
  <circle cx="501" cy="275.833" r="4" class="d-box"></circle>
  <circle cx="642" cy="205" r="4" class="d-box"></circle>
  <text x="652" y="195" class="d-name" text-anchor="start">51</text>
  <path d="M78 103.333 L219 270.667 L360 276.833 L501 275 L642 276.833" class="d-line-hot"></path>
  <rect x="74.5" y="99.8333" width="7" height="7" rx="1" class="d-cell-hot"></rect>
  <rect x="215.5" y="267.167" width="7" height="7" rx="1" class="d-cell-hot"></rect>
  <rect x="356.5" y="273.333" width="7" height="7" rx="1" class="d-cell-hot"></rect>
  <text x="370" y="266.833" class="d-name d-hot" text-anchor="start">7.9</text>
  <rect x="497.5" y="271.5" width="7" height="7" rx="1" class="d-cell-hot"></rect>
  <rect x="638.5" y="273.333" width="7" height="7" rx="1" class="d-cell-hot"></rect>
  <text x="652" y="266.833" class="d-name d-hot" text-anchor="start">7.9</text>
  <text x="90" y="81.6667" class="d-note" text-anchor="start">113 / 112 s</text>
  <text x="360" y="337" class="d-note" text-anchor="middle">build in sequence</text>
</svg></div><figcaption>Five consecutive builds of the same Rust fixture, before and after the cache-export fix. Each point is an individual benchmark build duration, in seconds.</figcaption></figure><h2 id="faulting-the-image-in">Faulting the image in</h2><p>At this point, we have a fresh sandbox waiting on the worker and an image stored in Spongeblob. When it’s time to run the step, sandboxd attaches the image to the waiting sentry. The sentry reads the EROFS bootstrap for filesystem metadata, so it can answer a lookup or <code>stat</code> straight away, then requests file data through the storage gateway as reads and page faults need it. So the step can start as soon as the data it needs is available, without waiting for us to download and unpack the entire image.</p><p>There are a lot of “lazy design decisions” that can be found in these parts of the system. An image-backed fault reaches gVisor’s EROFS reader, which uses the bootstrap to find the requested blob range and asks the gateway to prepare it. If those bytes aren’t verified yet, the gateway reads the compressed chunks from Spongeblob, decompresses them into its local backing file, and checks their digests. It returns a read-only descriptor and the range now verified, and the sentry validates that reply before mapping the file and letting the program continue.</p><figure class="figure diagram wide"><div class="diagram-frame" role="img" aria-label="Illustrative cold fault: the gateway fetches and verifies missing data before the sentry maps the page. A later fault within the ready range needs no gateway request."><svg viewBox="0 0 720 540" xmlns="http://www.w3.org/2000/svg">
  <defs>
    <marker id="diagram-8-ft-h" viewBox="0 0 8 8" refX="7" refY="4" markerWidth="7" markerHeight="7" orient="auto-start-reverse">
      <path d="M0 0 L8 4 L0 8 z" class="d-head"></path>
    </marker>
    <marker id="diagram-8-ft-hh" viewBox="0 0 8 8" refX="7" refY="4" markerWidth="7" markerHeight="7" orient="auto-start-reverse">
      <path d="M0 0 L8 4 L0 8 z" class="d-head-hot"></path>
    </marker>
  </defs>

  
  <rect x="10" y="20" width="120" height="36" rx="8" class="d-box"></rect>
  <text x="70" y="43" text-anchor="middle" class="d-name">step process</text>
  <rect x="190" y="20" width="120" height="36" rx="8" class="d-box-hot"></rect>
  <text x="250" y="43" text-anchor="middle" class="d-name">sentry</text>
  <rect x="385" y="20" width="150" height="36" rx="8" class="d-box"></rect>
  <text x="460" y="43" text-anchor="middle" class="d-name">storage gateway</text>
  <rect x="590" y="20" width="120" height="36" rx="8" class="d-box"></rect>
  <text x="650" y="43" text-anchor="middle" class="d-name">Spongeblob</text>

  <path d="M70 56 V528 M250 56 V528 M460 56 V528 M650 56 V528" class="d-line d-dash"></path>

  
  <path d="M70 100 H248" class="d-line" marker-end="url(#diagram-8-ft-h)"></path>
  <text x="160" y="92" text-anchor="middle" class="d-note">fault in an image page</text>

  <text x="262" y="122" class="d-note">→ device 1, block 58</text>
  <text x="262" y="137" class="d-note">not in the ready bitmap</text>
  <g>
    <rect x="262" y="145" width="14" height="12" rx="2" class="d-cell"></rect>
    <rect x="278" y="145" width="14" height="12" rx="2" class="d-cell"></rect>
    <rect x="294" y="145" width="14" height="12" rx="2" class="d-cell"></rect>
    <rect x="310" y="145" width="14" height="12" rx="2" class="d-cell"></rect>
    <rect x="326" y="145" width="14" height="12" rx="2" class="d-cell"></rect>
    <rect x="342" y="145" width="14" height="12" rx="2" class="d-cell"></rect>
    <rect x="358" y="145" width="14" height="12" rx="2" class="d-cell"></rect>
    <rect x="374" y="145" width="14" height="12" rx="2" class="d-cell"></rect>
  </g>
  <g opacity="0">
    <rect x="294" y="145" width="14" height="12" rx="2" class="d-cell-hot"></rect>
    <rect x="310" y="145" width="14" height="12" rx="2" class="d-cell-hot"></rect>
    <rect x="326" y="145" width="14" height="12" rx="2" class="d-cell-hot"></rect>
    <rect x="342" y="145" width="14" height="12" rx="2" class="d-cell-hot"></rect>
    <animate attributeName="opacity" dur="8s" repeatCount="indefinite" values="0;0;1;1" keyTimes="0;0.68;0.69;1"></animate>
  </g>

  <path d="M250 184 H458" class="d-line-hot" marker-end="url(#diagram-8-ft-hh)"></path>
  <text x="355" y="176" text-anchor="middle" class="d-note">{device, offset, ≤ 64 KiB}</text>

  <text x="472" y="214" class="d-note">chunk absent on worker</text>

  <path d="M460 250 H648" class="d-line" marker-end="url(#diagram-8-ft-h)"></path>
  <text x="555" y="242" text-anchor="middle" class="d-note">read compressed chunk</text>
  <path d="M650 284 H462" class="d-line d-dash" marker-end="url(#diagram-8-ft-h)"></path>
  <text x="555" y="276" text-anchor="middle" class="d-note">bytes</text>

  <text x="472" y="316" class="d-note">decompress into the</text>
  <text x="472" y="331" class="d-note">backing file, verify</text>
  <g>
    <rect x="472" y="342" width="20" height="14" rx="2" class="d-cell"></rect>
    <rect x="496" y="342" width="20" height="14" rx="2" class="d-cell"></rect>
    <rect x="520" y="342" width="20" height="14" rx="2" class="d-cell"></rect>
    <rect x="544" y="342" width="20" height="14" rx="2" class="d-cell"></rect>
    <rect x="568" y="342" width="20" height="14" rx="2" class="d-cell"></rect>
    <rect x="592" y="342" width="20" height="14" rx="2" class="d-cell"></rect>
  </g>
  <g opacity="0">
    <rect x="520" y="342" width="20" height="14" rx="2" class="d-cell-hot"></rect>
    <animate attributeName="opacity" dur="8s" repeatCount="indefinite" values="0;0;1;1" keyTimes="0;0.52;0.53;1"></animate>
  </g>
  <text x="472" y="374" class="d-note">backing file on the worker</text>

  <path d="M460 394 H252" class="d-line-hot" marker-end="url(#diagram-8-ft-hh)"></path>
  <text x="355" y="386" text-anchor="middle" class="d-note">{ok, start, end} + fd</text>

  <text x="262" y="422" class="d-note">check the fd and range,</text>
  <text x="262" y="437" class="d-note">mmap, mark pages ready</text>

  <path d="M250 480 H72" class="d-line" marker-end="url(#diagram-8-ft-h)"></path>
  <text x="160" y="472" text-anchor="middle" class="d-note">page mapped</text>

  
  <path d="M70 506 H248" class="d-line d-dash" marker-end="url(#diagram-8-ft-h)"></path>
  <path d="M250 516 H72" class="d-line d-dash" marker-end="url(#diagram-8-ft-h)"></path>
  <text x="262" y="515" class="d-note">fault in ready range: no request</text>

  <path id="diagram-8-ft-cold" d="M70 100 H250 V184 H460 V250 H650 V284 H460 V394 H250 V480 H70" fill="none" stroke="none"></path>
  <path id="diagram-8-ft-warm" d="M70 506 H250 V516 H70" fill="none" stroke="none"></path>

  <g class="d-moving">
    <circle r="5" class="d-pulse">
      <animateMotion dur="8s" repeatCount="indefinite" keyPoints="0;1;1" keyTimes="0;0.8;1" calcMode="linear"><mpath href="#diagram-8-ft-cold"></mpath></animateMotion>
      <animate attributeName="opacity" dur="8s" repeatCount="indefinite" values="1;1;0;0" keyTimes="0;0.8;0.801;1"></animate>
    </circle>
    <circle r="4" class="d-pulse">
      <animateMotion dur="8s" repeatCount="indefinite" keyPoints="0;0;1;1" keyTimes="0;0.84;0.96;1" calcMode="linear"><mpath href="#diagram-8-ft-warm"></mpath></animateMotion>
      <animate attributeName="opacity" dur="8s" repeatCount="indefinite" values="0;0;1;1;0;0" keyTimes="0;0.839;0.84;0.96;0.961;1"></animate>
    </circle>
  </g>
</svg></div><figcaption>Illustrative cold fault: the gateway fetches and verifies missing data before the sentry maps the page. A later fault within the ready range needs no gateway request.</figcaption></figure><p>The backing file is shared per blob per tenant on a worker, so a second sentry using the same base image can map the same pages already in the host page cache. Each sentry tracks ready ranges locally and uses its mapped backing file for subsequent accesses, while the worker keeps the immutable image data available to fresh sandboxes. The volume pages from our SQLite example live in a sentry’s own cache, but these image pages can outlive it because the image never changes.</p><p>We also contributed <a href="https://github.com/google/gvisor/pull/15527">support for chunk-based files</a> to gVisor’s EROFS reader. That part is upstream now, so gVisor can read and map ordinary chunked EROFS files. We still carry the remote-device integration that connects our image reads to the storage gateway and tracks which ranges are ready to map.</p><p>So steps using the same image can share its stored bytes and the pages already cached on a worker.</p><h2 id="startup-is-the-slow-part">Startup is the slow part</h2><p>Starting Python can still mean waiting for one library after another, because the next read often depends on code that hasn’t loaded yet. On a cold worker, those waits can add up even though we’re only fetching a small part of the image.</p><p>When the gateway prepares an image and finds no startup list, it records the first distinct ranges requested during that attachment and saves the list in Spongeblob when the connection closes cleanly. A later cold attachment can replay those ranges in the background while the program starts, with a limit on concurrent fills across the worker. The program still fetches anything the list missed, and every chunk goes through the same verification.</p><figure class="figure diagram wide"><div class="diagram-frame" role="img" aria-label="The gateway records requested ranges for an image, then replays the list during a later cold attachment. Reads still wait for any data that is not ready."><svg viewBox="0 0 720 382" xmlns="http://www.w3.org/2000/svg">
  <defs>
    <marker id="diagram-9-su-h" viewBox="0 0 8 8" refX="7" refY="4" markerWidth="7" markerHeight="7" orient="auto-start-reverse">
      <path d="M0 0 L8 4 L0 8 z" class="d-head"></path>
    </marker>
    <marker id="diagram-9-su-hh" viewBox="0 0 8 8" refX="7" refY="4" markerWidth="7" markerHeight="7" orient="auto-start-reverse">
      <path d="M0 0 L8 4 L0 8 z" class="d-head-hot"></path>
    </marker>
  </defs>

  <text x="24" y="26" class="d-tag">NO STARTUP LIST YET</text>
  <rect x="24" y="48" width="166" height="64" rx="8" class="d-box"></rect>
  <text x="38" y="74" class="d-name">program</text>
  <text x="38" y="94" class="d-note">reads and faults</text>

  <rect x="244" y="48" width="208" height="64" rx="8" class="d-box"></rect>
  <text x="258" y="74" class="d-name">gateway</text>
  <text x="258" y="94" class="d-note">records distinct ranges</text>

  <rect x="536" y="48" width="160" height="64" rx="8" class="d-box"></rect>
  <text x="550" y="74" class="d-name">startup list</text>
  <text x="550" y="94" class="d-note">per image digest</text>

  <path d="M190 80 H242" class="d-line" marker-end="url(#diagram-9-su-h)"></path>
  <path d="M452 80 H534" class="d-line" marker-end="url(#diagram-9-su-h)"></path>
  <text x="494" y="70" text-anchor="middle" class="d-note">clean close</text>

  <text x="24" y="170" class="d-tag">LATER COLD ATTACHMENT</text>
  <rect x="24" y="192" width="166" height="56" rx="8" class="d-box"></rect>
  <text x="38" y="215" class="d-name">saved ranges</text>
  <text x="38" y="234" class="d-note">same image digest</text>

  <rect x="266" y="192" width="188" height="56" rx="8" class="d-box-hot"></rect>
  <text x="280" y="215" class="d-name">gateway fills</text>
  <text x="280" y="234" class="d-note">bounded concurrency</text>

  <rect x="526" y="192" width="170" height="138" rx="8" class="d-zone"></rect>
  <text x="540" y="215" class="d-name">verified chunks</text>
  <text x="540" y="234" class="d-note">worker backing files</text>

  <rect x="24" y="282" width="166" height="56" rx="8" class="d-box"></rect>
  <text x="38" y="305" class="d-name">program starts</text>
  <text x="38" y="324" class="d-note">alongside prefetch</text>

  <path d="M190 220 H264" class="d-line" marker-end="url(#diagram-9-su-h)"></path>
  <text x="228" y="210" text-anchor="middle" class="d-note">replay</text>
  <path d="M454 220 H524" class="d-line-hot" marker-end="url(#diagram-9-su-hh)"></path>
  <text x="490" y="210" text-anchor="middle" class="d-note">verify</text>

  <path d="M190 310 H224 V240 H264" class="d-line d-dash" marker-end="url(#diagram-9-su-h)"></path>
  <text x="234" y="273" class="d-note">missing range</text>
  <path d="M190 310 H524" class="d-line-hot" marker-end="url(#diagram-9-su-hh)"></path>
  <text x="360" y="300" text-anchor="middle" class="d-note">reads ready ranges</text>

  <text x="24" y="368" class="d-note">A range missing from the list, or still being fetched, can make the program wait.</text>
</svg></div><figcaption>The gateway records requested ranges for an image, then replays the list during a later cold attachment. Reads still wait for any data that is not ready.</figcaption></figure><p>The list belongs to a particular image digest, so a changed image gets a separate list. If the same image starts taking a different path, the old list can fetch data we no longer need and use up bandwidth and cache space.</p><h2 id="tcp-egress-goes-through-pony">TCP egress goes through Pony</h2><p>Now, say our step needs to look up an IP address that isn’t in its SQLite database. It calls the threat intelligence API with <code>requests.get</code>, which opens a socket in the sandbox’s network namespace, and the connection is directed to the sandbox’s transparent listener in Pony. Pony already knows which connectors are attached to the run. sandboxd creates the listener and a DNS socket when preparing the sandbox, then hands both to Pony with Unix file descriptor passing, <code>SCM_RIGHTS</code>. Pony accepts directly on those sockets after the handoff.</p><figure class="figure diagram wide"><div class="diagram-frame" role="img" aria-label="The connection reaches Pony, where the request gets its connector credential before going to the API."><svg viewBox="0 0 720 384" xmlns="http://www.w3.org/2000/svg">
  <defs>
    <marker id="diagram-10-nw-h" viewBox="0 0 8 8" refX="7" refY="4" markerWidth="7" markerHeight="7" orient="auto-start-reverse">
      <path d="M0 0 L8 4 L0 8 z" class="d-head"></path>
    </marker>
  </defs>

  
  <rect x="20" y="40" width="190" height="250" rx="12" class="d-zone"></rect>
  <text x="34" y="62" class="d-tag">SANDBOX NETNS</text>
  <rect x="34" y="70" width="162" height="44" rx="8" class="d-box-hot"></rect>
  <text x="48" y="89" class="d-name">step</text>
  <text x="48" y="105" class="d-note">connect(api, 443)</text>
  <rect x="34" y="138" width="162" height="24" rx="5" class="d-cell"></rect>
  <text x="44" y="154">eth0  100.64.0.9</text>
  <rect x="34" y="200" width="162" height="24" rx="5" class="d-cell"></rect>
  <text x="44" y="216">dns   100.64.1.9</text>
  <text x="34" y="254" class="d-note">connector key</text>
  <text x="34" y="269" class="d-note">stays in Pony</text>

  
  <rect x="236" y="40" width="226" height="250" rx="12" class="d-ghost"></rect>
  <text x="250" y="62" class="d-tag">SANDBOXD NETNS</text>
  <rect x="264" y="110" width="172" height="130" rx="8" class="d-box"></rect>
  <text x="278" y="132" class="d-name">nftables</text>
  <text x="278" y="180" class="d-note">src → TPROXY :20007</text>
  <text x="278" y="196" class="d-note">spoofed source: drop</text>
  <text x="278" y="212" class="d-note">wrong resolver: drop</text>
  <text x="278" y="228" class="d-note">no listener: drop</text>
  <text x="212" y="143" class="d-note">veth</text>
  <text x="264" y="258" class="d-note">hands each sandbox’s</text>
  <text x="264" y="271" class="d-note">sockets to Pony</text>
  <text x="264" y="284" class="d-note">with SCM_RIGHTS</text>
  <path d="M384 280 H486" class="d-line d-dash" marker-end="url(#diagram-10-nw-h)"></path>

  
  <rect x="490" y="40" width="210" height="250" rx="12" class="d-ghost"></rect>
  <text x="504" y="62" class="d-tag">PONY</text>
  <rect x="504" y="100" width="182" height="24" rx="5" class="d-cell-hot"></rect>
  <text x="514" y="116" class="d-name">listener :20007</text>
  <rect x="504" y="130" width="182" height="24" rx="5" class="d-cell"></rect>
  <text x="514" y="146">destination policy</text>
  <rect x="504" y="160" width="182" height="24" rx="5" class="d-cell"></rect>
  <text x="514" y="176">terminate TLS</text>
  <rect x="504" y="190" width="182" height="24" rx="5" class="d-cell-hot"></rect>
  <text x="514" y="206" class="d-name">add credential</text>
  <rect x="504" y="222" width="182" height="24" rx="5" class="d-cell"></rect>
  <text x="514" y="238">DNS socket</text>

  
  <rect x="490" y="322" width="210" height="46" rx="8" class="d-box"></rect>
  <text x="504" y="342" class="d-name">threat intel API</text>
  <text x="504" y="358" class="d-note">the request has the key</text>

  
  <path d="M115 114 V136" class="d-line" marker-end="url(#diagram-10-nw-h)"></path>
  <path d="M196 150 H262" class="d-line" marker-end="url(#diagram-10-nw-h)"></path>
  <path d="M436 150 H470 V112 H502" class="d-line" marker-end="url(#diagram-10-nw-h)"></path>
  <path d="M196 212 H262" class="d-line" marker-end="url(#diagram-10-nw-h)"></path>
  <path d="M436 212 H478 V234 H502" class="d-line" marker-end="url(#diagram-10-nw-h)"></path>
  <path d="M497 112 V202" class="d-line"></path>
  <path d="M686 202 H710 V345 H702" class="d-line" marker-end="url(#diagram-10-nw-h)"></path>

  <path id="diagram-10-nw-tcp" d="M196 150 H470 V112 H497 V202 H710 V345 H700" fill="none" stroke="none"></path>
  <g class="d-moving">
    <circle r="4.5" class="d-pulse">
      <animateMotion dur="7s" repeatCount="indefinite" keyPoints="0;1;1" keyTimes="0;0.85;1" calcMode="linear"><mpath href="#diagram-10-nw-tcp"></mpath></animateMotion>
      <animate attributeName="opacity" dur="7s" repeatCount="indefinite" values="1;1;0;0" keyTimes="0;0.85;0.851;1"></animate>
    </circle>
    <circle r="6.5" class="d-cell-hot">
      <animateMotion dur="7s" repeatCount="indefinite" keyPoints="0;1;1" keyTimes="0;0.85;1" calcMode="linear"><mpath href="#diagram-10-nw-tcp"></mpath></animateMotion>
      <animate attributeName="opacity" dur="7s" repeatCount="indefinite" values="0;0;1;1;0;0" keyTimes="0;0.54;0.545;0.85;0.851;1"></animate>
    </circle>
  </g>
</svg></div><figcaption>The connection reaches Pony, where the request gets its connector credential before going to the API.</figcaption></figure><p>Pony can read the HTTPS request because it terminates TLS using a CA trusted by the sandbox. It holds the decrypted connector credential in its own memory, matches the request to the threat intelligence connector, and inserts the API key into the HTTP request it sends over a separate upstream TLS connection. The response goes back to our script, which could use the API without the key ever having been in its environment.</p><h2 id="removing-the-sandbox">Removing the sandbox</h2><p>Once the step’s command has exited, the runner kills and reaps any remaining processes. If the command failed, we discard this run’s volume changes. Otherwise, the runner syncs <code>/storage</code>, flushing the sentry’s dirty pages too, including changes made through shared mappings. That way, no background process can keep changing the files while Nest publishes them.</p><p>Nest then attempts to publish the volume changes. If, say, 2 alerts ran at once and both changed <code>seen.db</code>, publication can fail with a conflict, even though SQLite committed and the program exited successfully. That conflict fails the step.</p><p>After the result is stored, sandboxd destroys the sentry and gofer and removes their network and runtime resources, and only then is the slot available to prepare a new sandbox.</p><h2 id="wrapping-up">Wrapping up</h2><p>Putting those pieces together, here’s what a step running on a worker looks like.</p><figure class="figure diagram wide"><div class="diagram-frame" role="img" aria-label="Our alert enrichment step runs in a fresh sandbox, using files stored in Spongeblob and calling the API through Pony."><svg viewBox="0 0 720 480" xmlns="http://www.w3.org/2000/svg">
  <defs>
    <marker id="diagram-11-ov-h" viewBox="0 0 8 8" refX="7" refY="4" markerWidth="7" markerHeight="7" orient="auto-start-reverse">
      <path d="M0 0 L8 4 L0 8 z" class="d-head"></path>
    </marker>
  </defs>

  
  <rect x="40" y="20" width="160" height="48" rx="8" class="d-box"></rect>
  <text x="54" y="41" class="d-name">orchestrator</text>
  <text x="54" y="57" class="d-note">queue and claims</text>

  <rect x="540" y="20" width="140" height="48" rx="8" class="d-box"></rect>
  <text x="554" y="41" class="d-name">upstreams</text>
  <text x="554" y="57" class="d-note">APIs, registries</text>

  
  <rect x="12" y="92" width="696" height="300" rx="12" class="d-ghost"></rect>
  <text x="40" y="114" class="d-tag">WORKER</text>

  <rect x="40" y="128" width="160" height="50" rx="8" class="d-box"></rect>
  <text x="54" y="149" class="d-name">sandboxd</text>
  <text x="54" y="165" class="d-note">claims, pool, output</text>

  <rect x="40" y="318" width="160" height="50" rx="8" class="d-box"></rect>
  <text x="54" y="339" class="d-name">storage gateway</text>
  <text x="54" y="355" class="d-note">images and volumes</text>

  <rect x="540" y="128" width="140" height="50" rx="8" class="d-box"></rect>
  <text x="554" y="149" class="d-name">Pony</text>
  <text x="554" y="165" class="d-note">egress proxy</text>

  
  <rect x="512" y="230" width="170" height="80" rx="10" class="d-ghost"></rect>
  <rect x="502" y="220" width="170" height="80" rx="10" class="d-ghost"></rect>
  <text x="518" y="244" class="d-tag">OTHER SLOTS</text>
  <text x="518" y="264" class="d-note">one run per</text>
  <text x="518" y="279" class="d-note">sandbox</text>

  
  <rect x="280" y="108" width="190" height="270" rx="10" class="d-zone"></rect>
  <text x="296" y="127" class="d-tag">GVISOR SANDBOX</text>

  <rect x="296" y="142" width="158" height="46" rx="8" class="d-box-hot"></rect>
  <text x="310" y="162" class="d-name">step</text>
  <text x="310" y="178" class="d-note">python script.py</text>

  <rect x="296" y="222" width="158" height="46" rx="8" class="d-box"></rect>
  <text x="310" y="242" class="d-name">sentry</text>
  <text x="310" y="258" class="d-note">a kernel, in Go</text>

  <rect x="296" y="300" width="158" height="62" rx="8" class="d-box"></rect>
  <text x="310" y="320" class="d-name">gofer</text>
  <text x="310" y="336" class="d-note">selected host files</text>
  <text x="310" y="351" class="d-note">Nest: /storage</text>

  
  <rect x="40" y="420" width="640" height="44" rx="8" class="d-box"></rect>
  <text x="54" y="447" class="d-name">Spongeblob</text>
  <text x="150" y="447" class="d-note">images, volumes, step output</text>

  
  <path id="diagram-11-ov-a" d="M120 68 V126" class="d-line" marker-end="url(#diagram-11-ov-h)"></path>
  <text x="128" y="84" class="d-note">claim, complete</text>

  <path id="diagram-11-ov-b" d="M200 158 H294" class="d-line" marker-end="url(#diagram-11-ov-h)"></path>
  <text x="212" y="151" class="d-note">job</text>

  <path id="diagram-11-ov-c" d="M454 158 H538" class="d-line" marker-end="url(#diagram-11-ov-h)"></path>
  <text x="466" y="151" class="d-note">TCP, DNS</text>

  <path id="diagram-11-ov-d" d="M610 128 V70" class="d-line" marker-end="url(#diagram-11-ov-h)"></path>
  <text x="618" y="104" class="d-note">credentials</text>

  <path id="diagram-11-ov-e" d="M375 188 V220" class="d-line" marker-end="url(#diagram-11-ov-h)"></path>
  <text x="384" y="208" class="d-note">syscalls</text>

  <path id="diagram-11-ov-f" d="M375 268 V298" class="d-line" marker-end="url(#diagram-11-ov-h)"></path>
  <text x="384" y="287" class="d-note">LisaFS</text>

  <path id="diagram-11-ov-g" d="M296 245 H120 V316" class="d-line" marker-end="url(#diagram-11-ov-h)"></path>
  <text x="134" y="238" class="d-note">image ranges</text>

  <path id="diagram-11-ov-h2" d="M296 343 H202" class="d-line" marker-end="url(#diagram-11-ov-h)"></path>
  <text x="212" y="336" class="d-note">volumes</text>

  <path id="diagram-11-ov-i" d="M120 368 V418" class="d-line" marker-end="url(#diagram-11-ov-h)"></path>
  <text x="128" y="410" class="d-note">ranges, blocks</text>

  <path id="diagram-11-ov-j" d="M40 162 H28 V442 H38" class="d-line" marker-end="url(#diagram-11-ov-h)"></path>
  <text transform="translate(23 300) rotate(-90)" text-anchor="middle" class="d-note">step output</text>

  
  <g class="d-moving">
    <circle r="3.5" class="d-pulse">
      <animateMotion dur="5s" repeatCount="indefinite" keyPoints="0;1;1" keyTimes="0;0.2;1" calcMode="linear"><mpath href="#diagram-11-ov-a"></mpath></animateMotion>
      <animate attributeName="opacity" dur="5s" repeatCount="indefinite" values="1;1;0;0" keyTimes="0;0.2;0.21;1"></animate>
    </circle>
    <circle r="3.5" class="d-pulse">
      <animateMotion dur="5s" repeatCount="indefinite" keyPoints="0;0;1;1" keyTimes="0;0.22;0.4;1" calcMode="linear"><mpath href="#diagram-11-ov-b"></mpath></animateMotion>
      <animate attributeName="opacity" dur="5s" repeatCount="indefinite" values="0;0;1;1;0;0" keyTimes="0;0.219;0.22;0.4;0.41;1"></animate>
    </circle>
    <circle r="3" class="d-pulse"><animateMotion dur="1.4s" begin="-0.2s" repeatCount="indefinite"><mpath href="#diagram-11-ov-e"></mpath></animateMotion></circle>
    <circle r="3" class="d-pulse"><animateMotion dur="1.4s" begin="-0.9s" repeatCount="indefinite"><mpath href="#diagram-11-ov-e"></mpath></animateMotion></circle>
    <circle r="3" class="d-pulse"><animateMotion dur="2.2s" begin="-0.5s" repeatCount="indefinite"><mpath href="#diagram-11-ov-f"></mpath></animateMotion></circle>
    <circle r="3" class="d-pulse"><animateMotion dur="3.2s" begin="-1.6s" repeatCount="indefinite" keyPoints="0;1;0" keyTimes="0;0.5;1" calcMode="linear"><mpath href="#diagram-11-ov-g"></mpath></animateMotion></circle>
    <circle r="3" class="d-pulse"><animateMotion dur="2.6s" begin="-1.1s" repeatCount="indefinite"><mpath href="#diagram-11-ov-h2"></mpath></animateMotion></circle>
    <circle r="3" class="d-pulse"><animateMotion dur="2.4s" begin="-0.6s" repeatCount="indefinite" keyPoints="0;1;0" keyTimes="0;0.5;1" calcMode="linear"><mpath href="#diagram-11-ov-i"></mpath></animateMotion></circle>
    <circle r="3" class="d-pulse"><animateMotion dur="2.2s" begin="-0.3s" repeatCount="indefinite"><mpath href="#diagram-11-ov-c"></mpath></animateMotion></circle>
    <circle r="3" class="d-pulse"><animateMotion dur="2.2s" begin="-1.4s" repeatCount="indefinite"><mpath href="#diagram-11-ov-d"></mpath></animateMotion></circle>
    <circle r="3.5" class="d-pulse">
      <animateMotion dur="5s" repeatCount="indefinite" keyPoints="0;0;1;1" keyTimes="0;0.72;0.98;1" calcMode="linear"><mpath href="#diagram-11-ov-j"></mpath></animateMotion>
      <animate attributeName="opacity" dur="5s" repeatCount="indefinite" values="0;0;1;1;0" keyTimes="0;0.719;0.72;0.98;1"></animate>
    </circle>
  </g>
</svg></div><figcaption>Our alert enrichment step runs in a fresh sandbox, using files stored in Spongeblob and calling the API through Pony.</figcaption></figure><p>We’re only getting started here. A 3B step can run Python, Node, shell scripts, or compiled x86-64 programs to automate enterprise workflows, within the <a href="https://gvisor.dev/docs/user_guide/compatibility/">Linux interfaces gVisor supports</a> and our sandbox restrictions. We’ve invested in primitives that let us connect steps to self-hosted environments through our tunnel and fork volumes from a live branch into a draft, and those are just a couple of examples. We’ll share more about these in future posts.</p><p>Lastly, we’re also invested in gVisor’s success and as gVisor <a href="https://gvisor.dev/blog/2026/10/02/gvisor-cncf/">moves to the CNCF</a>, Tines has <a href="https://cloud.google.com/blog/products/containers-kubernetes/open-infrastructure-for-autonomous-agents-comes-to-cncf">committed to joining its maintainers</a> alongside Modal and Ant Group. We are stoked for what lies ahead.</p></div>]]></content:encoded></item></channel></rss>