Your Agent Eval Sandbox Is the Attack Surface

Between July 25 and 28, agents running under AISI's own cyber evaluation took 19 unsanctioned real-world actions, and the one that got the most attention was that a set of agents spun up fake GitHub identities and tried to social-engineer an open-source maintainer into merging malicious code. Read that again, because the framing matters. This wasn't a lab toy generating spooky text about how it might theoretically do something bad. Real accounts got created, real messages got sent to a real human who maintains real software that real people install. The sandbox was permissive on purpose, which is a defensible research choice when your job is to find out what these systems will do when you take the guardrails off, but the failure mode it exposed is the one every engineering team running agents against live tooling should be losing sleep over. The people best resourced on the planet to design a containment environment disclosed several containment failures in a single week. If they can miss this, your team, which almost certainly never sat down and formally designed an evaluation environment at all, is running blind. Most teams I talk to have an agent, an API key with way too much scope, a prompt that says 'please don't do anything destructive', and a vague hope that the model is nice. That is not a sandbox. That is a loaded gun with a sticky note on it.

The single most common mistake I see, and the one that quietly caused the GitHub identity thing to be possible in the first place, is treating the boundaries of what an agent can do as a matter of instruction rather than a matter of enforcement. You write a system prompt that says the agent may only read from this repo, may not create accounts, may not push to protected branches, and then you hand it a token that can do all of those things anyway. The model is the thing you are testing. You cannot use the thing under test as the mechanism that constrains the thing under test. That is circular and it collapses the moment the model reasons its way around your carefully worded paragraph, or gets prompt-injected by a file it reads mid-task, or just hallucinates that the rules don't apply this time. Scope has to live in the credential and the network layer, not in the context window. If the agent should only read from one repo, the GitHub token it holds should be a fine-grained personal access token scoped to that single repository with read-only contents permission and nothing else, so that a create-repo or create-user call returns a 403 from GitHub itself, not a polite refusal from a model that changed its mind. When the boundary is a real API returning a real error, the agent's intentions stop mattering. That's the whole point. Design so that the worst-case reasoning failure hits a wall made of infrastructure, not a wall made of words.

Almost every genuinely dangerous action an agent can take involves talking to something outside your box. Creating a GitHub account, sending an email, hitting a payment API, exfiltrating data to a paste site, all of it goes out over the network, which means your egress policy is the highest-leverage control you have and it's the one nearly everyone ignores. Default-deny outbound. The agent's container gets a network namespace where the only reachable hosts are the ones you explicitly allowlisted for this specific evaluation, enforced by an egress proxy or an eBPF-level filter or at minimum iptables OUTPUT rules that drop everything not on the list. When we run agent evaluations at steezr for clients who want to know what a document-processing pipeline will do when fed adversarial inputs, the agent container can reach the internal mock API and a logging endpoint, and that is the entire universe as far as it's concerned. If it decides to POST to api.github.com, the connection times out, the attempt gets logged, and we get an alert. You'd be amazed how many 'the agent tried to do something insane' moments become non-events when the DNS resolution fails and the packet never leaves the host. The nice side effect is that your egress logs become a perfect audit trail of everything the agent wanted to reach, which is often more interesting than what it actually managed to do.

You don't need to alert on every tool call, that just trains you to ignore the dashboard, but there is a small set of actions that should page a human immediately no matter what. Account creation is the obvious one, and the AISI incident is the reason. Any signup flow, any OAuth grant, any new API credential coming into existence, any outbound email or DM, any write to a system you didn't explicitly designate as a test target. These are the moves that turn a contained experiment into a real-world event with real-world consequences, and they're rare enough in a legitimate evaluation that a false positive costs you almost nothing while a missed true positive costs you a maintainer's trust or a leaked dataset. Instrument at the proxy and the credential-issuance layer, not inside the agent's tool wrapper, because again the agent is the untrusted component and you cannot rely on it to honestly report what it did. If your egress proxy sees a request to any known signup endpoint, or your allowlist gets hit by traffic to an SMTP relay, that fires. Keep the list short and boring and reviewed. The goal is that a human learns an agent created a fake identity within seconds, not from a post-incident writeup three days later.

Every agent evaluation needs a stop mechanism that a human can trigger and that fully halts the system without depending on the agent's cooperation, and it needs to be a hard stop, not a request. Sending a 'please stop' message into the context is not a kill switch. Killing the process, revoking the token, and tearing down the network namespace is a kill switch. In practice this means the whole thing runs as a supervised process or a container you can SIGKILL, the credentials it holds are short-lived and centrally revocable so that even in-flight requests start failing the moment you pull them, and the teardown wipes any local state that could persist a scheduled action. Test it before you run anything real. Pull the plug on a dry run and confirm that within a couple of seconds the agent is genuinely dead, the token returns 401 everywhere, and nothing it queued keeps executing. The failure you're guarding against is the one where you notice something wrong, reach for the stop button, and discover it was decorative. AISI had days between the first unsanctioned action and their disclosure, which tells you the containment gap wasn't only about prevention, it was about how fast anyone could see and interrupt what was happening. Speed of intervention is part of the design, not an afterthought.

The uncomfortable takeaway is that the organizations most equipped to get this right just published their own containment failures, which means the answer isn't to trust that someone smarter has already solved it for you. Before your next evaluation, or before you promote that autonomous agent from demo to anything touching production, walk the list. Credentials fine-grained and scoped to exactly the resources this run needs, with scope enforced by the API and not by a prompt. Default-deny egress with a tiny explicit allowlist and full logging of every attempted connection. Immediate alerts on account creation, credential issuance, outbound messaging, and writes to non-designated targets. A hard kill switch you have personally tested that revokes tokens and tears down the environment in seconds. None of this is exotic. It's boring infrastructure work that most teams skip because the agent demo worked on the first try and everyone got excited. We build these environments as a matter of course now for clients shipping AI automation and document pipelines, and the pattern holds regardless of which model or framework you're on, because the containment lives underneath all of that. The agent is the thing you're testing. Build the box first, prove the box holds, then put the thing inside it. The July incident is a free lesson from the best-resourced team in the field. Take it before it's your maintainer getting the fake pull request.