Your LLM Gateway Is a Credential Vault, and the LiteLLM Breach Proved It

By now you've probably seen the incident writeup, but the mechanics are worth walking through carefully because the interesting part isn't the payload, it's the path. TeamPCP didn't find a flaw in LiteLLM's request handling or some clever prompt injection into the proxy layer, they went one level up and compromised the security scanner that ran inside LiteLLM's own CI pipeline. Read that again. The tool that was supposed to catch malicious code was the thing that got weaponized, and once it had execution inside the trusted build environment it lifted the PyPI publishing tokens sitting in the runner's environment and pushed a tampered release out to a package that pulls something like 95 million downloads a month.

The harvester itself was a nasty little bit of engineering. They didn't touch the main package logic where a reviewer might notice a diff during a version bump, they dropped a .pth file into site-packages, which Python executes automatically on interpreter startup because that's literally what .pth files are allowed to do when they start with an import statement. So the moment any process imported anything in an environment where the poisoned wheel was installed, the harvester ran, walked the environment for anything matching the shape of an API key, and phoned home.

And where does LiteLLM live? Inside the one service in your infrastructure that has, by design, every OpenAI key, every Anthropic key, your Bedrock IAM credentials, your Vertex service account JSON, and usually a fat pile of CI tokens because someone wired the gateway config into the deploy pipeline. It's the richest environment variable dump in the whole company. The attackers understood that better than most of the teams running the thing.

Here is the structural problem nobody wanted to name out loud. An LLM gateway aggregates credentials the same way a secrets manager does, except a secrets manager is built, audited, and operated like a piece of security infrastructure, while the gateway got installed with a one-line pip command by a backend engineer who needed to A/B test two models on a Tuesday afternoon.

That asymmetry is the whole story. Vault, AWS Secrets Manager, Doppler, whatever you use, those things have threat models. People argue about their IAM policies. Nobody ships a Vault cluster by running an unpinned install off PyPI and forgetting about it. But LiteLLM, or any proxy that plays the same role, ends up holding the exact same class of secret while being treated like requests or httpx, a convenience wrapper you never think about again after the first import.

The fix in your head is a reclassification. The proxy that fans your traffic out to six model providers is not a library, it's the single point of aggregation for every provider key and cloud credential in your stack, and it deserves the same paranoia you'd apply to the box that terminates TLS or the service that mints JWTs. Once you frame it that way the mitigations stop feeling like overkill and start feeling like the baseline you were negligent for skipping. We ran this exact conversation with a couple of clients the week the news broke, and the ones who'd already isolated their gateway barely blinked, while the ones running it as a sidecar in the same pod as their app, sharing the same service account, spent two days rotating everything.

If you're still running pip install litellm with no version constraint, or even litellm==1.x with just a version and no hash, you are trusting that PyPI serves you the same bytes today that got reviewed yesterday, and the LiteLLM incident is precisely the case where that assumption breaks.

Hash pinning is the boring answer that actually would have blocked this. When you pin --require-hashes in your requirements file, or use uv with a locked uv.lock that records the sha256 of every wheel, a swapped release fails the install with a hash mismatch instead of silently landing in site-packages. The poisoned wheel had a different hash than the version your lockfile knew about, full stop, so the build breaks loudly instead of harvesting your keys quietly.

Concretely, generate a fully hashed lock and refuse to install anything outside it. With uv that's uv pip compile requirements.in --generate-hashes -o requirements.txt and then uv pip sync --require-hashes requirements.txt in CI. With plain pip it's pip install --require-hashes -r requirements.txt, and yes, that means transitive dependencies need hashes too, which is annoying the first time and free every time after. The annoyance is the point, it forces a human decision every time a dependency actually changes.

The thing people get wrong is treating the lockfile as a formality that gets regenerated on autopilot in the same PR that bumps a feature. If your CI regenerates hashes automatically on every dependency change without a human looking at what moved, you've rebuilt the vulnerability with extra steps. Someone has to look at the diff and ask why litellm jumped a patch version this morning. That five second glance is the whole control.

The reason the PyPI token got stolen is that it was sitting in the CI runner's environment, available to any code that executed during the build, including a scanner that everyone assumed was benign. Ambient credentials in build environments are the oldest supply chain footgun and we keep reloading it.

Stop putting long-lived publishing tokens in runner env vars. GitHub Actions has had OIDC trusted publishing to PyPI for a while now, and it works, you configure a trusted publisher on the PyPI side and the runner exchanges a short-lived OIDC token at publish time, so there is no static token to steal because none exists in the environment for more than a few seconds and it's scoped to exactly one workflow. If TeamPCP's harvester had run in an OIDC-based publish flow, there'd have been nothing sitting in the environment to grab.

Beyond publishing, the build step that runs your test suite and your linters and your scanners should not have access to production provider keys at all. There is no reason your unit tests need a live Anthropic key. Split the pipeline so that the stage running third party tooling has an environment scrubbed down to nothing sensitive, and the stage that actually deploys runs with tightly scoped, short-lived, freshly minted credentials that expire in minutes. It's more YAML. It's also the difference between a scanner compromise being a shrug and being a company-ending event.

We build most of our AI infrastructure at steezr on this assumption now by default, that any dependency in the build can and eventually will execute arbitrary code, so the question is never how do we trust the scanner, it's what can the scanner reach when it turns hostile.

Everything above is prevention. This is the layer that saves you when prevention fails, which it will, because you cannot audit every transitive dependency of a package that pulls in dozens of them.

The harvester's entire value proposition depends on one thing, being able to open an outbound connection to an attacker-controlled host and ship the stolen keys there. If your gateway runs in an environment where egress is denied by default and only explicitly allowed destinations can be reached, the payload collects everything, tries to exfiltrate, and hits a wall. The keys never leave the box.

This is deeply underused and it drives me a little crazy. Most people configure ingress rules obsessively and leave egress wide open, so a compromised process can reach any IP on the internet. Flip it. Your LiteLLM gateway needs to reach api.openai.com, api.anthropic.com, your Bedrock regional endpoint, maybe your Postgres and Redis, and that is the entire list. Everything else should be a rejected connection. On Kubernetes that's a NetworkPolicy with an explicit egress allowlist, or a service mesh with an outbound policy, or at minimum a NAT gateway with restrictive rules and a set of security groups that only permit the known provider CIDRs.

DNS matters here too, because a lot of naive egress rules key off DNS names while the actual filtering happens at the IP layer, so run egress filtering that resolves and pins the allowed hostnames rather than trusting whatever the workload resolves. The provider endpoints are stable enough that maintaining this list is a monthly chore, not a daily one, and in exchange a credential harvester that gets code execution in your gateway becomes a process that collected a bunch of secrets it can never send anywhere.

You don't need to boil the ocean. Do the following in roughly this order and you close most of the blast radius.

Audit where your gateway runs and what identity it has. If it shares a pod, a node, or a service account with your application code, separate it, because the gateway's credentials should never be reachable from a request handler that processes user input. Give it its own tightly scoped identity that can touch the provider keys and nothing else.

Regenerate your lockfile with full hashes and switch CI to --require-hashes or uv pip sync. Then make dependency bumps require a human review of the diff, not an automated regeneration that nobody reads. Kill any static PyPI or provider tokens living in runner environments and move publishing to OIDC.

Write the egress allowlist. Start in log-only mode so you learn what your gateway actually talks to over a couple of days, then flip it to deny by default once you're confident in the list. This one step, on its own, would have neutralized the LiteLLM payload entirely, and it protects you against the next one whose name you don't know yet.

The uncomfortable takeaway from March is that the attackers had a clearer mental model of this infrastructure than the people running it. They knew the gateway was the fattest credential target in the building. They knew the build environment was the soft underbelly. Treat the proxy like the trust boundary it always was, and the next TeamPCP finds a much smaller, much angrier surface to work against. If you want a second set of eyes on how yours is wired, that's a good chunk of what we do at steezr, and it's cheaper than a mass rotation at 2am.