As an enterprise architect, I’ve worked on many major banking and secure infrastructure public sector projects to make them as impermeable as possible. But AI has fundamentally changed the landscape. The LiteLLM breach shows how one compromised tool can carry an attack through build systems, software registries and into thousands of downstream environments. It also shows why removing the malicious package is only the beginning of the response.
Most cyberattacks require the victim to make an obvious mistake: click a malicious link, reuse a password or leave a server exposed.
A software supply-chain attack can punish an organization for doing what appears to be exactly the right thing, installing a legitimate product from its official source.
That is what happened in March when attackers published two malicious versions of LiteLLM, an open-source Python library and AI gateway used to connect applications with multiple model providers. The compromised versions, 1.82.7 and 1.82.8, appeared in the Python Package Index, or PyPI, where developers ordinarily obtain the package.
They did not merely contain a conventional software flaw. They contained credential-stealing malware.
According to PyPI’s incident report, the affected versions were downloaded more than 119,000 times during the two hours and 32 minutes between the first upload and quarantine. The malware harvested sensitive files and credentials and sent them to a remote server.
The apparent fallout became clearer this week. Security firm Hudson Rock says it obtained a 153GB compressed archive containing 433,909 files. It attributed 118,829 CI runner dumps—records from systems that execute automated software builds and tests—to 2,488 corporate domains. CloudSEK separately reported a reconstructed exposure dataset involving more than 2,500 organizations and roughly 434,000 continuous integration and deployment, or CI/CD, pipelines.
(Note: An organization appearing in an exposure dataset is not necessarily proof that attackers successfully compromised all of its systems, used its credentials or stole its data.) CloudSEK itself says its high-confidence matches should be treated as “potentially exposed” until malicious execution, exfiltration or unauthorized access is independently verified.
The evidence nevertheless points to a supply-chain compromise of extraordinary reach, and a useful case study in how these attacks work.
What is a software supply-chain attack?
A software supply chain includes far more than the company whose name appears on a product. It encompasses open-source libraries, build tools, external code actions, package registries, container images, cloud services, update systems and the people and credentials authorized to move code through them.
In a supply-chain attack, an attacker compromises one of those trusted suppliers or delivery mechanisms and uses it to reach downstream customers. CISA describes the basic pattern as a threat actor infiltrating a software vendor and using the vendor’s software or update process to attack its users. ENISA offers an even stricter definition: there is an attack on a supplier, followed by an attack on the customer through that trusted relationship.
The important feature is inherited trust. Instead of breaking into 1,000 organizations separately, the attacker compromises something those 1,000 organizations already allow into their systems.
That “something” may be:
● a widely used software dependency;
● a developer or maintainer account;
● a build or release pipeline;
● a package registry or update channel;
● a security tool with access to sensitive environments; or
● a service provider that can deploy code to many customers.
The customer may receive the malicious code through a correctly spelled package name, a familiar vendor and an official distribution channel. Traditional advice about avoiding suspicious downloads is therefore not enough.
How the LiteLLM attack moved through the chain
The LiteLLM incident was a sequence of compromises in which each trusted layer provided access to the next.
| Stage | What normally happens | What happened in this attack |
| 1. Security scanner | Trivy scans software and infrastructure for vulnerabilities. | Attackers compromised Trivy’s open-source release infrastructure and redirected numerous version tags to malicious code. |
| 2. Build pipeline | LiteLLM’s automated workflow used Trivy during its own security process. | The poisoned scanner ran with legitimate access inside the LiteLLM build environment and exposed publishing credentials. |
| 3. Package release | Authorized credentials publish official LiteLLM releases to PyPI. | Attackers used stolen authority to publish malicious LiteLLM versions 1.82.7 and 1.82.8. |
| 4. Downstream installation | Developers and automated systems install or update LiteLLM from PyPI. | Unpinned installations and transitive dependencies automatically pulled the compromised releases. |
| 5. Credential theft | The package operates as part of an AI application or gateway. | The malware searched for environment variables, SSH keys, cloud credentials, Kubernetes tokens, database passwords and AI provider keys. |
| 6. Expansion | Those credentials authorize normal development and production work. | Stolen credentials could open repositories, registries, cloud accounts, clusters and other services—or seed another supply-chain attack. |
Aqua Security’s incident report says attackers had altered 76 of 77 version tags in one Trivy GitHub Action repository and all seven tags in another. Many customers referenced readable version tags rather than immutable commit hashes, so their pipelines continued running code under familiar version names even after the code behind those names had changed.
LiteLLM version 1.82.8 added another dangerous feature: a malicious Python .pth startup file. It could execute when the Python interpreter started, without a developer explicitly importing LiteLLM. The payload then attempted to collect credentials, establish persistence and move into connected Kubernetes environments. LiteLLM’s own security advisory says the stolen material could include environment variables, SSH keys, AWS, Google Cloud and Azure credentials, Kubernetes tokens and database passwords.
The attack succeeded by abusing valid mechanisms at every step. A legitimate scanner ran in a legitimate workflow. A valid publishing credential placed files in the official registry. Automated systems then installed them as designed.
Why the AI connection matters
The technique is not uniquely “AI.” Software supply-chain attacks predate generative AI by decades.
AI infrastructure can, however, make the consequences larger. Gateways such as LiteLLM sit between applications and model providers. They may hold keys for several AI services at once, along with database access, logging systems, cloud accounts and internal tools. Agentic systems and MCP connections can add still more credentials and operational permissions.
The model is increasingly just one component inside a larger system. The surrounding gateway, harness, tools, connectors and deployment infrastructure may collectively possess far more authority than the model itself.
That makes AI middleware a particularly valuable junction. Compromising it can provide access not only to model usage and billing, but to the wider business systems connected around it. Rapid experimentation also means AI dependencies are sometimes introduced before they appear in a central asset inventory, leaving companies unable to answer the first incident-response question: Do we use this package anywhere?
How organizations can reduce supply-chain risk
No organization can eliminate third-party software. The practical goal is to make dependencies visible, constrain the authority they inherit and prevent one compromised component from reaching everything around it.
| Control | What it changes |
| Maintain a dependency inventory and SBOM | Shows which direct and transitive components are present, where they run and who owns them. AI gateways, agents, MCP servers and plugins should be included. |
| Lock packages and verify hashes | Prevents the same installation command from silently retrieving different code on a later run. A secure lock file records both versions and artifact hashes. |
| Pin CI/CD actions to immutable commit hashes | Stops an attacker from moving a familiar version tag so that it points to malicious code. |
| Add a short dependency cooldown | Gives registries and researchers time to detect a poisoned release before production systems install it. Urgent security patches should be able to bypass the delay. |
| Replace standing secrets with short-lived identity | OIDC and workload identity issue narrowly scoped, temporary credentials instead of leaving reusable publishing or cloud tokens inside a pipeline. |
| Separate build, test, publish and deploy authority | Prevents a scanner or test job from automatically inheriting permission to publish packages or reach production. |
| Isolate and restrict build runners | Clean, temporary runners, restricted cloud access and outbound network controls limit what a malicious dependency can read or exfiltrate. |
| Verify provenance and monitor behaviour | Signatures and attestations help establish an artifact’s origin, while runtime monitoring detects suspicious behaviour after installation. |
| Prepare a credential-compromise playbook | Allows teams to identify affected environments, revoke credentials, rebuild from trusted sources and examine logs quickly. |
An SBOM is especially useful, but it is not a shield. It tells a company what it has. It does not prove that every listed component is safe, stop a malicious package from running or limit the secrets available to it. Inventory must be paired with access control, artifact verification and monitoring.
The same is true of multifactor authentication. Strong MFA should protect every human maintainer and administrator account, but it does not neutralize a reusable machine token that malware has already copied. Human identity and machine identity require different controls.
Questions business buyers should ask suppliers
Supply-chain security cannot remain solely a developer concern. Procurement, risk and executive teams need evidence about how critical software is built and delivered.
For important suppliers, buyers should ask:
1. Can you provide an SBOM and verifiable provenance for each release?
2. Are third-party packages and CI/CD actions pinned to immutable versions and reviewed before updates?
3. Do build and publishing systems use short-lived, narrowly scoped credentials?
4. Are build, release and production environments separated?
5. What customer data, credentials and systems can the product or its updater access?
6. How quickly will you notify customers of a compromised dependency or release?
7. Can you identify every customer and environment that received a specific affected version?
These questions do not guarantee security. They establish whether a supplier can see and contain its own chain when something breaks.
What potentially affected LiteLLM users should do
Organizations should determine whether LiteLLM 1.82.7 or 1.82.8 entered any developer workstation, virtual environment, container image, build cache or CI/CD runner on March 24, including as a transitive dependency.
If either version was present, simply uninstalling it is not enough. The affected environment should be isolated and rebuilt from a known-clean source. Every credential the process could read, not only a LiteLLM or model API key, should be revoked and rotated. That may include cloud keys, repository and package-registry tokens, SSH keys, Kubernetes service accounts, database credentials, SaaS secrets and AI provider keys.
Security teams should also review cloud, source-control, package-registry, Kubernetes and network logs from March 24 onward for unusual token use, new accounts, altered repositories, unexpected releases and suspicious outbound connections. Stolen credentials remain useful after the malicious package disappears, which is why package removal ends distribution but does not end the incident.
The broader lesson is not that open source is unsafe, or that companies should stop using AI infrastructure. Modern software is necessarily assembled from shared components.
The lesson is that trust must have boundaries.
The most important question is not, Do we trust this supplier? It is: If one trusted layer turns hostile, how far can it reach, how quickly will we know and how fast can we revoke its authority?
That is the real measure of supply-chain resilience.
