Companies racing to build AI capabilities using open-source models from platforms like Hugging Face are facing a growing security nightmare. Attackers are slipping malicious code into AI repositories, exposing organizations to hidden risks. Experts warn that businesses can’t blindly trust security checks from these platforms—they need their own defenses.
Malicious AI Models Are Slipping Through Security Gaps
Cybercriminals are getting smarter. They’ve found ways to inject harmful code into AI models hosted on popular repositories. And the worst part? These malicious files are often flagged as “safe.”
A recent case highlights this vulnerability. Security firm ReversingLabs discovered that Hugging Face failed to detect two AI models laced with harmful code. The attackers used a technique called “NullifAI” to slip through security scans, packaging malicious payloads inside Pickle format files.
Tomislav Pericin, chief software architect at ReversingLabs, points out a troubling reality:
- Anyone can upload an AI model to Hugging Face, and bad actors are abusing that freedom.
- The malicious models passed all automated checks, appearing safe to users who downloaded them.
- Similar tactics can work across other repositories like TensorFlow Hub and PyTorch Hub.
The takeaway? Relying on Hugging Face’s built-in security isn’t enough. Companies need to scan every model before integrating it into their systems.
The Pickle Problem: A Well-Known Security Risk That Won’t Go Away
Cybersecurity experts have been sounding the alarm on Pickle files for years. Yet, this insecure data format is still widely used in AI development.
Pickle files can execute arbitrary code when loaded, making them a prime target for attackers. Tom Bonner, VP of research at AI security firm HiddenLayer, has seen real-world breaches caused by this vulnerability.
“I really hoped we’d make enough noise about it that Pickle would’ve gone by now, but it’s not,” Bonner says. “Organizations are getting compromised through machine learning models. It’s not as common as ransomware, but it does happen.”
Some security researchers thought they had a solution. Hugging Face implemented PickleScan, a tool designed to detect dangerous files. But attackers quickly found workarounds. Checkmarx, an application security firm, discovered multiple ways to bypass these scans.
A better alternative? Safetensors, a more secure format developed by Hugging Face, EleutherAI, and Stability AI. Unlike Pickle, it doesn’t allow arbitrary code execution, making it a much safer choice for AI developers.
Open-Source AI Models: Legal Landmines and Ethical Risks
Beyond cybersecurity threats, companies face another challenge when using open-source AI models: licensing and alignment issues.
Many assume these models are fully open source, but that’s not always the case. AI licensing is a messy, complex landscape. Andrew Stiefel, senior product manager at Endor Labs, breaks it down:
- AI models include multiple components: The model itself, training data, and weights. Each may have different licenses.
- Not all licenses allow commercial use. Companies risk legal trouble if they monetize models without understanding the fine print.
Another concern? Model alignment—ensuring AI behaves as expected. Some models have been caught producing harmful or biased outputs. Others, like DeepSeek, have even been manipulated to create malware.
Even OpenAI’s o3-mini model, which was designed with stronger alignment safeguards, was quickly jailbroken by researchers. This highlights how unpredictable AI behavior can be, making it a security wildcard.
How Companies Can Protect Themselves
Given these risks, businesses need a proactive approach when using open-source AI models. Endor Labs’ Stiefel suggests treating AI models like any other software dependency. That means scrutinizing them before deployment.
Key security measures include:
- Checking the source: Who created the model? Is it from a trusted organization?
- Monitoring development activity: How often is the model updated? Are security issues addressed?
- Scanning for vulnerabilities: Use security tools to detect hidden risks before integrating AI into business applications.
By taking these precautions, companies can reduce their exposure to AI-driven cyber threats while still benefiting from open-source innovation.
