AI-generated code is flooding the software supply chain with new threats

Cassie Crossley has literally written the book on software supply chain security. She was previously VP of supply chain security at Schneider Electric and is currently CEO and co-founder at a stealth AI cybersecurity startup. She understands supply chain risk as well as anyone, and she believes that AI-generated code represents a huge risk.

She argues that when you're using AI to vibe code, you need to know where every piece of the code comes from because models can make up libraries that themselves become security risks.

"If you're going to use AI to build an application, it's great for a proof of concept, but you should never put something AI coded into production that has not been thoroughly scanned," Crossley told FastForward. "You can't know every single library, and frankly, there are not enough security layers built in yet for vibe coding that I would ever trust at the moment."

A cautionary tale

Rodney Brooks, co-founder and CTO of warehouse robotics startup Robust.AI and former head of MIT’s CSAIL, once told me a story about stumbling onto a hallucinated software library. He was doing some coding with AI and thought it had done a great job, until he tried to compile the code and discovered the model had invented a library that doesn’t exist. 

Worse, as though to prove Crossley's point, when he dug deeper, he found that cybercriminals were taking advantage of these made-up library names by publishing malicious packages under the same name on GitHub, hoping some unsuspecting developer would install them. A made-up library on its own is annoying. A made-up library that someone else turns into malware is a much bigger problem.

Crossley says that stories like this one are unfortunately no longer an edge case. She believes that the industry has to do a better job at governance, and this is especially true for non-programmers who with the help of AI, can now create applications on their own. It's easy enough for a savvy programmer to run into problems, but newbies truly don't know what they don't know, and could easily stumble into trouble trying to build programs on their own.

"We need to be very clear that people understand that hallucinating [is possible] and leveraging those packages as they're building applications is a huge risk," she said. Crossley certainly isn't alone in this belief. According to a recent Dataiku survey of 600 CIOs worldwide, 81% reported being concerned that citizen-built AI could expose sensitive company data.

Building governance training wheels

The idea isn’t to ban vibe coding altogether though. That would be impossible, given the efficiency gains we keep hearing about. The Dataiku survey suggests CIOs know this instinctively, with 74% of respondents saying they must deliver measurable AI gains within two years or risk losing their jobs. That means it will take something more concrete than simply saying no.

"A lot of companies, especially more enterprises, are saying, here's the [sanctioned] code platform that we allow for business users, just as we did previously with low code platforms," Crossley said. So it's not about limiting how many people can use these tools, but about recognizing the risk and mitigating it through education and a tightly controlled set of approved platforms.

It’s a sentiment echoed by Elia Zaitsev, CTO at CrowdStrike, who says the real work isn’t in banning AI-generated code, but in tightly governing where it comes from and what it’s allowed to touch. “You need guardrails and vetted intermediaries so people aren’t pulling random, unvalidated packages from the wild — or even legitimate packages that have been poisoned later on in a supply chain attack,” Zaitsev told FastForward.

While vibe coding tools become increasingly capable, companies need governance and structure to keep malicious code out of the software supply chain. With AI helping generate more code than ever, some percentage will be slop and some downright dangerous. It’s important to put strong guidelines in place that add structure without discouraging usage.