FastForward #56: Sam and Dario's not-so-excellent AI adventure
ForwardThinking 🤔
Sam and Dario's not-so-excellent AI adventure
As the drama unfolded last week over OpenAI’s sudden partnership with the DoD, the department was also threatening to treat Anthropic as a supply chain risk (a designation that has since been made official). One of the key sticking points was whether these models should be used to conduct mass surveillance of U.S. citizens.
Chances are that neither of these companies’ chief executives is truly looking out for the best interests of the masses, but the debate itself speaks to the actual capabilities of these models and what's really possible.
I'll start with a research story that shows how difficult it is to use these tools to find specific information across multiple sources when there is no clear source of truth. I was trying to compile a list of all the panels and firesides I've done in my career, and there have been quite a few. This exercise pushed the AI models to their limits trying to build a list from agendas, preview stories, event articles and other sources.
No matter which model I used, whether Claude, ChatGPT or Gemini, they all failed to find many of the events I participated in, and worse, made up interviews or conflated article interviews with on-stage ones. Even when I tried to correct them, or add additional information, they universally did a poor job of compiling a reasonable list.
I tell this story because mass surveillance requires analyzing enormous amounts of data to answer questions or identify actions by individuals and groups. While the government clearly has far more powerful computers than I do, along with access to many other data sources like satellites and location data, the underlying limitation remains the same: these models still struggle to answer questions accurately across fragmented sources. I had to continually point out things I remembered as I went through the exercise, and if I'm feeding the model the information I hoped it would find for me, what is the point of relying on the model?
None of this means the current generation of AI couldn't be misused or prove dangerous in the wrong hands. But we should be honest about what these systems can actually do because it's entirely possible the DoD is reacting to hype more than to the real capabilities of the current technology.
Revving up the hype machine
If you pay attention to the statements of OpenAI CEO Sam Altman and Anthropic CEO Dario Amodei, you'll notice a pattern of exaggeration about what their models can do, along with plenty of scary rhetoric about what they might become capable of. But based on how these models actually perform today, these lofty statements are more about marketing than reality.
Consider these statements from the pair of CEOs:
- In an essay called Reflections published last year, Altman wrote: "We are now confident we know how to build AGI as we have traditionally understood it." Even though there is no universal definition of Artificial General Intelligence, when we hear the term, the implication is human-like intelligence, however you define that. Well-respected AI researchers like Andrew Ng say it could be years before anyone actually achieves it, yet Altman's comments suggest it could be imminent.
- In an interview last year, reported by Axios, Amodei predicted that up to half of white-collar jobs could be wiped out by AI over the next five years. It certainly sounds scary, but given my experience with this technology, a timeline that fast feels, let's just say, very unlikely.
A common example of a supposedly-doomed job is that of analyst, but a good analyst is talking to companies and customers every single day, compiling data and building a broad understanding of a market. All AI can do is peruse the internet for information that exists already. It can't access any new information because it's not capable of actually researching, just regurgitating.
Keeping humans involved in AI decision-making
Perhaps even more frightening, given the limitations of the current generation of AI, is the idea of handing over the keys to bombing missions to these models. Besides mass surveillance, Anthropic also specifically objected to launching any automated attacks without human control. They are correct, of course. The possibility of things going sideways is far too great to trust something as high stakes as a bombing mission to a large language model in its current state.
MIT professor Bryan Reimer, co-author of the book 'How To Make AI Useful,' says nobody should be entrusting this software with anything critical right now. "We need to be careful how we deploy AI in safety-critical situations such as defense, healthcare, transport, etc.," Reimer told FastForward.
Reimer is pro-AI, but he believes it requires a human being involved in any meaningful decisions. "Given the well-discussed limitations, there are areas where processing large amounts of data makes sense (e.g., looking for patterns in health care data), but until we can prove the reliability of these systems, human oversight as a co-pilot will be critical."
While AI clearly has real utility, it's not ready to operate alone in high-stakes situations. The real issue is the gap between CEO hype and the flawed reality many of us encounter on a regular basis using these tools. Until that gap closes, humans need to remain in the loop for decisions where the outcome actually matters.
~Ron
What's new on the blog 📰
AI-generated code is flooding the software supply chain with new threats
While it’s undeniable that AI coding tools make it easier to generate code faster and more efficiently, that new velocity is also introducing risks to the software supply chain that can’t be swept under the rug.
I spoke with Cassie Crossley, author of Software Supply Chain Security, about how AI is changing the threat landscape, and why caution should be the watchword.
"You can't know every single library, and frankly, there are not enough security layers built in yet for vibe coding that I would ever trust at the moment."
A SaaS pioneer confronts the rise of AI
As the SaaS relevancy debate in the age of AI rages on, I spoke to a SaaS pioneer, Tien Tzuo. He's CEO at subscription billing platform Zuora, and was employee #11 at Salesforce.
He doesn’t sugarcoat the threat AI poses to SaaS when he says, "Can AI versions of an enterprise software category replace a non-AI version? It's absolutely going to happen. How could it not?"
But he also argues that SaaS companies still have distinct advantages that vibe coding alone can’t match.
AI's not gonna kill SaaS, but it will force some changes
I sat down with good friend Alex Wilhelm and we wrote a post together, as we often used to when we worked together at TechCrunch. We took a look at the recent SaaS stock beat-down and looked at whether SaaS is getting what it deserves, or if, maybe, just maybe investors are overreacting a wee bit. Hope it's the first of many. Please let us know what you think of this collab at feedback@fastforward.blog.
Reports find agentic AI is running into limits of how work is organized
Anyone who has looked at digital transformation efforts over the last 15 years knows that older systems, institutional inertia and resistance from management and rank and file employees can undermine the best of intentions.
Perhaps it's not surprising then that two recent reports found that companies trying to implement agentic AI are running into broad organizational and technical challenges, and the vast majority of AI projects still remain stuck in experiment and PoC purgatory.
News of the Week 📣
When data centers become war targets
This week as bombs exploded across the Middle East, there were many human casualties, which should always remain the primary focus. At the same time, it's impossible to ignore that strategic infrastructure was also targeted, including several Amazon data centers in Bahrain and the United Arab Emirates, according to multiple reports.
The AWS health dashboard, which keeps a running tally on the status of data centers and Amazon services across the company, started reporting issues on March 1st, only saying that an "object" had struck several data centers "creating sparks and fire."
The following day, the company got more specific: "In the UAE, two of our facilities were directly struck, while in Bahrain, a drone strike in close proximity to one of our facilities caused physical impacts to our infrastructure," the company wrote. "These strikes have caused structural damage, disrupted power delivery to our infrastructure, and in some cases required fire suppression activities that resulted in additional water damage. We are working closely with local authorities and prioritizing the safety of our personnel throughout our recovery efforts."
As companies build data centers around the world, conflicts like this highlight a new reality where these facilities can become strategic targets. When that happens, customers that rely on these services are left with degraded or missing services unless they have backup plans.
Claude hits #1 in Apple App Store after OpenAI’s DoD deal sparks backlash
As I wrote in the ForwardThinking commentary today, the government took harsh action against Anthropic this week by labeling it a supply chain risk. As news of the conflict between Anthropic and the DoD spread far and wide last weekend, the general public responded by propelling the Claude app to number one on the Apple App Store this week.
As any parent of a teenager will tell you, the best way to make something attractive is to forbid it, but it was more than a case of questioning authority. A survey conducted by ITIF in the aftermath of the Anthropic-DoD brouhaha found that two thirds of respondents "believe private technology companies have a responsibility to set limits on how their products can be used, even if the government wants to use them differently."
Meanwhile, the public was pushing back against OpenAI, upset over the deal the company made in the aftermath of Anthropic's fight. That has given rise to a grassroots movement to boycott OpenAI with reports of 2.5 million joining and up to 1.5 million reportedly giving up their subscriptions.
While neither downloading a free app nor joining an online movement is likely to have a meaningful impact on either company, it does offer a glimpse into how the public feels about the positions each one has taken.
This week in startups
- Fig Security, yet another security startup out of Israel, announced a $38 million investment this week. The company maps how security tools and data pipelines connect so teams can spot breaks or gaps in their stack before they lead to missed detections or major incidents.
- WorkOS announced a $100 million Series C on a $2 billion valuation. The startup is building the plumbing to help agents communicate across systems including authentication, granular permissions, integrations, encryption, abuse detection, feature flags, MCP, all of which should help agents communicate more safely as they move across systems.
- Lio, an early-stage AI startup announced a $30 million Series A led by Andreessen Horowitz. The company is trying to upend the procurement category by building agents that sit on top of a company's procurement systems with the long-term goal of being able to place orders without humans involved.
What I'm reading 📚
AI overuse could spark "brain fry," new research finds
~By Emily Peck, Axios
The trillion-dollar question: Who pays when the industry’s AI bill comes due?
~By Grant Gross, CIO
SaaS in, SaaS out: Here’s what’s driving the SaaSpocalypse
~By Dominic-Madori Davis, TechCrunch
What is OpenAI going to do when the truth comes out?
~By Casey Newton, Platformer
Look who's talking 👄
"You need guardrails and vetted intermediaries so people aren’t pulling random, unvalidated packages from the wild — or even legitimate packages that have been poisoned later on in a supply chain attack,"
~CrowdStrike CTO Elia Zaitsev in my FastForward story this week on software supply chain risk from AI.
