Confluent's Jay Kreps returns to his large company roots with acquisition by IBM
When Jay Kreps was working at LinkedIn in 2010, the industry was just beginning to grapple with what at the time was being called big data. How could you reliably move massive amounts of event data — user activity, system logs, operational metrics — between a growing number of internal systems in near real time? We didn't have the mature cloud infrastructure we have today, and the existing messaging systems had trouble keeping up. The answer was what became Apache Kafka, an open source streaming data platform created by Kreps and two of his colleagues.
By 2014, the project had grown so large that Kreps and his two Kafka co-developers, Neha Narkhede and Jun Rao, left LinkedIn to build a startup they named Confluent around it. They raised over $450 million before going public in 2021, and last year IBM announced it was acquiring them for $11 billion, a price that represented a substantial premium over its market cap at the time. In other words, it was an offer too good to turn down.
That's the kind of arc that any startup founder group would be thrilled to achieve. But the reality is that even when it's a good decision for the company, it can still come with some real human emotion when it comes to pulling the trigger on a deal.
Today, Kafka moves general-purpose data streams between systems, which can be critical for giving AI models real-time context. That ability is a big part of the value proposition that appealed to IBM as it tries to be the AI orchestration layer.
When I spoke to Kreps at IBM Think in Boston earlier this month, I could sense that tension. He was happy to have delivered to his shareholders and to be solving big problems around AI for IBM customers using the technology he helped build, but that doesn't mean he wasn't a tad wistful about it too.
“It was one of the hardest decisions in the history of the company for me personally, even though I think it was the right call,” Kreps told FastForward. “In some sense it was kind of obvious, but that didn’t make it any easier.”
Ultimately, the deal ended up giving Confluent the best of both worlds. Kreps was able to maintain some autonomy inside the larger organization, while taking advantage of IBM's scale to take the company to heights it might not have reached on its own. But that doesn't mean he didn't recognize the inherent tradeoffs in a deal like this. As he said, "You don't get to see the path not traveled."
Moving back to a big corporation
Prior to the acquisition, Confluent boasted more than 6,500 enterprise customers, including 40% of the Fortune 500, but now Kreps is back inside a much larger firm, where the challenge is less about the technology than how to take advantage of IBM's scale.
The acquisition only closed in March, and he is still trying to navigate his way between what it means to work inside a larger organization with its strong sales and marketing motion, while still preserving some independence for his unit.
"We've kind of left this as an independent unit outside the rest of the IBM software org, but still spiritually associated with it," he said. In practice, they work together and cross-pollinate where it makes sense, while preserving continuity for existing Confluent customers.
"And then we've tried to do what we can to make sure the go-to-market organization is really plugged into what IBM has and is working closely there" with the goal of putting Confluent in front of more customers and making it part of more deals, accelerating reach that would have been hard to achieve on its own.
Made for the AI age
Confluent and Kafka's real-time data streaming has fit neatly into IBM's argument for being the middle layer for enterprise companies trying to implement AI. "Our customers have all this real time data flowing through Kafka," he said. "We bring the ability to do these very rich transformations for processing of that data, and the last mile in the chain is the ability to plug that into these AI models and serve up context off of that."
Today, a lot of organizations are simply moving the data into a data lake and processing it later, but that's not always good enough when you require it to be more timely. "You need something that's in sync with the current state of the business," Kreps said. But at the same time you need to have a high level of security because you can't just feed all the live business data into a model and expose it to the entire company. It requires controls, so it gets filtered inside the Kafka governance layer, and only the data that's allowed is pushed through.
“These models are very smart, but they're not smarter than the context data they're given,” Kreps said. “If they're given bad information about the state of the world, they make bad decisions.” Together, the two companies are betting that if the right data reaches the right model, it could minimize those bad decisions.
