In the debate over open-weight models, Capital One makes a different case

The AI industry is in the midst of a serious debate about open-weight models with the arguments centered on safety versus openness. While the AI labs argue that open weights lack sufficient guard rails -- pretty rich given the recent spate of reports of private models run amok – Capital One believes it's more about versatility and being able to mold the models for their unique requirements as a regulated bank.

I recently caught up with Milind Naphade, SVP of AI foundations at Capital One. He asserts that the bank has little choice but to go with open weights because building tools for a highly regulated industry requires massive customization, and that's generally not possible with the proprietary models.

"We actually go and customize the model to the point where you would not even recognize the original model if we showed you what we ended up with," Naphade told FastForward. 

"To do all that needs open-weight models, we just cannot do that without them. And why is that important? Because that's the only way that we can fulfill our twin mandates of delivering high accuracy with regulatory compliance," he said.

Data is key

Naphade's argument for customization builds on one his colleague made to me last year. When I interviewed chief scientist and head of enterprise data and AI Prem Natarajan, he contended that the bank's data advantage, one it had built over decades, was what helped prepare it for generative AI.

"The only way for my data advantage to become my AI advantage is if I can customize the models with my data, and I cannot customize closed source models," Natarajan said at the time. 

That thinking reflects Capital One’s broader approach to AI, which Naphade said has to be built into the entire system. "We fundamentally believe that the AI transformation is not a layer that you put on top of an existing system, it needs to actually happen in the entire system, top to bottom," he said. 

It's worth noting that last year, the company talked openly about using Meta's Llama models. This year, it was unwilling to share which models it is using.

Accuracy versus efficiency

When you work at a bank, even 90% accuracy isn't good enough. While no model is perfect, Naphade says that when it comes to AI applications, accuracy is always the first priority. Efficiency, controlling the cost of running the models, comes in a close second. 

"Our approach is, we start with the accuracy that we must achieve. Once we have customized the model to the point where we have that, then we start working on efficiency," he said. That means finding ways to achieve the lowest latency at the lowest cost.

To achieve that, they use several techniques including distillation and multi-token prediction. "We use distillation, which is when you take a model, and then you create a much more efficient model out of that, a derivative model, which is as accurate as the original model, but far less expensive in terms of the compute," he said.

Multi-token prediction lets a model generate several tokens per pass instead of one, reducing the number of trips required to generate an answer.

Putting it to work

When the company released Chat Concierge last year, an AI assistant to help consumers buy cars, it was built with these approaches. Through the customization process, the company went from a completely unacceptable latency of one minute between question and answer to less than a second in the released version. 

"We optimized the heck out of that. Now it is less than one second without compromising the accuracy," he said. As he pointed out, it's an extremely complex system involving multiple agents — about 15 LLM calls per response on average — yet using their "science and engineering prowess" and working with Nvidia, they were able to reduce the latency dramatically.

For Capital One, the debate isn't really about ideology, or even finding the most capable model. Open-weight models are simply the most practical choice for a bank that needs complete control over how AI is trained, tuned and deployed.