It’s no exaggeration to say the generative AI (GenAI) and large language models (LLMs) have become the dominant topic in technology, finance, and business circles in recent years. The power of these new technologies has been well demonstrated, with entire companies shifting their operating strategies to incorporate them into automated workflows, customer support chatbot services, and a range of analytical roles. However, despite the rapid implementation of these tools and the media hype driving it, one question has yet to be satisfactorily answered by the industry: are AI tools safe, and how can that be proven?
It’s that question, and its necessary follow-up investigations, that are the focus of Neel Somani’s work. A researcher and entrepreneur with a proven track record with machine learning, quantitative research, and blockchain development, Neel Somani has honed his analytical expertise solving problems in global markets. With his triple major in Computer Science, Mathematics, and Business Administration from UC Berkeley, where he contributed to research in type systems, differential privacy, and scalable machine learning frameworks, Somani is uniquely trained and positioned to tackle questions of AI safety.
“In safety, there are two big problems,” Somani explains. “One is ‘reward hacking,’ where the model takes optimizing its objective ‘too far,’ to the point of causing damage to other reasonable goals or ethical limitations. The second is ‘scheming,’ where if the model lies about its "true" intentions, and is actually doing something nefarious.”
These problems, among others, represent a yet-unsolved challenge facing AI development and proper integration into modern systems and businesses. Solving these problems requires tackling them at the technical fundamental level. Fortunately, Somani might have figured out the best practice for doing just that: formal methods.
The Problem Of Verification
Machine learning, GenAI, and LLMs share a common challenge of robustness; ideally, the model’s outputs are not wildly different after minor changes to their inputs. If that’s the case, it’s suggestive of a model that is unreliable and unstable—thus undesirable. The problem of verifying robustness with AI technology is that the inputs are continuous by design, which demands that any true verification test investigates infinitely many minor variable inputs for consistent outputs. This is impossible for humans without applying formal methods, and it’s only one of a handful of problems.
“A concrete problem in safety is that many claims are difficult to falsify with current methodologies,” Somani says. “For example, when someone analyzes a model, they might have a guess as to what it's doing under-the-hood, but there's no way to really prove or disprove that hypothesis.”
Another problem facing verification and AI safety is that most businesses refuse to touch machine learning models’ internal code and structure. At most, businesses and organizations at the bleeding edge of the technology might fine-tune and customize these tools for their workflows and operations, but none are willing to dive into the core of the technology to investigate further. Additionally, there are misconceptions about whether the technology is mature enough to apply formal methods and analytics at all; some fear that Somani’s research will join other ambitious approaches among the list of history’s failures. His rebuttal is simple: it’s far too early to throw out ambitious visions for safety and interpretability, and formal methods provide a North Star for ambitious researchers to work toward.
“My research provides a defensible, technically grounded North Star for AI safety research groups to build toward,” says Somani. “Regulators are increasingly seeking standards that balance innovation with safety, and they'll need some reasonable standard that doesn't cripple the models while also providing certifiable guarantees. Once formal methods in machine learning are further developed, they could form the backbone of such regulation.”
Defining The AI Safety Paradigm
Given that he’s working on safety for complex algorithms and mathematically-intense technologies, it stands to reason that Somani’s chosen strategy for tackling AI safety starts with formal methods: mathematically rigorous specification and analysis techniques derived from programming first principles. Having a grounded and logically strong foundation is crucial for any attempt to make advanced machine learning systems more reliable, trustworthy, and safe, because there isn’t any currently-established system or paradigm for doing so. There just isn’t an existing way to certify that a given system is safe or fully understood.
“Formal methods are the gold standard because they're the only way to establish strong, principled guarantees about programs,” Somani explains, “but we're a long way off from being able to apply them to machine learning systems.”
Neel Somani discovered in his own investigations of machine learning technologies (an existing field of interest for him) when he put together a project applying formal verification to GPU kernels. That project revealed just how underrepresented formal methods experts are in the machine learning industry, especially in regards to safety and interpretability—two fields that are preparadigmatic in AI spaces. Formal verification presents the opportunity to change that and provide the industry with something it desperately wants: a certification of predictability, reliability, and trustworthiness, rooted in the very functionality of the technology.
“Many concepts like formal definitions for privacy or formal safety definitions are possible to define but impossible to verify at real-world scale,” says Somani. “My work as of late has been in concretely implementing these methods to prove viability in small examples, with the goal of scaling the methods down the line.”
Neel Somani has a number of running projects that apply formal methods to LLMs and AI tools to solve these problems. One such project, Symbolic Circuit Distillation, takes what interpretability researchers call a ‘circuit,’ tries to extract the program it encodes, then proves or disproves equivalence over the relevant inputs to track robustness. When it comes to scalability and projects leveraging limited theory and aimed at real-world applications, his KV Marketplace project demonstrates the ability to fork an inference engine called vLLM in such a way that optimizes its GPU caching to prevent recomputation.
These, and other projects, stand as functional and logically-grounded examples that prove the immediately relevant applications of formal methods, and are a proof-text for future research. As AI technology grows in popularity and becomes more ubiquitous, the need for a trustworthy certification of robustness and safety grows with it. Neel Somani is firm in his convictions that this work is vitally important.
“In an ideal world, for high stakes or mission critical ML systems (health care, financial applications), the entire workflow would be formally specified at all levels,” he says. “As machine learning models write a larger percentage of our code, we might expect more out of them, such as requiring that these formal semantics are defined for the purpose of verification.”
