Skip to main content
The shift to building applications on top of Large Language Models (LLMs) fundamentally breaks traditional software testing. Methodologies designed for predictable, deterministic systems are ill-equipped to handle the non-deterministic nature of LLM-based apps, creating a critical gap in quality assurance.

​
The Challenges of Testing Gen AI Apps

Testing Gen AI applications is fundamentally different from testing traditional software. The core difficulty stems from two interconnected problems that create a cascade of complexity.
  • The Infinite Input Space: The range of possible user inputs is basically endless. You cannot write enough static test cases to cover every scenario or every subtle variation in how a user might phrase a question. This problem is compounded in conversational AI, where multi-turn chats branch out fast. Even small wording changes in an early turn can dramatically shift how later turns unfold, making the state space of a conversation explode.
  • Non-Deterministic & “Fuzzy” Outputs: Unlike a traditional function where 2 + 2 is always 4, a Gen AI model can produce a wide variety of responses to the same prompt. Outputs aren’t consistent, so a single clean run doesn’t mean much. On top of this, quality itself is fuzzy. There’s rarely a single right answer, and you’re often juggling competing priorities like helpfulness, safety, and tone all at once. Even human judges don’t always agree on what constitutes a “good” response.
These two fundamental challenges are amplified by the complex systems we build around the models. The integration of tools, memory, and RAG adds more moving parts, each a potential point of failure. External APIs or vector indexes drift over time, and even the underlying models can change quietly in the background. When you add operational concerns like costs, rate limits, and flaky integrations, it’s easy to think you’ve tested enough when you really haven’t. This reality means we must move beyond simple pass/fail checks and adopt a more robust, statistical approach to evaluation. The process becomes less like traditional software testing and more like a clinical trial. The goal isn’t to guarantee a specific outcome for every individual interaction, but to ensure the application is effective for a significant portion of users, within a defined risk tolerance.

​
Introducing SigmaEval

SigmaEval is a Python framework for the statistical, end-to-end evaluation of Gen AI apps, agents, and bots that helps you move from “it seems to work” to making rigorous, data-driven statements about your AI’s quality. It allows you to set and enforce objective quality bars by making statements like:
“We are confident that at least 90% of user issues coming into our customer support chatbot will be resolved with a quality score of 8/10 or higher.”
“With a high degree of confidence, the median response time of our new AI-proposal generator will be lower than our 5-second SLO.”
“For our internal HR bot, we confirmed that it will likely succeed in answering benefits-related questions in fewer than 4 turns in a typical conversation.”
This process transforms subjective assessments into quantitative, data-driven conclusions, giving you a reliable framework for building high-quality AI apps.
⌘I