Building a Model Evaluation Practice for Production AI

Most teams pick an AI model the same way they pick a coffee machine - the demo looks good or they hear about it from someone, so they assume it works. But in production, evaluation decisions directly impact cost, latency, accuracy, and user trust. Without a structured eval practice, you're flying blind when a model regresses, or a new release claims to be better. This session gives you a framework for evaluating LLMs against *your* workloads and operationalizing evaluation as a continuous practice, not a one-time exercise. What we'll cover: * Why standard benchmarks are often misleading for enterprise use cases * Building a golden dataset: how to sample, label, and version evaluation sets from real production traffic * Key metrics beyond accuracy: hallucination rates, precision/recall tradeoffs, cost-per-correct-answer, and latency percentiles * Running evals at scale: open-source tools (Ragas, PromptFoo, LangSmith) and using Amazon Bedrock Model Evaluation * Setting a model evaluation gate: how to enforce a promotion threshold before any model goes to production * Demo: running a side-by-side eval across two models on a real classification task Speaker Bio: Udaya is a Senior Solutions Architect at AWS and an AI-DLC Champion, with 18+ years of experience designing and building enterprise applications.