- ホーム
- > 洋書
- > 英文書
- > Computer / General
Full Description
Before you trust critical business systems to an AI model, you need to answer a few questions. Will it be fast enough? Will the system satisfy user expectations? Is it safe and can you trust the output? This book will help you answer these questions before you roll out an AI system, and make sure it runs smoothly after you deploy.
Learn simple ways to test how your model behaves before it is used in real systems.
Try your model with real data to see how it performs in real situations.
Design A/B tests that validate model impact on key product metrics.
Spot nuanced failures with human-in-the-loop feedback and qualitative evaluations.
Use LLMs to help review and test models more quickly.
AI Model Evaluation teaches you how to effectively evaluate and assess machine learning models for better scaling and integration. Each chapter looks at a different way to test a model, starting with offline evaluations and moving into live A/B tests, shadow traffic deployments and LLM-based feedback loops. The book uses a hands-on example grounded in a movie recommendation engine.
After reading this book, you will be able to evaluate both model behaviour and engineering system performance. You will have the tools to ensure your AI systems are effective and reliable in production. This book is for practitioners with experience in machine learning, data science, or software engineering.
Contents
PART 1: AI MODEL OFFLINE EVALUATIONS
1 SETTING THE STAGE FOR OFFLINE EVALUATIONS
2 ANATOMY OF AN OFFLINE EVALUATION
3 USING OFFLINE EVALUATIONS AS DIAGNOSTICS
4 ENGINEERING SYSTEM PERFORMANCE EVALUATIONS
5 COUNTERFACTUAL EVALUATIONS
PART 2: AI MODEL ONLINE EVALUATIONS
6 EVALUATING MODELS IN AN A/B TEST
7 FROM OFFLINE EVALUATION TO LIVE EXPERIMENT
8 PITFALLS OF ONLINE METRICS
PART 3: LLM-AS-A-JUDGE
9 LLM-AS-A-JUDGE FUNDAMENTS
10 DESIGN PATTERNS FOR LLM-AS-A-JUDGE
PART 4: HUMAN-IN-THE-LOOP OR QUALITATIVE EVALUATIONS FOR AI MODELS
11 WHY HUMAN EVALUATION MATTERS
12 HUMAN-IN-THE-LOOP EVALUATION TECHNIQUES
13 TRUST, SAFETY, AND RED TEAMING
14 OPERATIONALIZING HUMAN FEEDBACK
-
- 電子書籍
- ELLE Japon 2022年1月号
-
- 電子書籍
- ハワイスタイル No.36



