AI Evals For Engineers & PMs
Pre-course
Start here: Welcome to Maven
🚀 Two things to do to get started
0 videos
🤔 What makes cohort-based learning different?
0 videos
🛟 Where to get support
0 videos
Week 1
Start Here
📄 Orientation: Required Reading, Discord & Course Materials
0 videos
📄 How To Get The Most Out Of This Course
0 videos
📄 Meet our TA's!
0 videos
📄 This Week's Overview: Objectives, Readings, and HWs
1 video
Lesson 1: Fundamentals & Lifecycle of Application-Centric Evals
📄 Recorded Lecture: Fundamentals & Lifecycle of Application-Centric Evals
0 videos
📄 Chapter 1: Introduction
0 videos
📄 Chapter 2: What is LLM Evaluation?
0 videos
📄 Chapter 3: Why is LLM Development So Hard?
3 videos
📄 Chapter 4: The Evaluation Lifecycle
2 videos
📄 Chapter 5: Prompting Fundamentals
2 videos
📄 Chapter 6: Preview of Error Analysis
2 videos
📄 Optional Homework
0 videos
Lesson 2 & 3: Systematic Error Analysis
📄 Recorded Lecture: Systematic Error Analysis
0 videos
📄 Chapter 1: Introduction
0 videos
📄 Chapter 2: Create Synthetic Data
2 videos
📄 Chapter 3: Open Coding
2 videos
📄 Chapter 4: Axial Coding
0 videos
📄 Break & Reflection Question
0 videos
📄 Chapter 5: Error Analysis Recap
0 videos
📄 Chapter 6: Multi-Turn Evaluation
0 videos
📄 Chapter 7: Collaborative Evaluation
0 videos
📄 Chapter 8: Rubrics for Collaborative Evaluation
0 videos
📄 Optional HW
0 videos
FAQ and Links
Office Hours
OCT 8 Optional: Live Office Hours 1 WED 10/89:00 PM—10:00 PM (GMT+5:30) OPTIONAL Recording
0 videos
OCT 9 Optional: Live Office Hours 2 THU 10/99:00 AM—10:00 AM (GMT+5:30) OPTIONAL Recording
0 videos
OCT 10 Optional: Live Office Hours 3 FRI 10/109:00 PM—10:00 PM (GMT+5:30) OPTIONAL Recording
0 videos
Week 2
Start Here
Lesson 4&5: Automated Evaluators
📄 Recorded Lesson: Automated Evaluators
0 videos
📄 Chapter 1: Introduction
0 videos
📄 Chapter 2: Error Analysis Recap
0 videos
📄 Chapter 3: Code-based vs LLM-based Evaluators
2 videos
📄 Chapter 4: Overview of Creating a LLM-as-Judge Evaluator
0 videos
📄 Chapter 5: Example Criterion for LLM-as-a-Judge
0 videos
📄 Chapter 6: Crafting and Refining the LLM Judge Prompt
1 video
📄 Chapter 7: LLM as Judge Coding Demo
2 videos
📄 Chapter 8: Correcting Bias in LLM-as-Judge Evaluators
0 videos
📄 Chapter 9: Pitfalls to Avoid When Building Automated Evaluators
0 videos
📄 Optional HW
0 videos
Office Hours
OCT 14 Optional: Live Office Hours 4 TUE 10/149:00 PM—10:00 PM (GMT+5:30) OPTIONAL Recording
0 videos
OCT 15 Optional: Live Office Hours 5 WED 10/159:00 AM—10:00 AM (GMT+5:30) OPTIONAL Recording
0 videos
FAQ and Links
Week 3
Start Here
Lesson 6: Complex Architectures
📄 Complex Architectures (e.g., RAG)
0 videos
📄 Chapter 1: Retrieval-Augmented Generation
0 videos
📄 Chapter 2: Evaluating Retrieval Quality
0 videos
📄 Chapter 3: Synthetic Data for RAG
1 video
📄 Chapter 4: Retrieval Metrics
2 videos
📄 Chapter 5: Evaluating Generation Quality
0 videos
📄 Chapter 6: Common Pitfalls With RAG
3 videos
📄 Chapter 7: Complex Agents Beyond RAG
0 videos
📄 Optional HW
0 videos
Office Hours
OCT 21 Optional: Live Office Hours 6 TUE 10/219:00 PM—10:00 PM (GMT+5:30) OPTIONAL Recording
0 videos
OCT 22 Optional: Live Office Hours 7 WED 10/229:00 AM—10:00 AM (GMT+5:30) OPTIONAL Recording
0 videos
OCT 23 Optional: Live Office Hours 8 THU 10/239:00 PM—10:00 PM (GMT+5:30) OPTIONAL Recording
0 videos
OCT 24 Optional: Live Office Hours 9 FRI 10/249:00 AM—10:00 AM (GMT+5:30) OPTIONAL Recording
0 videos
Tools Workshops: Arize, Langsmith, Braintrust, Truesight - Homework 1&2
📄 Introduction
0 videos
📄 HW 1&2 walkthrough with Braintrust (pre-recorded)
0 videos
📄 HW 1&2 walkthrough with Phoenix (pre-recorded)
0 videos
📄 HW 1&2 walkthrough with LangSmith (pre-recorded)
0 videos
📄 HW 1&2 Walkthrough with Truesight (Pre-recorded)
2 videos
📄 Additional Links
1 video
Tools Workshops: Arize, Braintrust, Langsmith, Truesight - Homework 3
📄 HW 3 walkthrough with Phoenix (pre-recorded)
0 videos
📄 HW 3 walkthrough with Braintrust (pre-recorded)
2 videos
📄 HW 3 walkthrough with LangSmith (Notebook only)
0 videos
📄 HW3 walkthrough with Truesight (pre-recorded)
0 videos
Live Coding Workshops
FAQ and Links
Week 4
Start Here
Lessons 7 and 8: Advanced Topics
📄 Advanced Topics
0 videos
📄 Chapter 1: Introduction
0 videos
📄 Chapter 2: Why Build Custom UIs?
0 videos
📄 Chapter 3: Principles for Designing Effective Interfaces
0 videos
📄 Chapter 4: Sampling Traces to Review
0 videos
📄 Chapter 5: EvalGen Case Study
2 videos
📄 Chapter 6: DocWrangler Case Study
0 videos
📄 Chapter 7: Building A Custom Annotation UI
0 videos
📄 Chapter 8: Strategies to improve accuracy
2 videos
📄 Chapter 9: Final Thoughts
0 videos
Office Hours
OCT 28 Optional: Live Office Hours 10 TUE 10/289:00 PM—10:00 PM (GMT+5:30) OPTIONAL Recording
2 videos
OCT 29 Optional: Live Office Hours 11 WED 10/299:00 AM—10:00 AM (GMT+5:30) OPTIONAL Recording
0 videos
OCT 31 Optional: Live Office Hours 12 FRI 10/3110:00 PM—11:00 PM (GMT+5:30) OPTIONAL Recording
0 videos
Tools Workshops: Arize, Braintrust, Truesight - Homework 4
📄 HW 4 walkthrough with Braintrust (pre-recorded)
0 videos
📄 HW 4 walkthrough with Phoenix (pre-recorded)
2 videos
📄 HW 4 walkthrough with Truesight (pre-recorded)
0 videos
Live Coding Workshops
OCT 28 Optional: Live coding workshop HW 3 w/ Isaac Flath TUE 10/283:30 AM—5:00 AM (GMT+5:30) OPTIONAL Recording
1 video
OCT 30 Optional: Live coding workshop HW 4 w/ Isaac Flath THU 10/303:30 AM—5:00 AM (GMT+5:30) OPTIONAL Recording
0 videos
NOV 1 Optional: Live coding workshop HW 5 w/ Isaac Flath SAT 11/13:30 AM—5:00 AM (GMT+5:30) OPTIONAL Recording
0 videos
Live Guest Speaker
FAQ and Links
Post-course
Tools Workshops: Arize, Braintrust - Homework 5
📄 HW 5 walkthrough with Braintrust (pre-recorded)
0 videos
📄 HW 5 walkthrough with Phoenix (pre-recorded)
2 videos
Bonus
Using AI Assisted Coding For Homework Assignments: Walkthrough