AI Evals For Engineers & PMs
🚀 Two things to do to get started
Pre-course · 0 videos
🤔 What makes cohort-based learning different?
Pre-course · 0 videos
🛟 Where to get support
Pre-course · 0 videos
📄 Orientation: Required Reading, Discord & Course Materials
Week 1 · 0 videos
📄 How To Get The Most Out Of This Course
Week 1 · 0 videos
📄 Meet our TA's!
Week 1 · 0 videos
📄 This Week's Overview: Objectives, Readings, and HWs
Week 1 · 3 videos
📄 Recorded Lecture: Fundamentals & Lifecycle of Application-Centric Evals
Week 1 · 0 videos
📄 Chapter 1: Introduction
Week 1 · 0 videos
📄 Chapter 2: What is LLM Evaluation?
Week 1 · 0 videos
📄 Chapter 3: Why is LLM Development So Hard?
Week 1 · 0 videos
📄 Chapter 4: The Evaluation Lifecycle
Week 1 · 0 videos
📄 Chapter 5: Prompting Fundamentals
Week 1 · 0 videos
📄 Chapter 6: Preview of Error Analysis
Week 1 · 3 videos
📄 Optional Homework
Week 1 · 0 videos
📄 Recorded Lecture: Systematic Error Analysis
Week 1 · 0 videos
📄 Chapter 1: Introduction
Week 1 · 0 videos
📄 Chapter 2: Create Synthetic Data
Week 1 · 0 videos
📄 Chapter 3: Open Coding
Week 1 · 0 videos
📄 Chapter 4: Axial Coding
Week 1 · 0 videos
📄 Break & Reflection Question
Week 1 · 0 videos
📄 Chapter 5: Error Analysis Recap
Week 1 · 0 videos
📄 Chapter 6: Multi-Turn Evaluation
Week 1 · 0 videos
📄 Chapter 7: Collaborative Evaluation
Week 1 · 0 videos
📄 Chapter 8: Rubrics for Collaborative Evaluation
Week 1 · 0 videos
📄 Optional HW
Week 1 · 0 videos
📄 Frequently Asked Questions
Week 1 · 0 videos
📄 Optional Recommended Reading
Week 1 · 2 videos
OCT 8 Optional: Live Office Hours 1 WED 10/89:00 PM—10:00 PM (GMT+5:30) OPTIONAL Recording
Week 1 · 0 videos
OCT 8 Optional: Live Office Hours 1 WED 10/89:00 PM—10:00 PM (GMT+5:30) OPTIONAL Recording
Week 1 · 0 videos
OCT 9 Optional: Live Office Hours 2 THU 10/99:00 AM—10:00 AM (GMT+5:30) OPTIONAL Recording
Week 1 · 0 videos
OCT 9 Optional: Live Office Hours 2 THU 10/99:00 AM—10:00 AM (GMT+5:30) OPTIONAL Recording
Week 1 · 0 videos
OCT 10 Optional: Live Office Hours 3 FRI 10/109:00 PM—10:00 PM (GMT+5:30) OPTIONAL Recording
Week 1 · 0 videos
OCT 10 Optional: Live Office Hours 3 FRI 10/109:00 PM—10:00 PM (GMT+5:30) OPTIONAL Recording
Week 1 · 0 videos
📄 This Week's Overview
Week 2 · 1 video
📄 Recorded Lesson: Automated Evaluators
Week 2 · 0 videos
📄 Chapter 1: Introduction
Week 2 · 0 videos
📄 Chapter 2: Error Analysis Recap
Week 2 · 0 videos
📄 Chapter 3: Code-based vs LLM-based Evaluators
Week 2 · 2 videos
📄 Chapter 4: Overview of Creating a LLM-as-Judge Evaluator
Week 2 · 0 videos
📄 Chapter 5: Example Criterion for LLM-as-a-Judge
Week 2 · 0 videos
📄 Chapter 6: Crafting and Refining the LLM Judge Prompt
Week 2 · 0 videos
📄 Chapter 7: LLM as Judge Coding Demo
Week 2 · 0 videos
📄 Chapter 8: Correcting Bias in LLM-as-Judge Evaluators
Week 2 · 2 videos
📄 Chapter 9: Pitfalls to Avoid When Building Automated Evaluators
Week 2 · 0 videos
📄 Optional HW
Week 2 · 0 videos
OCT 14 Optional: Live Office Hours 4 TUE 10/149:00 PM—10:00 PM (GMT+5:30) OPTIONAL Recording
Week 2 · 0 videos
OCT 14 Optional: Live Office Hours 4 TUE 10/149:00 PM—10:00 PM (GMT+5:30) OPTIONAL Recording
Week 2 · 0 videos
OCT 15 Optional: Live Office Hours 5 WED 10/159:00 AM—10:00 AM (GMT+5:30) OPTIONAL Recording
Week 2 · 0 videos
OCT 15 Optional: Live Office Hours 5 WED 10/159:00 AM—10:00 AM (GMT+5:30) OPTIONAL Recording
Week 2 · 0 videos
📄 Frequently Asked Questions
Week 2 · 0 videos
📄 Optional Links
Week 2 · 0 videos
📄 This Week's Overview
Week 3 · 0 videos
📄 Complex Architectures (e.g., RAG)
Week 3 · 0 videos
📄 Chapter 1: Retrieval-Augmented Generation
Week 3 · 0 videos
📄 Chapter 2: Evaluating Retrieval Quality
Week 3 · 0 videos
📄 Chapter 3: Synthetic Data for RAG
Week 3 · 0 videos
📄 Chapter 4: Retrieval Metrics
Week 3 · 1 video
📄 Chapter 5: Evaluating Generation Quality
Week 3 · 0 videos
📄 Chapter 6: Common Pitfalls With RAG
Week 3 · 0 videos
📄 Chapter 7: Complex Agents Beyond RAG
Week 3 · 0 videos
📄 Optional HW
Week 3 · 0 videos
OCT 21 Optional: Live Office Hours 6 TUE 10/219:00 PM—10:00 PM (GMT+5:30) OPTIONAL Recording
Week 3 · 0 videos
OCT 21 Optional: Live Office Hours 6 TUE 10/219:00 PM—10:00 PM (GMT+5:30) OPTIONAL Recording
Week 3 · 0 videos
OCT 22 Optional: Live Office Hours 7 WED 10/229:00 AM—10:00 AM (GMT+5:30) OPTIONAL Recording
Week 3 · 0 videos
OCT 22 Optional: Live Office Hours 7 WED 10/229:00 AM—10:00 AM (GMT+5:30) OPTIONAL Recording
Week 3 · 0 videos
OCT 23 Optional: Live Office Hours 8 THU 10/239:00 PM—10:00 PM (GMT+5:30) OPTIONAL Recording
Week 3 · 0 videos
OCT 23 Optional: Live Office Hours 8 THU 10/239:00 PM—10:00 PM (GMT+5:30) OPTIONAL Recording
Week 3 · 0 videos
OCT 24 Optional: Live Office Hours 9 FRI 10/249:00 AM—10:00 AM (GMT+5:30) OPTIONAL Recording
Week 3 · 0 videos
OCT 24 Optional: Live Office Hours 9 FRI 10/249:00 AM—10:00 AM (GMT+5:30) OPTIONAL Recording
Week 3 · 0 videos
📄 Introduction
Week 3 · 0 videos
📄 HW 1&2 walkthrough with Braintrust (pre-recorded)
Week 3 · 0 videos
📄 HW 1&2 walkthrough with Phoenix (pre-recorded)
Week 3 · 2 videos
📄 HW 1&2 walkthrough with LangSmith (pre-recorded)
Week 3 · 0 videos
📄 HW 1&2 Walkthrough with Truesight (Pre-recorded)
Week 3 · 0 videos
📄 Additional Links
Week 3 · 0 videos
📄 HW 3 walkthrough with Phoenix (pre-recorded)
Week 3 · 0 videos
📄 HW 3 walkthrough with Braintrust (pre-recorded)
Week 3 · 3 videos
📄 HW 3 walkthrough with LangSmith (Notebook only)
Week 3 · 0 videos
📄 HW3 walkthrough with Truesight (pre-recorded)
Week 3 · 0 videos
OCT 23 Optional: Live coding workshop HW 1+2 w/ Isaac Flath THU 10/233:30 AM—5:00 AM (GMT+5:30) OPTIONAL Recording
Week 3 · 0 videos
OCT 23 Optional: Live coding workshop HW 1+2 w/ Isaac Flath THU 10/233:30 AM—5:00 AM (GMT+5:30) OPTIONAL Recording
Week 3 · 0 videos
📄 Frequently Asked Questions
Week 3 · 0 videos
📄 Other (Optional) Links
Week 3 · 0 videos
📄 This Week's Overview
Week 4 · 0 videos
📄 Advanced Topics
Week 4 · 0 videos
📄 Chapter 1: Introduction
Week 4 · 0 videos
📄 Chapter 2: Why Build Custom UIs?
Week 4 · 0 videos
📄 Chapter 3: Principles for Designing Effective Interfaces
Week 4 · 0 videos
📄 Chapter 4: Sampling Traces to Review
Week 4 · 0 videos
📄 Chapter 5: EvalGen Case Study
Week 4 · 0 videos
📄 Chapter 6: DocWrangler Case Study
Week 4 · 0 videos
📄 Chapter 7: Building A Custom Annotation UI
Week 4 · 1 video
📄 Chapter 8: Strategies to improve accuracy
Week 4 · 0 videos
📄 Chapter 9: Final Thoughts
Week 4 · 1 video
OCT 28 Optional: Live Office Hours 10 TUE 10/289:00 PM—10:00 PM (GMT+5:30) OPTIONAL Recording
Week 4 · 0 videos
OCT 28 Optional: Live Office Hours 10 TUE 10/289:00 PM—10:00 PM (GMT+5:30) OPTIONAL Recording
Week 4 · 0 videos
OCT 29 Optional: Live Office Hours 11 WED 10/299:00 AM—10:00 AM (GMT+5:30) OPTIONAL Recording
Week 4 · 0 videos
OCT 29 Optional: Live Office Hours 11 WED 10/299:00 AM—10:00 AM (GMT+5:30) OPTIONAL Recording
Week 4 · 0 videos
OCT 31 Optional: Live Office Hours 12 FRI 10/3110:00 PM—11:00 PM (GMT+5:30) OPTIONAL Recording
Week 4 · 0 videos
OCT 31 Optional: Live Office Hours 12 FRI 10/3110:00 PM—11:00 PM (GMT+5:30) OPTIONAL Recording
Week 4 · 0 videos
📄 HW 4 walkthrough with Braintrust (pre-recorded)
Week 4 · 0 videos
📄 HW 4 walkthrough with Phoenix (pre-recorded)
Week 4 · 1 video
📄 HW 4 walkthrough with Truesight (pre-recorded)
Week 4 · 0 videos
OCT 28 Optional: Live coding workshop HW 3 w/ Isaac Flath TUE 10/283:30 AM—5:00 AM (GMT+5:30) OPTIONAL Recording
Week 4 · 0 videos
OCT 28 Optional: Live coding workshop HW 3 w/ Isaac Flath TUE 10/283:30 AM—5:00 AM (GMT+5:30) OPTIONAL Recording
Week 4 · 0 videos
OCT 30 Optional: Live coding workshop HW 4 w/ Isaac Flath THU 10/303:30 AM—5:00 AM (GMT+5:30) OPTIONAL Recording
Week 4 · 0 videos
OCT 30 Optional: Live coding workshop HW 4 w/ Isaac Flath THU 10/303:30 AM—5:00 AM (GMT+5:30) OPTIONAL Recording
Week 4 · 0 videos
NOV 1 Optional: Live coding workshop HW 5 w/ Isaac Flath SAT 11/13:30 AM—5:00 AM (GMT+5:30) OPTIONAL Recording
Week 4 · 0 videos
NOV 1 Optional: Live coding workshop HW 5 w/ Isaac Flath SAT 11/13:30 AM—5:00 AM (GMT+5:30) OPTIONAL Recording
Week 4 · 0 videos
OCT 30 Live: AI Evals: The Next Discovery Habit w/ Teresa Torres THU 10/3010:30 PM—11:15 PM (GMT+5:30) OPTIONAL Recordin
Week 4 · 0 videos
OCT 30 Live: AI Evals: The Next Discovery Habit w/ Teresa Torres THU 10/3010:30 PM—11:15 PM (GMT+5:30) OPTIONAL Recordin
Week 4 · 0 videos
📄 Frequently Asked Questions
Week 4 · 0 videos
📄 Helpful Links
Week 4 · 5 videos
📄 HW 5 walkthrough with Braintrust (pre-recorded)
Post-course · 0 videos
📄 HW 5 walkthrough with Phoenix (pre-recorded)
Post-course · 0 videos
📄 Using AI Assisted Coding To Tackle Homework Problems
Bonus · 0 videos