Notes

• Home
• Inbox
• People

Channels

• [

  announcements

  ](/parlance-labs/evals/2025-3/channel?channelId=283916d6-d0ce-4b62-9af9-b308a8eceecf)
  - [

  general

  ](/parlance-labs/evals/2025-3/channel?channelId=6aa0dbf0-46ed-47e7-9ae8-cbbb4db531c9)
  - [

  intros

  ](/parlance-labs/evals/2025-3/channel?channelId=c4b50f08-c5b5-4f3f-a9bd-c9684d5c7b3d)
  - [

  💭

  questions

  ](/parlance-labs/evals/2025-3/channel?channelId=0458bb33-ed98-42cf-b593-4161ac4cd2cd)

Messages

• Inbox
• People

Channels

• [

  announcements

  ](/parlance-labs/evals/2025-3/channel?channelId=283916d6-d0ce-4b62-9af9-b308a8eceecf)
  - [

  general

  ](/parlance-labs/evals/2025-3/channel?channelId=6aa0dbf0-46ed-47e7-9ae8-cbbb4db531c9)
  - [

  intros

  ](/parlance-labs/evals/2025-3/channel?channelId=c4b50f08-c5b5-4f3f-a9bd-c9684d5c7b3d)
  - [

  💭

  questions

  ](/parlance-labs/evals/2025-3/channel?channelId=0458bb33-ed98-42cf-b593-4161ac4cd2cd)

Messages

Back

Start Here

📄

This Week's Overview

Lesson 1

This week covers advanced topics, focusing on scaling human review and connecting evaluation results to pipeline improvements. First, we will explore interfaces for human review and error analysis. You will learn why dedicated annotation tools can speed up your analyze and measure processes (moving beyond spreadsheets or off-the-shelf tools).

Second, we will briefly cover the "Improve" phase of the development lifecycle. After identifying failures (Week 1) and measuring them (Weeks 2-3), the final step is fixing them. We will survey some accuracy improvement techniques: e.g., prompt engineering, adding more context, fine-tuning.

Objectives

• Understand the importance of dedicated interfaces for human review

• Learn principles for designing effective human-in-the-loop (HITL) annotation workflows

• Survey the main strategies for improving pipeline accuracy (prompting, RAG, fine-tuning)

• Connect the "Analyze" and "Measure" stages to the "Improve" stage

Readings (link to course reader)

• Chapter 10: Interfaces for Human Review and Error Analysis

• Chapter 11: Focus on the accuracy optimization sections. The sections on cost optimization are supplemental.

Optional Coding Homework

• There is no homework for this week.

Code Snippets from Lessons

• Custom failure mode inspection interface

• Model cascades for cost optimization (we don't cover this in lecture, but we cover this in the reader)

Slides: Lesson 7 | Lesson 8

CompleteComplete this lesson

0%

• [

  📄

  Lesson 1

  This Week's Overview

  ](/parlance-labs/evals/2025-3/syllabus/modules/576020?item=9wguaq0xsn)

[

Home

](/parlance-labs/evals/2025-3/home)[

Community

](/parlance-labs/evals/2025-3)