Week 2

📄 Chapter 8: Correcting Bias in LLM-as-Judge Evaluators

Resources

linkhttps://github.com/ai-evals-course/judgy

Notes

Channels

[

announcements

](/parlance-labs/evals/2025-3/channel?channelId=283916d6-d0ce-4b62-9af9-b308a8eceecf)
- [

general

](/parlance-labs/evals/2025-3/channel?channelId=6aa0dbf0-46ed-47e7-9ae8-cbbb4db531c9)
- [

intros

](/parlance-labs/evals/2025-3/channel?channelId=c4b50f08-c5b5-4f3f-a9bd-c9684d5c7b3d)
- [

💭

questions

](/parlance-labs/evals/2025-3/channel?channelId=0458bb33-ed98-42cf-b593-4161ac4cd2cd)

Messages

Inbox
People

Channels

[

announcements

](/parlance-labs/evals/2025-3/channel?channelId=283916d6-d0ce-4b62-9af9-b308a8eceecf)
- [

general

](/parlance-labs/evals/2025-3/channel?channelId=6aa0dbf0-46ed-47e7-9ae8-cbbb4db531c9)
- [

intros

](/parlance-labs/evals/2025-3/channel?channelId=c4b50f08-c5b5-4f3f-a9bd-c9684d5c7b3d)
- [

💭

questions

](/parlance-labs/evals/2025-3/channel?channelId=0458bb33-ed98-42cf-b593-4161ac4cd2cd)

Messages

Lesson 4&5: Automated Evaluators

📄

Chapter 8: Correcting Bias in LLM-as-Judge Evaluators

Lesson 9

Open source version https://github.com/ai-evals-course/judgy pip install judgy

[

Previous

](/parlance-labs/evals/2025-3/syllabus/modules/b51ca3?item=pt4ba03ydnb)

CompleteComplete this lesson

[

Next

](/parlance-labs/evals/2025-3/syllabus/modules/b51ca3?item=4tnvqupsv1g)

0%

[

📄

Lesson 1

Recorded Lesson: Automated Evaluators

](/parlance-labs/evals/2025-3/syllabus/modules/b51ca3?item=oumjjqd57c)
- [

📄

Lesson 2

Chapter 1: Introduction

](/parlance-labs/evals/2025-3/syllabus/modules/b51ca3?item=ger3pdhumf)
- [

📄

Lesson 3

Chapter 2: Error Analysis Recap

](/parlance-labs/evals/2025-3/syllabus/modules/b51ca3?item=1kesnlso95e)
- [

📄

Lesson 4

Chapter 3: Code-based vs LLM-based Evaluators

](/parlance-labs/evals/2025-3/syllabus/modules/b51ca3?item=l0d8vp059j)
- [

📄

Lesson 5

Chapter 4: Overview of Creating a LLM-as-Judge Evaluator

](/parlance-labs/evals/2025-3/syllabus/modules/b51ca3?item=kwvk6ff9exn)
- [

📄

Lesson 6

Chapter 5: Example Criterion for LLM-as-a-Judge

](/parlance-labs/evals/2025-3/syllabus/modules/b51ca3?item=rpbapruasgn)
- [

📄

Lesson 7

Chapter 6: Crafting and Refining the LLM Judge Prompt

](/parlance-labs/evals/2025-3/syllabus/modules/b51ca3?item=j51tj9fyzsc)
- [

📄

Lesson 8

Chapter 7: LLM as Judge Coding Demo

](/parlance-labs/evals/2025-3/syllabus/modules/b51ca3?item=pt4ba03ydnb)
- [

📄

Lesson 9

Chapter 8: Correcting Bias in LLM-as-Judge Evaluators

](/parlance-labs/evals/2025-3/syllabus/modules/b51ca3?item=wtd458l9bub)
- [

📄

Lesson 10

Chapter 9: Pitfalls to Avoid When Building Automated Evaluators

](/parlance-labs/evals/2025-3/syllabus/modules/b51ca3?item=4tnvqupsv1g)
- [

📄

Lesson 11

Optional HW

](/parlance-labs/evals/2025-3/syllabus/modules/b51ca3?item=o8wmm10ftsg)

[

Home

](/parlance-labs/evals/2025-3/home)[

Community

](/parlance-labs/evals/2025-3)