Notes

• Home
• Inbox

• People

• Inbox

• People

Back

Recording

Optional: Live coding workshop HW 5 w/ Isaac Flath

Nov 1, 20253:30 AM - 5:00 AM GMT+5:30

Audio Transcript

Chat Messages

Isaac Flath

Hey, how's it going?00:00:22

Hamel Husain

Hold on a sec, let me just get it.00:00:23

Connected.00:00:26

Can you hear me?00:00:34

Isaac Flath

Yep, I can hear you.00:00:36

Hamel Husain

I think this time we'll have you kind of… I'll try to, like, minimize questions.00:00:40

There was, like, tons of… When I was looking at the recording, it was like… Maybe too many interruptions?00:00:45

Isaac Flath

Okay.00:00:54

Hamel Husain

like, field the questions. How do you feel about that?00:00:54

Isaac Flath

I'm good either way. I mean, there's less content in this homework, but, I mean, we could kind of get through it, and then…00:00:57

do questions, you know, most questions at the end. Obviously, if somebody's like, I don't understand what you're doing right now, that seems like a good thing to raise, but…00:01:05

Makes sense. Okay.00:01:13

And, you know, if I can get through it fairly quick, and if we have more time, I'll just… we can look at the data. That's what we're here for.00:01:17

Hamel Husain

Yeah.00:01:23

Nice joke.00:01:26

Alright, welcome back, everybody! This is the last… Session of the course.00:01:27

So, today we have Isaac.00:01:35

Who is going to… he's been doing live walkthroughs of the homework.00:01:39

Isaac already has recorded walkthroughs of the homework, but he's doing a different flavor live, where he's using AI.00:01:43

kind of, like, leaning on AI.00:01:50

to help…00:01:53

do the homework as much as possible? Like, how would you do that, if you're interested in that kind of thing?00:01:54

Targeted at people who may not be as familiar with coding?00:02:00

So I'll hand it off to Isaac.00:02:05

Isaac Flath

Alright, cool.00:02:09

So,00:02:10

Just as a little bit of a recap, week 1 and 2, the first session, we did weeks one and two, where we looked at,00:02:12

We looked at creating a prompt, what you might look at there, how do you create some synthetic data, kind of getting started with that, getting started with error analysis, looking at the data for both open and axial coding.00:02:22

We then moved into… the next session was on LLM as a judge. How do you look at, kind of, the data going in, the data going out? How do you think about this? How do you measure it? And last week, we really talked a lot about,00:02:36

the RAG use case, and evaluating that, processing it, all that kind of stuff, synthetic data for that.00:02:52

And so then this week is the, last homework.00:03:01

And, that's what we're gonna do here. Go ahead and ask questions in,00:03:05

you know, Discord, or… or ask questions in the Zoom chat, and then Hamel will raise them. We're gonna get through some of the material, and then have some more time for questions, you know, at the end, but if there's anything that you don't understand that I'm doing right now,00:03:12

you know, as I do it, please ask those so that we can… Clarify those.00:03:27

Alright, let me find… share button…00:03:34

Perfect.00:03:41

Alright, everyone should see my screen.00:03:43

So this is in Homeworks, Homework 5, and this is the README for Homework 5.00:03:46

And so, every conversation as a trace contains one failure, so,00:03:52

Mostly important to understand is what data's provided. There's,00:03:59

Labeledraces.json, which is a list of 100 traces.00:04:04

There's, okay, a conversation ID, a bunch of messages, so a chat back and forth, maybe an agent.00:04:08

We'll see.00:04:14

What was the last success state and the first failure rate? So we'll look at that.00:04:16

And what that means here in a bit, but let's skim it. Okay, so we've got some states.00:04:21

So, we have a parse request, and this is kind of… Ideally, if you're doing evals.00:04:27

you know, if you did evals from the start, you kind of defined these already, so you would already be familiar. If you're coming in and doing it on a product that, maybe is new to your team, or you're looking at something else, or you're consulting, these are things you might have to define. Like, what are all the things that happen in your agent?00:04:33

Llm interprets the message, Okay, there's a planning step, deciding which tools to invoke.00:04:50

There's generating arguments for the customer.00:04:58

There's, you know, a tool to get recipes, to get a search tool.00:05:03

Let's see, there's… okay, there's customer profile tools. I know I did not go in order.00:05:08

Generating arguments, for the recipeDB.00:05:12

Oh, there's a web search tool?00:05:16

Okay, there's, like, a final answer and delivering response. Okay, so this seems like it's a, like, the gist here is, like, once you understand a little bit here, you can think, see, like, alright, so it's… it's gonna get some customer information and profile, I don't know what exactly, we'll look at the traces, maybe it's getting stuff like what kinds of,00:05:20

dietary restrictions they have. Maybe they have, like, a profile page, it can do some web search, it can pull recipes from a database,00:05:40

So that's cool.00:05:48

And so, what do we need to do?00:05:50

Familiarize yourself with the JSON structure.00:05:52

Okay, so we familiarized ourselves a little bit with this.00:05:55

Let's look at the JSON structure. We got this, but let's look at the JSON.00:05:59

We can look at… data… Labeledraces.json? Okay.00:06:05

So, now we can see a little bit more. We have our conversation ID.00:06:12

And we see these messages,00:06:18

If you've been in the last few, you could probably see where this is going. It's not the nicest format to read it in.00:06:24

So, what I did, and I won't make you, you know, watch the agent do this.00:06:30

But what I did is I said, create a self-contained HTML file. Hopefully you guys aren't tired of seeing that, like, 3 times every session. I told it where the JSON was, I'm like, I just want to view this JSON file in a nice way. I described what I wanted.00:06:39

from this, I gave it all the states.00:06:56

It gave me some stuff.00:07:00

I said I couldn't load it, I gave it the error.00:07:06

I didn't understand something. I was like, why do I need… it was telling me to start a server. I was like, why do I need that? Can't it just read the file?00:07:09

It explained…00:07:18

a bit, and I said, like, well, I could just add an input for you to upload it, instead of starting a whole server, and I was like, okay, yeah, let's just do the upload box.00:07:20

And it did it.00:07:29

So, this is the whole thing I did, is 4 prompts.00:07:32

To get it created.00:07:37

And this created, this… this here.00:07:39

Let's pretend like we haven't seen that transition matrix yet.00:07:43

And so here I can look through, and I can see them. This may not even… this isn't, like, the perfect one, like, this tool called ParseRequest might be nice to see that differently. But, you know, this gives us a little bit nicer way to read the JSON, so now we can see.00:07:47

We have… Start and an end…00:08:01

And we can see what vegetarian meal… okay, so now we… inside of this chat, we have these traces that are…00:08:05

Parsing a request, it's planning some tool calls, It's… now it's failing.00:08:11

And then it carries on, so it tried to continue, even though there was a failure.00:08:20

And etc. So these are the… these are the conversations.00:08:24

So I think we're familiar with roughly what it is. Let's see what else we're supposed to do.00:08:30

We want to build a transition matrix.00:08:36

And to do that, we count how many times from last success to first failure.00:08:39

So… And we visualize that in the heat map.00:08:44

This is the… Heat map provided.00:08:50

And this is a really useful thing,00:08:56

for kind of a simple visualization, it's kind of, like, unreasonably helpful to look at. And what this is, is that…00:08:59

For example, we can see that, like, what was the last successful thing that happened in this trace?00:09:07

And in this case, let's pick this 10, because that's the biggest number.00:09:14

For 10… there was 10 traces where the last thing that succeeded was planned tool calls, and then the first thing that fails was generate recipe args.00:09:19

So, that's interesting, so we might want to dive into that.00:09:30

This 9, the last thing that was successful is it looked up to get the customer profile, cool.00:09:34

And then it failed to get recipes.00:09:41

So that's interesting.00:09:45

Okay.00:09:47

Great. So we can look at this.00:09:48

And this tells us, like, okay, well, maybe there's something between these steps, maybe one of these steps fail, like, why are these all lit up?00:09:52

Very red.00:10:01

So that's what that is.00:10:04

And then we just want to analyze it, you know? Which… which fails? Do clusters happen? Any… anything surprising? And this is just mostly, like, get curious.00:10:08

Write notes, write things you observe, etc.00:10:17

I wanted not just to…00:10:21

look at this, but, like, once I see this, I can see patterns, but then I might want to, like.00:10:26

actually look at the traces and look at the data underneath, and that's why I created this.00:10:31

So this is the view where…00:10:37

I can click on one of the boxes, and I can see, okay, here are the 10 traces.00:10:40

That fell in here, and I can just kind of look through them, if I'd like, just to understand a little bit more.00:10:47

Before we do that, we can go ahead and look and see if we see any trends here.00:10:53

What I see here is we have a 10, a 9,00:11:02

an 8, 7, 7, 7… like, okay, so the first thing that sticks out is over here, there's, like, very light colors.00:11:06

Very few things that are going wrong over here.00:11:15

Hamel Husain

So tell me what happened here. You, you said you…00:11:18

you wanted… you exported a self-contained HTML file, and then, from that code.00:11:21

Isaac Flath

Exactly, yes, I created a…00:11:28

Hamel Husain

Do you prefer doing that, let's say…00:11:29

You know, versus a notebook? Just out of curiosity.00:11:32

Isaac Flath

I find that I have, for looking at simple stuff, or JSON files or CSV, I kind of…00:11:36

tend to like it, because it's sometimes making… I find sometimes making, like, display widgets in…00:11:45

Notebooks can be tricky at times. I know you're, like, really good at them, so if you are, great.00:11:53

Hamel Husain

No, I kinda like this, self-contained HTML thing, actually, to be honest.00:12:01

Isaac Flath

But my plan is, after I look at this and get a high level, I was gonna create a notebook, and then, look at some of the things, in more detail as well. So, I don't think it's one or the other.00:12:06

But yeah, this is kind of a high level, so basically what I see is, like.00:12:18

Two columns, going pretty well.00:12:23

Lots of dark colors.00:12:26

Lots of good stuff, so… What do these have in common?00:12:29

We have our failure traces, okay, we're…00:12:34

Hamel Husain

And is this Monster UI? It looks very monster… it's somewhat Monster UI-ish, kind of. Or maybe I'm just making it up in my mind. To be honest, I didn't even look, but I'm sure… I don't think this is Monster UI. Okay. I don't think so, but…00:12:37

Isaac Flath

Could be.00:12:51

So, I guess what I'm seeing here is I think, like, okay, so we have something about generating recipe arguments.00:12:55

Generating customer arguments.00:13:04

Getting a customer profile, getting recipes.00:13:08

So, like, can I try a thread through these? And sometimes you can't… maybe things are completely isolated, and maybe things can be… and they're… they truly are separate issues. Sometimes they come together.00:13:12

I find it a little suspicious right off the bat.00:13:24

that, okay, Git probably means I'm looking something up, right? I'm getting something.00:13:31

If I get the customer profile, that's probably in my own database, I would guess.00:13:37

If I'm getting recipes.00:13:42

you know, that could be from the web, or it could be from my own database. I would think it's probably from the web, because there's a separate… from my database, because there's a separate git web info.00:13:45

So this immediately, I'm like, Why would getting customer profile information and getting recipes from my database00:13:56

have the highest number of failures. I would expect getting random information from the web to be harder, because, like, I can control. So, like, maybe this isn't even an LLM issue at all, maybe it's an issue with querying our database, or maybe our tool doesn't have the right definition for querying our database. So that's kind of interesting.00:14:03

And then, we also see… Generating…00:14:21

Going from this planning step, it's gonna plan the tool calls, and then gen customer args and gin recipeArgs are both extremely high.00:14:28

And that seems suspicious,00:14:37

So, these are kind of things that I might start to just think about. Like, why…00:14:43

are there common threads? You know, like, I could think about them in isolation, but at first I want to say, like, okay, maybe I should be looking at some sort of database issue, maybe I need to look at this planning step, maybe this planning step has some…00:14:49

bad information about, like, the tools. Like, maybe we misdefined our tool calls, or maybe we updated and removed an argument and didn't remove it from the prompt. Maybe that's why planning to getting our database info is really bad.00:15:01

I don't know, but…00:15:14

now that I kind of have some possible ideas in mind, I would just read some of these. I'd say, like, well, 10 is the biggest category.00:15:17

So I need a gluten-free dinner idea for 4.00:15:25

I can look through… So it parses the request,00:15:29

And it plans the tool calls.00:15:35

It gets… tries to generate recipe args, okay.00:15:38

So, that's just something to look at.00:15:45

Vegetarian high-profile meal. Okay, so this is, like…00:15:48

a very different kind of query, I guess.00:15:52

You can see parse, plan, GenRecipe args.00:15:57

This was unable to generate recipe search parameters.00:16:01

for the request.00:16:08

And I think this is often what you want to do, is just, like, look through some of them, to start to get an idea. A healthy breakfast using oatmeal.00:16:11

I'll analyze your request to identify your preferences in the main ingredient. And so, as I'm looking through this, the first thing that's coming to my mind is I'm lacking in information in these traces.00:16:23

This is kind of going beyond the homework. The homework was just to look at the transition matrix, but, you know, while I'm here. So I look at this, and I say, like, okay, well, we have this tool called Polish Requests and Plan Tools. All I see is it, like, worked, worked, failed.00:16:35

I really want to know, like, well, what was passed into this tool call, and what came out? Like, I want to be able to, like, click on this.00:16:50

and see, like, what… what actually happened. Playing tool calls, like, maybe I don't need to see it all the time, but I'd like to say, like.00:16:57

this worked, like, how can I… like, I want to right-click this or something and see, okay, well, what was the plan? Like, what plan was created? Does that plan even make sense?00:17:04

And so, I think this is a good start.00:17:13

And we don't have all this information in the traces, but I look at this, and this is a really good start.00:17:16

And where I'm saying, like, you know.00:17:24

I need to figure out how I can drill into these more.00:17:27

like, unable to generate recipe arguments from the provider request, like, that seems odd. Like, well, I… it's hard for me to know what happened if I can't see00:17:30

the plan.00:17:39

So, cool, so we know we need some more information.00:17:41

Hamel Husain

There's a question in the chat. Yeah. So, like, let's say this particular circle right there, what does that mean? It failed when going from plan tool calls to gen recipe arcs? Is that the right way to interpret that?00:17:43

Isaac Flath

Exactly. So we can see that this… I'm clicking on this 10, so all these over here are those 10 traces. We see that 10 matches this 10.00:17:56

And here we see… User, the assistant responded successfully, the parse request happened successfully, planned tools happened successfully.00:18:04

And then, gin recipeArgs had an error here.00:18:15

So we can say…00:18:19

Hamel Husain

I think the underrated part about what you're showing is how you created this interface, sort of, on the fly right now.00:18:20

And… There is no off-the-shelf tool that does what you're…00:18:27

Showing, but you just created it in, like… A minute.00:18:32

Yeah, I mean, it's actually, it's pretty good.00:18:37

Isaac Flath

Yeah, I think it took… it took me less than 5, just because the agent takes some time. But yeah, I mean…00:18:40

And the thing is, is, like, with how little time it took.00:18:46

If all I did was do this, and an hour later, I delete it, like, it was worth it. Like, 100% worth it.00:18:49

Like, could you imagine spending an hour00:18:57

Literally, like, scrolling through this and trying to understand, like, what are these success rates?00:19:03

Like, going from, like… and I see people actually do this sometimes, where they're like, oh, these 10, these plan tool calls to Gen Recipe ARGs, and they go over here, and they're like, okay, well, it was plan tool calls.00:19:09

And then they might try and find these, and they say, like, okay, well, let me try and, like.00:19:24

Like, I honestly see people do this, where they're looking for, like, the failures, and they scroll up. It's like, it's so painful, you know? You're just much better off spending, like, 3 minutes making a little interface for yourself.00:19:30

Hamel Husain

I really like this, like…00:19:44

arc you have of creating self-contained HTML files, so I feel like that's a little bit…00:19:46

unique to Isaac, but I really like it.00:19:51

I think we talked about this. It's, like, very portable, you can, like, give it to your colleague, they can just, like, open it.00:19:54

They don't have to, like, install anything. It's pretty… It's pretty cool.00:20:01

Isaac Flath

Yeah, and I think… I don't think, like, these are the… these are necessarily, like, the end, but,00:20:08

what I like is that it doesn't mean… like, if you can't code, and you don't feel like you know how to…00:20:15

build an interface. You don't have to, like, submit a ticket and wait for an engineer to be prior… like, you can just do it, you know?00:20:23

And then, you know, you're probably going to get a better output, even when you get someone, because, like, you can then go and say, like, hey, I like this. This is kind of big, like, I hate scrolling, and I wish I could, like, drill into these.00:20:35

But this is what I want. And that's a pretty good, like, ask for someone to help you build something.00:20:47

So, yeah.00:20:54

Okay.00:20:58

So, we've kind of understood it, we've kind of looked, we have a better idea what we want with our interface. Maybe you have the ability to fix it yourself. If so, do it. If not, that's okay, you just have something in the back pocket to improve upon. Now I want to kind of, like, dive a little bit more into my…00:21:00

A little bit more into my… like, okay, sorry, sorry. So, like, this is interesting, too. I can't help but looking… looking at these traces when they're in front of me. That's… that's what you should do. Any specific ingredients you'd like to include in your dinner? No beans, please? Great! I'll keep beans out of an option. Let me know if you have any other trace preferences.00:21:22

And then it just stops? Like, what…00:21:40

I feel like this should have given a recipe without beans.00:21:44

Now, we're not really focusing on that, because we had a failure a little bit higher up in this git recipes, but it seems kind of weird that the git recipes failed.00:21:48

It failed, so it asked them for information and then didn't give them anything.00:21:59

So, like, that's something I'd keep in mind. So I was just thinking about things.00:22:04

Okay, before I distract myself too much, I'll quit looking at the data.00:22:10

Okay.00:22:14

So,00:22:16

So the next thing, I wanted to try and… I want to try and see if I can understand some of these. Like, I mentioned that…00:22:19

you know, there's get customer, get recipes, like, it was a little suspicious that this is, like, clustered.00:22:26

like, if I saw, like, 2s everywhere across the board, like, 1 to 3 on every cell, all the way over, I would say, like, oh, that's kind of weird, it feels like I've got just, like, lots of little issues to fix in lots of paces.00:22:32

It doesn't really feel like that to me, because it's like…00:22:44

Like, I've got these, like, few columns that have almost all of the errors.00:22:51

And I'd like to know why that is.00:22:58

And so this is where I might go to a Jupyter notebook.00:23:03

To try and understand that a little bit more.00:23:08

And by Jupyter Notebook, I mean, let's… You can either, write it yourself.00:23:11

Or we can use an agent to do it.00:23:20

And I'm gonna go ahead and transcribe.00:23:23

Please look in the Homework 500:23:28

data labeled underscore traces dot JSON file. I want to read that into a Jupyter notebook.00:23:30

What I'm interested in… is… Patterns in, like… What's causing failures?00:23:37

I made a transition matrix, and I looked at it, and in this transition matrix, I noticed that, like.00:23:48

Get customer profile, get recipes, were all very high.00:23:54

I was thinking maybe that's, like, some sort of, like, database querying or connection issue,00:24:00

like, Gen Recipe Args had some high stuff, I'm not sure what that is. Getting web info seemed to be successful a lot, which tells me, like, it's not necessarily just a tool calling thing, because it can successfully reach the web. But anyway, these are just, like, thoughts, and I want you to help me think through these.00:24:07

And do some analysis, group them in different ways, if there's any visuals,00:24:25

create some text that, you know, might explain some of these, like, clusters in it.00:24:31

In the, in the heat maps.00:24:38

And,00:24:40

Yeah, let me, just give me something to help think through these, transitions, which are kind of from the last failure state to the new failure state in that JSON file.00:24:43

And so, we'll see what it comes up with.00:24:54

This might take a bit, so this might be a good, good time for questions.00:25:05

If there are any…00:25:09

Hamel Husain

I've been answering them in the chat.00:25:14

Isaac Flath

Oh, great.00:25:16

Hamel Husain

Is there… one question was, do you…00:25:18

Do you find that these tool calls fail in certain ways, in your experience? Like…00:25:21

Arguments being wrong, the tool itself having some kind of, like.00:25:28

bugging it, etc. Any kind of things that you…00:25:33

Or is it, like, all over the map? I said that it was all over the map, in my experience, but…00:25:37

Isaac Flath

Yeah, I think it's all over the map. I think, like, when I… when I find if…00:25:42

I find when it's most consistent is when something is, like, brand new, like, just a brand new AI feature. It's very common, I think, that people just haven't read the prompts, and, like, the tool call instructions and stuff are just wrong in the prompts.00:25:47

And so, like, there's usually one iteration of, like, reading the prompt and just fixing it. But yeah, then it's kind of all over the map.00:26:01

I know it sounds obvious to just, like, read the prompt that AI creates for you, but it doesn't.00:26:12

Hamel Husain

No, a lot of people don't read it. A lot of people do not read the prompts.00:26:17

Isaac Flath

So…00:26:20

There's, like, a… there's a first round of, like, just read it, and then you fix a lot of really obvious things.00:26:22

Hamel Husain

I'm really curious to see if this notebook comes out with anything good.00:26:37

Isaac Flath

I have not tried to generate a notebook based on this JSON before.00:26:41

Hamel Husain

Yeah.00:26:45

Isaac Flath

I usually try and do dry runs, but in this case, I didn't, so we'll see.00:26:46

Hamel Husain

I think data analysis is… Like, the hidden gem of AI.00:26:58

Like, it's pretty… like…00:27:05

people get, like, really excited about coding, but I think this niche… there's, like, this niche of, like, data analytics.00:27:07

That is a little bit underrated.00:27:14

Isaac Flath

Yeah, it's, it's like,00:27:17

It's like everything seems to come back to this data analysis, data analytics, because it's like…00:27:21

that's what tells you what to do. You know, like, should I be building this feature or not? Should I be…00:27:26

like, what should I be worried? Like, it all comes back to, like, you do data analytics to figure out what you're supposed to do, basically.00:27:34

Hamel Husain

Network X, which makes me slightly skeptical, like…00:27:40

The complexity that might be there, but let's see.00:27:44

Isaac Flath

Yeah, spatial distance.00:27:48

Hamel Husain

Hmm.00:27:52

Isaac Flath

Cosine distances… yeah, this is making me a little skeptical, I'm not gonna lie.00:27:52

There's a lot there. It's okay, we'll try it, and then we'll iterate.00:27:58

Oh, you know what? Okay.00:28:10

I'm gonna get rid of this exception, because if we don't have widgets, I want it to… Tell me.00:28:13

Like, all this stuff, okay.00:28:21

Hamel Husain

Yeah, AI loves to do this defensive nonsense.00:28:23

Isaac Flath

Like, and sometimes it's okay, usually it's not. Like, I… if it doesn't have the tools it needs, it can just tell me, and I'll give it the tools it needs.00:28:26

I don't know why it would need…00:28:36

Hamel Husain

This, like, try… this aggressive try-catch is, like, some kind of… Weird.00:28:38

post-training… Thing that's been baked into the model, like, it's really hard to steer them out of this…00:28:45

You know, over-eager try-catch stuff.00:28:53

Isaac Flath

Yeah, I have a couple things in my normal AgentsMD that is, like, there's a few things on this earth that I hate more, and then config files is one, and then error handling is another.00:28:58

Alright, so let's see here…00:29:11

Hamel Husain

I don't know, man, they shouldn't really be using NetworkX.00:29:14

Might be worth… Telling it… not to.00:29:18

Isaac Flath

What's…00:29:22

Hamel Husain

Why the hell would it? What?00:29:23

I don't think you'd want to do network analysis.00:29:25

Okay, graph node, number of edges…00:29:28

Oh, yeah. Okay, I guess, like, transition count… I guess, kind of makes sense, maybe. It wants to…00:29:32

He wants to graph?00:29:41

It has a graph.00:29:43

Huh.00:29:46

Yeah, it's kind of interesting, potentially. I don't know, I'm curious now.00:29:47

Isaac Flath

Yeah.00:29:51

Boop.00:29:53

In general, I would say,00:29:54

If you think you need a graph database or a graph structure, you probably don't, and so you… that's why it's, like, immediately a red flag.00:29:58

Hamel Husain

Yeah, that's why this is definitely a red flag. That's why, yeah, when we see graph database, Isaac and I are like, no, this is not good.00:30:06

Or even graph, structure.00:30:14

Like, maybe not.00:30:17

I guess you could tell it… can you select that whole thing and say, can you please install the libraries or something?00:30:29

You can tell AI.00:30:36

Oh, no.00:30:37

Isaac Flath

I wanted to add the air handling in.00:30:39

Please install if… Are missing a depth.00:30:44

So…00:30:56

I'll iterate a little bit, and if I, don't get to somewhere, then I'll go back up, I'll change the prompt, and I'll ask it to keep things very, very simple, and have it, created again. I'm a little skeptical that this is gonna work,00:30:58

And you don't want to spend too much time trying to get it to work.00:31:12

If you try a few things and it doesn't, then it's probably not gonna… just prompting it more probably isn't gonna help.00:31:16

I'm gonna help.00:31:22

Hamel Husain

Is it actually able to run those notebook sales for you?00:31:23

Isaac Flath

It should be able to, yeah.00:31:27

Hamel Husain

It said, like, installing and running notebook cells, which is interesting.00:31:29

Isaac Flath

Yeah, it tried, but there was a failure here.00:31:33

Hamel Husain

You saw a failure?00:31:39

Isaac Flath

Well, it tried to do this run…00:31:41

Hamel Husain

Yeah, right, yeah.00:31:43

Isaac Flath

Has an X, I think it means failure.00:31:44

Okay, so we see BIMF, BIN Python…00:31:48

So I'm just gonna help it here. Network X.00:31:54

Hamel Husain

I guess you are… wait,00:32:12

Are you still installing the right VM? Because… is it a local VM that you have in this directory?00:32:18

Isaac Flath

It should be…00:32:26

Hamel Husain

Because… This message down here makes me think it's in, like, your root environment.00:32:29

You see that?00:32:38

Isaac Flath

Oh, yeah.00:32:40

I'll just install it here. So pip is the way you install packages.00:32:43

It says I'm still missing iPy widgets.00:32:47

So I'm just gonna do pip install iPywidgets.00:32:53

And it's trying to do this here, but I gave up on it. Okay, so we successfully got our dependencies installed with pip.00:32:59

Let's see what it's doing. It uses file, it loaded it…00:33:09

Figuring out the memory footprint of the file is definitely excessive.00:33:16

Hamel Husain

Yeah, it, like, even the import thing of, like.00:33:20

Confirming that the import happened, that was excessive.00:33:24

Isaac Flath

Yeah, for sure. And, like, honestly, looking at this, I would,00:33:28

I'll probably start over and have it really simplify this notebook, because.00:33:34

Like, I look at this, and I'm like, I don't understand this enough.00:33:39

And so, I'll probably, like, run it through, see what it creates, look at the charts, see if there's just any quick value without spending too much time on it.00:33:42

And then I can, you know, try and create another one that's more targeted, because I think…00:33:55

spending too much time in this, knowing how overcomplicated this is, I don't feel like it's gonna be worth a huge amount of my time.00:34:00

But let's see here. Okay, first two messages, it got them.00:34:08

Two extractions… what is it doing here?00:34:15

Hamel Husain

Yeah, this is like, okay, this is producing the transition matrix, like the node, the… whatever.00:34:19

Isaac Flath

Yep.00:34:27

Doing it.00:34:29

Hamel Husain

edges.00:34:30

Isaac Flath

Okay. This is saying here are the top, you know.00:34:32

Top failures, percentage-wise, so 31%.00:34:36

Coming from Plan Tool Calls.00:34:41

Yeah, so I got scanned through this… Doesn't seem that useful.00:34:51

Great.00:34:57

So let's start over here.00:34:58

Let's see, drastically simplify.00:35:00

I'll just voice transcribe. Drastically simplify. I don't need Network X, I don't need anything fancy, I don't need the transition matrix, I want to analyze with just pandas, that's the only dependency. Just looking at tables.00:35:03

Load something minimal, I'm gonna delete almost everything in this notebook for you to start over. Just load in the JSON and do a little bit of analysis with just pandas in a super minimal, simple way, so that I am comfortable jumping in and being hands-on.00:35:17

So we'll do this.00:35:38

I'm gonna delete it.00:35:39

And I think this is actually, like, you know, it would have been cool if it just worked, but I'm actually not that upset that it didn't work.00:35:43

Because… I think it's pretty important to know that When you see this complication.00:35:51

And I'm like, I can't understand this.00:35:59

It seems way overkill. It went far beyond me.00:36:02

Usually the best thing to do is just to, like, delete it and take a step back and simplify. It's usually not the best call.00:36:08

to…00:36:16

just keep prompting it, and keep asking it to fix it, and keep giving it errors, and keep doing it.00:36:18

You can. You can kind of keep prompting it and see where you get.00:36:26

keep vibe coding, so to speak. In my experience, I don't tend to learn a whole lot from it.00:36:31

And I usually just end up frustrated, so…00:36:37

If you get the vibes that it's, like, gone too far off the… off the wheels, and you look a little bit…00:36:43

It's better to, like, pair it back than… Try to go.00:36:50

Any other questions while we're waiting for the new one?00:37:02

Hamel Husain

Hmm… I don't see any questions in the chat, really.00:37:08

I guess, like, one question might be.00:37:20

I'm just trying to put myself in the shoes.00:37:24

Of a student.00:37:26

What is the best way one should develop the sense of whether something is too complicated or not? Do you think you need to learn all the fundamentals, like.00:37:28

Okay, you know it's… like, we know it's too complicated, because we've done this manually so many times ourselves, like, before AI.00:37:37

Like, we just know right away that, like, okay, this doesn't make any sense. Like, why is there so much code? Why is there so many dependencies?00:37:44

Do you think someone starting anew can develop that sense without00:37:53

Going through the same journey we did, or what is your feeling?00:38:00

Isaac Flath

Yeah, I mean, learning some of the fundamentals is helpful, for sure. There's no denying that learning more is valuable, you know, of anything. But yeah, I think so. I think… I think when you…00:38:04

try to look and see, like, what is the point of this stuff here? We identified that Network X was doing this graph stuff.00:38:18

If you… you could use AI to ask about it, and if it's like, I don't understand this at all.00:38:27

Like, it makes no sense to me.00:38:34

Then you took too big of a step.00:38:37

And you should kind of get rid of it.00:38:40

if you see something, and you're like, I kinda see where it might be going with that, that could be interesting. I don't really understand it, but like…00:38:45

Okay, it's trying to… it's trying to make some sort of chart.00:38:54

That's cool. Then you might want to understand it. So, yeah, it's very much, I guess an intuition thing about, like.00:38:58

I feel like it's okay if AI is… if you use AI and it gets, like, one or two steps ahead of you.00:39:08

It's still okay, because you can learn, you can…00:39:15

dive into it, you can ask questions, you can learn. If you ever look and think, like, wow, it's, like, 15 steps ahead of me.00:39:18

Then you should probably just delete the notebook, and ask it to do a smaller step.00:39:27

And what you'll see is this first prompt, I didn't really set myself up for very good success, because I just said, like, do a bunch of stuff.00:39:36

And so, when I created this error, the way I said is I said, okay, first, drastically simplify. I said, I don't need… and I said some specific things I didn't need.00:39:45

And I said I don't need the transition matrix, because I already looked at that. And then I said what I did want. I want pandas. That's the only dependency.00:39:59

Just looking at tables. Load something minimal.00:40:10

And so…00:40:14

I think, if it gets too far ahead, where you're like, I don't understand any of this, and I just, like, I ran the stuff, and I saw, and I was like, I don't know what these things are doing, this isn't helpful.00:40:16

That's when you say, okay, I need to be definitely more in control than that, and think about, well, what might I want? And I can go back to the heat maps that we… the heat map that we had.00:40:26

Here is the…00:40:40

Here. We can go back to here and say, like, okay, well, what… what would I want to see?00:40:42

And one of the things I want to see, and we'll see if the notebook creates it, otherwise we'll create it.00:40:47

is I wanna see, like, it seems like the first 3… Letters?00:40:53

of the, of these codes are meaningful.00:41:00

Meaning gin, gin, gin.00:41:04

Git, get, get.00:41:09

So, like, what if I grouped by those first 3 letters?00:41:12

And you can ask very specifically for that, and, have quite a lot of success getting that. And so, I think it's, like, try to get more specific with what you want, and yeah, if it feels like it's just, like, just beyond you, that's totally fine. If you feel like it's, like.00:41:15

Way beyond you, depending on wherever you're at, then you probably just want to delete it and work your way up there incrementally.00:41:32

And maybe you say, like, I don't understand Network X.00:41:43

like, what, what value would I get? Like, what would this tell… what could this potentially tell me about it, and have it explained? And even if you don't use it, then you're like, okay, well, I know what network X is now.00:41:47

And so, I guess we can do that. I'll just jump over to, like…00:41:57

I'll jump over to Ann.00:42:07

Hamel Husain

It's taken quite a long time.00:42:12

Isaac Flath

Yeah, it really is. I don't know if they're having issues here. So I might say, like, if I'm like, oh, I don't know what…00:42:15

What is NumPy?00:42:22

Like, maybe Seabourn.00:42:25

Well, I see solaces, okay. So we can say something like…00:42:29

I created a notebook to analyze,00:42:35

This transition heat map, and, like, failure transitions.00:42:39

I'm trying to do something simple, and there's some imports I don't understand. Could you explain, like, what NumPy and Pandas are for in Python? And, like, why…00:42:43

why those might be helpful for analysis, and what kinds of things I would be interested in learning if I were using those tools.00:42:55

Oh, looks like my transcription got cut off.00:43:08

And so I guess just being a little curious, let's see here… Fast…00:43:17

arrays and matrices and math operations. That could be useful.00:43:22

Pandas is structured data, like spreadsheets and tables. That sounds exactly what I'm looking for.00:43:28

Manipulating tabular data, filtering and aggregating and grouping data, Pivot tables and cross-tabulations.00:43:34

Time… time series, maybe not so much, but…00:43:43

Cool, so that's… that's good. Pandas can count transitions between states and create frequency tables, that sounds useful.00:43:47

NumPy handles underlying math for probability and rates.00:43:55

And they both work with visualization libraries really well, Matpotlib and Seabourn, which I can see it imported.00:43:59

So, I would say… if I see that, start with doing that, asking AI this thing,00:44:06

get kind of curious, understand what these are for, and then now I can say, like, oh.00:44:17

That might give you enough to do pandas. It will be…00:44:23

a long process, but I think… I think you can use AI to ask lots of questions about, like.00:44:27

Why would I want this?00:44:33

I don't know, what do you think, Hamel? Does that make sense?00:44:38

Hamel Husain

Sorry, I was, can you repeat.00:44:43

Isaac Flath

Oh, that's okay.00:44:45

I was saying that, like, if you see stuff, one thing you can do that you don't understand is you can ask AI to, like.00:44:46

Explain things.00:44:53

Hamel Husain

Oh, right.00:44:55

Isaac Flath

tell you what the dependencies are for and why, and that hopefully that means that, like, you can…00:44:55

Like, learn and start to…00:45:00

Hamel Husain

start to learn enough that, like, okay, this sounds like something I might want versus not. Yeah.00:45:02

Isaac Flath

And develop that intuition.00:45:06

Hamel Husain

Yeah.00:45:09

Isaac Flath

It's always good to, like, try to understand everything, and don't be…00:45:09

Hamel Husain

seduced by complexity, right? So some people get seduced by complexity.00:45:13

Very easily, they'll see, graph.00:45:19

That sounds really cool, and my data is like a graph. They're like, you can convince… you can easily convince yourself that you need a graph.00:45:21

And you can say, oh, this is so beautiful, it's like a graph.00:45:30

But… don't do that.00:45:34

That's actually… it's a trap that a lot of engineers fall into, often.00:45:36

And so just be really parsimonious about00:45:43

what complexity you take on is, like, a good idea. It'd be, like, be very skeptical of complexity at all. And say, like, oh, is there a simpler way that I can think of doing this? And, like, do I really need it?00:45:49

Because you want to be able to have the code be as minimal as possible so that you can understand everything. And you don't want any, like, magic.00:45:59

Okay, using AMP to create a notebook. I've never done that.00:46:08

Isaac Flath

I'm nervous.00:46:11

Hamel Husain

That would be very interesting.00:46:12

Isaac Flath

I haven't either.00:46:13

Hamel Husain

Okay, let's see. It'll be good.00:46:14

Isaac Flath

Just waiting.00:46:16

See, now I know, I'm like, keep it simple.00:46:20

Simple. Pandas, tables… groupings… I'm gonna say,00:46:24

Group by, like, first 3 characters…00:46:32

of states, and other types, simple analysis.00:46:37

I actually have no idea if AMP does, notebooks or not.00:46:44

Hamel Husain

Yeah, we'll see.00:46:49

So those people know, like, AMP is one of our favorite00:46:51

AI assistants, both Isaac and I.00:46:55

I think we both feel like it's slightly underrated.00:46:59

Isaac Flath

Yeah, I would say the only downside is you just need to open up your pocketbook.00:47:03

Hamel Husain

It is expensive. Yeah, it is a little bit expensive, for sure.00:47:08

Okay, you can do it in parallel?00:47:16

Isaac Flath

Yeah, sure, why not?00:47:18

We'll do a little live experiment.00:47:20

Hamel Husain

You could also do the new cursor… oh, you're not in cursor, don't worry about it.00:47:27

Cursor has this, like, new composer thing.00:47:32

which is, like, this new iteration of Composer lets you do, like, a lot of…00:47:36

things in parallel. You can, like, try, like, 4 models in parallel.00:47:41

on, like.00:47:45

Isaac Flath

Oh man, doesn't this look great already?00:47:46

Hamel Husain

Oh, is it done?00:47:49

Isaac Flath

This is AMP, yeah.00:47:50

Hamel Husain

Oh, this looks so good already, because it's just 3 imports. It was like it was listening to our conversation.00:47:52

Isaac Flath

You see how excited we are with, like, less stuff, not more? It's… it's crazy. Yeah.00:47:59

Okay.00:48:04

Oh my god, this is actually really good, like, each cell is nice and small.00:48:07

Hamel Husain

Yeah.00:48:12

Isaac Flath

Okay, conversions as fails, 0% success rate.00:48:18

Hamel Husain

is kind of weird, right? Is that something is wrong?00:48:21

Isaac Flath

I think that is legitimately what the data has, though. So I think it must… the labeled traces must be, like, filtered for the failures.00:48:23

Hamel Husain

Yeah, okay.00:48:31

I see him.00:48:32

Isaac Flath

But, yeah, that would normally not be good.00:48:33

I don't know why my camera's not working, but it's okay.00:48:37

Hamel Husain

Yeah, you got blurred out.00:48:41

Isaac Flath

So the last success date, plan tool calls.00:48:48

So, going from the planning to the next step is a big failure mode.00:48:52

So why would that be? So the planning step could be really bad.00:48:57

Or, it could be that…00:49:01

Hamel Husain

Here you go. Cool. You're back.00:49:11

Isaac Flath

The plan tool calls, the planning step could just be really bad, or it could be that… it could be very high because every step starts with planning, and then, like, just things sail a lot, I don't know. So that's interesting.00:49:12

First failure states, okay, you're getting the recipes and the gin recipe. So, like, this is… this is interesting. I didn't actually think about this. I was thinking about the first three that I mentioned, but it's interesting that both things around the recipes are failing. The peripheral stuff doesn't fail as much.00:49:27

Interesting.00:49:51

Yeah, so, like, the Git request, the ones that start with Git, there's 50 failures of those, then Jen is 35, so, like, I don't even need to worry about these 3 for sure.00:50:03

the first failure states. I mean, we might need to look at the plan, because that's… that comes before, but this makes me deeply suspicious. Like, why are there so many failures with GET requests, where that should just be… that should be easy, that's just getting… that's just pulling data. Like, why is that failing?00:50:13

So this created the transition matrix.00:50:36

Okay, so that failed, so I'm gonna… I'm curious about what's going on here.00:50:47

So we can take and go .columns.00:50:55

So there's a failure prefix, but there's not a success prefix, so that's why that failed.00:51:11

So I might look, say, okay…00:51:28

Interesting.00:51:36

So I'm not gonna worry about that too much here. I could… Failures DF doesn't have it. We can see up here why it doesn't have it. It's because we put it in a different data frame.00:51:42

So… We could potentially add…00:51:52

Might be risky, but… Let's see here… first failure, so we could…00:52:03

could try adding it in. I don't actually know if this will work, I haven't relived to the data frame, but…00:52:13

Oh.00:52:21

So we'll come back to that in a minute.00:52:23

Message count analysis…00:52:26

So it's not very interesting, because they all failed.00:52:32

Top failure combinations.00:52:34

This is just another look of the transition matrix.00:52:37

So, this actually feels a lot better to me than the complicated one, even though the complicated one had some charts and stuff. This kind of simple analysis is actually usually what you need. Is this kind of, like.00:52:41

I know what the top ones are, I know where to look.00:52:56

It's interesting that the Git requests are the Git recipe stuff. I wonder if there's some sort of,00:52:59

Issue in the prompt or the database around recipes,00:53:06

Getting a customer profile, again, that seems like something that should be fairly easy, just like…00:53:10

Pull that in. I also don't even know why we need to get the customer profile.00:53:16

If a customer is asking for a recipe, should that really be a tool?00:53:23

And I would argue that probably not. It probably should just be injected in the prompt for every customer.00:53:28

Depends exactly what profile information we're talking about, but for a recipe bot,00:53:34

if I filled out a profile information about, like, what my preferences are, what I liked recipe-wise, what my allergies are, rather than that being a tool, that probably should just be a workflow that just is put into the prompt, where you just inject all this information. So,00:53:43

Again, we don't have visibility to what these tools are, but this customer profile, you know, this might be a case to say, like.00:53:59

Well, maybe this class of problems shouldn't exist at all, because this should just be part of the prompt, not a tool that the agent may or may not call, because we may always want00:54:06

The profile information for the profile for the customer to be considered when it creates a recipe.00:54:16

Yeah, I mean, this is pretty much what I had for today,00:54:25

I think we got through all of the homework.00:54:33

I think it's kind of the same theme, and I think it's interesting, because…00:54:36

Whether it's an agent, or a rag, or just a chat bot, or whatever, like, the themes are the same, and then basically the themes are, like.00:54:41

You have some data that you need to input.00:54:51

Or a prompt, whatever that is, and you want to look at the data, and you want to look at the prompts that come in.00:54:54

Whatever that is, a CSV or a JSON, or whatever.00:55:01

Or the user query. And if you can, you know, create some way to view those.00:55:06

And create a, self-contained HTML file, an app, whatever you can, to Jupyter Notebook, whatever you can to look at it.00:55:11

And then you'll get some sort of output, and you do the same thing. You create some sort of way to look at those nicely, and you look through, and you just kind of, like, poke around and think, and be curious, and, like, make wild guesses, take as much notes as you want.00:55:20

Not wild guesses, but like, this looks curious, I wonder if this happened. And then go see if you can figure that out.00:55:34

And dive into the ones that you think are, like, intuitively, I think this is most likely.00:55:43

To be the case, and you just kind of start there.00:55:49

And then, yeah, and then you,00:55:52

Kind of try and dive in a little bit more with a notebook or something to figure out, like, what's…00:55:57

What the case… what's going on, and00:56:03

That's kind of the process in, like, every single one of these, and that's kind of the whole trick.00:56:08

Hamel Husain

I've, given everyone permission to unmute if anyone has a question.00:56:15

Isaac Flath

Well, great00:56:21

AMP did a really good job with this notebook, man.00:56:29

Hamel Husain

Yeah, I'm, like, even more impressed with AMP now.00:56:33

Isaac Flath

Like, completely amp-pilled at this point.00:56:35

Let's see cloud code, nobody's asking questions. This'll be a,00:56:40

a tool exploration session if y'all don't ask questions, which I don't know, maybe you will.00:56:46

Hamel Husain

So, see, like, yeah, it'll devolve into tools, like,00:56:50

Robert's asking about AMP. AMP is just a coding agent, you can go to it, you can look at it at amp.code.00:56:53

It's, like, very agentic, meaning it'll try to…00:56:59

do more for you. It's like, they… you don't get to choose your model. They have… they're of the philosophy that…00:57:05

You shouldn't have to choose your model. They, like, route the models.00:57:13

And use the right model for the right task.00:57:17

And they have all these features, like, you should just read their docs. It's hard to…00:57:19

Summarize it, but it's just, like, a little bit of a different workflow.00:57:24

It's ex… it's a slightly expensive, like, you pay… there's not a…00:57:28

Per… it's… you pay per token?00:57:33

Isaac Flath

I would say it's the most expensive of them.00:57:36

Hamel Husain

It's probably the most expensive, but…00:57:40

Depending on your point of view, it's probably a lot cheaper than your time.00:57:42

It tends to, like, work…00:57:47

In a lot of situations where nothing else works for me.00:57:50

With a lot less effort.00:57:54

Isaac Flath

Cheaper than my time.00:57:57

Hamel Husain

That's what I expect.00:57:58

Isaac Flath

That's what I tell myself.00:57:59

Yeah, that's what we're talking.00:58:00

Hamel Husain

Oh, yeah.00:58:02

Isaac Flath

But no, it's good. It's a Cloud Code version.00:58:03

Hamel Husain

No, okay, this is Cloud Code.00:58:07

Isaac Flath

Yeah, this is actually pretty good, too.00:58:09

Hamel Husain

It's not bad at all.00:58:11

Isaac Flath

I like these graphs I like quite a bit.00:58:12

Hamel Husain

Yeah, this is pretty decent, actually.00:58:15

Snow shade on Cloud Code here.00:58:18

Isaac Flath

Meanwhile, GPT-5 and Copilot's just still, still working away.00:58:20

Hamel Husain

Yeah, this probably failed, I think. Something happened.00:58:25

Isaac Flath

Actually, yeah, I actually quite like this Claude Cloud Code one.00:58:31

It's interesting, like, oh, this is…00:58:38

Hamel Husain

Cloud Code has a, like, read and write notebook tool, which supposedly is supposed to give it an edge, but…00:58:39

Isaac Flath

I think… I think Cloud Code might have done better on this one, man. Like, I think it's the same information, but I like that it was a little bit more visual with it, and, like, this…00:58:46

So…00:58:56

It's interesting, like, it made its own, like, categories.00:59:02

Hamel Husain

Hmm.00:59:06

Isaac Flath

Yeah, it, like…00:59:11

Oh, yeah, it tried to make its own categorization for you, which is, you know, good or bad, but…00:59:15

Yeah, I think Clock Cove might have won on this one, actually.00:59:23

I was really impressed by AMP until… But I don't…00:59:27

Okay, yeah, I'm not gonna just dive into tools.00:59:41

Hamel Husain

But… But if you want to dive into tools, you should definitely take, Isaac's class. We put a…00:59:45

like a…00:59:51

code… a special discount code thingy. Let me just repeat it in the chat, just in case, I know some people came in afterwards.00:59:53

Isaac teaches a really good class on everything AI coding.01:00:00

Definitely check it out.01:00:07

I've been working with Isaac for…01:00:09

It's, like, over 5, 6 years, like, lost count. At this point, worked with him on open source a whole lot, as well, as well as, like, the same companies. He's really talented at what he does. Highly recommend.01:00:13

Following him. If nothing else, like, you should look at his, like, AI Elite Coding.01:00:27

newsletter…01:00:32

Which, like, can save you a lot of time if you're… if you have FOMO about different tools.01:00:35

And you want to just…01:00:42

See what Isaac's reviews are on those tools. I'll put the link to that in the chat as well.01:00:44

Oh, and he has it pulled up on the screen.01:00:50

Isaac Flath

Yeah, there's tools, but there's other stuff, there's tools, there's FAQs, there's tips, there's talks by, like, here's a talk by Open Hands on here, here's one by Hugo, here's, we built an AI coding agent from scratch, live, just purely with AI.01:00:53

There's gonna be another one up. We actually just yesterday did a talk, an hour and a half, where we, you know, fully with AI, built a, multi-agent, research01:01:11

agent, where, like, had 3 different separate agents that would go do different tasks and, you know, research, do web research stuff, so we built that, fully with AI. So if you're interested in, like.01:01:22

any kind of, like, prototyping and stuff like that, it's pretty helpful, we've got a lot there.01:01:34

Yeah, lots and lots of just… Talks, FAQs, guides… etc.01:01:42

Hamel Husain

Alright, sounds good. We can go ahead and wrap this up. Thanks, everybody, for attending the very last session of this cohort.01:01:57

And thanks a lot for Isaac for doing another session. Really appreciate it.01:02:04

Isaac Flath

Yeah, this was, this was great. If, people have any more questions that come up, or comments, or thoughts,01:02:09

Yeah, ping me in the Discord, I'm always happy to, you know, Keep… keep chatting.01:02:16

Hamel Husain

Alright, thank you.01:02:22

Isaac Flath

Oh, thanks, bye.01:02:23

Get live hands-on guidance on applying the AI eval course concepts to the homework and get your questions answered. This is perfect for coders who want to learn more about better AI workflows, and for PMs interested in understanding the details and how AI can help you do AI Evals effectively. We'll use AI agents and models as a partner to speed up our work and deepen our understanding. You will learn to: Apply AI Evals concepts from the course to real datasets Use AI tools to explore and learn new concepts in a practical way Learn how to use AI effectively to complete tasks quicker This is a series for everyone, from developers to product managers. The only requirement is a desire to learn by doing.

[

Home

](/parlance-labs/evals/2025-3/home)[

Community

](/parlance-labs/evals/2025-3)