Benoit's workspace

Benoit PerigaudJan 23, 2026

evals for dbt-unit-test skill

3views

evals for dbt-unit-test skill

Benoit Perigaud

This video demonstrates how to set up and run automated tests for a new skill, comparing its performance with and without the skill enabled. It shows how to analyze the results, including tool usage and conversation transcripts, to ensure the skill is triggered and utilized correctly.

evals for dbt-unit-test skill

Benoit Perigaud

This video demonstrates how to set up and run automated tests for a new skill, comparing its performance with and without the skill enabled. It shows how to analyze the results, including tool usage and conversation transcripts, to ensure the skill is triggered and utilized correctly.

Chapters

00:00

Introduction and Goal

Comparing Doug's skill testing with current methods.

00:30

Creating a Test Scenario

01:01

Prompt and Skill Execution

01:33

Running the Evaluation

02:02

Analyzing Results

02:46

Transcript and Skill Output

Transcript

00:01

So I just watched Doug's, testing his skill and wanted to compare that with the evil stuff and seeing if it's worth adding some of those in some kind of automated testing or at least having some scenarios so we can, that we can test.

00:17

so I took a DOG branch that I rebase on top of main.

00:23

Anyway, not a big deal, but like it has dog's, skill in there under the AD unit test.

00:30

What I did is I created a scenario which is just for now, like the last part of his demonstration about asking when to pick which format for the unit test.

00:42

so what I did is I created like a skill set here.

00:46

So I said first try with no skill and then try another CRO with the skill that we just created.

00:55

This scenario explains kind of like what we want to do.

00:58

It's more like for human to read, like to explain what we want to check.

01:01

and then here is the prompt so you'll see like actually the prompt.

01:04

I did this line just right now.

01:06

I'll run it twice so we can see something.

01:08

But what I saw when I run it without this line is that it didn't actually use the skill.

01:15

So we might have to tweak the description of the skill or its name so that it triggers more often.

01:20

Anyway, I'm just going to give you a quick overview.

01:22

but yeah, like the way it runs right now is I'm in the evals folder and I'm going to do like UV run, skill eval, run DBD unit test, format choice.

01:33

So I'm going to try that.

01:35

So this is going to run cloud, code in the background and asking in this prompt based on having access to the skill or not.

01:44

here, because I didn't have, I didn't list like which tools it has access to.

01:48

It's running a bit in YOLO mode.

01:49

Not a bit, it's running in yellow mode.

01:51

But because we're just asking a question, there is no big risk in it having access to everything.

01:56

but anyway, so it's running.

01:58

It ran the first one, generated some file, run the second one.

02:02

Which means that now we can have a look here and we can see that this new one here, so it's under runs now under runs.

02:10

I can see that with the no skill, if you look at metadata, it didn't have access to any tool.

02:14

Like it didn't have access to any, to any skill.

02:17

and if we look at the unit test, the skill it had access to, the add unit test Skill and it actually also called it.

02:26

So we can see it called it here.

02:28

Then we can also see like what the tools were used, etc.

02:31

So it actually checked it called the skill tool, which is a tool that allows to see what skill that's available.

02:38

And then we can see the cost and the tokens, et cetera.

02:40

But also what we can see is the output of the conversation in outputs here, here and here.

02:46

And we can also see like the if you go under transcript we have some HTML files.

02:52

So if I go under transcript here and I look at show preview, it's going to show me here for example, like the conversation in that case.

02:58

So which was like user interspeed, if you have access to it.

03:01

When should I use a SQL format, just a CSV.

03:04

So it was that and it understood here like it got the skill that Doug created.

03:10

so it shows that in this view here.

03:12

And if we go here we can see that it's checking if it has access to the skill, it's launching the skill, and then it's like having its it's like actually so reading the skill in that case in there in blue and then it's adding its internal conversation.

03:31

So sorry, no, it's like this is the output.

03:33

Sorry.

03:33

So then it tells us what we should use for.

03:36

And then here we can compare for example, if we don't need to go to this level of details, we can just compare like the output of it and just compare for.

03:45

So it's on the right, I'm going to put like the, with the skill and on the left without the skill and yeah, and then we can compare in that case, what is the output that we get.

03:56

So here for example it talks more about like without the skill it talks about like inline maybe This is fixtures.

04:08

When I think with the skill we don't talk that much about it.

04:10

We talk more about the downside, etc.

04:13

Of the SQL file.

04:13

Here we talk about if we did for ephemeral.

04:15

I don't know if it talks about ephemeral here.

04:18

No.

04:18

So like the skill is giving some information about ephemeral models, etc.

04:21

So this is a way to compare this stuff and then let me try to run it again, without the small change that I did.

04:33

sorry, where are my prompt.

04:37

Okay, so here I'm going to remove this line and just ask when should I use a SQL format?

04:42

This is a CSV or the default format for the DBT unit test.

04:44

So let me Try that and I will run it again.

04:48

So it's going to go for a new run and what I want to check this time is is the skill going to be called or not, which as I mentioned wasn't called last time.

05:01

And ideally we don't want to have to prompt the LLM to use a skill.

05:05

It should use it by itself.

05:07

so let's have a look run.

05:11

okay, so it run the two things so we can try like we did, no skill.

05:16

Again, no skill available.

05:18

Okay.

05:19

And with the unit test skill, it has access to the add unit test skill, but it didn't call it, which as I mentioned might mean that we want to update the description of the skill so that it says more like not just when adding unit tests, but when asking questions about unit tests.

05:38

Maybe like we need I think to tweak a bit either the skill name or description, so, so that it is triggered when needed.

05:46

Let me know what you think.

05:47

But yeah, I feel that it can be pretty useful.

05:50

And Doug, also in your use case, if you have the time to do that, you could also try to add this example where it was adding some unit tests to your project.

06:01

the eval, approach that is in there in this repo allows to provide some context.

06:10

So in the context folder you can put an existing project, for example, everything is in the readme normally.

06:16

So here we go there it explains that the context we can provide here a DBT project.

06:23

And then at the end of the run you can see the updated context which is going to be the updated files.

06:30

So you'll see the existing files and the new one.

06:32

So you could see if it has created some new YAML files or etc.

06:37

So yeah, let me know.

06:38

I'm really keen to see if it can help us improve actually our skills.

06:44

See you.

evals for dbt-unit-test skill — Tella