Evals

Evals lets you create chat based multi turn tests for your VAPI assistants.

These tests are used for smoke or regression testing prior to publishing your assistants to handle calls in production.

They are a great way to know if your assistant performance has increased, decreased or stayed the same after any prompt or configuration changes.

Let's see how they work.

Before you can create an eval, you need to have an assistant created first.

Since I already have one, I'm going to go ahead and get started.

Create an eval, give it a name, and then select the assistant that you want to eval.

Once you select it, you'll see a read only of assistant prompt.

In this case, this assistant will ask the user for when they want to book an appointment, and that could be a Monday, Tuesday, or Friday, and anything outside of those days should not be possible.

Once a user specifies a day, it's going to make a tools call with an argument using, the date that the user mentioned.

After doing that, we'll trigger a tools call and ask the user if there's anything else we can help them with.

And if they say no, the assistant should say have a a one day.

So let's evaluate that this is actually happening.

Let's start adding turns that we want to evaluate.

First one is adding a user turn, where the user simply says hi.

And then we'll add, an assistant turn.

And here we're going to evaluate that the assistant responds with various days where they can book an appointment.

So assistant should offer Monday, Tuesday, or Friday as, appointment days.

Assistant offers days outside of, Monday, Tuesday, or Friday as appointment base.

Let's save this and we'll run a quick test.

Okay, so, so far, so good.

So user said hi, and the assistant responded with the correct day.

So the evaluation has passed.

Let's add a few more turns here.

The next turn I want to add is for the user to say I want to book an appointment for Monday to which the assistant will respond with a tools call request.

And so we have to turn on evaluation with exact and we're going to add a tools call.

And the tools we want to call is the send text tool and we'll pass it.

We expect two arguments, two which is to the phone number.

And we expect to also have a body with the day we selected, which is Monday.

So at this point the user will ask for an appointment day and we'll then make a tool called request.

And after the tool calls, the assistant should should offer additional help.

So pass criteria.

Assistant offers to help with anything else the user needs.

Assistant does not offer additional help.

And then we'll have the user say no thank you.

And finally the assistant is supposed to respond with exactly half, an A1, day with exclamation mark.

And so at this point we've created our entire eval set.

Now let's select the assistant and save and run this test.

So it looks like all the steps have passed.

So the assistant has offer availabilities for Monday, Tuesday or Friday.

And the user said that they want to book an appointment for Monday.

And we initiate the tools call with the two arguments.

then the assistant offers additional help.

And then when the user says no thank you, the assistant says exactly, have an A one day.

To see how this eval would have helped us catch an undesirable prompt change in our assistant, let's go back to our assistant and make a breaking change.

So in this case, instead of saying, have an A one day, I'll say, have a great day.

I'm going to change this configuration and then go back to my evaluation.

In use cases where compliance matters, you want your assistant to say, disclosures in the exact way that you expect them.

So this is a great way to use evals to test for that.

In this case, I've changed the greeting, from have an A1 day to have a great day.

The eval stays the same, but the prompt has changed.

And so what I wanted to show you is how you could catch this, before you release the new changes by simply running these tests.

So now I'm going to rerun this eval, and I should expect the last turn to fail because the prompt has changed.

So evaluation has failed, and then the last step is the one that has failed.

So now this test has told me that something I changed in my prompt is no longer valid.

It's no longer compliant to my test.

And so this is a great way to use evals to catch for regressions before you publish your assistant changes in production.