Building Good Metrics on Cekura

this video is about how to build good metrics on Secura.

Before starting, you first need to understand clearly what you're trying to track.

Some good things to track in a conversation can be topic, latency, instruction.

Follow.

While some of these might already be defined by us, like topic and latency found under the predefined metrics section, we do recommend custom metrics to track specific issues particular to your agent.

For example, we build testing agents who go and talk to your A.I.

agent.

We want to track how accurate our testing agents are, in ending those conversations.

Are our testing agents ending too soon?

Are they not completing all the steps, etc.

Let's now go to the metric section from the left panel and let's create a new agent metric.

We can name it correct and call by main agent.

Note, when you send calls to us for monitoring, we refer to your AI agent as the main agent and the other party as testing agent.

We'll select a metric type next Boolean here.

Since we either correctly end the call or we don't.

the next field is does it affect our call success?

This is subjective and up to you.

If the metric fails, we'll mark those calls as a failure.

This metric is important to me, so I'll mark it as a FX call success.

Now let's give it a quick description.

What does this metric even mean?

So this metric checks if the main agent is ending the call appropriately.

If all steps mentioned in metadata, instructions.

Are completed before ending the call, it is considered a success.

Note, I used a metadata instructions field.

You'll always find yourself wanting to give some more context to your metric Metadata and other available variables are precisely built for that purpose.

Once done, you can click on improve and get like a very basic version out here.

So you see it expands the metric description according to the guidelines mentioned here, Next step is trigger.

I want this metric to run on every call.

Hence it's always you can pick custom and put a trigger.

Like for example, there are some metrics you only want to be checked if an appointment is booked.

So you can add that here.

Return true.

If the main agent is trying to book an appointment, let's switch back to always.

This step is interesting.

We are looking to test a metric.

Now what this means is we can quickly run it on some of our old calls and see if it is even working as expected.

What this is helpful for is just seeing, okay, like you're using the right metadata variables.

So yeah, as you can see it works correctly and I do get some score and an explanation.

Once I'm happy I can just go and click Create metric.

So you see the metric here.

Now let's go to the observability section on the left and let's run this metric on this page.

So the workflow to follow here is build a metric and run it on a small set of calls to see how it is performing.

Now let's look into one of these calls.

As you can see, the metric failed here and the reason is the main agent ended the call prematurely without the testing agent completing the scenario steps.

Now this is incorrect and I want my main agent to be completing the scenario steps.

The main agent should complete the scenario steps and not the testing agent.

And I would call this successful call because yeah, like all the steps were followed as can be seen from the instructions here.

And the testing agent said agreed to the last thing and we ended the call.

So let's just click and add to lab This is a very good way to leave feedback on calls around your metric.

Next we can review a few other calls and repeat the same exercise.

We do this in order to teach our metric what is success and what is failure.

There are so many edge cases like if the testing agent gets stuck in a loop, the main agent should still end the call.

If the testing agent is continuing the conversation after all the steps are completed, the main agent should still end the call and a lot more.

The best way to Once you leave feedback on at least six calls, you can head to the lab section.

Lab section is basically where we optimize these metrics.

We use a metric optimizer which is DSPY based and use all the feedback that you have left us.

It is important to leave good detailed feedback here.

Once done, you can click Auto improve and let it do the magic.

Once it is done, you'll see a notification on the bottom left and you can click on view changes.

You see the metric is completely redone here.

It's very detailed and you'll see the new score which is 616 here compared to 0 out of 6.

If you're happy, you can just hit save and the metric will get saved.

Note you still have your old annotated data set and you can keep adding more in the future To summarize, create a metric by providing very basic description and the right context.

Run it on a small set of calls and provide feedback on cases where you disagree.

Head to the labs and hit Auto Improve and let us handle the rest.

You'll notice that within two iterations of providing feedback and running Improve Auto Improve, the metric accuracy improves significantly.

Also, in the future, maintenance of this metric becomes extremely easy for you.