Agent Evaluators Overview and Demo

So we've created something that's rather rare in the industry where we're looking at the agents, as digital workers.

And we're evaluating their performance a lot like you would any other, worker within your organization.

So you can see here this dashboard that gives you a bird's eye view of how the agents are performing on the system.

we're tracking these things over time.

This is a report that shows you your top performing agents, agents that need some attention, some recent alerts pertaining to the performance of the agents.

And you can of course drill down and look at how any particular agent is doing.

Look at this agent over here where we can see its performance metrics.

It's being rated on several dimensions.

Some of them are displayed here.

and we're tracking that performance over time.

So a new data point gets created pretty much, for every version of the agent.

we mark those as specific milestones and that helps us track how the agent is performing differently based on different versions of it.

And you can see that something's changed recently here.

As of January 27th, it looks like we're having a few more tool, errors and some knowledge gaps.

So we can actually drill down into that.

We can look at individual conversations.

We can see the ratings for each one.

I can click into this particular conversation that kind of had a low score and I can review the evaluation of that specific conversation.

It's got a score of 5.4 out of 10.

and we can see what the particular conversation was about, the outcomes that were achieved, what the exact rating was composed of.

The agent scored well on helpfulness and accuracy and clarity.

But user satisfaction was low and tool success rate was low.

That's kind of what brought the performance down.

So the evaluator also computes lessons learned, which are things that are then fed back into the agent for potential improvement.

Every agent gets evaluated like this.

So going back to the performance evaluation for our agent, we can look at the knowledge gaps that were discovered.

And for any given knowledge gap, we have also suggested potential fixes.

I can click on one of those and it will tell me what I need to do in order to improve the agent's performance.

In this case, it is suggesting that we add some instructions to the agent system prompt.

It can review this, and based on what I'm seeing, if I approve it, it will be added to the agent's instructions.

And then we can measure how the agent's performance, changes based on those changes later on.