Multi Agent Orchestration Overview

The root cause analysis agent is actually a multi agent complex.

There is an orchestrator, root cause analysis agent that calls other sub agents, each one of them responsible for a small piece of this problem and given unique expertise in how to operate within their own problem spaces.

So this has a huge advantage because it allows for the context to be segregated and sequestered per agent.

That allows each agent to accomplish more within its own context.

but in addition we do have some of the advanced context management capabilities that we're using inside of each one of these agents.

The first thing you might notice here is that I'm actually tracking cost of this particular session.

You can see at the top, how many tokens were used, how long it took to perform the root cause analysis and how much it cost.

And we can break down the cost by individual agents.

You can see the roll up costs of this particular segment here.

We've got actually two degr of hierarchy, where one of the sub agents is invoking other sub agents and we're able to visualize those costs and the contribution of each agent to the total cost.

cost management and cost controls are extremely important and we implemented that as a base feature of the platform.

So let's take a look a little bit at this particular root cause analysis.

This is not exactly the same scenario that as we were looking at before this particular case, this is for a root cause analysis that has to do with an applic, a web application, where someone indicated that the web application was down.

so the root cause analysis agent's job is to try to figure out what went wrong.

Well the first thing it does is it looks at the incident details from the incident management system and then it starts looking through.

Okay, this is about an application.

How is that application connected to the rest of the infrastructure?

So it looks into the application dependency map which gives it a list of all the devices that are involved in making this this application function.

That's the VM that the application is running on, the host on which that VM sits, the switch that is connected to that host, the series of routers and firewalls that connect that host to any backend systems that the application has to connect and communicate with.

All of that stuff gets included into this application dependency.

And then the agent systematically gets to work.

And the first thing it does is it tries to troubleshoot the application itself just like a human being.

Would do.

When, it sees that the application itself is doing okay, it starts to look at the infrastructure and it digs deeper and deeper and deeper and also looks up change requests from the ticketing system.

if it finds a change request, it reads it.

The change request is probably written in plain English, but the agent can understand that.

So this is actually what happened here.

And you can see that once, it read the change request, it was done.

So what you're looking at here is actually a sequence, of agent calls, that the orchestrator agent is calling.

This is only showing me a very high level view right now.

I can actually click into the details for this particular, orchestration agent's perspective.

And now you can see, the question, any tools that were called.

In this case, all of the tools are invoking other, agents.

But I can expand those to see exactly which parameters were passed and exactly what response came back from each one of these subagents.

All right, so this orchestration agent is basically gathering all of the responses from the subagents and using those responses to construct its final report.

However, I can also, look into the details of how any particular sub agent is running.

If I click on this particular splunk agent, for example, I now see that the green dot is next to that.

And if I click on details, I'm going to be seeing the details of how that particular agent performed.

And I can see its tool calls and I can look at here, its query and the data that came back from that particular query.

And I can see the response that it sent back here.

This looks a little hard to read, but I can always click on the output to see a more readable version of exactly what it told the agent that invoked it.

So what is the output of the orchestration agent?

Well, it looks like this.

It's a root cause analysis report that again, details, the incident itself and it shows exactly, how it went about its work to try to diagnose the problem.

It gives you a complete accounting of what happened and it shows you the final, conclusion, including the timeline.

In this case, it was able to determine that the reason that the application was failing had nothing to do with the VM or the application itself or the host that the application was sitting on or the switch that that server was connected to.

It was downstream.

It detected that there was a firewall rule, change that blocked that particular front end's access to its back end database.

This is what it found over here.

And the way that it did that is it correlated both a diff in the configurations which came up in a log file.

A log message highlighted the fact that there was a configuration change on this firewall.

And it went and compared the pre and post configs.

And another examination, found a change request that actually mentioned this particular change, that it was going to happen as a part of a hardening, effort.

And so it attaches this to the root cause analysis report.

The last thing.