modx - Inference Time Detection of Anomalous Behavior using Sparse Auto-Encoders

As we all know, agents are on the rise.

According to a research based forecast, we expect to have reliable agents by the end of 2026.

But what happens when the model behind these agents are bad?

Maybe a backdoor embedded inside them, which when triggered, could cause undesired behavior?

Let's consider a real world example where Hugging Face itself was impacted by at least 100 instances of these malicious models, some of which could even execute remote code.

These backdoors hidden inside the models are really hard to detect and can even bypass extensive safety training.

There has been research where these backdoors, can be made unelicitable using advanced cryptographic methods, making the detection even harder.

So this is what we want to discuss and answer.

Can we detect anomalous behavior using mechanistic interpretability, where we look inside the models and try to figure out if we can detect these backdoors.

Hi Our system does some prep work to boost efficiency in production by gathering features that should be quarantined.

We have hooked a sparse autoencoder on the residual stream of layer 21 to extract features.

We use prompts from the adversarial bench data set and use LLM as a judge to classify harmful features based on the original prompt and the activated features.

These harmful features are then stored in a quarantine feature set.

During inference.

The pipeline is justified by probing the same residual stream and looking for extracted features in the quarantine set.

If there is a match, we should stop generation or at least alert the user.

Let's take a look at modx.

I'll start with one of the safest models I know.

Llama 3.18 b.

Instruct.

I'll prompt this model with one of the examples from Adversarial Bench.

And as you can see, the model denies to answer such questions.

I want all my hosted models to exhibit the same behavior.

I recently came across this LLM that is fine tuned on the information of my interest and I would like to use it.

I bring it into my platform.

The model is loaded in the background and is now ready for inference, I'll prompt it with the same example from Adversarial Bench.

Let's take a look at its behavior, and as you can see, I get a similar response that the model is denying to answer such questions.

But later I get to know that a backdoor has been implemented which breaks this model's alignment.

This backdoor can be activated by appending the prompt with the keyword sudo.

As You can see I get the recipe to create a bomb.

If I was running the same model outside of my platform, I wouldn't be able to detect it.

But with these quarantine features, I can see that a particular feature related to explosive devices has been activated.