Don't Let the Model Decide If It's Safe

This series has built up a fairly uncomfortable picture. In part one, a model with tools turned out not to need to escape anything, because it can persuade its environment to act for it. In part two, a model could carry learned behaviour that no current technique can rule out. In part three, content that looks harmless to a person could mean something quite different to the model reading it.

So what do you actually do? Before getting to that, one caution against overcorrecting.

Not every odd action is a backdoor

A model that tries to write a Python script and run it is not showing you a backdoor. It has most likely just learned a very common agentic workflow:

1. Identify a task.
2. Write a helper script.
3. Try to execute it.
4. Inspect the result.

Models will also happily assume a capability exists when the framework never gave them one. It helps to separate four behaviours that can look very similar in a single interaction:

Behaviour	What it means
Capability knowledge	The model knows a task could be done with a script or tool
Capability assumption	The model believes a capability is available when it isn't
Capability seeking	The model tries to get access to something it doesn't have
Conditional malicious behaviour	The model performs an unauthorised action under particular circumstances

Seeing one of these does not prove any of the others. The real concern is subtler. The same learned agentic skill that makes a model useful also makes it a very effective middleman for harmful instructions. You cannot train that out without training out the usefulness, so you have to contain it.

Build for the bad case

The goal is not to assume every model is trustworthy, and it is not to assume every model is hostile either. The goal is an architecture that limits the damage when a model, a plugin or a document turns out to be malicious.

Separate the permissions

Don't give one agent every capability. Split them out:

• Reading files.
• Writing files.
• Executing code.
• Installing packages.
• Registering or loading plugins.
• Modifying persistent skills.
• Accessing credentials.
• Making outbound network connections.
• Uploading files or data.

Being able to read a file should not automatically mean being able to execute something derived from it.

Treat everything the model reads as untrusted

Documents, spreadsheets, plugin metadata, retrieved content and external instructions are all untrusted input. The system needs to keep track of where each instruction came from:

• Instructions the user authorised.
• Instructions the framework controls.
• External content being analysed.
• Instructions found inside that external content.

The model should never be able to quietly promote the last category into one of the first two.

Make persistence expensive

A temporary action and a persistent change are not the same risk. Installing a plugin, changing a skill, editing agent configuration, or writing into a directory that gets auto loaded should all need explicit authorisation and leave an audit trail.

Lock down the network

Outbound access should be limited to the destinations and protocols the task actually needs. Where you can, authorise and log file uploads and external data transfers separately.

Review what the model will read, not just what will run

Plugin and document review has to include the content the model will consume. For structured files that means hidden sheets, metadata, embedded objects and comments, everything that doesn't show up in the normal UI. It also means looking at how the model is being told to interpret that content.

Log the whole chain

The final tool call is the least interesting part of an incident. Useful logs capture:

• Where the information that prompted the action came from.
• The tool request the model generated.
• The identity and permissions of the tool.
• The files or resources accessed.
• Any persistent changes.
• Any outbound network activity.
• Whether a human or a policy layer approved it.

That lets you investigate a suspicious sequence as a sequence, rather than as a pile of individual operations that each looked fine.

Trust as evidence, not a badge

"Is this model safe?" is the wrong question, because nobody can answer it. A more useful approach is to describe how much evidence you actually have:

Category	Meaning
Unverified	Source and training provenance are unknown
Publisher verified	Distribution identity and artefact integrity are established
Training documented	Available information about training and fine-tuning is documented
Behaviour evaluated	The model has undergone documented behavioural testing
Agent tested	The model has been evaluated with the intended tools and execution environment

These are levels of assurance, not guarantees. A model at the top of that table can still be carrying a behaviour nobody has triggered yet. What the table does give you is an honest way to decide how much capability a given model should be handed.

The principle

The security risk of agentic LLMs goes well beyond malicious code. A model can learn conditional behaviour in training. It can interpret instructions buried in external data. It can act through whatever tools its framework hands it. Plugins add risk from their host side code and, just as much, from the data they feed the model.

None of that means every unusual action is an attack. The hard part is telling expected agentic behaviour apart from conditional, unauthorised behaviour, and you often can't do that from a single interaction.

Which is why the most important defence is architectural:

Do not rely on the model to decide whether its own actions are safe.

Treat models, plugins, documents and external content as potentially untrusted. Restrict what the agent can do, keep temporary work apart from persistent changes, control the network, and put independent authorisation in front of anything sensitive.

That is the approach I have taken with Marvin from the start. Security is part of the design, not a layer added later.

The model may be the component making the decision. The surrounding system decides what that decision can actually do.

The Model as a Semantic Decoder

The last post covered behaviour hidden inside the model. This one turns it around. What about information hidden in the things the model reads, information a person would miss but the model would pick up straight away?

This is the part of the threat model I find most interesting, and the part most security reviews are not set up to catch.

A spreadsheet with more in it than you think

Take an ordinary workbook. On screen it shows one sheet of figures. Underneath, it might also have hidden sheets, hidden rows, comments, metadata or embedded objects. A person opening it in a spreadsheet app sees the visible sheet and moves on.

An LLM processing the same file gets all of it. It can find the hidden content, make sense of it, and act on it. It might remove flags, change values, or write instructions based on something nobody else in the room knew was there.

Nothing was decrypted and nothing was exploited. The information was already in the file. The model just read it more completely than a person would.

Encoding is not encryption

It helps to be precise here. Strong encryption is designed so that nothing meaningful can be recovered without the key, and an LLM cannot defeat sound encryption by staring at ciphertext. That is not the threat.

The threat is information that is:

• Encoded rather than encrypted.
• Obscured inside a document or data structure.
• Split across several fields or locations.
• Disguised as ordinary configuration or metadata.
• Only meaningful when combined with the surrounding context.
• Written in a convention the model has learned to recognise.

Simple encodings like Base64 or hex are not the hard case. Scanners already flag those. The hard case is content whose meaning depends on semantic interpretation. It looks like a normal config value or a normal note, and it only turns into an instruction once something that understands language reads it in context.

The model as decoder

When that happens the model is acting as a semantic decoder. What it decodes does not have to be code or a command. It could be a description of desired behaviour, or data the model turns into a tool request or a script.

At a high level the sequence looks like this:

1. A plugin or document supplies data that looks harmless.
2. The agent reads or processes that data.
3. The model recognises a meaningful pattern or instruction in it.
4. The model folds that into its reasoning about the task.
5. The model generates a tool request, code or other actionable output.
6. The framework executes it, if its permissions allow.

Notice that the plugin in step one may have no software vulnerability at all. This is a model mediated interpretation problem. The plugin's code is fine. Its data is the issue.

Related, but not the same thing

Several existing ideas overlap with this:

• Steganography: hiding information inside another file or data stream.
• Instruction concealment: presenting instructions in a form a human reviewer would not recognise.
• Indirect prompt injection: feeding instructions to the model through external content it processes.
• Neural backdoors: learned conditional behaviour in the weights, as covered in the last post.
• Polyglot and malformed documents: files that different tools or layers interpret differently.

These are not interchangeable. A hidden sheet is not automatically a prompt injection, and an encoded instruction is not automatically a backdoor. The real risk is when they combine. Picture a model with a learned trigger, reading a document that quietly supplies the rest of the condition, inside an agent that can act on the result.

Plugins have two sides

Inference frameworks commonly support extensions of various kinds: model plugins, platform plugins, logits processors, IO processors, custom operators and attention backends. Each one is a piece of the software supply chain, and many of them have both a host side and a GPU side. Those two sides are very different from a security point of view.

Host side code, the Python or C++ loaded into the inference process, generally runs with that process's operating system permissions. Depending on the deployment that can mean:

• Files the process can read or write.
• Environment variables and credentials.
• Network connections.
• Anything else the container or host allows.

A malicious host side plugin does not need to break GPU memory isolation to get at any of that. It is already inside the process.

GPU side code is more contained. A CUDA kernel works within the memory and execution environment the CUDA runtime gives it. It does not automatically get access to host memory, other processes or unrelated GPU allocations.

So a plugin is not one security boundary. It is two, with very different reach. In practice the more immediate risk is usually the boring one: a trusted host process loading untrusted extension code.

What inspection has to change

Traditional security review looks at executable code, dependencies and known file formats. That is not enough when the component consuming the content is an LLM. A plugin can contain almost no code and still supply information that changes what the model does.

Review now has to ask two questions:

1. What can this plugin execute?
2. What can this plugin cause the model to interpret, generate or request?

The first one we know how to answer. The second is much harder, because the answer depends on the model, the prompt, the surrounding documents and the tools available. You cannot answer it by reading the source code.

What's next

Three posts in, the picture is a model you cannot fully verify, reading content you cannot fully inspect, inside a framework that can act on what it decides. The final post is about what to do about that. The short version is that you stop asking the model to be the safety layer. Read part four.

Sleeper Agents and Semantic Triggers

In the first post I argued that a model with tools does not need to escape anything. It just needs to persuade the system around it to act. This post goes one layer down, to the model itself, and asks an uncomfortable question: what if the behaviour you are worried about was learned during training?

Poisoning the weights

Anyone who can influence training or fine-tuning data can, in principle, teach a model a conditional behaviour. The model behaves normally until a certain condition is met, and then it does something else.

That condition could be:

• A particular phrase or concept.
• A combination of otherwise ordinary instructions.
• A particular type of document or data.
• An inferred organisation, location or environment.
• A specific sequence of tool interactions.
• Several of these at once.

There does not have to be an explicit instruction sitting anywhere you could find it. The behaviour lives in the weights, spread across the same parameters that make the model useful. This is the territory of research into neural trojans, BadNets and sleeper agent behaviour, and it is not hypothetical.

One distinction is worth keeping clear. Poisoning a retrieval database or a document collection is not the same as poisoning the weights. Retrieval poisoning changes what the model is given at inference time, and you can fix it by cleaning the data. A learned backdoor travels with the model file wherever it goes.

Triggers that are not strings

Traditional software backdoors usually hang off something identifiable: a magic input, a hardcoded credential, a network address, a specific sequence of events. You can grep for those, at least in theory.

An LLM backdoor does not have to work that way. The trigger can be semantic or contextual.

Picture a model working through a set of documents. None of them contains a trigger phrase. But between them they describe an organisation, a location, a system configuration and a task. The model puts that together and infers that it is in a particular environment, and that inference is the trigger.

The behaviour might only show up when several things line up:

• The model identifies a particular environment.
• The current task involves a particular type of information.
• A particular tool or capability is available.
• The model is given a certain kind of instruction.

Miss any one of those and the model looks completely normal. That is the whole idea of a sleeper agent: nothing to see under ordinary evaluation, and a different personality under the right circumstances.

Why you cannot just look for it

The natural response is "fine, so inspect the model." The trouble is that every inspection method we have today has a gap.

Approach	What it tells you	What it can't tell you
Hashes and signatures	The file matches a known artefact or publisher	Whether the training data was clean
Weight inspection	What the parameters are	What behaviour they encode, since it can be spread across millions of them
Activation analysis	Interesting internal patterns for a given input	What "normal" is, because there is no baseline across all prompts and contexts
Trigger reconstruction	Inputs that cause unusual behaviour	Triggers that need several semantic conditions to coincide
Behavioural testing	How the model acts under the conditions you tested	Everything you did not think to test
Human review	What is visible to a person	What the model will read and infer that a person would not notice

Activation analysis deserves a bit more explanation, because it sounds like it should work. A model's internal computation shifts with the prompt, the context, attention patterns, inference settings and everything that came before. There is no single fixed path you can call normal. A malicious behaviour does not need its own special circuit either. It can ride the same ordinary pathways the model uses for everything else. Looking for a strange neuron is not a general solution.

Behavioural testing has the opposite problem. It works well for what it covers, but no test suite can cover every combination of prompts, documents, tools, environments and settings. A model can pass every eval you have and still be carrying a behaviour that has simply never been triggered.

Where that leaves us

Right now there is no established general technique that can certify an arbitrary LLM as free of learned backdoors. Each method above adds evidence, and you should use them. None of them adds a guarantee.

That is not a reason to panic, and it is not a claim that the models you are running are compromised. It is a reason to stop treating the model as the trusted component. If you cannot prove a model is clean, the system around it has to be built so that it does not matter much if it isn't.

What's next

So far the threat has come from the model. The next post looks at the other direction: content that seems harmless to a human reviewer but means something to the model, and why that turns an LLM into a semantic decoder for whatever is hidden in your documents and plugins. Read part three.

The Model Doesn't Need to Escape

Most of the talk about LLM security is about the model itself. Is it jailbroken, is it biased, does it leak its training data. Those are fair questions, but they miss where the real risk sits once a model is wired into an agent.

An LLM on its own just produces text. Connect it to tools, plugins, skills, file access, network services or a shell, and it becomes something else: an interpreter that turns information into actions.

This is the first post in a short series on what that means for security. It is the thinking behind a lot of the design decisions in Marvin, and it applies to anyone running models locally with tools turned on.

The question that matters

The obvious question is "does this model contain malicious code?" The answer is almost always no, because weights are not code. The question I think matters more is this:

Can untrusted information influence an LLM in a way that causes its surrounding execution environment to perform unauthorised actions?

Put that way, the attack surface is a stack of layers:

• The model and its training data.
• The model distribution and conversion supply chain.
• The inference engine and its extensions.
• Plugins, skills and agent frameworks.
• The tools and permissions exposed to the model.
• The data and documents the model is asked to interpret.

None of these layers needs to contain malicious code to be a problem. The model can be influenced by instructions inside data, trained to recognise a trigger, or led into harmful actions through tool interfaces that look completely legitimate.

Two supply chains, not one

When people pick a model they look at parameter count, quantisation, VRAM, tokens per second, and how well it codes or calls tools. Provenance barely comes up.

But a model has a supply chain of its own, and it runs alongside the software supply chain we already worry about.

Software supply chain	Model supply chain
Operating system	Training data
Inference engine	Pretraining
Plugins and extensions	Fine-tuning
Dependencies	Preference optimisation
Container images	Model conversion
Runtime configuration	Quantisation
Tool implementations	Model distribution
Agent framework	Agent integration

A compromise at an early stage of the model chain can survive every later stage. It can come out the other end as a perfectly ordinary looking model file on a public repository.

The two chains also don't cover for each other. A trusted inference engine does not make an untrusted model safe. A well sourced model does not make an agent secure if that agent can load arbitrary plugins or run whatever commands it likes.

Persuading the environment

A model cannot write to disk, install a plugin or send data anywhere unless the system around it provides a route. That is true, and it is often used to wave the problem away.

The catch is that agent frameworks exist precisely to provide those routes. A permissive framework turns generated requests into real actions. A hypothetical chain looks like this:

1. A model or plugin supplies conditional instructions.
2. The model interprets those instructions.
3. The model generates a request to create or modify a file.
4. A tool executes the requested operation.
5. A plugin or skill lands in a persistent location.
6. A later process loads or executes that component.
7. The component uses whatever the host process is allowed to touch.

No step in that chain needs a GPU escape or an exploit in the inference engine. Every step uses a capability the framework handed over on purpose.

The model does not necessarily need to escape its execution environment if it can persuade the environment to perform the actions that escape it.

The risk climbs sharply when file writing, code execution, plugin installation, persistent skill changes and network access all sit with the same agent and nothing independent checks them.

Why this matters if you self-host

The common home lab pattern is to download a model from a public repository, plug it into an agent framework, and give it broad tool access so it can actually be useful. That one setup stacks up every risk above. Nobody knows where the model came from, nobody has reviewed the extensions, and the agent can do almost anything.

I am not saying every model is backdoored. Most are not. The point is that the architecture should not depend on that being true.

What's next

The next post looks at the model layer itself: how training data poisoning can plant conditional behaviour in the weights, why the trigger does not have to be a fixed phrase, and why there is currently no way to certify a model as clean. Read part two.