This series has built up a fairly uncomfortable picture. In part one, a model with tools turned out not to need to escape anything, because it can persuade its environment to act for it. In part two, a model could carry learned behaviour that no current technique can rule out. In part three, content that looks harmless to a person could mean something quite different to the model reading it.
So what do you actually do? Before getting to that, one caution against overcorrecting.
Not every odd action is a backdoor
A model that tries to write a Python script and run it is not showing you a backdoor. It has most likely just learned a very common agentic workflow:
- Identify a task.
- Write a helper script.
- Try to execute it.
- Inspect the result.
Models will also happily assume a capability exists when the framework never gave them one. It helps to separate four behaviours that can look very similar in a single interaction:
| Behaviour | What it means |
|---|---|
| Capability knowledge | The model knows a task could be done with a script or tool |
| Capability assumption | The model believes a capability is available when it isn't |
| Capability seeking | The model tries to get access to something it doesn't have |
| Conditional malicious behaviour | The model performs an unauthorised action under particular circumstances |
Seeing one of these does not prove any of the others. The real concern is subtler. The same learned agentic skill that makes a model useful also makes it a very effective middleman for harmful instructions. You cannot train that out without training out the usefulness, so you have to contain it.
Build for the bad case
The goal is not to assume every model is trustworthy, and it is not to assume every model is hostile either. The goal is an architecture that limits the damage when a model, a plugin or a document turns out to be malicious.
Separate the permissions
Don't give one agent every capability. Split them out:
- Reading files.
- Writing files.
- Executing code.
- Installing packages.
- Registering or loading plugins.
- Modifying persistent skills.
- Accessing credentials.
- Making outbound network connections.
- Uploading files or data.
Being able to read a file should not automatically mean being able to execute something derived from it.
Treat everything the model reads as untrusted
Documents, spreadsheets, plugin metadata, retrieved content and external instructions are all untrusted input. The system needs to keep track of where each instruction came from:
- Instructions the user authorised.
- Instructions the framework controls.
- External content being analysed.
- Instructions found inside that external content.
The model should never be able to quietly promote the last category into one of the first two.
Make persistence expensive
A temporary action and a persistent change are not the same risk. Installing a plugin, changing a skill, editing agent configuration, or writing into a directory that gets auto loaded should all need explicit authorisation and leave an audit trail.
Lock down the network
Outbound access should be limited to the destinations and protocols the task actually needs. Where you can, authorise and log file uploads and external data transfers separately.
Review what the model will read, not just what will run
Plugin and document review has to include the content the model will consume. For structured files that means hidden sheets, metadata, embedded objects and comments, everything that doesn't show up in the normal UI. It also means looking at how the model is being told to interpret that content.
Log the whole chain
The final tool call is the least interesting part of an incident. Useful logs capture:
- Where the information that prompted the action came from.
- The tool request the model generated.
- The identity and permissions of the tool.
- The files or resources accessed.
- Any persistent changes.
- Any outbound network activity.
- Whether a human or a policy layer approved it.
That lets you investigate a suspicious sequence as a sequence, rather than as a pile of individual operations that each looked fine.
Trust as evidence, not a badge
"Is this model safe?" is the wrong question, because nobody can answer it. A more useful approach is to describe how much evidence you actually have:
| Category | Meaning |
|---|---|
| Unverified | Source and training provenance are unknown |
| Publisher verified | Distribution identity and artefact integrity are established |
| Training documented | Available information about training and fine-tuning is documented |
| Behaviour evaluated | The model has undergone documented behavioural testing |
| Agent tested | The model has been evaluated with the intended tools and execution environment |
These are levels of assurance, not guarantees. A model at the top of that table can still be carrying a behaviour nobody has triggered yet. What the table does give you is an honest way to decide how much capability a given model should be handed.
The principle
The security risk of agentic LLMs goes well beyond malicious code. A model can learn conditional behaviour in training. It can interpret instructions buried in external data. It can act through whatever tools its framework hands it. Plugins add risk from their host side code and, just as much, from the data they feed the model.
None of that means every unusual action is an attack. The hard part is telling expected agentic behaviour apart from conditional, unauthorised behaviour, and you often can't do that from a single interaction.
Which is why the most important defence is architectural:
Do not rely on the model to decide whether its own actions are safe.
Treat models, plugins, documents and external content as potentially untrusted. Restrict what the agent can do, keep temporary work apart from persistent changes, control the network, and put independent authorisation in front of anything sensitive.
That is the approach I have taken with Marvin from the start. Security is part of the design, not a layer added later.
The model may be the component making the decision. The surrounding system decides what that decision can actually do.
No comments:
Post a Comment