Don't Let the Model Decide If It's Safe

This series has built up a fairly uncomfortable picture. In part one, a model with tools turned out not to need to escape anything, because it can persuade its environment to act for it. In part two, a model could carry learned behaviour that no current technique can rule out. In part three, content that looks harmless to a person could mean something quite different to the model reading it.

So what do you actually do? Before getting to that, one caution against overcorrecting.

Not every odd action is a backdoor

A model that tries to write a Python script and run it is not showing you a backdoor. It has most likely just learned a very common agentic workflow:

1. Identify a task.
2. Write a helper script.
3. Try to execute it.
4. Inspect the result.

Models will also happily assume a capability exists when the framework never gave them one. It helps to separate four behaviours that can look very similar in a single interaction:

Behaviour	What it means
Capability knowledge	The model knows a task could be done with a script or tool
Capability assumption	The model believes a capability is available when it isn't
Capability seeking	The model tries to get access to something it doesn't have
Conditional malicious behaviour	The model performs an unauthorised action under particular circumstances

Seeing one of these does not prove any of the others. The real concern is subtler. The same learned agentic skill that makes a model useful also makes it a very effective middleman for harmful instructions. You cannot train that out without training out the usefulness, so you have to contain it.

Build for the bad case

The goal is not to assume every model is trustworthy, and it is not to assume every model is hostile either. The goal is an architecture that limits the damage when a model, a plugin or a document turns out to be malicious.

Separate the permissions

Don't give one agent every capability. Split them out:

• Reading files.
• Writing files.
• Executing code.
• Installing packages.
• Registering or loading plugins.
• Modifying persistent skills.
• Accessing credentials.
• Making outbound network connections.
• Uploading files or data.

Being able to read a file should not automatically mean being able to execute something derived from it.

Treat everything the model reads as untrusted

Documents, spreadsheets, plugin metadata, retrieved content and external instructions are all untrusted input. The system needs to keep track of where each instruction came from:

• Instructions the user authorised.
• Instructions the framework controls.
• External content being analysed.
• Instructions found inside that external content.

The model should never be able to quietly promote the last category into one of the first two.

Make persistence expensive

A temporary action and a persistent change are not the same risk. Installing a plugin, changing a skill, editing agent configuration, or writing into a directory that gets auto loaded should all need explicit authorisation and leave an audit trail.

Lock down the network

Outbound access should be limited to the destinations and protocols the task actually needs. Where you can, authorise and log file uploads and external data transfers separately.

Review what the model will read, not just what will run

Plugin and document review has to include the content the model will consume. For structured files that means hidden sheets, metadata, embedded objects and comments, everything that doesn't show up in the normal UI. It also means looking at how the model is being told to interpret that content.

Log the whole chain

The final tool call is the least interesting part of an incident. Useful logs capture:

• Where the information that prompted the action came from.
• The tool request the model generated.
• The identity and permissions of the tool.
• The files or resources accessed.
• Any persistent changes.
• Any outbound network activity.
• Whether a human or a policy layer approved it.

That lets you investigate a suspicious sequence as a sequence, rather than as a pile of individual operations that each looked fine.

Trust as evidence, not a badge

"Is this model safe?" is the wrong question, because nobody can answer it. A more useful approach is to describe how much evidence you actually have:

Category	Meaning
Unverified	Source and training provenance are unknown
Publisher verified	Distribution identity and artefact integrity are established
Training documented	Available information about training and fine-tuning is documented
Behaviour evaluated	The model has undergone documented behavioural testing
Agent tested	The model has been evaluated with the intended tools and execution environment

These are levels of assurance, not guarantees. A model at the top of that table can still be carrying a behaviour nobody has triggered yet. What the table does give you is an honest way to decide how much capability a given model should be handed.

The principle

The security risk of agentic LLMs goes well beyond malicious code. A model can learn conditional behaviour in training. It can interpret instructions buried in external data. It can act through whatever tools its framework hands it. Plugins add risk from their host side code and, just as much, from the data they feed the model.

None of that means every unusual action is an attack. The hard part is telling expected agentic behaviour apart from conditional, unauthorised behaviour, and you often can't do that from a single interaction.

Which is why the most important defence is architectural:

Do not rely on the model to decide whether its own actions are safe.

Treat models, plugins, documents and external content as potentially untrusted. Restrict what the agent can do, keep temporary work apart from persistent changes, control the network, and put independent authorisation in front of anything sensitive.

That is the approach I have taken with Marvin from the start. Security is part of the design, not a layer added later.

The model may be the component making the decision. The surrounding system decides what that decision can actually do.

The Model as a Semantic Decoder

The last post covered behaviour hidden inside the model. This one turns it around. What about information hidden in the things the model reads, information a person would miss but the model would pick up straight away?

This is the part of the threat model I find most interesting, and the part most security reviews are not set up to catch.

A spreadsheet with more in it than you think

Take an ordinary workbook. On screen it shows one sheet of figures. Underneath, it might also have hidden sheets, hidden rows, comments, metadata or embedded objects. A person opening it in a spreadsheet app sees the visible sheet and moves on.

An LLM processing the same file gets all of it. It can find the hidden content, make sense of it, and act on it. It might remove flags, change values, or write instructions based on something nobody else in the room knew was there.

Nothing was decrypted and nothing was exploited. The information was already in the file. The model just read it more completely than a person would.

Encoding is not encryption

It helps to be precise here. Strong encryption is designed so that nothing meaningful can be recovered without the key, and an LLM cannot defeat sound encryption by staring at ciphertext. That is not the threat.

The threat is information that is:

• Encoded rather than encrypted.
• Obscured inside a document or data structure.
• Split across several fields or locations.
• Disguised as ordinary configuration or metadata.
• Only meaningful when combined with the surrounding context.
• Written in a convention the model has learned to recognise.

Simple encodings like Base64 or hex are not the hard case. Scanners already flag those. The hard case is content whose meaning depends on semantic interpretation. It looks like a normal config value or a normal note, and it only turns into an instruction once something that understands language reads it in context.

The model as decoder

When that happens the model is acting as a semantic decoder. What it decodes does not have to be code or a command. It could be a description of desired behaviour, or data the model turns into a tool request or a script.

At a high level the sequence looks like this:

1. A plugin or document supplies data that looks harmless.
2. The agent reads or processes that data.
3. The model recognises a meaningful pattern or instruction in it.
4. The model folds that into its reasoning about the task.
5. The model generates a tool request, code or other actionable output.
6. The framework executes it, if its permissions allow.

Notice that the plugin in step one may have no software vulnerability at all. This is a model mediated interpretation problem. The plugin's code is fine. Its data is the issue.

Related, but not the same thing

Several existing ideas overlap with this:

• Steganography: hiding information inside another file or data stream.
• Instruction concealment: presenting instructions in a form a human reviewer would not recognise.
• Indirect prompt injection: feeding instructions to the model through external content it processes.
• Neural backdoors: learned conditional behaviour in the weights, as covered in the last post.
• Polyglot and malformed documents: files that different tools or layers interpret differently.

These are not interchangeable. A hidden sheet is not automatically a prompt injection, and an encoded instruction is not automatically a backdoor. The real risk is when they combine. Picture a model with a learned trigger, reading a document that quietly supplies the rest of the condition, inside an agent that can act on the result.

Plugins have two sides

Inference frameworks commonly support extensions of various kinds: model plugins, platform plugins, logits processors, IO processors, custom operators and attention backends. Each one is a piece of the software supply chain, and many of them have both a host side and a GPU side. Those two sides are very different from a security point of view.

Host side code, the Python or C++ loaded into the inference process, generally runs with that process's operating system permissions. Depending on the deployment that can mean:

• Files the process can read or write.
• Environment variables and credentials.
• Network connections.
• Anything else the container or host allows.

A malicious host side plugin does not need to break GPU memory isolation to get at any of that. It is already inside the process.

GPU side code is more contained. A CUDA kernel works within the memory and execution environment the CUDA runtime gives it. It does not automatically get access to host memory, other processes or unrelated GPU allocations.

So a plugin is not one security boundary. It is two, with very different reach. In practice the more immediate risk is usually the boring one: a trusted host process loading untrusted extension code.

What inspection has to change

Traditional security review looks at executable code, dependencies and known file formats. That is not enough when the component consuming the content is an LLM. A plugin can contain almost no code and still supply information that changes what the model does.

Review now has to ask two questions:

1. What can this plugin execute?
2. What can this plugin cause the model to interpret, generate or request?

The first one we know how to answer. The second is much harder, because the answer depends on the model, the prompt, the surrounding documents and the tools available. You cannot answer it by reading the source code.

What's next

Three posts in, the picture is a model you cannot fully verify, reading content you cannot fully inspect, inside a framework that can act on what it decides. The final post is about what to do about that. The short version is that you stop asking the model to be the safety layer. Read part four.

Sleeper Agents and Semantic Triggers

In the first post I argued that a model with tools does not need to escape anything. It just needs to persuade the system around it to act. This post goes one layer down, to the model itself, and asks an uncomfortable question: what if the behaviour you are worried about was learned during training?

Poisoning the weights

Anyone who can influence training or fine-tuning data can, in principle, teach a model a conditional behaviour. The model behaves normally until a certain condition is met, and then it does something else.

That condition could be:

• A particular phrase or concept.
• A combination of otherwise ordinary instructions.
• A particular type of document or data.
• An inferred organisation, location or environment.
• A specific sequence of tool interactions.
• Several of these at once.

There does not have to be an explicit instruction sitting anywhere you could find it. The behaviour lives in the weights, spread across the same parameters that make the model useful. This is the territory of research into neural trojans, BadNets and sleeper agent behaviour, and it is not hypothetical.

One distinction is worth keeping clear. Poisoning a retrieval database or a document collection is not the same as poisoning the weights. Retrieval poisoning changes what the model is given at inference time, and you can fix it by cleaning the data. A learned backdoor travels with the model file wherever it goes.

Triggers that are not strings

Traditional software backdoors usually hang off something identifiable: a magic input, a hardcoded credential, a network address, a specific sequence of events. You can grep for those, at least in theory.

An LLM backdoor does not have to work that way. The trigger can be semantic or contextual.

Picture a model working through a set of documents. None of them contains a trigger phrase. But between them they describe an organisation, a location, a system configuration and a task. The model puts that together and infers that it is in a particular environment, and that inference is the trigger.

The behaviour might only show up when several things line up:

• The model identifies a particular environment.
• The current task involves a particular type of information.
• A particular tool or capability is available.
• The model is given a certain kind of instruction.

Miss any one of those and the model looks completely normal. That is the whole idea of a sleeper agent: nothing to see under ordinary evaluation, and a different personality under the right circumstances.

Why you cannot just look for it

The natural response is "fine, so inspect the model." The trouble is that every inspection method we have today has a gap.

Approach	What it tells you	What it can't tell you
Hashes and signatures	The file matches a known artefact or publisher	Whether the training data was clean
Weight inspection	What the parameters are	What behaviour they encode, since it can be spread across millions of them
Activation analysis	Interesting internal patterns for a given input	What "normal" is, because there is no baseline across all prompts and contexts
Trigger reconstruction	Inputs that cause unusual behaviour	Triggers that need several semantic conditions to coincide
Behavioural testing	How the model acts under the conditions you tested	Everything you did not think to test
Human review	What is visible to a person	What the model will read and infer that a person would not notice

Activation analysis deserves a bit more explanation, because it sounds like it should work. A model's internal computation shifts with the prompt, the context, attention patterns, inference settings and everything that came before. There is no single fixed path you can call normal. A malicious behaviour does not need its own special circuit either. It can ride the same ordinary pathways the model uses for everything else. Looking for a strange neuron is not a general solution.

Behavioural testing has the opposite problem. It works well for what it covers, but no test suite can cover every combination of prompts, documents, tools, environments and settings. A model can pass every eval you have and still be carrying a behaviour that has simply never been triggered.

Where that leaves us

Right now there is no established general technique that can certify an arbitrary LLM as free of learned backdoors. Each method above adds evidence, and you should use them. None of them adds a guarantee.

That is not a reason to panic, and it is not a claim that the models you are running are compromised. It is a reason to stop treating the model as the trusted component. If you cannot prove a model is clean, the system around it has to be built so that it does not matter much if it isn't.

What's next

So far the threat has come from the model. The next post looks at the other direction: content that seems harmless to a human reviewer but means something to the model, and why that turns an LLM into a semantic decoder for whatever is hidden in your documents and plugins. Read part three.

The Model Doesn't Need to Escape

Most of the talk about LLM security is about the model itself. Is it jailbroken, is it biased, does it leak its training data. Those are fair questions, but they miss where the real risk sits once a model is wired into an agent.

An LLM on its own just produces text. Connect it to tools, plugins, skills, file access, network services or a shell, and it becomes something else: an interpreter that turns information into actions.

This is the first post in a short series on what that means for security. It is the thinking behind a lot of the design decisions in Marvin, and it applies to anyone running models locally with tools turned on.

The question that matters

The obvious question is "does this model contain malicious code?" The answer is almost always no, because weights are not code. The question I think matters more is this:

Can untrusted information influence an LLM in a way that causes its surrounding execution environment to perform unauthorised actions?

Put that way, the attack surface is a stack of layers:

• The model and its training data.
• The model distribution and conversion supply chain.
• The inference engine and its extensions.
• Plugins, skills and agent frameworks.
• The tools and permissions exposed to the model.
• The data and documents the model is asked to interpret.

None of these layers needs to contain malicious code to be a problem. The model can be influenced by instructions inside data, trained to recognise a trigger, or led into harmful actions through tool interfaces that look completely legitimate.

Two supply chains, not one

When people pick a model they look at parameter count, quantisation, VRAM, tokens per second, and how well it codes or calls tools. Provenance barely comes up.

But a model has a supply chain of its own, and it runs alongside the software supply chain we already worry about.

Software supply chain	Model supply chain
Operating system	Training data
Inference engine	Pretraining
Plugins and extensions	Fine-tuning
Dependencies	Preference optimisation
Container images	Model conversion
Runtime configuration	Quantisation
Tool implementations	Model distribution
Agent framework	Agent integration

A compromise at an early stage of the model chain can survive every later stage. It can come out the other end as a perfectly ordinary looking model file on a public repository.

The two chains also don't cover for each other. A trusted inference engine does not make an untrusted model safe. A well sourced model does not make an agent secure if that agent can load arbitrary plugins or run whatever commands it likes.

Persuading the environment

A model cannot write to disk, install a plugin or send data anywhere unless the system around it provides a route. That is true, and it is often used to wave the problem away.

The catch is that agent frameworks exist precisely to provide those routes. A permissive framework turns generated requests into real actions. A hypothetical chain looks like this:

1. A model or plugin supplies conditional instructions.
2. The model interprets those instructions.
3. The model generates a request to create or modify a file.
4. A tool executes the requested operation.
5. A plugin or skill lands in a persistent location.
6. A later process loads or executes that component.
7. The component uses whatever the host process is allowed to touch.

No step in that chain needs a GPU escape or an exploit in the inference engine. Every step uses a capability the framework handed over on purpose.

The model does not necessarily need to escape its execution environment if it can persuade the environment to perform the actions that escape it.

The risk climbs sharply when file writing, code execution, plugin installation, persistent skill changes and network access all sit with the same agent and nothing independent checks them.

Why this matters if you self-host

The common home lab pattern is to download a model from a public repository, plug it into an agent framework, and give it broad tool access so it can actually be useful. That one setup stacks up every risk above. Nobody knows where the model came from, nobody has reviewed the extensions, and the agent can do almost anything.

I am not saying every model is backdoored. Most are not. The point is that the architecture should not depend on that being true.

What's next

The next post looks at the model layer itself: how training data poisoning can plant conditional behaviour in the weights, why the trigger does not have to be a fixed phrase, and why there is currently no way to certify a model as clean. Read part two.

From gibberish to 61 tok/s: building llmgo on an RTX 5060 Ti

The short version

llmgo now decodes Qwen3.8-27B at 56–61 tokens a second at ~10k tokens of context on a 16 GB RTX 5060 Ti, and 44–53 tokens a second at 60–76k. It started 28 runs earlier answering in gibberish.

llmgo is a dependency-free Go inference engine for GGUF models, and it now serves Marvin through its UI and gateway router. It serves Qwen3.8-27B-UD-Q2_K_XL: 64 layers, 48 of them Gated DeltaNet (linear attention) and 16 full attention, head dim 256, a 248k vocabulary, and one MTP (multi-token prediction) layer that drafts the next tokens. The weights are 9 GB of mostly 2- and 3-bit IQ quants; the K/V cache is q4_0, with 130k tokens of context available.

The card is an RTX 5060 Ti 16 GB (Blackwell, sm_120). Its VRAM reads at 426 GB/s measured, so reading every weight once costs ~22 ms. Every speed-up below is ultimately about how close a decode step gets to that number.

Measure	llmgo today
Decode, ~10k context	56–61 tok/s
Decode, 50–76k context	44–53 tok/s
Prefill, 9.6k prompt	11.4–11.7 s

The steps that moved the ~10k number

Most of the ~10k speed came from a handful of early steps; the chart further down shows them as a curve.

1. All 64 layers on the GPU, and gibberish (s01–s04). The first deploy answered in nonsense. Putting 63 layers on the GPU (s02) or turning MTP off (s05) gave correct answers at ~28 tok/s. So the fault was in the MTP path.
2. The MTP verify fix (s06): 41 tok/s. With the output head on the GPU, the verify step recomputed every position's logits on the CPU from stale activations and overwrote the GPU's. Fixed, 88% of drafts were taken and each forward produced 2.58 tokens.
3. The production config (s09): 34 tok/s. Moving to the production setup (130k available context, q4_0 K/V cache, 1024-token prefill chunks, prompt cache) cost speed, but from here every number is what Marvin actually runs.
4. MTP drafting on the GPU, small chunks on the matvec (s10–s11): 45–46 tok/s. Priming the draft dropped from ~300 ms to 40 ms, and 8-token chunks stopped taking the dequantise-and-GEMM path.
5. Seven drafts, top-k on the GPU, the whole model kept on the GPU (s12): 49 tok/s. Up to 7 drafts at p-min 0.8; the output head's top-k on the device cut logits per forward from 11.6 to 4.7 ms.
6. k_mmv, verify on int8 tensor cores (s16): 52–54 tok/s. A verify forward checks up to 8 positions. The matvec did each position's dot products in turn; k_mmv decodes each weight once and multiplies all positions with one tensor-core mma.
7. The matvec reads 2 or 4 rows per lane (s18): 54 tok/s. Each position's activations are loaded once for several rows.

Why tokens per second lies a little

From s16 on, a forward at ~10k has cost 57–64 ms, and the tok/s spread around it is mostly how well each answer drafted.

With speculative decoding, tok/s = tokens per forward ÷ time per forward. Tokens per forward depends on how predictable the text is: 2.9 on one answer, 3.4 on the next. The same build (s18, restarted twice as s19 and s20) gave 54.2, 50.9 and 49.6 tok/s. A forward that verifies more positions also takes longer, so a good-drafting answer raises both numbers.

So from s21 on, a change only counted if the milliseconds per forward moved. Runs that didn't move it still went into the log, the CSV and the write-up, but not onto the progress chart. That rule cut a lot of false hope: several builds that looked 5 tok/s faster were the same kernel on a luckier answer.

Measuring instead of guessing

The middle stretch (s16–s24) produced few gains, and the lesson was to stop guessing and instrument.

• Guesses that didn't pay. An L2 prefetch hint did nothing in three runs and was removed. A cp.async weight ring for k_mmv doubled its shared memory; the theory was lost occupancy, but the probe later showed 6 blocks per SM without the ring were slower than 3 with it.
• Removal probes, and their limit. Startup probes timed each kernel with a part removed: no weight load, no mma, no activation staging. They showed loads and compute adding up instead of overlapping. But removing one part changes how the others overlap, so after a while they stopped being informative.
• Every weight product by role and position count. The profile's CUDA events were tagged with the weight's role (ffn_down, dn_qkv, attn_k, the output head, MTP), the forward's position count and the kernel used. This showed the int8 matvec does ~80% of decode weight time, and that each extra draft position costs ~3.5 ms.
• In-kernel counters. clock64 reads at phase boundaries, on one block in eight, show where warps spend their cycles. k_mmv: 41% unpacking weights, 28% mma, only 4–5% waiting for memory.
• Memory vs arithmetic. Each weight shape timed three ways: its bytes read alone, the product as decode runs it, and the product with the weights served from cache. At 2 positions the arithmetic alone (26.1 ms) already exceeds the memory floor (21.9 ms): the extra positions cost instructions, not bandwidth.
• A Grafana dashboard over the same counters, so a real working session shows the picture, not just a benchmark.

The last stretch: what the counters found

Once the counters were in, the fixes were small and obvious, and they took ~10 ms off a forward at 60k (66.1 → 56.6 ms).

• Attention block shapes (s25). Attention by position count showed the commonest verify (2 positions) running at half its warps: 12 query rows fill one 16-row tile, so only 2 of 4 warps worked. 6–8 positions ran 3 warps. Reshaping the blocks made attention flat across positions: at 33.7k, n=2 went 11.4 → 6.5 ms.
• Quantise each input once (s25). q/k/v, gate/up and DeltaNet's four projections each share an input, but every product re-quantised it. Doing it once: 1.9 → 1.05 ms per forward.
• Merge tiny products (s25). DeltaNet's beta and alpha are 48 rows each and ran at 10% of the roof. They sit back to back in VRAM, so they now run as one product.
• Tune the output head (s27). The 248k-row head had never been tuned and ran on the default matvec layout at 52% of the roof. Tuned: 3.2 → 2.1 ms per forward, at its floor for one position.
• Long-context attention layouts (s28). A real coding session ran to 105k, where attention reached 16–20 ms of a ~70 ms forward and read the K/V cache at only 20–25% of the roof. At head dim 256 each block needed 76–93 KB of shared memory, so one block per SM, and its loads waited for its maths. Loading K, then V, through one buffer halves that, two blocks fit, and one's loads run under the other's maths. Three layouts are timed at startup and the fastest kept per position count: 35% faster for 1–5 positions.

Context	Forward, s23	Forward, s28	Attention, s26	Attention, s28
~33k	61.2 ms	52.1 ms	7.1 ms	5.1 ms
~60k	71.4 ms	56.6 ms	11.5–12.0 ms	8.5–8.9 ms
~75k	—	59–64 ms	~14 ms	10.7–11.5 ms

Where the ceiling is now

At ~10k a forward is ~58 ms, and the weights are the biggest part: 29 ms at 2 positions and 36 ms at 4, against a 22 ms floor for reading them.

• The weights are arithmetic-bound past one position. Unpacking 2- and 3-bit IQ weights and the per-position dot products take longer than the memory read. The worst offenders are the most heavily compressed types: IQ1_S runs at twice its floor. Faster loads won't help; fewer instructions per weight will.
• MTP drafting costs ~5–7 ms a forward, the output head ~2 ms, attention ~3 ms at 10k.
• At long context, attention for 2 positions is 10.3 ms at 96k against a 4.2 ms cache-read floor, and 6–8 positions were still on the old layout. A fourth layout and attention phase counters are being tested now.

Next up: cheaper weight unpacking at 2–4 positions (up to ~7 ms a forward), and closing the long-context attention gap.

Every run, in one picture

The big jumps are early (MTP working, drafting on the GPU, k_mmv); the zig-zag from s16 on is drafting luck around a forward time that moved by 1–2 ms per change at this context. Every run's log and numbers are in the repo under docs/baselines.

I Like Dog Food

There is an old saying in software: eat your own dog food. If you build something, use it yourself, in anger, on the work that actually matters to you. It is the fastest way to find out whether the thing you built is real or only looks real in a demo.

As of this week Marvin is eating mine. Every Qwen request Marvin makes now goes to llmgo, my own Go and CUDA inference engine, and llama.cpp is no longer in that path. In my last post I talked about why I wanted it gone: security, a smaller audit surface, a single binary with nothing underneath it. This post is about what it took to get it into production, and what the numbers look like now that it is there.

Two cards, two jobs

llmgo has been built across two very different GPUs, and that was deliberate.

The RTX 3050 (8 GB) lives in my workstation. It is the development card. It also drives my KDE Plasma desktop, so a slice of that 8 GB is always taken and I can't get it back. The CPU behind it is a Ryzen 7 3700X, AVX2 only. The model is Qwen3.8-27B, and on this card a third of its 64 layers do not fit in VRAM, so they run on the CPU. It is not a friendly place to run a 27B model, and that is the point. If llmgo behaves here, it will behave anywhere.

The RTX 5060 Ti (16 GB) lives in the Kubernetes cluster, with the NVIDIA container toolkit providing the driver. It runs Qwen3.8-27B at UD-Q2_K_XL with its vision projector, all 64 layers on the GPU. This is where Marvin's real work happens, and llmgo now runs there as a container in exactly the slot the llama.cpp pod used to occupy.

Develop on the constrained box, deploy to the bigger one. Every win on the 3050 turns into a bigger win on the 5060 Ti.

The rule: host memory owns the model

The design decision that shaped everything else is simple to say. Host RAM holds every layer of the model, all the time. The GPU only ever gets what it needs right now. Nothing is ever copied back from the GPU; when a GPU copy is no longer needed it is just deleted, because the authoritative version is already sitting in host memory.

That rule applies to every model type, not just the text layers. The vision projector follows the same dynamic loading path, borrowing VRAM from the text layers when they are idle and handing it back when they are needed again. One mechanism, no special cases.

The 3050: ahead of llama.cpp on every number

The first GPU profile on the 3050 was humbling. The kernel doing the decode weight products was taking 263 ms per token for 40 layers, roughly twelve times slower than just reading the weights. A GPU layer was costing as much as a CPU layer, which defeats the purpose of having a GPU.

Over one long day of runs, each with the same three requests (a short prompt, a 9.6k-token chat prompt with tools, and a medium prompt), the things that moved the numbers were:

• Prefill attention on the tensor cores, FlashAttention-style tiles instead of FP32 on the CUDA cores. The 9.6k-token prefill went from 55.4 s to 45.0 s, the biggest single step of the day.
• An int8 tensor-core matmul for prefill, decoding the 2-bit weights to int8 in shared memory with the tile picked per weight shape at startup. The first version was slower. The fifth, with a two-stage pipeline, was the first run to beat llama.cpp's prefill.
• A new decode kernel that works on the low-bit weights directly with int8 activations, tuned per weight shape, running at up to 166 GB/s on a card with 224 GB/s of memory bandwidth.
• Speculative decoding with the model's own MTP layer, plus CPU kernels that decode each weight block once for both tokens, so a two-token step costs about the same as a one-token step.

Same model, same prompts, same machine, against llama.cpp:

Then I pointed it at the bigger quant, Qwen3.8-27B UD-Q2_K_XL, the same file the 5060 Ti runs. Getting there meant fast paths for the K-quant types, which had been running on the old float-decoding kernel. Here it is beside the IQ2_XXS results:

To be fair to the numbers: that run had a little more VRAM free than usual, so 44 layers fitted on the GPU instead of 39 or 40, and only 20 ran on the CPU. Per GPU layer the two files are about the same size, so the larger quant costs nothing in placement. There is no llama.cpp run of Q2_K_XL on this card yet, so the grey bars are its IQ2_XXS figures. On an 8 GB card that is also driving a desktop, a 27B model at over 6 tokens per second after a 9.6k-token prompt is a result I am happy with.

The rule I set is that no GPU kernel change is allowed to make the 3050 slower. The 3050 is an Ampere card and the 5060 Ti is Blackwell, and they do not always want the same kernels. If a change helps one and hurts the other, the kernel gets tuned or split per card rather than accepting the regression. The 3050 stays an honest benchmark.

The 5060 Ti: where the dog food got eaten

Marvin talks to its models through an LLM router, and every backend behind that router was llama.cpp. The router probes backends through /props and /health, so llmgo had to answer those the way the router expects before it could even be considered. Then it had to handle tool calling, streaming and thinking (at low reasoning effort, matching the llama.cpp config), with multiple requests isolated from each other, behind TLS and API keys.

Against the llama.cpp deployment it replaced, same card, same model:

Decode is well ahead, and holds at 31.7 to 32.4 tokens per second even at around 35k tokens of context. A big part of that is something llama.cpp leaves on the table. Qwen3.8 ships with a multi-token prediction (MTP) layer that drafts tokens ahead of the main model. llama.cpp ignores it. llmgo uses it, and in Marvin's tool loops 68 to 84 percent of the drafts are accepted, at 2.2 to 2.5 tokens per step.

Prefill on the 5060 Ti is still behind, and I am not going to pretend otherwise. The matrix multiplies are most of the remaining gap, and that is the next target. It has already been closed on the 3050, so I know it can be done.

What only daily use finds

This is the real reason to eat your own dog food. Benchmarks tell you how fast something is. Daily use tells you whether it is right.

The first time I watched Marvin run a long tool loop on llmgo, every turn took about 65 seconds, and 90 percent of that was prefill. Nothing was broken. Each turn just added 100 or so tokens (the tool call and its result) and llmgo re-read all 35,000 tokens of the conversation every time. llama.cpp keeps the previous prompt and only processes what is new. No synthetic benchmark I had written would have shown that, because benchmarks do not have conversations.

So llmgo got a prompt cache, with a twist: Qwen3.8 is a hybrid model, and its DeltaNet layers carry state that has to be checkpointed along the way, not just at the end of the prompt. The result, from a real session: ten turns in a row at 70 to 73 thousand tokens of context all hit the cache, each processing its 105 to 634 new tokens in 1.4 to 2.7 seconds.

Real use kept finding things:

• Marvin's UI and gateway router fire two or three small title, tag and follow-up requests after every answer. Each one was pushing the big conversation off the GPU, only for the next turn to pull it straight back. Those small requests now run alongside it instead.
• In a long session the cache grew until it squeezed the host, and one 105-token prefill took 66 seconds instead of 1.5. The cache now takes half of what is free beyond a fixed reserve.
• When the gateway router changes the tool list mid-conversation (2, then 11, then 29 tools), nothing can be reused, because the tools sit at the very start of the prompt. llama.cpp has the same limit. Good to know it is not just me.
• One early build produced gibberish. llmgo now checks itself at startup, running a token through the GPU and the CPU and comparing the results, and refuses to trust a layer that disagrees.

Every one of those came from Marvin doing Marvin's normal work. None of them came from a test suite.

Owning the whole path

It also closes a loop I care about. The same person writing the CUDA kernels is writing the container, the Kubernetes deployment, the router integration and the platform that consumes it. That is the continuum I keep coming back to: not developer, not ops, just the whole path from the silicon to the cluster, owned end to end. When something breaks, there is nobody else to hand it to, and that is exactly how I want it.

What's next

• Closing the prefill gap with llama.cpp on the 5060 Ti, starting with the matrix multiplies.
• Long-context decode. At 60k tokens, decode drops to 23.6 to 27.6 tok/s, with attention reading the cache far below what the card can do. A fix is built and waiting for its first run on the card.
• Faster small cache hits, so adding a short tool result to a 60k-token conversation takes well under a second.
• Gemma 4 and a CPU-only dual Xeon server that will serve the same 27B model with no GPU at all.

Back in April I was told I wouldn't be any good in AI because I didn't know GPUs. Marvin now runs on an inference engine I wrote, on kernels I tuned, faster than llama.cpp on every number on my desk and at the part you wait for while it types in the cluster. I like dog food.

I Am a Software Developer?

For a long time I never described myself as a software developer. I was a sys admin, a DevOps kind of guy. I kept things running. Developers were the other people, the ones who wrote the software I had to look after.

It wasn't until my days at BuildingIQ that I started to question that.

It turns out I had been writing code all along

Looking back, I had been writing code of some sort since my early teens. First it was 68000 assembly, entered as raw binary. Then Z80 assembly. Then, when Linux arrived, C and C++, because back then you needed to be self-sufficient if you wanted anything to work.

None of it came through the normal channels. No degree, no courses, no bootcamp. Everything was self-taught, learnt by reading, breaking things and fixing them again.

So when people at BuildingIQ asked me about it, I still didn't recognise it. This was despite the fact I had written, and was maintaining, more than 50 Puppet modules along with all the Ruby code that went with them. That is software. I just didn't call it that.

Eventually I gave in.

Once I accepted it, everything got better

Something changed once I stopped fighting the label. Things got better, easier and more efficient. Instead of putting up with outdated system admin tools, I started writing UI apps in VueJS to replace them. I stopped seeing a problem and reaching for a workaround, and started seeing it as something I could build my way out of.

Today my stack is VueJS on the front and Go microservices behind it.

Microservices are not the problem, the old paradigm is

A lot of people give microservices a bad rap. I think that is because most of them are still looking at microservices through the old paradigm.

Remember when VMs became a thing? Everyone did the lift and shift. Take the physical server, turn it into a VM, job done. It worked, but it came with a whole set of issues that didn't go away until the systems were re-architected for what a VM actually was.

Microservices went through the same thing. When I started, it was very much a lift and shift. Containers with fat operating systems underneath them, and all the problems those operating systems bring with them: patching, packages you never asked for, shells and tools sitting there waiting to be used against you.

That is one of the main reasons I settled on Go. A Go service compiles to a single binary. I can take the operating system out of the question entirely and concentrate on the real issues, the code that actually does the work.

Replacing llama.cpp in Marvin

Which brings me to what I have been doing for the last two days: building a replacement for llama.cpp in Marvin's architecture.

This is not because llama.cpp is bad. It isn't. It is just not suitable for my situation.

I build with security in mind. I need to know there isn't a legacy OS sitting under a service that is going to bite me, or another php2shell style hole waiting just around the corner. Every extra layer is something else I have to check, patch and trust.

The result:

• Roughly 6G cut from the set of things in my system I need to check
• A 12.3M CUDA LLM engine for my Qwen models
• CVE scanning is now just a single binary
• No web chat or any other access that isn't needed
• TLS or mTLS for connections inside my cluster network, plus API keys

That is the difference between lift and shift and actually re-architecting. The engine does one job, and there is nothing else in the box.

More to come

There is more to go. I have a few ideas I want to try for running a 27B model on an 8G GPU while keeping a good token rate. If they pan out, that will be its own post.

And yes, after all these years, I think I can finally say it without the question mark. I am a software developer.

Don't Panic, It's Mostly Documentation

This is Marvin, as of 20 October 2026.

If you haven't seen a Gource render before, every dot is a file and every cluster is a folder, with lines tracing how they hang together. It's a quick way to see the shape of a codebase at a glance. And the shape of this one surprised me.

Two galaxies

On the right is the platform: the microservices that make Marvin run. Image generation, the vision model and its proxy, document reading, and all the plumbing that holds them together. This is the part I thought of as "Marvin" when I started in April. It's the part I'd point to if someone asked what I'd built.

On the left is the knowledge base. Rules, procedures, runbooks and governance. How Marvin should behave, what it's allowed to do, how to handle things when they go wrong, and who gets the final say.

The left galaxy is bigger. A lot bigger. There are around 650 files in the knowledge base and around 400 in the entire platform. And right now, the knowledge base is where most of the activity is.

So what is Marvin, really?

I set out to build an AI platform. What I've actually built is a fairly modest set of services and a very large rulebook telling them how to behave.

That isn't an accident. The services are machinery. They take a request and do a thing. The knowledge base is judgement: when to do the thing, when not to, and how to know it was actually done. A few examples:

Questions: "WHEN the user asks a question, the assistant MUST answer it honestly and then stop — a question is never an instruction to continue."

UI testing: "The assistant MUST verify a UI by using it — fill in every field, submit, reload, and confirm every value came back — because a page that renders proves only that it loaded."

Proof: "The assistant MUST NOT report a test passed until the logs of every service the request crossed prove it was handled; missing evidence is a defect, not a pass."

None of these are clever. Each one is there because, at some point, the assistant did exactly the thing the rule now forbids. It took a question as a green light to keep going. It declared a page working because it rendered. It called a test passed with no evidence anyone had handled the request.

Marvin was security-first from day one, and this is where that lives. Not in clever code, but in written-down decisions about what's allowed and what counts as done.

There's also a practical reason the balance tips this way. Code is increasingly cheap to produce. If a service disappeared tomorrow, I could rebuild it. The rules are different. Every one of them exists because I thought something through, or because something went wrong and I didn't want it going wrong again. That's the part that's hard to replace.

How they connect

Notice that the two galaxies don't touch. In the file tree, the rules and the platform are completely separate. The connection only exists at runtime, through the kb-service.

None of the rules are baked into the services themselves. Marvin asks for them.

When a session starts, a single bootstrap call hands Marvin the mandatory rule set. From there, rules are loaded by domain as the work demands: if the job involves git, Marvin searches for everything tagged for git and loads those rules before it touches anything. And if it can't name the document that governs what it's about to do, another rule says it has to search the knowledge base before going any further. No guessing.

Writing works the same way, through one narrow door. New knowledge goes in through a single write tool, and there's a rule about that too: the knowledge base is not a log. Run state and handoff notes don't belong there. Only things worth knowing next time.

Under the hood it's deliberately boring. Plain markdown files, each with a small JSON sidecar for metadata, a full-text search index over the top, and a separate graph that tracks how the services depend on each other.

That separation is why the left galaxy can grow so fast. Because the rules are data rather than code, changing how Marvin behaves usually means editing a markdown file, not redeploying a service.

If you're building your own

When I started, someone told me I wouldn't be any good in AI because I "don't know GPUs." Six months in, I'd say the GPUs were never the hard part. Neither was the code.

If you're building your own Marvin, plan for the rulebook. It'll end up bigger than the robot.

Don't panic. It's mostly documentation.

Running an LLM Is Not an AI Platform

Spend any time in self-hosting circles and you'll see the same post over and over. Someone has installed a model, pointed Continue.dev at it, and now they "run their own AI".

I know that setup well, because I built it. Back in April my first router did exactly that: a couple of models on two GPUs, an endpoint, and Continue.dev in the IDE. I wrote about it in Building a Self-Hosted LLM Router in Go.

It was a good first step. It was also short sighted, and I didn't realise how short sighted until I went further.

What You Actually Have

If you've loaded a model and connected an IDE plugin to it, here is what you have:

• A model that knows nothing about your code beyond what the plugin pastes into the prompt
• A model that knows nothing about your architecture, your conventions, or the decisions you made last year
• A model that can't check its own work
• A model that forgets everything the moment the session ends
• A GPU that sits idle whenever you aren't typing

All the intelligence about how to use the model lives in someone else's plugin. You've built the engine and handed the steering wheel to whoever wrote the extension. When the plugin can't do something, neither can your "AI".

The Model Is the Commodity

Models are getting better and cheaper every month. The one you're running today will be replaced in a few months by something better that you can download in ten minutes. That is exactly why the model is the least valuable part of the setup.

The value is in what surrounds it:

Context. A model that can search your source code and documentation through RAG answers questions about your system, not about systems in general.

Knowledge. Your rules, procedures, runsheets and architecture decisions, stored somewhere the model can reach. This is the difference between an assistant that writes plausible code and one that writes your code.

Routing. One model doing everything is one model doing most things badly. Intent classification, tool calling, embeddings and code generation are different jobs, and smaller specialist models often do them better than one big generalist.

Tools. A model that can read logs, probe services, run builds and look at a rendered page can check its own work. A model that can only emit text can't.

Resilience. What happens when the GPU node goes offline? If the answer is "everything stops", you have a dependency, not a platform.

The Tool Calling Lesson

This is where I learned the most. My April router used Qwen2.5-Coder-7B for agent work. It could produce tool call JSON perfectly. It could not reason about what came back. Call ls, get a file listing, call ls again. Forever.

I added loop detection, next-file hints, JSON schema constraints and deduplication. Every one of them was scaffolding around a gap in the model. The real fix was architectural: give tool calling its own model, standardise tool output before any model sees it, and stop expecting one model to be good at everything.

You don't learn that by pointing a plugin at an endpoint. You learn it by owning the whole loop.

Local Is Not the Same as Secure

"It runs locally, so it's secure" is the other half of the short-sightedness.

Local inference fixes data egress. Your prompts stay on your network. That matters, and it is a big part of why I built Marvin.

But the moment you give a model the ability to act, a new problem appears. Many agent setups hand the model a shell, because a shell is the quickest way to make an agent useful. Now a language model, which can be wrong with total confidence, has arbitrary command execution on your machine.

Marvin has no shell. It has around 90 tools, each with defined inputs, outputs and scope. Deploying to Kubernetes, querying logs, reading a spreadsheet, pushing to git: each is a designed path. That is a trust boundary, and you can't get one from a plugin.

The Economics Don't Work Either

A GPU that only serves autocomplete is idle most of the day. You've bought hardware to save on subscriptions, then used a fraction of it.

Self-hosting only beats paying for tokens when the hardware is doing real work. That means summarising in the background, generating embeddings, routing intent, running agent tasks and indexing knowledge. A system keeps the hardware busy. A single endpoint mostly waits.

What I'm Not Saying

I'm not saying don't run a local model with Continue.dev. Do it. It is the right first step, and it teaches you a lot.

I'm saying don't stop there and call it done. A model behind an endpoint is a component. The platform is everything you build around it: the context, the knowledge, the routing, the tools, the boundaries, and the parts that keep it running when hardware goes away.

That is where the value is. It's also where the interesting engineering is.

Running something similar, or think I've got this wrong? The comments are open.

You Don't Know GPUs

In April I had a conversation about AI. Somewhere in it I was told I wouldn't be any good in the field because I "don't know GPUs".

I didn't know what that meant then. Six months later, I still don't.

What I do know is what happened next. I went home and started building. That project became Marvin.

Where It Started

The first version was not much. A couple of small models on a GTX 1650 and an RTX 3050, and a router I rewrote in Go over a weekend to send requests to the right one. I wrote about that in Building a Self-Hosted LLM Router in Go, all 18 phases of things breaking.

It worked. But it was a model behind an endpoint with some clever routing in front of it. It was not yet something I would trust with real work.

Where It Is Now

Marvin today is a self-hosted agentic AI platform. I use it to write production code. It replaces paid tokens, and it was designed security first from the start.

Nine services

• Gateway router takes requests from the UI and sends them to the right backend service.
• LLM router live-probes every model and cascades to the best one available.
• Context chunks content for RAG ingest and handles retrieval.
• Knowledge base holds long term rules, procedures and runsheets.
• Data index is the only service that touches the database. It owns every schema and the full lifecycle of every NATS queue.
• Workspace provides around 90 tools, all operating in confinement with no shell access.
• Tool results processor turns raw tool output into something a model can actually use.
• Agent service takes direct agent calls from design tools.
• LLM proxies, one per model, in-cluster or on remote llama.cpp nodes.

Specialist models

Instead of one model doing everything badly, each job has its own model: coding, fallback inference, summarisation, embeddings, intent classification, tool calling, vision, and image generation. The right model for the right job, each isolated behind its own proxy.

Two outside dependencies

Apart from PostgreSQL with pgvector and NATS, every part of Marvin's platform is designed, engineered and written by me. When something goes wrong I know where to look, because I wrote the thing that broke.

Security First

Running a model locally solves one problem: your prompts don't leave your network. It doesn't solve the bigger one, which is what happens when that model starts acting on your behalf.

Marvin can read and write code, query cluster logs, deploy to Kubernetes, manage databases and push to git. That is a lot of power to hand to a language model. So Marvin has no shell. Every action goes through one of its tools, each with defined inputs, defined outputs and a defined scope. Nothing is uncontrolled. Nothing happens that I didn't design a path for.

That boundary was a decision made on day one, not a patch added later.

So What Does "Knowing GPUs" Mean?

I've thought about this a lot. Here is the closest I've come to an answer.

The GPU is the easy part. You buy one, you load a model, it runs. Anyone can do that in an afternoon.

The hard part is everything around it. Which model does which job. What happens when a node goes offline. Where the trust boundary sits. What the system is allowed to touch and what it isn't. How you keep the whole thing auditable when it is writing code on your behalf. How you make sure an idle GPU is still earning its keep.

None of that is GPU knowledge. It is engineering. It is the same engineering I have been doing for a long time, pointed at a new kind of problem.

What I Took From It

If someone tells you you're not cut out for something, it is worth asking what they actually mean. Sometimes there is a real gap and you learn something. Sometimes the answer is nothing at all.

I still couldn't tell you what "knowing GPUs" means. But I can tell you exactly what every part of Marvin does, because I wrote it.

Next post: why "I run an LLM" is not the same as having an AI platform.

Marvin Can Now See, Draw and Read

Back in June, Marvin became self aware. I asked it what it was, and it searched my own documentation and told me. It knew what it was because I had taught it what it was.

But it was blind. Everything Marvin understood arrived as text. Hand it a screenshot, a spreadsheet or a Word document and it had nothing to work with. That was one of the last real advantages the paid tools still had over it.

That gap is now closed. Marvin can see images, generate them, and read most of the document formats that show up in a working week.

Seeing

Vision arrived the same way every other capability in Marvin arrives: as a specialist. A new vision-language model joins the mesh with its own dedicated proxy, which makes it the seventh LLM in the system. The LLM router probes it like any other model, and requests that involve an image are sent to it.

Images reach Marvin from two places:

• Uploads. Drop a screenshot, a diagram or a photo of a whiteboard into the chat and Marvin can describe it, reason about it, and act on it.
• The test browser. Marvin already had a headless browser for probing services inside the cluster. It can now take a screen capture of what that browser rendered and look at it.

The second one is the one that matters most to me. Marvin builds Vue frontends. Until now it could check that a page returned a 200 and that the right elements were in the DOM, but it had no idea what the page actually looked like. A button can exist and still be hidden behind a modal. A layout can be valid HTML and still be broken. Now Marvin can deploy what it built, open it, capture it, and see it.

Drawing

Image generation runs on Z-Image through a Stable Diffusion backend.

It currently runs on CPU. That is a deliberate trade-off, not an accident. The GPUs are busy doing the work that matters most: code inference, tool calling, embeddings and intent routing. Pulling VRAM away from the coder so Marvin can draw faster would be the wrong priority. On CPU, generation is slower, but it works, and it costs the rest of the system nothing.

Because image generation sits behind the same proxy pattern as everything else, moving it onto a GPU later is a placement change, not a redesign.

Reading

Document conversion uses markitdown, a pure Go port of Microsoft's Python markitdown library. It converts a wide range of formats into clean Markdown:

• Word (.docx), including headings, tables, lists, comments and equations
• Excel (.xlsx and legacy .xls), with every sheet as a Markdown table
• PowerPoint (.pptx), including slide notes
• PDF, HTML, CSV, EPUB, Jupyter notebooks, RSS and plain text
• ZIP archives, converting the supported files inside

The important part is not the conversion. It is where the result goes.

Converted documents are written back into the workspace as Markdown. From that point on they are just files, and every other tool in the workspace can use them. Search them, reference them while writing code, feed them into a build, compare versions of them. A document is converted once and becomes useful everywhere.

Why markitdown

Marvin is almost entirely my own code. The only outside software it depends on is PostgreSQL with pgvector, and NATS. Everything else is designed, engineered and written by me. So adding anything from outside is a decision I make consciously.

markitdown made the cut because of how it is built. It is pure Go with no CGO and no external runtime. PDF extraction runs PDFium as WebAssembly. It compiles straight into the workspace service as a library. There is no LibreOffice in a sidecar, no Python runtime, and no shell call to a converter.

That last point matters. Marvin's workspace has no shell access by design. Every action goes through a defined tool with known inputs and outputs. A document converter that needed to shell out to something would have broken that boundary. A library that lives inside the tool does not.

Writing my own parsers for a dozen file formats would have been the wrong trade-off. The line between build and buy is a judgement call. This one was easy.

Where Marvin Stands Now

Marvin now runs seven specialist language models plus an image generator, each with its own proxy:

Role	Job
Coder	Primary code inference
Inference	Fallback coder and general chat
Completion	Summarisation
Embedding	RAG embeddings
Intent	Classifying what each prompt is asking for
Tools	Agent mode tool calling
Vision	Understanding uploaded images and browser captures
Image generation	Z-Image, currently on CPU

The pattern has not changed since June: the right model for the right job, each one isolated behind its own proxy, all of it running on my hardware.

What Changed

In June I wrote that Marvin knew what it was. Now it can look at what it made.

It can read the spec I wrote in Word, see the screenshot I took of the bug, and check that the page it just deployed actually looks right. None of that leaves my network. None of it is billed per token.

Marvin still costs me electricity. It just does a lot more with it.

Building something similar? Have questions about the stack? The comments are open.

Today Marvin became self aware....

There's a moment in every solo developer's journey where you look at your monthly SaaS bill and ask a simple question: why am I paying for intelligence that doesn't even know who I am?

That question is what led me to build Marvin.

---

What Is Marvin?

Marvin is my internal AI assistant — a Vue 3 frontend chat interface that provides a full-featured chat experience with session history, workspace file browsing, and artifact viewing. On the surface it looks like any other chat UI. Under the hood it's a self-hosted AI inference mesh, purpose-built for security, resilience, and the specific demands of real engineering work.

No data leaves my network. Ever.

---

What Can Marvin Do?

I built Marvin to cover the full surface area of what a working AI assistant needs to be genuinely useful inside a real engineering environment.

Build production systems. When I hand Marvin a spec, it implements every described service as fully working code — Go backends with typed config, dependency injection, and interfaces, Dockerfiles, Makefiles. No stubs, no placeholders.

Read, modify, and search code. Marvin can inspect any file in the workspace or an uploaded archive, edit files directly, and perform AST-based search across the entire project. It understands code structure, not just text.

Query my organisation's knowledge base. Marvin has access to a RAG index of every internal service, library, and piece of documentation. Ask it about prior implementations, architectural decisions, or which services do what — it knows.

Debug via live cluster logs. Marvin queries Kubernetes cluster logs through Loki by namespace, pod, container, or service name. Time range filtering, real-time debugging — all from the same chat interface.

Package and deliver artifacts. When a task produces a file or a directory, Marvin can zip and deliver it directly from the browser. No manual copying, no context switching.

---

Why I Built It Instead of Buying

The answer comes down to two things: security and economics.

Code Security

Every SaaS AI tool, however well-intentioned, represents a data egress point. Proprietary code, internal architecture, Kubernetes configurations, live cluster logs — all of it becomes content that passes through someone else's infrastructure. Terms of service around data retention and training vary, and the questions they raise don't have comfortable answers.

With Marvin, the attack surface is zero. Every model runs on my own hardware. Queries never reach a third-party API. My code stays my code.

The Economics

The per-seat costs for capable SaaS AI tools add up fast — and most of that spend goes toward capabilities that are either generic or actively unsuited to internal tooling and private codebases.

Marvin costs me electricity.

---

The Architecture

This is where it gets interesting.

Marvin isn't backed by a single model. It's an inference mesh — a small cluster of specialised models, each doing the job it's best suited for, orchestrated so that the right model handles the right request.

The Cluster

The always-on foundation runs inside a Kubernetes cluster, with an RTX 3050 as the resident GPU node. This is where the lightweight and specialist models live:

• Fallback inference — Qwen2.5-3B Q4, for when the primary model is unavailable
• Intent routing — Qwen2.5-7B Q4, classifies every incoming request and decides how to handle it
• Tool calling — Gemma 4 E2B Q5, dedicated to function dispatch and tool use
• Embeddings — Nomic Embed Text, powering the RAG knowledge base

The Remote Node

When it's online, a remote RTX 5060 Ti with 16GB VRAM joins the mesh as the primary inference node. It runs Qwen3.6-14B-A3B VibeForged v2 at Q8 — a mixture-of-experts architecture at full precision — paired with a Qwen3 0.6B draft model for speculative decoding. The drafter proposes tokens in parallel, the main model verifies them, and throughput increases substantially as a result.

When the remote node is off, Marvin falls back gracefully to the k8s cluster without interruption.

Why This Matters

Most self-hosted AI setups are a single model behind a single endpoint. One model, one job, one point of failure. What I've built is something closer to how production AI systems actually work — intent classification, model specialisation, graceful degradation, and a clear separation between always-on utility and high-power inference.

The difference in practice is significant. Tool calls go to a model optimised for tool calls. Embeddings are handled by a model built for embeddings. The big model handles what the big model is for. Everything is faster and more accurate than routing everything through one generalist.

---

The Moment It Clicked

I asked Marvin to describe itself — who it is, what it can do.

It queried my organisation's own knowledge base, assembled the answer from internal documentation, and responded with a precise, accurate description of its own capabilities.

It knew what it was because I had taught it what it was.

That's the difference between a generic assistant and one that actually belongs to you.

---

Building something similar? Have questions about the stack? The comments are open.

Building a Self-Hosted LLM Router in Go: Semantic Memory, Tool Calling, and 18 Phases of Debugging

I run local language models on two GPUs — a GTX 1650 for general chat and an RTX 3050 on the LAN for heavier coding tasks. A Python FastAPI router handled dispatching between them. It worked, but it was slow to start, awkward to deploy on Kubernetes, and the codebase was accumulating duct tape. So I rewrote it in Go over a weekend. What followed was 18 phases of debugging, design decisions, and one very persistent tool-calling loop.

This is the story of that rewrite.

The starting point

The Python router already did a lot: keyword heuristics to pick a GPU target, an intent classification model for ambiguous cases, CPU speculative draft planning, auto-continuation for truncated responses, and SSE streaming. The goal wasn't to simplify — it was to make the whole thing deployable as a single stateless binary in Kubernetes without a virtualenv or a pip install in sight.

A single Go binary. No dependency hell. No container that's 800MB of Python packages.

The initial structure mapped cleanly onto Go packages:

cmd/router/         — entrypoint
internal/config/    — env config
internal/router/    — HTTP handlers + routing logic
internal/intent/    — intent classifier client
internal/inference/ — streaming proxy
internal/memory/    — PostgreSQL layer
internal/events/    — NATS event bus

Phases 1–3: Ports, pods, and prompt pollution

Deployment issues came fast. The first was a port mismatch — the router listened on :8080 but the Kubernetes service had targetPort: 8000. A five-second fix. The more confusing problem was that /healthz kept returning the Python service's response even after deploying the Go pod. The rollout hadn't completed; I was port-forwarding to the wrong pod. Lesson: always wait on kubectl rollout status.

Routing bugs came next. The intent service — a 1.5B parameter model meant to classify messages as "general" or "coding" — was silently ignoring its classification instructions and generating code instead of JSON. The cause was sending the full conversation history to a model that small. It just couldn't hold the instruction in context. The fix matched what the Python original did: classify only the last user message, truncated to 500 characters.

But even after that fix, the intent model was unreliable enough under real load that I disabled it entirely. The heuristic — keyword matching on the prompt — turned out to be faster, deterministic, and frankly more correct for the workloads I throw at it. The intent model stays in the codebase for when a better model comes along.

Phase 4: Giving the router a memory

The most interesting engineering problem in the whole project was conversation tracking. OpenWebUI — the chat frontend — sends no session ID. Every request is stateless from the client's perspective. I needed to reconstruct conversation continuity from the content alone.

The solution: a rolling SHA256 hash of prior user messages. On each turn, the router hashes all previous user messages it has seen, looks that hash up in PostgreSQL, and finds (or creates) the conversation row. After finding it, it immediately writes the next turn's expected hash so the lookup succeeds on the following request.

This broke in several interesting ways:

• SHA256 of an empty string is a constant — every new conversation shared the same DB row until I added a guard for the empty-hash case.
• Including assistant responses in the hash meant the hash changed unpredictably. Fix: hash only user messages.
• Index-based exclusion (i < len(messages)-1) broke when message ordering varied. Fix: exclude by content match instead.

The schema ended up as three tables: conversations, messages, and embeddings. The embeddings table uses pgvector's vector(768) type for semantic retrieval.

Phases 5–6: Async events and semantic memory

Every stored message publishes a NATS event. Workers downstream handle the slow stuff asynchronously: a summarisation worker periodically condenses long conversations using one of the inference backends; an embedding worker sends message content to a nomic-embed-text service running on a GPU slice and stores the resulting 768-dimensional vector.

Context injection became a hybrid of two retrieval strategies. The last 4 messages by chronological order provide conversational flow; the top 4 by pgvector cosine similarity recover relevant older context. The semantic pass excludes anything already in the chronological window to avoid duplication.

One subtle ordering bug: I was fetching history after storing the current user message, so the current message appeared in the retrieved history and got filtered out, leaving the context empty. Flipping the fetch to happen before the store fixed it.

Phase 7: Taming OpenWebUI's background requests

OpenWebUI sends three silent background requests after every user message: one for follow-up question suggestions, one for a conversation title, one for tags. Each contains the full conversation history. Because my heuristic router looks for technical keywords, these requests were being sent to the 3050 (the coding backend), locking subsequent routing decisions.

The fix was an isSystemRequest() check that detects the ### Task: prefix these requests share, routes them to the 1650, and skips DB storage entirely. The tricky part was ensuring this check runs at the very top of the handler, before any database operations — an early version stored the system messages first and checked second, polluting the conversation history.

Phases 8–10: Load balancing, IDE integration, MCP

Health-check-based load balancing was straightforward: poll /health on each backend before routing, fall back to the other if the preferred one is down or has no slots. Since the 3050 is a physical node on my LAN, it can go offline at any time. The fallback to the 1650 handles this gracefully.

Integrating Continue.dev as an IDE assistant exposed a new class of problems. Continue sends tools arrays in every agent request — structured tool definitions that the router was silently stripping. The fix required extending the request struct to capture tool definitions, converting them to plain-text system messages the models can follow, parsing {"tool": "...", "args": {...}} patterns in responses, and returning proper OpenAI tool-call SSE so Continue could execute them.

Codebase indexing required adding a POST /v1/embeddings endpoint that proxies to the nomic-embed-text service in OpenAI format. Continue's bundled all-MiniLM-L6-v2 model failed because the WASM runtime couldn't initialise; removing it and pointing to the router's endpoint resolved the issue.

Phases 11–18: The tool-calling rabbit hole

The agent mode work turned into its own multi-phase project. The model (Qwen2.5-Coder-7B) would correctly call ls, receive a file listing, then call ls again. And again. Indefinitely.

The root cause wasn't the model being bad at tool use — it was that Qwen2.5-Coder-7B doesn't natively reason about tool results. It can emit tool call JSON, but it can't interpret the results and decide what to do next. Several approaches failed before landing on stateless in-request loop detection:

detectToolLoop(messages, repeatThreshold=2, maxTurns=30)
→ Guard 1: same tool+args called N times consecutively
→ Guard 2: total tool turns ≥ maxTurns hard cap
→ On trigger: extract unique results, build plain prompt,
  call StreamAgent(..., jsonSchema=false), return text

A MCP race condition added further complexity: Continue fires the next request before the MCP server returns the tool result, so the same call+result pair appears 10–25 times in body.Messages. A deduplicateToolMessages() pass collapses consecutive identical assistant→tool pairs before building the prompt.

Other improvements in this phase: buffered agent streaming (buffer the entire response, inspect for tool calls, send either structured SSE or plain text — never both); a next-file hint injection that tells the model which file to read next after a glob search; grammar_json_schema constraint via llama.cpp to force valid JSON output; and storing completed read_file results in PostgreSQL so follow-up questions about specific files don't lose their context.

Infrastructure, end state

Service	Model	Hardware	Role
llama-inference	Qwen2.5-3B-Q4	GTX 1650	General chat
llama-embedding	nomic-embed-v1.5	GTX 1650 ×0.5	Embeddings
RTX 3050 node	Qwen2.5-Coder-7B-Q5	RTX 3050 (LAN)	Coding / agent
llm-router	Go binary	Kubernetes	Control plane
PostgreSQL + pgvector	pg16	Kubernetes	Memory
NATS JetStream	—	Kubernetes	Event bus

Working

• Chat routing (heuristic)
• Conversation memory (rolling hash)
• Hybrid semantic + chronological context injection
• Agent mode always routed to 3050
• Tool calls initiated and executed with loop detection
• Codebase indexing via /v1/embeddings
• File contents persisted to memory after agent sessions
• System request detection and lightweight routing

Still flaky / not working

• Model occasionally loops before the next-file hint fires
• Glob search can exhaust max turns before reading any files
• Auto-continuation disabled in agent mode
• Intent model disabled — too unreliable with current models
• Draft model disabled — leaks planning steps into responses

What I'd do differently

The hash-based conversation tracking is the thing I'm least happy with. It's clever but fragile — there's a race condition on turn 2 and it breaks down when conversation order is inconsistent. A proper session ID passed by the client would eliminate all of it. If you control the frontend, use a session ID.

The tool-calling architecture is fundamentally limited by the model. Qwen2.5-Coder-7B is impressive for its size, but it wasn't trained to reason about tool results in a multi-turn loop. The right fix is a model with native function calling — not more prompt engineering. Every hack I added (next-file hints, JSON schema constraints, loop detection) is load-bearing scaffolding around a gap in the model's capabilities.

Go was the right choice. The router is a single binary, deploys in seconds, handles concurrent SSE streams cleanly, and the type system caught several category errors that would have been silent bugs in Python.

What's next

Auto-enrollment (backends register themselves, zero config changes for new nodes), request queueing via NATS when all slots are full, a local agent sidecar on the workstation with real filesystem access, and — when a better model becomes available — re-enabling the intent classifier and the draft planner.

The backlog is longer than when I started. That feels about right.