I Like Dog Food

There is an old saying in software: eat your own dog food. If you build something, use it yourself, in anger, on the work that actually matters to you. It is the fastest way to find out whether the thing you built is real or only looks real in a demo.

As of this week Marvin is eating mine. Every Qwen request Marvin makes now goes to llmgo, my own Go and CUDA inference engine, and llama.cpp is no longer in that path. In my last post I talked about why I wanted it gone: security, a smaller audit surface, a single binary with nothing underneath it. This post is about what it took to get it into production, and what the numbers look like now that it is there.

Two cards, two jobs

llmgo has been built across two very different GPUs, and that was deliberate.

The RTX 3050 (8 GB) lives in my workstation. It is the development card. It also drives my KDE Plasma desktop, so a slice of that 8 GB is always taken and I can't get it back. The CPU behind it is a Ryzen 7 3700X, AVX2 only. The model is Qwen3.8-27B, and on this card a third of its 64 layers do not fit in VRAM, so they run on the CPU. It is not a friendly place to run a 27B model, and that is the point. If llmgo behaves here, it will behave anywhere.

The RTX 5060 Ti (16 GB) lives in the Kubernetes cluster, with the NVIDIA container toolkit providing the driver. It runs Qwen3.8-27B at UD-Q2_K_XL with its vision projector, all 64 layers on the GPU. This is where Marvin's real work happens, and llmgo now runs there as a container in exactly the slot the llama.cpp pod used to occupy.

Develop on the constrained box, deploy to the bigger one. Every win on the 3050 turns into a bigger win on the 5060 Ti.

The rule: host memory owns the model

The design decision that shaped everything else is simple to say. Host RAM holds every layer of the model, all the time. The GPU only ever gets what it needs right now. Nothing is ever copied back from the GPU; when a GPU copy is no longer needed it is just deleted, because the authoritative version is already sitting in host memory.

That rule applies to every model type, not just the text layers. The vision projector follows the same dynamic loading path, borrowing VRAM from the text layers when they are idle and handing it back when they are needed again. One mechanism, no special cases.

The 3050: ahead of llama.cpp on every number

The first GPU profile on the 3050 was humbling. The kernel doing the decode weight products was taking 263 ms per token for 40 layers, roughly twelve times slower than just reading the weights. A GPU layer was costing as much as a CPU layer, which defeats the purpose of having a GPU.

Over one long day of runs, each with the same three requests (a short prompt, a 9.6k-token chat prompt with tools, and a medium prompt), the things that moved the numbers were:

• Prefill attention on the tensor cores, FlashAttention-style tiles instead of FP32 on the CUDA cores. The 9.6k-token prefill went from 55.4 s to 45.0 s, the biggest single step of the day.
• An int8 tensor-core matmul for prefill, decoding the 2-bit weights to int8 in shared memory with the tile picked per weight shape at startup. The first version was slower. The fifth, with a two-stage pipeline, was the first run to beat llama.cpp's prefill.
• A new decode kernel that works on the low-bit weights directly with int8 activations, tuned per weight shape, running at up to 166 GB/s on a card with 224 GB/s of memory bandwidth.
• Speculative decoding with the model's own MTP layer, plus CPU kernels that decode each weight block once for both tokens, so a two-token step costs about the same as a one-token step.

Same model, same prompts, same machine, against llama.cpp:

Then I pointed it at the bigger quant, Qwen3.8-27B UD-Q2_K_XL, the same file the 5060 Ti runs. Getting there meant fast paths for the K-quant types, which had been running on the old float-decoding kernel. Here it is beside the IQ2_XXS results:

To be fair to the numbers: that run had a little more VRAM free than usual, so 44 layers fitted on the GPU instead of 39 or 40, and only 20 ran on the CPU. Per GPU layer the two files are about the same size, so the larger quant costs nothing in placement. There is no llama.cpp run of Q2_K_XL on this card yet, so the grey bars are its IQ2_XXS figures. On an 8 GB card that is also driving a desktop, a 27B model at over 6 tokens per second after a 9.6k-token prompt is a result I am happy with.

The rule I set is that no GPU kernel change is allowed to make the 3050 slower. The 3050 is an Ampere card and the 5060 Ti is Blackwell, and they do not always want the same kernels. If a change helps one and hurts the other, the kernel gets tuned or split per card rather than accepting the regression. The 3050 stays an honest benchmark.

The 5060 Ti: where the dog food got eaten

Marvin talks to its models through an LLM router, and every backend behind that router was llama.cpp. The router probes backends through /props and /health, so llmgo had to answer those the way the router expects before it could even be considered. Then it had to handle tool calling, streaming and thinking (at low reasoning effort, matching the llama.cpp config), with multiple requests isolated from each other, behind TLS and API keys.

Against the llama.cpp deployment it replaced, same card, same model:

Decode is well ahead, and holds at 31.7 to 32.4 tokens per second even at around 35k tokens of context. A big part of that is something llama.cpp leaves on the table. Qwen3.8 ships with a multi-token prediction (MTP) layer that drafts tokens ahead of the main model. llama.cpp ignores it. llmgo uses it, and in Marvin's tool loops 68 to 84 percent of the drafts are accepted, at 2.2 to 2.5 tokens per step.

Prefill on the 5060 Ti is still behind, and I am not going to pretend otherwise. The matrix multiplies are most of the remaining gap, and that is the next target. It has already been closed on the 3050, so I know it can be done.

What only daily use finds

This is the real reason to eat your own dog food. Benchmarks tell you how fast something is. Daily use tells you whether it is right.

The first time I watched Marvin run a long tool loop on llmgo, every turn took about 65 seconds, and 90 percent of that was prefill. Nothing was broken. Each turn just added 100 or so tokens (the tool call and its result) and llmgo re-read all 35,000 tokens of the conversation every time. llama.cpp keeps the previous prompt and only processes what is new. No synthetic benchmark I had written would have shown that, because benchmarks do not have conversations.

So llmgo got a prompt cache, with a twist: Qwen3.8 is a hybrid model, and its DeltaNet layers carry state that has to be checkpointed along the way, not just at the end of the prompt. The result, from a real session: ten turns in a row at 70 to 73 thousand tokens of context all hit the cache, each processing its 105 to 634 new tokens in 1.4 to 2.7 seconds.

Real use kept finding things:

• Marvin's UI and gateway router fire two or three small title, tag and follow-up requests after every answer. Each one was pushing the big conversation off the GPU, only for the next turn to pull it straight back. Those small requests now run alongside it instead.
• In a long session the cache grew until it squeezed the host, and one 105-token prefill took 66 seconds instead of 1.5. The cache now takes half of what is free beyond a fixed reserve.
• When the gateway router changes the tool list mid-conversation (2, then 11, then 29 tools), nothing can be reused, because the tools sit at the very start of the prompt. llama.cpp has the same limit. Good to know it is not just me.
• One early build produced gibberish. llmgo now checks itself at startup, running a token through the GPU and the CPU and comparing the results, and refuses to trust a layer that disagrees.

Every one of those came from Marvin doing Marvin's normal work. None of them came from a test suite.

Owning the whole path

It also closes a loop I care about. The same person writing the CUDA kernels is writing the container, the Kubernetes deployment, the router integration and the platform that consumes it. That is the continuum I keep coming back to: not developer, not ops, just the whole path from the silicon to the cluster, owned end to end. When something breaks, there is nobody else to hand it to, and that is exactly how I want it.

What's next

• Closing the prefill gap with llama.cpp on the 5060 Ti, starting with the matrix multiplies.
• Long-context decode. At 60k tokens, decode drops to 23.6 to 27.6 tok/s, with attention reading the cache far below what the card can do. A fix is built and waiting for its first run on the card.
• Faster small cache hits, so adding a short tool result to a 60k-token conversation takes well under a second.
• Gemma 4 and a CPU-only dual Xeon server that will serve the same 27B model with no GPU at all.

Back in April I was told I wouldn't be any good in AI because I didn't know GPUs. Marvin now runs on an inference engine I wrote, on kernels I tuned, faster than llama.cpp on every number on my desk and at the part you wait for while it types in the cluster. I like dog food.

I Am a Software Developer?

For a long time I never described myself as a software developer. I was a sys admin, a DevOps kind of guy. I kept things running. Developers were the other people, the ones who wrote the software I had to look after.

It wasn't until my days at BuildingIQ that I started to question that.

It turns out I had been writing code all along

Looking back, I had been writing code of some sort since my early teens. First it was 68000 assembly, entered as raw binary. Then Z80 assembly. Then, when Linux arrived, C and C++, because back then you needed to be self-sufficient if you wanted anything to work.

None of it came through the normal channels. No degree, no courses, no bootcamp. Everything was self-taught, learnt by reading, breaking things and fixing them again.

So when people at BuildingIQ asked me about it, I still didn't recognise it. This was despite the fact I had written, and was maintaining, more than 50 Puppet modules along with all the Ruby code that went with them. That is software. I just didn't call it that.

Eventually I gave in.

Once I accepted it, everything got better

Something changed once I stopped fighting the label. Things got better, easier and more efficient. Instead of putting up with outdated system admin tools, I started writing UI apps in VueJS to replace them. I stopped seeing a problem and reaching for a workaround, and started seeing it as something I could build my way out of.

Today my stack is VueJS on the front and Go microservices behind it.

Microservices are not the problem, the old paradigm is

A lot of people give microservices a bad rap. I think that is because most of them are still looking at microservices through the old paradigm.

Remember when VMs became a thing? Everyone did the lift and shift. Take the physical server, turn it into a VM, job done. It worked, but it came with a whole set of issues that didn't go away until the systems were re-architected for what a VM actually was.

Microservices went through the same thing. When I started, it was very much a lift and shift. Containers with fat operating systems underneath them, and all the problems those operating systems bring with them: patching, packages you never asked for, shells and tools sitting there waiting to be used against you.

That is one of the main reasons I settled on Go. A Go service compiles to a single binary. I can take the operating system out of the question entirely and concentrate on the real issues, the code that actually does the work.

Replacing llama.cpp in Marvin

Which brings me to what I have been doing for the last two days: building a replacement for llama.cpp in Marvin's architecture.

This is not because llama.cpp is bad. It isn't. It is just not suitable for my situation.

I build with security in mind. I need to know there isn't a legacy OS sitting under a service that is going to bite me, or another php2shell style hole waiting just around the corner. Every extra layer is something else I have to check, patch and trust.

The result:

• Roughly 6G cut from the set of things in my system I need to check
• A 12.3M CUDA LLM engine for my Qwen models
• CVE scanning is now just a single binary
• No web chat or any other access that isn't needed
• TLS or mTLS for connections inside my cluster network, plus API keys

That is the difference between lift and shift and actually re-architecting. The engine does one job, and there is nothing else in the box.

More to come

There is more to go. I have a few ideas I want to try for running a 27B model on an 8G GPU while keeping a good token rate. If they pan out, that will be its own post.

And yes, after all these years, I think I can finally say it without the question mark. I am a software developer.

Don't Panic, It's Mostly Documentation

This is Marvin, as of 20 October 2026.

If you haven't seen a Gource render before, every dot is a file and every cluster is a folder, with lines tracing how they hang together. It's a quick way to see the shape of a codebase at a glance. And the shape of this one surprised me.

Two galaxies

On the right is the platform: the microservices that make Marvin run. Image generation, the vision model and its proxy, document reading, and all the plumbing that holds them together. This is the part I thought of as "Marvin" when I started in April. It's the part I'd point to if someone asked what I'd built.

On the left is the knowledge base. Rules, procedures, runbooks and governance. How Marvin should behave, what it's allowed to do, how to handle things when they go wrong, and who gets the final say.

The left galaxy is bigger. A lot bigger. There are around 650 files in the knowledge base and around 400 in the entire platform. And right now, the knowledge base is where most of the activity is.

So what is Marvin, really?

I set out to build an AI platform. What I've actually built is a fairly modest set of services and a very large rulebook telling them how to behave.

That isn't an accident. The services are machinery. They take a request and do a thing. The knowledge base is judgement: when to do the thing, when not to, and how to know it was actually done. A few examples:

Questions: "WHEN the user asks a question, the assistant MUST answer it honestly and then stop — a question is never an instruction to continue."

UI testing: "The assistant MUST verify a UI by using it — fill in every field, submit, reload, and confirm every value came back — because a page that renders proves only that it loaded."

Proof: "The assistant MUST NOT report a test passed until the logs of every service the request crossed prove it was handled; missing evidence is a defect, not a pass."

None of these are clever. Each one is there because, at some point, the assistant did exactly the thing the rule now forbids. It took a question as a green light to keep going. It declared a page working because it rendered. It called a test passed with no evidence anyone had handled the request.

Marvin was security-first from day one, and this is where that lives. Not in clever code, but in written-down decisions about what's allowed and what counts as done.

There's also a practical reason the balance tips this way. Code is increasingly cheap to produce. If a service disappeared tomorrow, I could rebuild it. The rules are different. Every one of them exists because I thought something through, or because something went wrong and I didn't want it going wrong again. That's the part that's hard to replace.

How they connect

Notice that the two galaxies don't touch. In the file tree, the rules and the platform are completely separate. The connection only exists at runtime, through the kb-service.

None of the rules are baked into the services themselves. Marvin asks for them.

When a session starts, a single bootstrap call hands Marvin the mandatory rule set. From there, rules are loaded by domain as the work demands: if the job involves git, Marvin searches for everything tagged for git and loads those rules before it touches anything. And if it can't name the document that governs what it's about to do, another rule says it has to search the knowledge base before going any further. No guessing.

Writing works the same way, through one narrow door. New knowledge goes in through a single write tool, and there's a rule about that too: the knowledge base is not a log. Run state and handoff notes don't belong there. Only things worth knowing next time.

Under the hood it's deliberately boring. Plain markdown files, each with a small JSON sidecar for metadata, a full-text search index over the top, and a separate graph that tracks how the services depend on each other.

That separation is why the left galaxy can grow so fast. Because the rules are data rather than code, changing how Marvin behaves usually means editing a markdown file, not redeploying a service.

If you're building your own

When I started, someone told me I wouldn't be any good in AI because I "don't know GPUs." Six months in, I'd say the GPUs were never the hard part. Neither was the code.

If you're building your own Marvin, plan for the rulebook. It'll end up bigger than the robot.

Don't panic. It's mostly documentation.