Saturday, 8 November 2025

Updating the Package Server – Auth, Probes, and a Bit of Cleanup

It’s been a while since I last wrote about the package server project. In that post, I’d just finished stabilizing the upload flow and getting the dynamic mirror logic working. Since then, I’ve pushed a fair number of updates — tightening authentication, cleaning up CI, and finally adding the small operational touches that make it more comfortable to run day to day.

Cleaning up after August

After I wrapped up the mirror work in August, I spent some time chasing edge cases and wiring up the CI. That work landed in a couple of late-September releases — mostly housekeeping and the first experiments toward a proper auth stack.
By November, the branch had grown into a decent fall cleanup: new auth modes, proper health endpoints, saner logs, and the usual round of dependency bumps and formatting fixes.

Authentication done right (or at least, done)

I finally formalized authentication. You can now run the server with no auth, basic file-based auth, an API backend that returns a JWT, or full OIDC integration. That gives me a smooth path from local testing through production, all using the same configuration block.

auth:
  mode: basic
  basic_htpasswd_file: ./.htpasswd
  basic_api_backends:
    - https://{rest server auth}

I spent more time than I’d like to admit making sure those combinations actually worked — the “none/basic/OIDC” wiring was surprisingly fiddly. The payoff, though, is that it now supports the same JWT-based flow I use elsewhere for CI jobs and other internal services.

Health endpoints and quieter logs

While working through container orchestration setups, I finally added proper /health and /ready endpoints. They exist mainly so Kubernetes probes don’t clutter the logs with noise.
At the same time, I reworked logging to use a standard Apache-style format and correctly handle X-Forwarded-For, so it’s finally possible to see who’s actually talking to the server through a proxy chain.

Mirrors that behave like mirrors

The dynamic pull-through mirrors I introduced last time are now more flexible. You can chain mirrors — useful when running a local cache in front of a site-wide one — and control the upstream via environment variables.
Set REGISTRY_UPSTREAM_MIRROR and flip enableProxy to true in the config, and the server will behave as a caching proxy without additional plumbing.

It’s a small feature, but it makes it much easier to run the server as part of a layered mirror setup.

CI and releases

Most of the September commits were about getting CI to publish images reliably. That’s all cleaned up now — image pushes happen automatically, and local builds match what I push from CI.
If you just want to run the latest build, you can grab it directly from Docker Hub:

docker pull jlcox1970/package-server:<tag>

I also standardized the module layout so it’s easier to build locally without wrestling with paths or dependencies.

Upgrading in place

If you’re already running an older instance, upgrades should be painless:

  • Keep your existing mirror and storage settings.
  • Switch your auth mode from “none” to “basic” (and point it to an .htpasswd file).
  • Add basic_api_backends if you want to authenticate against an external service.
  • Redirect liveness and readiness checks to the new endpoints.
  • Optionally enable REGISTRY_UPSTREAM_MIRROR to save bandwidth on cold pulls.

The default behavior hasn’t changed — just more options where they make sense.

Looking ahead

Next up is tightening the OIDC path and publishing a short “cookbook” of operational examples. I’ve had a few requests for topologies that mix public mirrors, internal caches, and private registries, so I’ll document what I’m using in production once it settles down.

As always, the code’s on GitLab at
https://gitlab.com/jlcox70/repository-server

and the Docker image builds are automatically pushed to jlcox1970/package-server on Docker Hub.

Sunday, 31 August 2025

Fixing Our Package Server: Upload Flow, OCI Compliance, and Dynamic Mirroring

Running your own package server and mirror can be a powerful way to control your software supply chain, but it comes with sharp edges. Recently, we hit several breaking issues in our registry implementation that highlighted how easy it is to get things almost working — but not correctly enough for the ecosystem tools.

What Was Broken & Why

  • Image push 404/commit errors: Our upload flow mixed up upload UUIDs vs. blob digests, and paths weren’t repo-scoped. When finalizing with PUT …/uploads/<uuid>?digest=sha256:…, the server couldn’t find the temp file → 404.
  • crictl pull size validation: HEAD/GET for blobs didn’t return Content-Length/ETag/Accept-Ranges, so unpack failed.
  • Manifests vs indexes: Server always served one content type, but clients need the correct application/vnd.oci.image.index.v1+json vs application/vnd.oci.image.manifest.v1+json.
  • Nested repos & tags: Our path parsing assumed user/repo, but real-world needs include deeper hierarchies (e.g., kubecve/api/kubecve-api:…).

Server Changes (Registry API v2)

We implemented repo-aware routing with a new parseV2Path, allowing any depth of repository naming. Storage layout now separates blobs and manifests cleanly, while maintaining legacy fallbacks:

<repo>/
  blobs/<sha256-hex>
  manifests/
    by-digest/<sha256-hex>
    by-tag/<tag>              # JSON: {"digest":"sha256:<hex>"}
  manifests/<tag>.json        # legacy fallback

Upload Flow

  1. POST /blobs/uploads/ → 202 + Docker-Upload-UUID
  2. PATCH /blobs/uploads/<uuid> → append chunk
  3. PUT /blobs/uploads/<uuid>?digest=sha256:<hex> → verify, rename, return 201
  4. Monolithic PUT supported for entire blob

Blob GET/HEAD

  • Now return Docker-Content-Digest, Content-Length, ETag, Accept-Ranges: bytes.

Manifests

  • PUT stores under by-digest and links via by-tag.
  • GET/HEAD resolves by digest or tag with correct headers and media type.

Pull-Through Mirror (Dynamic)

We added dynamic mirror capability: if a requested manifest/blob is missing, the server fetches it upstream, caches it, and serves it immediately — without extra client config.

Mirror Logic

  • Uses containerd’s ?ns=<original-host> when available.
  • Else checks X-Registry-Host header.
  • Else tries fallback list: ["docker.io", "registry.k8s.io", "ghcr.io"].

Special-casing for Docker Hub normalizes single-component repos (alpinelibrary/alpine). Storage is host-namespaced:

__mirror__/<host>/<repo>/
  blobs/<hex>
  manifests/
    by-digest/<hex>
    by-tag/<tag>

Client Configs

  • Podman / CRI-O: Per-registry mirror configs in /etc/containers/registries.conf.d/.
  • containerd: Configured in /etc/containerd/config.toml with /etc/containerd/certs.d/_default/hosts.toml to route all pulls through the mirror.
  • Docker Engine: Limited to mirroring Docker Hub, so we don’t rely on it for dynamic routing.

CI & Versioning

In GitLab CI, we refined .release handling:

  • Increment/write .release only in the job workspace.
  • Publish it as an artifact for downstream jobs.
  • Never push it back to git.

This keeps tags consistent ($PACKAGE_SERVER_DOCKER, $NAME, $VERSION + .release) without polluting the repo.

Validated Fixes

  • Manifest misses now trigger upstream fetch instead of 404.
  • Blob HEAD/GET now return correct sizes → crictl pull succeeds.
  • Mirror logs show host resolution, upstream attempts, and cache writes.

Takeaway

Building a standards-compliant registry isn’t just about storing blobs — every detail of the API matters. By fixing path parsing, header responses, manifest media types, and adding dynamic mirroring, we now have a robust package server that integrates cleanly with modern container tooling. It’s a good reminder that correctness is the real feature.

Repository: https://gitlab.com/jlcox70/repository-server


Container: https://hub.docker.com/r/jlcox1970/package-server


Tags: containers, registry, oci, mirror, kubernetes, devops, supply-chain, security

Sunday, 20 July 2025

From 290 CVEs to Zero: Rebuilding the Repository Server the Hard Way

The container image backing my repository server had quietly accumulated over 290 CVEs. Each of those is not just a statistic—they’re potential entry points on the attack surface.

Let’s be clear: just because this service ran inside Kubernetes doesn't mean those vulnerabilities were somehow magically mitigated. Kubernetes may abstract deployment and orchestration, but it does nothing to shrink the surface exposed by the containers themselves. A vulnerable container in Kubernetes is still a vulnerable system.

This image was built on Rocky Linux 9. While updates were technically available, actually applying them was more difficult than it should have been. Patching wasn't just a matter of running dnf update—dependency entanglements and version mismatches made the process fragile.

I attempted a move to Rocky Linux 10, hoping for a cleaner slate. Unfortunately, that path was blocked: the DEB repo tooling I rely on couldn’t be installed at all. The package dependencies for the deb-dev utilities were broken or missing entirely. At that point, the problem wasn’t patching—it was the platform itself.

That left one real option: rebuild the entire server as a pure Go application. No more relying on shell scripts or external tools for managing Debian or RPM repository metadata. Instead, everything needed—GPG signing, metadata generation, directory layout—was implemented natively in Go.

The Result

  • Container size dropped from 260MB to just 7MB
  • Current CVE count: zero
  • Dependencies are explicit and pinned
  • Future updates are under my control, not gated by an OS vendor

In practical terms, the entire attack surface is now reduced to a single statically-linked Go binary. No base image, no package manager, no lingering system libraries to monitor or patch.

This is one of those changes that doesn’t just feel cleaner—it is objectively safer and more maintainable.

Lesson reinforced: containers don’t remove the need for security hygiene. They just make it easier to ignore it—until it’s too late.

Source on GitLab