Fixing Our Package Server: Upload Flow, OCI Compliance, and Dynamic Mirroring

Running your own package server and mirror can be a powerful way to control your software supply chain, but it comes with sharp edges. Recently, we hit several breaking issues in our registry implementation that highlighted how easy it is to get things almost working — but not correctly enough for the ecosystem tools.

What Was Broken & Why

• Image push 404/commit errors: Our upload flow mixed up upload UUIDs vs. blob digests, and paths weren’t repo-scoped. When finalizing with PUT …/uploads/<uuid>?digest=sha256:…, the server couldn’t find the temp file → 404.
• crictl pull size validation: HEAD/GET for blobs didn’t return Content-Length/ETag/Accept-Ranges, so unpack failed.
• Manifests vs indexes: Server always served one content type, but clients need the correct application/vnd.oci.image.index.v1+json vs application/vnd.oci.image.manifest.v1+json.
• Nested repos & tags: Our path parsing assumed user/repo, but real-world needs include deeper hierarchies (e.g., kubecve/api/kubecve-api:…).

Server Changes (Registry API v2)

We implemented repo-aware routing with a new parseV2Path, allowing any depth of repository naming. Storage layout now separates blobs and manifests cleanly, while maintaining legacy fallbacks:

<repo>/
  blobs/<sha256-hex>
  manifests/
    by-digest/<sha256-hex>
    by-tag/<tag>              # JSON: {"digest":"sha256:<hex>"}
  manifests/<tag>.json        # legacy fallback

Upload Flow

1. POST /blobs/uploads/ → 202 + Docker-Upload-UUID
2. PATCH /blobs/uploads/<uuid> → append chunk
3. PUT /blobs/uploads/<uuid>?digest=sha256:<hex> → verify, rename, return 201
4. Monolithic PUT supported for entire blob

Blob GET/HEAD

• Now return Docker-Content-Digest, Content-Length, ETag, Accept-Ranges: bytes.

Manifests

• PUT stores under by-digest and links via by-tag.
• GET/HEAD resolves by digest or tag with correct headers and media type.

Pull-Through Mirror (Dynamic)

We added dynamic mirror capability: if a requested manifest/blob is missing, the server fetches it upstream, caches it, and serves it immediately — without extra client config.

Mirror Logic

• Uses containerd’s ?ns=<original-host> when available.
• Else checks X-Registry-Host header.
• Else tries fallback list: ["docker.io", "registry.k8s.io", "ghcr.io"].

Special-casing for Docker Hub normalizes single-component repos (alpine → library/alpine). Storage is host-namespaced:

__mirror__/<host>/<repo>/
  blobs/<hex>
  manifests/
    by-digest/<hex>
    by-tag/<tag>

Client Configs

• Podman / CRI-O: Per-registry mirror configs in /etc/containers/registries.conf.d/.
• containerd: Configured in /etc/containerd/config.toml with /etc/containerd/certs.d/_default/hosts.toml to route all pulls through the mirror.
• Docker Engine: Limited to mirroring Docker Hub, so we don’t rely on it for dynamic routing.

CI & Versioning

In GitLab CI, we refined .release handling:

• Increment/write .release only in the job workspace.
• Publish it as an artifact for downstream jobs.
• Never push it back to git.

This keeps tags consistent ($PACKAGE_SERVER_DOCKER, $NAME, $VERSION + .release) without polluting the repo.

Validated Fixes

• Manifest misses now trigger upstream fetch instead of 404.
• Blob HEAD/GET now return correct sizes → crictl pull succeeds.
• Mirror logs show host resolution, upstream attempts, and cache writes.

Takeaway

Building a standards-compliant registry isn’t just about storing blobs — every detail of the API matters. By fixing path parsing, header responses, manifest media types, and adding dynamic mirroring, we now have a robust package server that integrates cleanly with modern container tooling. It’s a good reminder that correctness is the real feature.

Repository: https://gitlab.com/jlcox70/repository-server

Container: https://hub.docker.com/r/jlcox1970/package-server

Tags: containers, registry, oci, mirror, kubernetes, devops, supply-chain, security