A Kubernetes platform from scratch in GitOps

In the first article, the LLM gateway ran on my computer with Docker Compose. One file, one command, and the teams of the fictional company could call their models without leaking personal data. For a demo, that is perfect. For a company, much less so: nobody knows what really runs, a change made by hand on a Friday evening is forgotten by Monday, and the day the machine dies, someone has to remember everything that was installed.

This second block of the platform answers that problem. I built a Kubernetes cluster where nothing is installed by hand. Everything that runs on it is described in a Git repository, and an agent inside the cluster keeps comparing what runs with what is written. If someone changes something directly on the cluster, the agent puts it back the way Git says. If everything is deleted, it rebuilds everything.

The cluster runs on my machine and, on every pull request, on a GitHub runner where it is recreated from scratch. Every number in this article comes from those runs, and can be reproduced with the commands of the repo.

Git as the single source of truth

This way of working is called GitOps. The idea fits in one sentence: the desired state of the cluster is written in Git, and a program makes reality match what is written.

A comparison helps to see the difference. Without GitOps, running a cluster is like a kitchen where every cook adds a pinch of salt their own way. The dish ends up good, but nobody could make it again. With GitOps, the recipe is written down, and a chef checks the kitchen every minute. An ingredient is missing? The chef adds it. Someone changed the cooking time without changing the recipe? The chef puts it back. To change the dish, you change the recipe, and that change goes through review, like any code.

In practice, a change to the platform is a pull request. The CI tests it on a brand new cluster, someone reviews it, and it is merged. The agent sees the new commit and applies it. The repository keeps the history of every change, with its author and its reason, and going back means reverting a commit.

ArgoCD rather than Flux

Two tools dominate this agent role, ArgoCD and Flux. Both are mature projects of the CNCF, the foundation that hosts Kubernetes, and both do the basic job very well. What sets them apart is what you can see.

One component, one folder

The whole platform fits in one folder per component. Each describes what it installs with Kustomize, the standard Kubernetes tool to assemble configuration files, and one layer per environment adds what is specific to that environment, such as the local certificate authority.

apps/  # one ArgoCD Application per component
└── values.yaml  # the list of components and their order
platform/
├── argocd/  # ArgoCD manages itself
├── envoy-gateway/
├── cert-manager/
├── external-secrets/
├── cloudnative-pg/
├── monitoring/
├── llm-gateway/  # the gateway of the first block
└── job-queue/
scripts/  # the checks of just test
docs/adr/  # one record per decision

Components cannot start in just any order. A PostgreSQL cluster cannot be created until the operator that knows how to run it is there. So ArgoCD deploys them in waves: it waits for each wave to be healthy before starting the next. A single Application, the root, creates all the others. This is called the app of apps pattern. And ArgoCD is on the list: it upgrades itself when its folder changes.

What the platform runs

The list of components looks like that of a production platform, at a smaller scale. I will not go through all of them here, but each one has a story.

HTTP traffic comes in through Envoy Gateway, never through an Ingress. For years the default answer was ingress-nginx. On November 11, 2025, the Kubernetes networking team announced its retirement: no more security fixes after March 2026. Starting a platform in 2026 on ingress-nginx meant choosing a component that was already abandoned. I went with the Gateway API, its official successor, implemented by Envoy Gateway, and a hook rejects any Ingress before it is even committed.

No secret is in Git. Team keys, database passwords and Langfuse keys are generated when the cluster starts, then handed to the applications by External Secrets. The day a real secret manager arrives, only the configuration of that source changes, not the applications. gitleaks checks the whole history of the repository on every pull request.

PostgreSQL is run by an operator, CloudNativePG. Each database has a primary and a replica on another node, and its transaction logs stream continuously to S3 storage. On every run, the CI kills the primary and checks that the replica takes over without losing the last write, in 22 seconds on the runner. It also restores a backup into a brand new cluster, from the storage alone.

Observability links metrics, traces and logs. On the gateway’s dashboard, a dot on the latency graph is a real request. One click opens its trace, a second one its logs. The CI repeats those two clicks through the APIs on every run, to check that the links hold.

Two SLOs protect the gateway, its availability and its latency, with alerts that warn when the error budget burns too fast and a runbook for each. The CI stops a model server and checks that the alert fires, which takes 107 seconds on the runner.

A job queue takes the long-running work. A job that fails comes back after 5 seconds, then 10, and goes to a separate queue after its third failure. This behavior is configured entirely in RabbitMQ, without a line of code, and tested on every pull request.

From commit to cluster, without copying a tag

Having everything in Git is not enough if every new version of an application means copying a number by hand. The gateway lives in its own repository, and I wanted a fix merged there to reach the cluster without anyone touching the platform’s repository.

The key link is the platform’s CI. Before it lets a new version through, it checks that the images were signed by the project’s build workflow, with the list of their components (the SBOM), and that the configuration files match that version. Then it creates a cluster, deploys the platform on it and runs the gateway’s 46 integration tests, those of the version being deployed.

The signature is not a theoretical precaution. On March 19, 2026, 76 of the 77 tags of trivy-action, a security scanning GitHub Action used everywhere, were moved to code that stole the secrets of the pipelines. Since then, every action in my workflows is pinned to the exact hash of a commit, and the platform refuses an image it cannot trace back to its build. The first scan found plenty to do: the two gateway images had 10 and 9 fixable critical vulnerabilities, one of them in LiteLLM itself, which allowed bypassing authentication.

Here is the real path of version 0.2.1, a small fix to the tests.

1. 20:33

   Fix merged in llmops-gateway

2. 20:37

   Release PR merged, tag v0.2.1

   release-please computes the version from the commit messages.

3. 20:40

   0.2.1 images published, signed, with their SBOM

4. 20:51

   Renovate opens the PR on llmops-platform

5. +17 min

   CI green

   Signature, configuration, whole cluster and the 46 tests of the gateway.

6. 09:46

   Renovate merges the PR itself

   The next morning, on its following run. A fix is merged without human review, once the whole CI has passed.

7. 09:47

   ArgoCD deploys 0.2.1

   87 seconds after the merge.

Two merges by hand in total, both in the gateway’s repository: the fix and its release. The only idle time is Renovate, hosted by Mend, which only merges on its next run. The PR was green in 17 minutes, and it waited until the next morning, when I ran Renovate again. A minor version waits for a human merge, and a major version for one more approval.

Why no cloud cluster

The original plan was to deploy the same platform on Kapsule, Scaleway’s managed Kubernetes, with Terraform. I gave that up along the way, and I would rather say why.

A managed cluster would mostly have taught me to click, or to write Terraform, to get three nodes. The hard part of a platform is what runs on it and how you prove it works. The local cluster already runs exactly the same manifests, and it is recreated from scratch on every pull request. The install script was also checked on minikube, and nothing in it is specific to k3d. Finally, I already run a production cluster at work. Paying for a cluster for this portfolio would have shown nothing more.

Scaleway stays in the project, for what a laptop cannot do: serving real models. Its inference APIs, hosted in Paris and billed per token, will soon replace the small Ollama models behind the same usage names. Personal data will then leave the cluster, which is exactly where the masking of the first article earns its keep.

The pitfalls

This is the part I would read first if I started a project like this one again. None of these problems shows up in the tools’ documentation.

Envoy cuts a request after 15 seconds by default. When the gateway’s large model does not answer, it falls back to the small one, and the whole thing can take up to 20 seconds. On the cluster, those requests failed where they worked with Docker Compose. The route to LiteLLM now has a 90-second timeout.

Kustomize skipped a namespace, and LiteLLM stopped restarting. Kustomize adds a suffix to every ConfigMap it generates, a hash of its content, so that a configuration change restarts the pods that use it. But it leaves alone the resources rendered from a Helm chart when they have no namespace. The LiteLLM Deployment therefore pointed to the ConfigMap’s name without its hash, and a new configuration never restarted it. A patch that sets the namespace fixed it.

k3s already installs the Gateway API, in a version that cannot be disabled, and it clashed with Envoy Gateway’s. So the local cluster tells k3s to skip that file at startup.

ArgoCD saw differences that did not exist. The Kubernetes API fills HTTP routes with default values, and ArgoCD took them for changes made by hand. Making it compare against a server-side dry run made those false differences go away.

The gateway’s tests had two mysterious failures. The test that checks that no personal data reaches the model reads LiteLLM’s detailed logs. On the cluster, it first found only 18 requests out of 100. Kubernetes splits a container’s logs every 10 MB, and kubectl logs only reads the last piece. Once the size was raised, it found none at all: ArgoCD put LiteLLM back into its normal configuration in the middle of the test, because a sync was already running when the test switched it to detailed mode. The test now suspends ArgoCD’s reconciliation, lets the running one finish, and checks that detailed mode is still on before concluding. That is the flip side of an agent that repairs everything: it also repairs what you changed on purpose.

The numbers

These measurements come from the last CI run on main, on a standard GitHub runner with 4 cores and 16 GB, except the rebuild of the job queue, measured on my machine.

The local cluster costs nothing, but it has a price in memory. The platform takes between 13 and 14 GB, of which 3 for Presidio and LiteLLM, 2 for Langfuse and 1.5 for observability. The CI runner has 16: the peak of the last run was 14.1 GB. That is the constraint that weighs most on what comes next.

just bootstrap k3d-llmops local main

== ArgoCD Applications
33 s: argocd synced and healthy
85 s: external-secrets synced and healthy
176 s: cert-manager synced and healthy
209 s: envoy-gateway synced and healthy
321 s: job-queue synced and healthy
332 s: cloudnative-pg synced and healthy
381 s: monitoring synced and healthy
453 s: langfuse synced and healthy
631 s: llm-gateway synced and healthy
631 s: root synced and healthy
Every Application is synced and healthy

An excerpt of the CI output. Each component is reported when it becomes healthy, and the LLM gateway comes last: it is in the last wave, after Langfuse, where it creates the teams’ projects.

The limits

• A single machine. The cluster has three nodes, but they all run on the same computer. PostgreSQL failing over from one node to another is real; a real machine failing is not.
• Memory is tight. With 1.8 GB of headroom on the runner, every new component has a cost. The next saving will be to replace, in the CI only, the second Ollama model with a smaller one.
• ArgoCD polls GitHub instead of being notified. That explains the minute and a half between a merge and the deployment. A webhook would remove it, but GitHub would need to reach the cluster.
• Signatures are checked before the merge, not at the cluster’s door. A kubectl apply run by hand could deploy an unsigned image. Admission control, with Kyverno for example, would close that door.

What’s next

The third milestone is starting. The job queue is in place; next come the workers that consume it and KEDA to scale them from zero to many with the length of the queue, then the real models on Scaleway, with the cost of each job measured on the same dashboard as the queue and the latency. That will be the subject of the next article.

The code, the architecture decisions and the test scripts are on GitHub. Two commands rebuild everything: just up, then just test.