GitLab Bootstrap Rollback

This page is an incident retrospective for the April 10, 2026 self-hosted GitLab experiment. Lumie's surviving Git hosting path is documented in the infrastructure GitOps Gitea reference page.

What Was Added

6136a0c2 introduced two Argo CD applications:

• bootstrap/gitlab/argocd.yaml at sync wave 3 for the GitLab chart.
• applications/gitlab-runner/argocd.yaml at sync wave 4 for the runner chart.

The stack reused shared platform services instead of bundled subcharts:

• PostgreSQL from storage/infra-db/manifests/cluster.yaml
• Redis Sentinel from lumie-cache
• MinIO-backed object storage for GitLab and runner cache
• seven GitLab VaultStaticSecret objects plus one runner-token VaultStaticSecret
• Kyverno namespace exclusions so non-Zot upstream images could start

Why It Was Rolled Back

The same day, the bootstrap exposed three kinds of failure:

• Day-zero chart assumptions were wrong. 74352715 had to disable upgradeCheck, 7426094e had to disable Redis auth against an unauthenticated lumie-cache, and 8590841c had to switch the chart onto KEDA-backed scaling instead of rendering HPAs.
• Memory tuning was brittle. 9e7a17d6 cut several limits for solo use, but a85d67af had to restore webservice and Sidekiq memory after the real baselines stayed high.
• Chart values were easy to place at the wrong path. 2c1c0926 showed that gitlab.webservice.extraEnv was ignored by the chart, and Sidekiq tuning had to move under gitlab.sidekiq.pods[0]. f2d114ad then had to restate memoryKiller defaults because the nested map was replaced, not merged.

The important operational lesson is that Argo CD can reconcile cleanly even when a Helm values edit renders byte-identical manifests. For this stack, rendered output had to be checked alongside sync status.

Rollback Boundary

1e18f276 removed more than the visible applications:

Surface	Rollback action
bootstrap/gitlab/**	Deleted the GitLab Argo CD app, values, and Vault manifest source
applications/gitlab-runner/**	Deleted the runner app, values, and token secret source
bootstrap/kustomization.yaml and applications/kustomization.yaml	Removed both app registrations from the live GitOps graph
storage/infra-db/manifests/cluster.yaml	Removed bootstrap-time gitlab user and gitlabhq_production database creation
security/kyverno/manifests/policies/require-zot-registry.yaml	Removed the gitlab namespace exemption

The commit message also states an important limit: removing postInitSQL does not drop database objects that were already created on an earlier bootstrap. The repo proves future bootstraps no longer create GitLab objects; it does not prove historical database cleanup happened.

Current Repo State

The current repo no longer contains bootstrap/gitlab/** or applications/gitlab-runner/**, bootstrap/kustomization.yaml now registers gitea instead of gitlab, and the current storage/infra-db/manifests/cluster.yaml keeps gitea bootstrap SQL but no GitLab entries.

Verification

cd lumie-infra
find bootstrap applications -path '*gitlab*' -print

rg -n "gitlab|gitlabhq_production" \
  bootstrap/kustomization.yaml \
  applications/kustomization.yaml \
  storage/infra-db/manifests/cluster.yaml \
  security/kyverno/manifests/policies/require-zot-registry.yaml

Success means the file search returns no GitLab app paths and the targeted repo grep finds no surviving GitLab registration, bootstrap SQL, or Kyverno exemption in the current manifests.