Skip to main content

Loki

Loki is Lumie's log store. It is intentionally configured as a small, local-retention service rather than a durable long-history archive.

Source paths​

  • lumie-infra/observability/loki/argocd.yaml
  • lumie-infra/observability/loki/helm-values.yaml
  • lumie-infra/observability/opentelemetry/manifests/collector.yaml
  • lumie-infra/applications/teleport-event-handler/manifests/vector-configmap.yaml

Runtime contract​

  • chart: grafana/loki
  • deployment mode: SingleBinary
  • replicas: 1
  • storage backend: filesystem
  • persistence: disabled
  • retention: 72h
  • auth: disabled

Ingestion flow​

The main log path is OpenTelemetry:

  • the collector tails /var/log/pods/*/*/*.log
  • enriches records with Kubernetes attributes
  • exports logs to http://loki.loki.svc.cluster.local:3100/otlp

Teleport event logs are a special case and go directly to Loki through a Vector sink.

Operational boundaries​

  • Loki is internal-only; no ingress is defined.
  • The Helm chart's canary and self-monitoring agent are disabled.
  • Because storage is emptyDir, Loki is a short-term troubleshooting surface, not a compliance archive.

Failure modes​

  • Restarting or rescheduling the Loki pod removes locally stored history.
  • If the OpenTelemetry collector loses file access or the exporter path, application logs disappear from Grafana even though the applications keep running.
  • If operators expect Promtail, they will miss that observability/kustomization.yaml explicitly removed it in favor of the collector filelog receiver.

Verification​

kubectl get applications.argoproj.io -n argocd loki
kubectl get pods -n loki
kubectl describe pod -n loki loki-0
kubectl logs -n opentelemetry daemonset/otel-collector-collector --tail=200