OpenTelemetry Collector Architecture

This retrospective explains the January 13, 2026 observability design shift from Prometheus-first scraping toward an OpenTelemetry collector-first model.

The original note is still valuable, but the checked-in topology has evolved. This page separates the durable architecture decision from details that have drifted in the repo.

Source Paths

• todo/incidents/trouble/opentelemetry-collector-architecture-2026-01-13.md
• lumie-infra/observability/kustomization.yaml
• lumie-infra/observability/opentelemetry/manifests/collector.yaml
• lumie-infra/observability/opentelemetry/manifests/targetallocator-rbac.yaml
• lumie-infra/observability/opentelemetry-operator/helm-values.yaml
• lumie-infra/observability/prometheus/helm-values.yaml
• lumie-infra/security/kyverno/manifests/policies/remove-cpu-limits.yaml

Context

The old mental model made Prometheus the default collector. That worked for scrape-oriented metrics, but Lumie also needed:

• application OTLP traces;
• application and container logs;
• Kubernetes metrics discovered through monitor resources;
• a clearer path to Loki for logs and Tempo for traces;
• less duplicate scraping logic across observability components.

The durable decision was to make OpenTelemetry the shared collection layer and let each backend own storage/query for its signal type.

Durable Design Decisions

The January architecture note made three decisions that still match the repo:

Decision	Current meaning
OpenTelemetry is the shared collection layer	app OTLP traffic, Kubernetes scrape targets, and container logs pass through the collector
Prometheus is not the default scraper for most workload metrics	Prometheus stores and serves metrics, but collector-managed pipelines do much of the collection work
storage stays signal-specific	Prometheus for metrics, Loki for logs, Tempo for traces

The repo-level observability kustomization preserves one important consequence:

# promtail removed - OTel filelog receiver handles log collection

Source: lumie-infra/observability/kustomization.yaml

Current Repo-Managed Contract

The present topology is defined by one OpenTelemetryCollector custom resource:

Component	Current mode	Checked-in responsibility
otel-collector	daemonset	receives OTLP traffic, tails /var/log/pods, and scrapes targets discovered through the Target Allocator
Target Allocator	enabled inside the collector spec	reads ServiceMonitor and PodMonitor resources and assigns scrape work to collector pods
Prometheus	single replica	enables the OTLP receiver and limits its own scraping to scrape-by: prometheus-only targets
Loki	OTLP HTTP endpoint	receives logs from the collector
Tempo	OTLP gRPC endpoint	receives traces from the collector

These excerpts capture the live contract:

spec:
  mode: daemonset
  targetAllocator:
    enabled: true
    allocationStrategy: per-node

Source: lumie-infra/observability/opentelemetry/manifests/collector.yaml

enableOTLPReceiver: true
enableRemoteWriteReceiver: false
serviceMonitorSelector:
  matchLabels:
    scrape-by: prometheus-only

Source: lumie-infra/observability/prometheus/helm-values.yaml

Drift From The January Note

Several original details are no longer the checked-in contract:

January note	Current repo state
separate otel-metrics and otel-logs collectors	one otel-collector DaemonSet handles metrics, logs, and traces
Target Allocator consistent hashing across a metrics StatefulSet	collector manifest sets allocationStrategy: per-node
Prometheus as remote-write receiver	Prometheus enables native OTLP receiver and disables remote write receiver
Thanos as active query path	observability kustomization comments out thanos/argocd.yaml; current footprint plan keeps Thanos disabled

There is also a smaller comment drift in lumie-infra/observability/prometheus/helm-values.yaml: a comment still mentions collector remote write, but the active exporter in collector.yaml is otlphttp/prometheus and enableRemoteWriteReceiver is false. The active exporter and receiver settings are the source of truth.

Operational Implications

Collector-first observability changes the failure domains:

Implication	Operational meaning
one collector DaemonSet handles multiple signals	a collector rollout problem can degrade metrics, logs, and traces together
Target Allocator owns scrape assignment	a ServiceMonitor alone does not prove a target is being scraped by a specific collector pod
Prometheus has a narrower scrape role	targets that need Prometheus-native scraping must opt into scrape-by: prometheus-only
CPU throttling matters	collector/operator values omit CPU limits, and Kyverno strips ordinary pod CPU limits through remove-cpu-limits

The collector became an infrastructure dependency, not a sidecar convenience.

Verification

Use read-only checks to verify the current contract:

kubectl get applications.argoproj.io -n argocd opentelemetry opentelemetry-operator prometheus
kubectl get opentelemetrycollectors.opentelemetry.io -n opentelemetry
kubectl get pods -n opentelemetry -o wide
kubectl logs -n opentelemetry daemonset/otel-collector-collector --tail=200
kubectl get clusterpolicy remove-cpu-limits -o yaml

Success means:

• the collector and operator Applications exist;
• the OpenTelemetryCollector custom resource exists;
• the DaemonSet is running on nodes;
• collector logs show live pipelines;
• the CPU-limit removal policy remains installed.

What Went Well

• The design separated collection from storage/query backends.
• Logs, metrics, and traces moved toward one collection plane instead of three unrelated mechanisms.
• Prometheus retained a clear role without staying responsible for every scrape path.
• The repo now documents enough manifest state to distinguish current contract from historical notes.

What Went Poorly

• The original writeup and current manifests drifted in collector count, target allocation strategy, Prometheus receiver mode, and Thanos status.
• Collector-first design increases the blast radius of collector outages.
• Some comments remained stale after implementation changed.
• Operators need Target Allocator visibility in addition to normal Kubernetes monitor resources.

Action Items

Action	Owner / surface	Status	Evidence
Keep this page aligned with collector.yaml and Prometheus values when observability topology changes.	lumie-document, lumie-infra	Ongoing	This retrospective records known drift from the January note.
Fix stale Prometheus values comments when editing observability manifests.	lumie-infra/observability/prometheus/helm-values.yaml	Planned	Active config uses OTLP receiver, not remote write receiver.
Preserve CPU-limit removal policy or explicitly document exceptions.	lumie-infra/security/kyverno	Ongoing	remove-cpu-limits is part of the current operational contract.

Lessons

• A collector-first design is simpler to reason about only if the collector's failure domain is treated as shared infrastructure.
• Historical architecture notes need drift sections when manifests evolve.
• Prometheus can remain the metrics backend without remaining the universal metrics collector.
• Comments in values files can become misleading; active manifests and receiver/exporter settings own the truth.

Related Pages

• OpenTelemetry operations reference
• OpenTelemetry Operator operations reference
• Prometheus operations reference