본문으로 건너뛰기

Kyverno

Kyverno는 Kubernetes Admission Webhook 기반의 정책 엔진입니다. YAML로 정의된 ClusterPolicy를 통해 모든 리소스 생성·수정 요청을 검증하고 보안 정책을 강제합니다.

아키텍처​

구성 요소​

Admission Controller​

  • 역할: 리소스 생성/수정 요청 가로채기
  • 이미지: zot.lumie-infra.com/kyverno/kyverno:v1.13.4
  • 리소스: CPU 25m, Memory 128Mi

Background Controller​

  • 역할: 기존 리소스에 대한 정책 검증 (백그라운드)
  • 이미지: zot.lumie-infra.com/kyverno/background-controller:v1.13.4
  • 리소스: CPU 15m, Memory 128Mi

Cleanup Controller​

  • 역할: 정책 위반 리소스 정리
  • 이미지: zot.lumie-infra.com/kyverno/cleanup-controller:v1.13.4
  • 리소스: CPU 25m, Memory 128Mi

Reports Controller​

  • 역할: 정책 위반 리포트 생성
  • 이미지: zot.lumie-infra.com/kyverno/reports-controller:v1.13.4
  • 리소스: CPU 25m, Memory 128Mi

클러스터 정책​

1. require-zot-registry​

모든 컨테이너 이미지는 내부 Zot 레지스트리에서만 허용됩니다.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-zot-registry
spec:
  validationFailureAction: Audit
  background: true
  rules:
  - name: validate-image-registry
    match:
      any:
      - resources:
          kinds: [Pod]
    exclude:
      any:
      - resources:
          namespaces:
          - kube-system
          - kube-public
          - kube-node-lease
          - argocd
          - cert-manager
          - external-secrets
          - haproxy
          - minio
          - vault
          - zot
          - kyverno
    validate:
      message: "Container images must be from zot.lumie-infra.com registry."
      foreach:
      - list: "request.object.spec.containers"
        deny:
          conditions:
            all:
            - key: "\{\{element.image\}\}"
              operator: NotEquals
              value: "zot.lumie-infra.com/*"

적용 범위:

  • 모든 Pod의 containers 및 initContainers
  • 시스템 네임스페이스 제외

2. require-resource-limits​

모든 컨테이너는 리소스 제한을 설정해야 합니다.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-resource-limits
spec:
  validationFailureAction: Audit
  background: true
  rules:
  - name: require-memory-limits
    match:
      any:
      - resources:
          kinds:
          - Pod
    exclude:
      any:
      - resources:
          namespaces:
          - kube-system
          - kube-public
          - kube-node-lease
          - kyverno
    validate:
      message: "Memory limits are required for all containers."
      pattern:
        spec:
          containers:
          - resources:
              limits:
                memory: "?*"
  - name: require-memory-requests
    match:
      any:
      - resources:
          kinds:
          - Pod
    exclude:
      any:
      - resources:
          namespaces:
          - kube-system
          - kube-public
          - kube-node-lease
          - kyverno
    validate:
      message: "Memory requests are required for all containers."
      pattern:
        spec:
          containers:
          - resources:
              requests:
                memory: "?*"
  - name: require-memory-requests-equal-limits
    match:
      any:
      - resources:
          kinds:
          - Pod
    exclude:
      any:
      - resources:
          namespaces:
          - kube-system
          - kube-public
          - kube-node-lease
          - kyverno
    validate:
      message: "Memory requests must equal memory limits."
      foreach:
      - list: "request.object.spec.containers"
        deny:
          conditions:
            any:
            - key: "\{\{ element.resources.requests.memory \}\}"
              operator: NotEquals
              value: "\{\{ element.resources.limits.memory \}\}"
  - name: require-cpu-requests
    match:
      any:
      - resources:
          kinds:
          - Pod
    exclude:
      any:
      - resources:
          namespaces:
          - kube-system
          - kube-public
          - kube-node-lease
          - kyverno
    validate:
      message: "CPU requests are required for all containers."
      pattern:
        spec:
          containers:
          - resources:
              requests:
                cpu: "?*"
  - name: disallow-cpu-limits
    match:
      any:
      - resources:
          kinds:
          - Pod
    exclude:
      any:
      - resources:
          namespaces:
          - kube-system
          - kube-public
          - kube-node-lease
          - kyverno
    validate:
      message: "CPU limits should not be set. They cause CPU throttling."
      pattern:
        spec:
          containers:
          - resources:
              limits:
                X(cpu): "?*"

정책 내용:

  • 메모리 limits 필수
  • 메모리 requests 필수
  • 메모리 requests = limits (QoS Guaranteed)
  • CPU requests 필수
  • CPU limits 금지 (CPU throttling 방지)

3. require-security-context​

컨테이너 보안 컨텍스트를 강제합니다.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-security-context
spec:
  validationFailureAction: Audit
  background: true
  rules:
  - name: require-run-as-non-root
    match:
      any:
      - resources:
          kinds:
          - Pod
    exclude:
      any:
      - resources:
          namespaces:
          - kube-system
          - kube-public
          - kube-node-lease
          - argocd
          - cert-manager
          - external-secrets
          - haproxy
          - minio
          - vault
          - zot
          - falco
          - kyverno
    validate:
      message: "Containers must set securityContext.runAsNonRoot to true."
      pattern:
        spec:
          containers:
          - securityContext:
              runAsNonRoot: true
  - name: deny-privileged-containers
    match:
      any:
      - resources:
          kinds:
          - Pod
    exclude:
      any:
      - resources:
          namespaces:
          - kube-system
          - kube-public
          - kube-node-lease
          - argocd
          - cert-manager
          - external-secrets
          - haproxy
          - minio
          - vault
          - zot
          - falco
          - kyverno
          - mas # 추가된 제외 네임스페이스
    validate:
      message: "Privileged containers are not allowed."
      deny:
        conditions:
          any:
          - key: "\{\{ request.object.spec.containers[*].securityContext.privileged || `[false]` \}\}"
            operator: AnyIn
            value: [true]
  - name: deny-privilege-escalation
    match:
      any:
      - resources:
          kinds:
          - Pod
    exclude:
      any:
      - resources:
          namespaces:
          - kube-system
          - kube-public
          - kube-node-lease
          - argocd
          - cert-manager
          - external-secrets
          - haproxy
          - minio
          - vault
          - zot
          - falco
          - kyverno
    validate:
      message: "Privilege escalation is not allowed."
      pattern:
        spec:
          containers:
          - securityContext:
              allowPrivilegeEscalation: false

보안 요구사항:

  • runAsNonRoot: true 필수
  • privileged: false 필수 (일부 시스템/인프라 네임스페이스 제외)
  • allowPrivilegeEscalation: false 필수

4. require-goldilocks-label​

VPA 연동을 위한 Goldilocks 레이블을 강제합니다.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-goldilocks-label
spec:
  admission: false
  background: true
  validationFailureAction: Audit
  rules:
  - name: check-goldilocks-label
    match:
      any:
      - resources:
          kinds: [Namespace]
    exclude:
      any:
      - resources:
          names:
          - kube-system
          - kube-public
          - kube-node-lease
          - default
    validate:
      message: "Namespace is missing goldilocks.fairwinds.com/enabled label"
      pattern:
        metadata:
          labels:
            goldilocks.fairwinds.com/enabled: "true"

5. remove-cpu-limits​

모든 컨테이너에서 CPU limits를 자동으로 제거합니다(Mutate 정책). CPU는 압축 가능 리소스로 limits 설정 시 CPU throttling이 발생하므로, Kyverno가 입력 Pod의 CPU limit 필드를 자동으로 삭제합니다.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: remove-cpu-limits
spec:
  rules:
  - name: remove-cpu-limits
    match:
      any:
      - resources:
          kinds: [Pod]
    exclude:
      any:
      - resources:
          namespaces:
          - kube-system
          - kube-public
          - kube-node-lease
    mutate:
      foreach:
      - list: "request.object.spec.containers"
        patchesJson6902: |-
          - path: /spec/containers/\{\{elementIndex\}\}/resources/limits/cpu
            op: remove

정책 예외​

특정 네임스페이스나 워크로드에 대한 정책 예외를 설정할 수 있습니다.

부트스트랩 예외​

apiVersion: kyverno.io/v2
kind: PolicyException
metadata:
  name: bootstrap-exceptions
  namespace: kyverno
spec:
  exceptions:
  - policyName: require-zot-registry
    ruleNames:
    - validate-image-registry
    - validate-init-container-registry
  - policyName: require-security-context
    ruleNames:
    - require-run-as-non-root
    - deny-privileged-containers
    - deny-privilege-escalation
  match:
    any:
    - resources:
        namespaces:
        - argocd
        - cert-manager
        - external-secrets
        - kong
        - minio
        - vault
        - zot
        - keda-system

Falco 특권 예외​

Falco는 eBPF 드라이버 로드를 위해 특권 모드가 필요하므로 예외 처리됩니다.

apiVersion: kyverno.io/v2
kind: PolicyException
metadata:
  name: falco-privileged-exception
  namespace: kyverno
spec:
  exceptions:
  - policyName: require-security-context
    ruleNames:
    - require-run-as-non-root
    - deny-privileged-containers
    - deny-privilege-escalation
  match:
    any:
    - resources:
        namespaces: [falco]
        kinds: [Pod]
  conditions:
    any:
    - key: "\{\{request.object.metadata.labels.app\}\}"
      operator: Equals
      value: falco

Tekton DinD 예외​

Tekton CI의 DinD(Docker-in-Docker) 사이드카 컨테이너는 /var/lib/docker 접근을 위해 특권 모드가 필요합니다. 이 예외는 Tekton TaskRun Pod 전체에 적용됩니다:

apiVersion: kyverno.io/v2
kind: PolicyException
metadata:
  name: tekton-dind-exception
  namespace: kyverno
spec:
  exceptions:
  - policyName: require-security-context
    ruleNames:
    - require-run-as-non-root
    - deny-privileged-containers
    - deny-privilege-escalation
  match:
    any:
    - resources:
        kinds:
        - Pod
        namespaces:
        - tekton-pipelines

CNPG 예외​

CloudNativePG 클러스터의 일부 Pod는 runAsNonRoot 정책에서 예외 처리됩니다.

apiVersion: kyverno.io/v2
kind: PolicyException
metadata:
  name: cnpg-exception
  namespace: kyverno
spec:
  exceptions:
  - policyName: require-security-context
    ruleNames:
    - require-run-as-non-root
  match:
    any:
    - resources:
        kinds:
        - Pod
  conditions:
    any:
    - key: "{{request.object.metadata.labels.cnpg.io/cluster || ''}}"
      operator: NotEquals
      value: ""

모니터링​

Prometheus 메트릭​

Kyverno는 각 컨트롤러별로 ServiceMonitor를 제공합니다:

serviceMonitor:
  enabled: true
  interval: 30s
  labels:
    release: prometheus

주요 메트릭:

  • kyverno_policy_rule_info_total: 정책 규칙 정보
  • kyverno_policy_rule_execution_duration_seconds: 정책 실행 시간
  • kyverno_admission_review_duration_seconds: Admission 검토 시간

로그 설정​

features:
  logging:
    format: json
    verbosity: 2

리소스 필터​

시스템 리소스는 정책 검증에서 제외됩니다:

config:
  excludeGroups:
    - system:nodes
  resourceFilters:
    - '[Event,*,*]'
    - '[*/*,kube-system,*]'
    - '[*/*,kube-public,*]'
    - '[*/*,kube-node-lease,*]'
    - '[Node,*,*]'
    - '[Node/*,*,*]'
    - '[APIService,*,*]'
    - '[APIService/*,*,*]'
    - '[TokenReview,*,*]'
    - '[SubjectAccessReview,*,*]'
    - '[SelfSubjectAccessReview,*,*]'
    - '[Binding,*,*]'
    - '[Pod/binding,*,*]'
    - '[ReplicaSet,*,*]'
    - '[ReplicaSet/*,*,*]' # 추가된 필터

배포 설정​

ArgoCD Application​

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: kyverno
  namespace: argocd
spec:
  project: default
  sources:
  - repoURL: https://kyverno.github.io/kyverno
    chart: kyverno
    targetRevision: 3.3.7
    helm:
      valueFiles:
      - $values/security/kyverno/helm-values.yaml
  - repoURL: https://github.com/Lumie-Edu/lumie-infra.git
    targetRevision: main
    ref: values
    path: security/kyverno
    kustomize: {}
  destination:
    server: https://kubernetes.default.svc
    namespace: kyverno
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
    - CreateNamespace=true
    - SkipDryRunOnMissingResource=true

CRD 설치​

Kyverno CRD는 별도로 설치해야 합니다:

kubectl create -f https://github.com/kyverno/kyverno/releases/download/v1.13.4/install.yaml

문제 해결​

정책 위반 확인​

# 정책 위반 리포트 확인
kubectl get cpol

# 특정 정책 상세 확인
kubectl describe cpol require-zot-registry

# 정책 위반 이벤트 확인
kubectl get events --field-selector reason=PolicyViolation

정책 테스트​

# 정책 시뮬레이션 (dry-run)
kubectl apply --dry-run=server -f test-pod.yaml

# Kyverno CLI로 정책 테스트
kyverno apply policy.yaml --resource test-pod.yaml

관련 문서​