Configuring logging in a kubernetes cluster

#loki #fluentbit #grafana #chainlink

Requirements

• helm 3

Stack used

• fluent-bit

• loki

• grafana

Introduction

The fluent-bit -> loki <- grafana stack is great if you are using prometheus, alertmanager, grafana, etc. for monitoring. Why fluent-bit and not promtail? fluent-bit is a very lightweight, high performance log collector. Out of the box it supports the main data sources for reading and sending, and has integration with kubernetes.

Configuring loki

helm repo add grafana https://grafana.github.io/helm-charts

The default loki chart is pretty heavyweight and includes prometheus operator, grafana, the loki cluster deployment, and a few other things. We will stick to configuring these services separately, as well as a single-node loki configuration to make operation and understanding easier. To do this, we need to change some default values via values.yaml

Example of a values.yaml file to configure using nfs storage

Note! We are using Basic Auth. Change your password. gateway.basicAuth.password

# disable service monitor from internal prometheus operator
serviceMonitor:
  enabled: false

# disable enterprise feature
enterprise:
  enabled: false

loki:
  auth_enabled: false

  server:
    http_listen_port: 3100
    grpc_listen_port: 9095

  podAnnotations:
    ### if you have annotations configured for prometheus, uncomment below
    # prometheus.io/scrape: "true"
    # prometheus.io/port: "3100"
    # prometheus.io/path: "metrics"
    # prometheus.io/scheme: "http"

  limits_config:
    enforce_metric_name: false
    reject_old_samples: true
    reject_old_samples_max_age: 168h
    max_cache_freshness_per_query: 10m
    split_queries_by_interval: 15m
    max_global_streams_per_user: 10000


  commonConfig:
    # using single binary loki instance, not cluster
    replication_factor: 1

  storage:
    type: "filesystem"

  readinessProbe:
    httpGet:
      path: /ready
      port: 3100
    initialDelaySeconds: 30
    timeoutSeconds: 1

gateway:
  # Enable basic auth
  basicAuth:
    enabled: true
    username: "admin"
    password: "your_password_here"

singleBinary:
  # Use nfs storage
  persistence:
    enabled: true
    type: pvc
    storageClass: ssd-nfs-storage
    accessModes:
      - ReadWriteOnce
    size: 2Gi
  # Disable cluster
  targetModule: "all"

test:
  enabled: false

# Disable internal prometheus operator
monitoring:
  dashboards:
    enabled: false

  rules:
    enabled: false
    alerting: false

  serviceMonitor:
    enabled: false

  selfMonitoring:
    enabled: false

    grafanaAgent:
      installOperator: false

  lokiCanary:
    enabled: false

Uploading the chart

helm upgrade --install loki grafana/loki \  --values ./loki/values.yaml \  --namespace YOUR_NAMESPACE

Fluentbit

Fluent Bit allows you to collect logs and metrics from multiple sources, enrich them with filters, and send them to any specific destination.

Adding a chart repository

helm repo add fluent https://fluent.github.io/helm-charts

Configuring log collection

Use the values.yaml file to modify the default chart values.

image:
  repository: cr.fluentbit.io/fluent/fluent-bit
  # Overrides the image tag whose default is {{ .Chart.AppVersion }}
  # tag: "1.9.10"
  pullPolicy: IfNotPresent

service:
  type: ClusterIP
  port: 2020
  annotations:
    ### if you have annotations configured for prometheus, uncomment below
    # prometheus.io/scrape: "true"
    # prometheus.io/port: "2020"
    # prometheus.io/path: "/api/v1/metrics/prometheus"

metricsPort: 2020

flush: 1

serviceMonitor:
  enabled: false

prometheusRule:
  enabled: false

livenessProbe:
  httpGet:
    path: /
    port: http

readinessProbe:
  httpGet:
    path: /api/v1/health
    port: http

resources:
  limits:
    cpu: 50m
    memory: 128Mi
  requests:
    cpu: 100m
    memory: 128Mi

config:
  service: |
    [SERVICE]
      Daemon Off
      Flush {{ .Values.flush }}
      Log_Level {{ .Values.logLevel }}
      Parsers_File parsers.conf
      Parsers_File custom_parsers.conf
      HTTP_Server On
      HTTP_Listen 0.0.0.0
      HTTP_Port {{ .Values.metricsPort }}
      Health_Check On

  inputs: |
    [INPUT]
      Name tail
      Tag kube.*
      Path /var/log/containers/*.log
      Mem_Buf_Limit 5MB
      Skip_Long_Lines On
      Skip_Empty_Lines Off
      Refresh_Interval 60

  filters: |
    [FILTER]
      Name parser
      Match kube.*
      Key_Name log
      Parser chainlink
      Parser cri
      Preserve_Key On
      Reserve_Data On

    [FILTER]
      Name kubernetes
      Match kube.*
      Kube_URL         https://kubernetes.default.svc:443
      Kube_CA_File     /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      Kube_Token_File  /var/run/secrets/kubernetes.io/serviceaccount/token
      Kube_Tag_Prefix kube.var.log.containers.
      Merge_Log On
      Merge_Log_Trim On
      Keep_Log Off
      tls.verify On
      K8S-Logging.Parser On
      K8S-Logging.Exclude On
      Labels On
      Annotations On
      Use_Kubelet Off

  outputs: |
    [OUTPUT]
      name                   loki
      match                  *
      host                   loki.your-namespace.svc
      port                   3100
      http_user              admin
      http_passwd            your_loki_password
      labels                 $service_logtag
      auto_kubernetes_labels on

  customParsers: |
    [PARSER]
      Name chainlink
      Format regex
      Regex ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<container_logtag>[^ ]*) (?<service_time>[^ ]+) (?<service_logtag>[^ ]*) (?<service_message>.*)$
      Time_Key time
      Time_Format %Y-%m-%dT%H:%M:%S.%L%z

    [PARSER]
      Name cri
      Format regex
      Regex ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<message>.*)$
      Time_Key time
      Time_Format %Y-%m-%dT%H:%M:%S.%L%z

podAnnotations:
  ### if you have annotations configured for prometheus, uncomment below
  # prometheus.io/scrape: "true"
  # prometheus.io/port: "2020"
  # prometheus.io/path: "/api/v1/metrics/prometheus"
  # prometheus.io/scheme: "http"

logLevel: "inf

Note!

• Change config.outputs[name=loki].loki.http_passwd to the password used for Basic Auth in the loki service

• Change config.outputs[name=loki].loki.host to match your namespace

• In the config.customParsers section, the order in which the parsers are defined matters!

For a better understanding of how log processing works in fluent-bit, visit this page

Uploading

helm upgrade --install fluentbit fluent/fluent-bit \  --values ./fluentbit/values.yaml \  --namespace YOUR_NAMESPACE

Visualizing your data in grafana

It is assumed that you have already deployed grafana.

• Create a datasource loki with the data to connect

• Create or find a dashboard to display the data

Sources

• https://docs.fluentbit.io/manual/installation/kubernetes

• https://habr.com/ru/company/otus/blog/487118/

• https://github.com/grafana/loki/blob/main/production/helm/loki/values.yaml

• https://grafana.com/docs/loki/latest/installation/helm/install-monolithic/

• https://github.com/fluent/helm-charts/blob/main/charts/fluent-bit/values.yaml#L280

• https://docs.fluentbit.io/manual/pipeline/outputs/loki#getting-started