Blog Articles Configuring logging in a kubernetes cluster #loki #fluentbit #grafana #chainlink
Requirements
Stack used
Introduction
The fluent-bit -> loki <- grafana stack is great if you are using prometheus, alertmanager, grafana, etc. for monitoring. Why fluent-bit and not promtail? fluent-bit is a very lightweight, high performance log collector. Out of the box it supports the main data sources for reading and sending, and has integration with kubernetes.
Configuring loki
Copy helm repo add grafana https://grafana.github.io/helm-charts The default loki chart is pretty heavyweight and includes prometheus operator, grafana, the loki cluster deployment, and a few other things. We will stick to configuring these services separately, as well as a single-node loki configuration to make operation and understanding easier. To do this, we need to change some default values via values.yaml
Example of a values.yaml file to configure using nfs storage
Note! We are using Basic Auth. Change your password. gateway.basicAuth.password
Copy # disable service monitor from internal prometheus operator
serviceMonitor:
enabled: false
# disable enterprise feature
enterprise:
enabled: false
loki:
auth_enabled: false
server:
http_listen_port: 3100
grpc_listen_port: 9095
podAnnotations:
### if you have annotations configured for prometheus, uncomment below
# prometheus.io/scrape: "true"
# prometheus.io/port: "3100"
# prometheus.io/path: "metrics"
# prometheus.io/scheme: "http"
limits_config:
enforce_metric_name: false
reject_old_samples: true
reject_old_samples_max_age: 168h
max_cache_freshness_per_query: 10m
split_queries_by_interval: 15m
max_global_streams_per_user: 10000
commonConfig:
# using single binary loki instance, not cluster
replication_factor: 1
storage:
type: "filesystem"
readinessProbe:
httpGet:
path: /ready
port: 3100
initialDelaySeconds: 30
timeoutSeconds: 1
gateway:
# Enable basic auth
basicAuth:
enabled: true
username: "admin"
password: "your_password_here"
singleBinary:
# Use nfs storage
persistence:
enabled: true
type: pvc
storageClass: ssd-nfs-storage
accessModes:
- ReadWriteOnce
size: 2Gi
# Disable cluster
targetModule: "all"
test:
enabled: false
# Disable internal prometheus operator
monitoring:
dashboards:
enabled: false
rules:
enabled: false
alerting: false
serviceMonitor:
enabled: false
selfMonitoring:
enabled: false
grafanaAgent:
installOperator: false
lokiCanary:
enabled: false Uploading the chart
Copy helm upgrade --install loki grafana/loki \ --values ./loki/values.yaml \ --namespace YOUR_NAMESPACE Fluentbit
Fluent Bit allows you to collect logs and metrics from multiple sources, enrich them with filters, and send them to any specific destination.
Adding a chart repository
Copy helm repo add fluent https://fluent.github.io/helm-charts Configuring log collection
Use the values.yaml file to modify the default chart values.
Copy image:
repository: cr.fluentbit.io/fluent/fluent-bit
# Overrides the image tag whose default is {{ .Chart.AppVersion }}
# tag: "1.9.10"
pullPolicy: IfNotPresent
service:
type: ClusterIP
port: 2020
annotations:
### if you have annotations configured for prometheus, uncomment below
# prometheus.io/scrape: "true"
# prometheus.io/port: "2020"
# prometheus.io/path: "/api/v1/metrics/prometheus"
metricsPort: 2020
flush: 1
serviceMonitor:
enabled: false
prometheusRule:
enabled: false
livenessProbe:
httpGet:
path: /
port: http
readinessProbe:
httpGet:
path: /api/v1/health
port: http
resources:
limits:
cpu: 50m
memory: 128Mi
requests:
cpu: 100m
memory: 128Mi
config:
service: |
[SERVICE]
Daemon Off
Flush {{ .Values.flush }}
Log_Level {{ .Values.logLevel }}
Parsers_File parsers.conf
Parsers_File custom_parsers.conf
HTTP_Server On
HTTP_Listen 0.0.0.0
HTTP_Port {{ .Values.metricsPort }}
Health_Check On
inputs: |
[INPUT]
Name tail
Tag kube.*
Path /var/log/containers/*.log
Mem_Buf_Limit 5MB
Skip_Long_Lines On
Skip_Empty_Lines Off
Refresh_Interval 60
filters: |
[FILTER]
Name parser
Match kube.*
Key_Name log
Parser chainlink
Parser cri
Preserve_Key On
Reserve_Data On
[FILTER]
Name kubernetes
Match kube.*
Kube_URL https://kubernetes.default.svc:443
Kube_CA_File /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
Kube_Token_File /var/run/secrets/kubernetes.io/serviceaccount/token
Kube_Tag_Prefix kube.var.log.containers.
Merge_Log On
Merge_Log_Trim On
Keep_Log Off
tls.verify On
K8S-Logging.Parser On
K8S-Logging.Exclude On
Labels On
Annotations On
Use_Kubelet Off
outputs: |
[OUTPUT]
name loki
match *
host loki.your-namespace.svc
port 3100
http_user admin
http_passwd your_loki_password
labels $service_logtag
auto_kubernetes_labels on
customParsers: |
[PARSER]
Name chainlink
Format regex
Regex ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<container_logtag>[^ ]*) (?<service_time>[^ ]+) (?<service_logtag>[^ ]*) (?<service_message>.*)$
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L%z
[PARSER]
Name cri
Format regex
Regex ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<message>.*)$
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L%z
podAnnotations:
### if you have annotations configured for prometheus, uncomment below
# prometheus.io/scrape: "true"
# prometheus.io/port: "2020"
# prometheus.io/path: "/api/v1/metrics/prometheus"
# prometheus.io/scheme: "http"
logLevel: "inf Note!
Change config.outputs[name=loki].loki.http_passwd to the password used for Basic Auth in the loki service
Change config.outputs[name=loki].loki.host to match your namespace
In the config.customParsers section, the order in which the parsers are defined matters!
For a better understanding of how log processing works in fluent-bit, visit this page
Uploading
Copy helm upgrade --install fluentbit fluent/fluent-bit \ --values ./fluentbit/values.yaml \ --namespace YOUR_NAMESPACE Visualizing your data in grafana
It is assumed that you have already deployed grafana.
Create a datasource loki with the data to connect
Create or find a dashboard to display the data
Sources
