Kubernetes host logs, journals, and logfiles

Kubernetes host tailer allows you to tail logs like kubelet, audit logs, or the systemd journal from the nodes.

Host-tailer

Create host tailer

To tail logs from the node’s host filesystem, define one or more file tailers in the host-tailer configuration.

kubectl apply -f - <<EOF
apiVersion: logging-extensions.banzaicloud.io/v1alpha1
kind: HostTailer
metadata:
  name: multi-sample
spec:
  # list of File tailers
  fileTailers:
    - name: nginx-access
      path: /var/log/nginx/access.log
    - name: nginx-error
      path: /var/log/nginx/error.log
  # list of Systemd tailers
  systemdTailers:
    - name: my-systemd-tailer
      maxEntries: 100
      systemdFilter: kubelet.service
EOF

Create file tailer

When an application (mostly legacy programs) is not logging in a Kubernetes-native way, Logging operator cannot process its logs. (For example, an old application does not send its logs to stdout, but uses some log files instead.) File-tailer helps to solve this problem: It configures Fluent Bit to tail the given file(s), and sends the logs to the stdout, to implement Kubernetes-native logging.

Host-tailer

However, file-tailer cannot access the pod’s local dir, so the logfiles need to be written on a mounted volume.

Let’s assume the following code represents a legacy application that generates logs into the /legacy-logs/date.log file. While the legacy-logs directory is mounted, it’s accessible from other pods by mounting the same volume.

kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: test-pod
spec:
  containers:
  - image: busybox
    name: test
    volumeMounts:
    - mountPath: /legacy-logs
      name: test-volume
    command: ["/bin/sh", "-c"]
    args:
      - while true; do
          date >> /legacy-logs/date.log;
          sleep 1;
        done
  volumes:
  - name: test-volume
    hostPath:
      path: /legacy-logs
EOF

To tail the logs of the previous example application, you can use the following host-tailer custom resource:

kubectl apply -f - <<EOF
apiVersion: logging-extensions.banzaicloud.io/v1alpha1
kind: HostTailer
metadata:
  name: file-hosttailer-sample
spec:
  fileTailers:
    - name: sample-logfile
      path: /legacy-logs/date.log
      disabled: false
EOF

Logging operator configure the environment and start a file-tailer pod. It’s also able to deal with multi-node clusters, since is starts the host-tailer pod through a daemonset.

Check the created file tailer pod:

kubectl get pod

The output should be similar to:

NAME                                       READY   STATUS    RESTARTS   AGE
file-hosttailer-sample-host-tailer-5tqhv   1/1     Running   0          117s
test-pod                                   1/1     Running   0          5m40s

Checking the logs of the file-tailer's pod. You will see the logfile’s content on stdout. This way Logging operator can process those logs as well.

kubectl logs file-hosttailer-sample-host-tailer-5tqhv

The logs of the sample application should be similar to:

Fluent Bit v1.9.5
* Copyright (C) 2015-2022 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2022/09/13 12:26:02] [ info] [fluent bit] version=1.9.5, commit=9ec43447b6, pid=1
[2022/09/13 12:26:02] [ info] [storage] version=1.2.0, type=memory-only, sync=normal, checksum=disabled, max_chunks_up=128
[2022/09/13 12:26:02] [ info] [cmetrics] version=0.3.4
[2022/09/13 12:26:02] [ info] [sp] stream processor started
[2022/09/13 12:26:02] [ info] [output:file:file.0] worker #0 started
[2022/09/13 12:26:02] [ info] [input:tail:tail.0] inotify_fs_add(): inode=418051 watch_fd=1 name=/legacy-logs/date.log
Tue Sep 13 12:22:51 UTC 2022
Tue Sep 13 12:22:52 UTC 2022
Tue Sep 13 12:22:53 UTC 2022
Tue Sep 13 12:22:54 UTC 2022
Tue Sep 13 12:22:55 UTC 2022
Tue Sep 13 12:22:56 UTC 2022

File Tailer configuration options

Variable NameTypeRequiredDefaultDescription
namestringYes-Name for the tailer
pathstringNo-Path to the loggable file
disabledboolNo-Disable tailing the file
containerOverrides*types.ContainerBaseNo-Override container fields for the given tailer

Tail systemd journal

This is a special case of file-tailer, since it tails the systemd journal file specifically.

kubectl apply -f - <<EOF
apiVersion: logging-extensions.banzaicloud.io/v1alpha1
kind: HostTailer
metadata:
  name: systemd-tailer-sample
spec:
  # list of Systemd tailers
  systemdTailers:
    - name: my-systemd-tailer
      maxEntries: 100
      systemdFilter: kubelet.service
EOF

Systemd tailer configuration options

Variable NameTypeRequiredDefaultDescription
namestringYes-Name for the tailer
pathstringNo-Override systemd log path
disabledboolNo-Disable component
systemdFilterstringNo-Filter to select systemd unit example: kubelet.service
maxEntriesintNo-Maximum entries to read when starting to tail logs to avoid high pressure
containerOverrides*types.ContainerBaseNo-Override container fields for the given tailer

Example: Configure logging Flow to route logs from a host tailer

The following example uses the flow’s match term to listen the previously created file-hosttailer-sample Hosttailer’s log.

kubectl apply -f - <<EOF
apiVersion: logging.banzaicloud.io/v1beta1
kind: Flow
metadata:
  name: hosttailer-flow
  namespace: default
spec:
  filters:
  - tag_normaliser: {}
  # keeps data matching to label, the rest of the data will be discarded by this flow implicitly
  match:
  - select:
      labels: 
        app.kubernetes.io/name: file-hosttailer-sample
      # there might be a need to match on container name too (in case of multiple containers)
      container_names:
        - nginx-access
  outputRefs:
    - sample-output
EOF

Example: Kubernetes host tailer with multiple tailers

kubectl apply -f - <<EOF
apiVersion: logging-extensions.banzaicloud.io/v1alpha1
kind: HostTailer
metadata:
  name: multi-sample
spec:
  # list of File tailers
  fileTailers:
    - name: nginx-access
      path: /var/log/nginx/access.log
    - name: nginx-error
      path: /var/log/nginx/error.log
  # list of Systemd tailers
  systemdTailers:
    - name: my-systemd-tailer
      maxEntries: 100
      systemdFilter: kubelet.service
EOF

Set custom priority

Create your own custom priority class in Kubernetes. Set its value between 0 and 2000000000. Note that:

  • 0 is the default priority
  • To change the default priority, set the globalDefault key.
  • 2000000000 and above are reserved for the Kubernetes system
  • PriorityClass is a non-namespaced object.
kubectl apply -f - <<EOF
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: hosttailer-priority
value: 1000000
globalDefault: false
description: "This priority class should be used for hosttailer pods only."
EOF

Now you can use your private priority class name to start hosttailer/eventtailer, for example:

kubectl apply -f -<<EOF
apiVersion: logging-extensions.banzaicloud.io/v1alpha1
kind: HostTailer
metadata:
  name: priority-sample
spec:
  controlNamespace: default
  # Override podSpecBase variables here
  workloadOverrides:
    priorityClassName: hosttailer-priority
  fileTailers:
    - name: nginx-access
      path: /var/log/nginx/access.log
    - name: nginx-error
      path: /var/log/nginx/error.log
EOF

Configuration options

Variable NameTypeRequiredDefaultDescription
fileTailers[]FileTailerNo-List of file tailers
c[]SystemdTailerNo-List of systemd tailers
enableRecreateWorkloadOnImmutableFieldChangeboolNo-EnableRecreateWorkloadOnImmutableFieldChange enables the operator to recreate the
fluentbit daemonset and the fluentd statefulset (and possibly other resource in the future)
in case there is a change in an immutable field
that otherwise couldn’t be managed with a simple update.
workloadMetaOverrides*types.MetaBaseNo-Override metadata of the created resources
workloadOverrides*types.PodSpecBaseNo-Override podSpec fields for the given daemonset

Advanced configuration overrides

MetaBase

Variable NameTypeRequiredDefaultDescription
annotationsmap[string]stringNo-
labelsmap[string]stringNo-

PodSpecBase

Variable NameTypeRequiredDefaultDescription
tolerations[]corev1.TolerationNo-
nodeSelectormap[string]stringNo-
serviceAccountNamestringNo-
affinity*corev1.AffinityNo-
securityContext*corev1.PodSecurityContextNo-
volumes[]corev1.VolumeNo-
priorityClassNamestringNo-

ContainerBase

Variable NameTypeRequiredDefaultDescription
resources*corev1.ResourceRequirementsNo-
imagestringNo-
pullPolicycorev1.PullPolicyNo-
command[]stringNo-
volumeMounts[]corev1.VolumeMountNo-
securityContext*corev1.SecurityContextNo-