Cluster Logging Architecture

We define 3 logging categories

The logging system breaks down into 4 logical components:

The cluster logging operator (CLO) implements the following custom resources:

The elasticsearch logging operator (ELO) implements the following custom resources:

The container runtime interface (CRI-O) on each node writes container logs to files. The file names include the container’s UID, namespace, name and other data. We also collect per-node logs from the Linux journald.

The CLO deploys a Fluentd daemon on each node which acts both as a collector (reading log files) and as a forwarder (sending log data to configured outputs)

The term log is overloaded so we’ll use these terms to clarify:

Kubernetes does not enforce a uniform format for logs. Anything that a containerized process writes to stdout or stderr is considered a log. This "lowest common denominator" approach allows pre-existing applications to run on the cluster.

Traditional log formats write entries as ordered fields, but the order, field separator, format and meaning of fields varies.

Structured logs write log entries as JSON objects on a single line. However names, types, and meaning of fields in the JSON object varies between applications.

The Kubernetes Structured Logging proposal will standardize the log format for some k8s components, but there will still be diverse log formats from non-k8s applications running on the cluster.

Status: Needs write-up as enhancment proposal(s)

TODO: Updated diagram - CRIO to fluentd end-to-end.

Goal: 1) Sustain a predictable average load without log loss up to retention limits. 2) Handle temporary load spikes predictably: drop or back-pressure. 3) Handle long-term overload with alerts and predictable log loss at source.

Problems now: - Uncontrolled (file-at-a-time) log loss from slow collection + node log rotation. - Large back-up in file buffers under load: very high latencies, slow recovery.

Propose 2 qualities of service: - fast: priority is low latency, high throughput, no effect on apps. May drop data. n- reliable: Priority is to avoid data loss, but may allow loss outside of defined limits. May slow application progress.

In traditional terminology: - fast: at-most-once - reliable: at-least-once with limits. Even users who want reliable logging may have a breaking point where they’d rather let the application progress and lose logs. We may need configurable limits on how hard we try to be reliable.

Architecture: - conmon writes container log files on node, log rotation (retention) - fluentd on node: file vs. memory buffers - forwarder target: throughput - store target: retention - Future: separate normalizer/forwarder, fluentbit/fluentd

Must consider data loss by forwarding protocol also: - store (elasticsearch) review options. - fluent-forward need to enable at-least-once acks (we don’t) - others need to review case by case if its possible.

Throughput and latency: - evaluate throughput of each stage: node log to store/target. - end-to-end latency, expected/acceptable variation.

Buffer sizes - all components must maintain bounded buffers. - without end-to-end back-pressure we cannot guarantee no data loss. - we should be able to give better sizing/capacity guidelines.

Configuration: - Enable backpressure by pod label and/or namespace. - Can’t impose backpressure everywhere? - Enable rate limiting in low-latency mode (back-pressure always limits rate)

The logging system itself can encounter errors that need to be diagnosed, examples: * Invalid JSON received where structured logs are reuqired. * Hard (no retry possible) errors from store or other target causing unavoidable log loss.

Alerts are a key component, but alerts must be actionable, they can’t be used to record ongoing activity that might or might not be reviewed later. For that we need logs.

The CLO and fluentd collector logs can be captured just like any other infrastructure log. However, if the logging system itself is in trouble, users need a simple, direct path to diagnose the issue. This path might have a simpler implementation that is more likely to survive if logging is in trouble.

Proposal: add a 4th logging category [application, infrastructure, audit, logging] This category collects logs related to errors in the logging system, including fluentd error messages and errors logged by the CLO.

Decide on the supported set of envelope metadata fields and document them.

Some of our format decisions are specifically for elasticsearch (e.g. flattening maps to lists) We need to separate the ES-specifics, either:

Optionally copy metadata copied to syslog STRUCTURED-DATA

TODO