Fluentd is a CNCF-graduated log collector whose job is to gather events from many sources, transform and route them, and ship them to many destinations - without coupling the application to any one logging backend. It treats every event as a tagged triple of timestamp, tag, and JSON record, and the tag is what drives routing. An application logs to a file or a socket, Fluentd picks the events up, attaches a tag, and matches that tag against output rules. Switching from Elasticsearch to S3 to Kafka becomes a config change, not an application change.

Sources, matches, and the unified log layer

A Fluentd config is a set of plugin blocks. A <source> defines where events come in and assigns a tag; a <match> selects events by tag pattern and sends them to an output. Between them, <filter> blocks parse, enrich, or redact records. This source-to-match pipeline is what people mean by Fluentd as a "unified logging layer" - one place that all logs flow through and out of.

<source>
  @type tail
  path /var/log/app/*.log
  pos_file /var/log/fluentd/app.pos
  tag app.access
  <parse>
    @type json
  </parse>
</source>

<filter app.**>
  @type record_transformer
  <record>
    environment "#{ENV['DEPLOY_ENV']}"
  </record>
</filter>

<match app.**>
  @type elasticsearch
  host logs.internal
  port 9200
  logstash_format true
  <buffer>
    flush_interval 5s
    retry_max_times 10
  </buffer>
</match>

Fluentd vs. Fluent Bit

The same project ships a lighter sibling, Fluent Bit, written in C with a much smaller memory footprint. The common pattern in Kubernetes is Fluent Bit as the per-node DaemonSet that tails container logs cheaply, forwarding to a smaller pool of Fluentd aggregators that do the heavier parsing, buffering, and fan-out to multiple backends. Buffering is the reliability mechanism in both: events queue to memory or disk and retry on backend failure, so a downed Elasticsearch does not drop logs.

Frequently asked questions

What problem does Fluentd solve? It decouples applications from logging backends - apps emit logs once, and Fluentd routes them anywhere by tag, so changing or adding a destination is a config edit, not a code change.

What is the difference between source, filter, and match? <source> ingests and tags events, <filter> transforms or enriches them, and <match> routes tagged events to an output plugin.

When should I use Fluent Bit instead? When footprint matters - as a lightweight per-node collector. A common setup pairs Fluent Bit collectors with Fluentd aggregators.

How does Fluentd avoid losing logs? Output plugins buffer events to memory or disk and retry on failure, so a temporarily unavailable destination causes a queue, not a drop.

See the monitoring & logging overview and compare with Grafana Loki.