Grafana is an open-source dashboarding and alerting tool built for time-series and operational data. In a data engineering stack it sits in front of Prometheus, InfluxDB, Graphite, and increasingly SQL warehouses, turning metrics into panels that answer "is this pipeline healthy and how does it compare to last week." It is the default choice for monitoring rather than business reporting: the unit of work is a metric over time, not a modeled fact table.
Data sources and how queries work
Grafana does not store data. Each dashboard panel issues a query to a configured data source, and the source's native query language is used directly. Against Prometheus you write PromQL; against InfluxDB, Flux or InfluxQL; against Postgres or a warehouse, plain SQL with time macros that Grafana expands to the dashboard's selected range.
# Prometheus panel: 95th percentile Airflow task duration over 5m windows
histogram_quantile(0.95,
sum(rate(airflow_task_duration_seconds_bucket[5m])) by (le, dag_id)
)A single dashboard can mix sources, so one panel can chart Kafka consumer lag from Prometheus next to a panel querying warehouse row counts in SQL. Variables (template variables) let a viewer switch DAG, environment, or cluster without editing the query.
Common data-engineering dashboards
Teams use Grafana to watch the operational health of pipelines and infrastructure that feed the warehouse. Typical panels:
| Signal | Source | Why it matters |
|---|---|---|
| Airflow task duration / failure rate | Prometheus (statsd exporter) | Catch slow or failing DAGs before SLAs slip |
| Kafka consumer lag | Prometheus (kafka-exporter) | Detect consumers falling behind producers |
| Warehouse query queue / credits | SQL or vendor exporter | Spot runaway queries and cost spikes |
| Ingest rows-per-minute | Postgres / warehouse SQL | Confirm pipelines are still delivering data |
Alerting and provisioning as code
Grafana's unified alerting evaluates a query on a schedule and fires to Slack, PagerDuty, or email when a threshold or absence-of-data condition holds. An alert on "no rows ingested in the last hour" catches a silently dead pipeline that an error-free log would miss. Dashboards, data sources, and alert rules can be provisioned from YAML or JSON checked into git, so the monitoring layer is reproducible alongside the rest of the infrastructure rather than clicked together by hand.
Frequently asked questions
Is Grafana a BI tool? Not primarily. It excels at time-series monitoring and alerting. For ad-hoc analyst exploration over modeled warehouse tables, a BI tool like Looker, Metabase, or Mode fits better, though Grafana can query SQL sources for simple operational reports.
Does Grafana store the data it charts? No. It is a stateless query and visualization layer; data lives in the configured sources (Prometheus, InfluxDB, a warehouse). Grafana's own database only holds dashboards, users, and alert state.
Grafana or Kibana for logs? Kibana is purpose-built for searching log documents in Elasticsearch. Grafana can display logs via Loki and is the better choice when you want logs and metrics on one dashboard, but for deep full-text log search Kibana is stronger.
How do alerts avoid noise? Use "for" durations so a condition must hold for several minutes before firing, group related alerts, and add no-data and error handling so flaky scrapes do not page anyone unnecessarily.