Spline (from "Spark Lineage") is a tool for capturing data lineage automatically from Apache Spark applications. It hooks into Spark's execution and records how each output dataset was derived from its inputs - down to the transformations involved - without requiring developers to annotate their jobs. For Spark-heavy platforms, it answers "how was this table produced?" automatically.

How it works

Spline attaches an agent to Spark that intercepts the logical plan of each job as it runs. From that plan it reconstructs lineage: which sources were read, which transformations were applied, and which outputs were written, including attribute-level detail. The lineage is sent to a Spline server backed by a graph store, with a UI to explore it. Because it reads Spark's own plan, the lineage reflects what the job actually did.

What it captures

• Job-level lineage - inputs, outputs, and the chain of transformations between them.
• Attribute-level detail - how output columns trace to input columns through the plan.
• Execution history of Spark runs.

Where it fits

• Spark-centric data platforms needing automatic, code-free lineage for Spark pipelines.
• Auditing and impact analysis where Spark transformations are the main processing layer.

How it compares

Spline is specialized for Spark, capturing richer Spark-internal detail than a generic collector. The broader, vendor-neutral approach is OpenLineage, which has a Spark integration and feeds general backends like Marquez and catalogs. Teams standardizing lineage across many tools tend toward OpenLineage; those wanting deep, Spark-specific lineage sometimes use Spline. Full catalogs (DataHub, OpenMetadata) then consume lineage for discovery and governance.