JSON (JavaScript Object Notation) is the most common text format for exchanging structured data. It represents records as nested objects of key-value pairs and arrays, is human-readable, and is supported by every language and API. In data engineering, JSON is the format data usually arrives in - from REST APIs, webhooks, and application logs - rather than the format you store it in for analytics.

Why it dominates ingestion

JSON's flexibility is its appeal: no schema to declare up front, nested structures map cleanly to application objects, and producers can add fields without breaking older consumers. That makes it the default for APIs and event payloads. The same flexibility is its weakness for analytics - there is no enforced schema, types are loose (a field can be a number in one record and a string in the next), and the format is verbose.

JSON Lines for data pipelines

A single JSON array of millions of records can't be streamed or split - you must parse the whole thing. The pipeline-friendly variant is JSON Lines (NDJSON): one JSON object per line. It is splittable, append-friendly, and streamable, which is why log shippers and bulk-load APIs use it.

{"id": 1, "event": "click", "ts": "2026-06-01T10:00:00Z"}
{"id": 2, "event": "view",  "ts": "2026-06-01T10:00:01Z"}

The cost of querying JSON at scale

Storing analytic data as JSON is slow and expensive: it is uncompressed text, the entire record must be read and parsed even to access one field, and there are no column statistics to skip data. A query over a terabyte of JSON logs reads and parses the full terabyte. The standard pattern is therefore to land JSON, then convert to Parquet - flatten the fields you query into columns and keep the rare/variable parts in a semi-structured column.

# Land raw, convert once to columnar
raw = spark.read.json("s3://lake/raw/events/*.jsonl")
raw.write.partitionBy("event_date").parquet("s3://lake/events_parquet/")

Semi-structured columns

Modern warehouses meet JSON halfway with native semi-structured types - Snowflake's VARIANT, BigQuery's JSON type, Postgres JSONB. These store JSON in an optimized, partly-columnar form and let you query nested fields with path syntax while still indexing or pruning. They are the right home for genuinely variable payloads; for fields you query constantly, promote them to real typed columns for speed.

-- Snowflake: query a field inside a VARIANT column
SELECT payload:user.id::string AS user_id, count(*)
FROM raw_events
GROUP BY 1;

Schema drift, the real operational risk

Because JSON has no enforced schema, upstream producers change payloads silently - a renamed field, a string that becomes an object, a new nesting level. Pipelines that assume a fixed shape break or silently drop data. Defend against it: validate against a JSON Schema at ingestion, store the raw payload alongside the parsed columns so nothing is lost, and alert on parse-failure rates.

Practical guidance

• Use JSON/NDJSON for ingestion and APIs; don't keep it as your analytic storage layer.
• Convert to Parquet (or load into a warehouse) early in the pipeline.
• Promote frequently-queried fields to typed columns; keep variable parts in a VARIANT/JSONB column.
• Validate schema at the boundary and retain raw payloads for reprocessing.