AWS has announced the general availability of AWS Glue 6.0, a fully managed ETL service update that comes with a 30% price reduction compared to earlier versions and complete support for Apache Iceberg v3 features. The new release is built on a modernized runtime including Apache Spark 4.1, Python 3.12, and Scala 2.13, which the company says delivers faster performance.

According to AWS, Glue 6.0 now offers the most complete Iceberg v3 implementation on any fully serverless managed Spark service. A key addition is the VARIANT data type with shredding support, which allows users to store and query semi-structured data like JSON, logs, and event data without flattening schemas. This removes the need for duplicate data copies, custom parsing code, and prevents pipeline breaks when schemas change.

Other Iceberg v3 capabilities include geometry and geography data types for spatial processing, nanosecond-precision timestamps for high-frequency workloads, and handling of unknown data types to withstand upstream schema changes without pipeline failures.

The runtime upgrade to Spark 4.1 brings several improvements: declarative pipelines that let engineers specify desired transformations while the engine handles execution order and optimization, Arrow-native Python UDFs and UDTFs that reduce serialization overhead for better PySpark performance, and a real-time streaming mode achieving single-digit millisecond latency for stateless processing.

No API changes are required to adopt Glue 6.0. Users can select the new version through the existing --glue-version parameter in create-job or update-job APIs via CLI, SDK, or AWS Glue Studio. Existing jobs can be migrated using the Spark upgrade agent or an auto-upgrade feature. AWS Glue 6.0 is now generally available in all AWS Regions where Glue operates, with pricing billed per second for crawlers and ETL jobs, plus a monthly fee for Data Catalog metadata storage and access.

Source: AWS