Apache Spark 4.3 adds stateful real-time processing and quarterly releases
Overview
Apache Spark 4.3 extends Real-Time Mode to stateful workloads, with P99 end-to-end latency below 150 ms in three tested benchmarks, according to a Databricks blog post.
The release also adds SCD Type 2 Auto CDC, Spark Connect observability features, and a nanosecond timestamp preview, and Spark moves to quarterly feature releases.
The single-task optimization and Arrow cache are opt-in. Auto CDC applies each micro-batch to the target using MERGE, so the target table must be in a catalog whose connector implements Data Source V2 row-level operations. The timestamp preview supports TIMESTAMP_NTZ(p) and TIMESTAMP_LTZ(p) at precisions from 7 through 9.
Spark 4.3 also adds a SQL ASOF JOIN, which the post describes as a preview feature where breaking changes may happen.
Written by AI from the articles below · updated Nov 4, 7:52 PM ET
Check the sources:
Article timeline
The articles in this story. Times are ET.
- Databricks BlogOfficialPickApache Spark 4.3 adds stateful real-time processing and a quarterly release cadence
AIApache Spark 4.3 extends Real-Time Mode to stateful workloads, with P99 end-to-end latency below 150 ms in three tested benchmarks. The release also adds SCD Type 2 Auto CDC, Spark Connect observability features, and a nanosecond timestamp preview, and Spark moves to quarterly feature releases.
Heat Trend
Not enough continuous observations to show a trend yet.