Databricks releases UCDataset beta to stream Unity Catalog tables to GPU training
Overview
Databricks releases UCDataset in Beta on AI Runtime, a PyTorch data loader that streams Unity Catalog tables directly to every GPU in a training job without exports or copies.
Databricks reports an average 10x training-time improvement over reading an exported Parquet copy, measured on a simulated 300 GB recommendation workload across 8 NVIDIA H100 GPUs.
The Beta reads Delta tables only. It cannot yet read tables with row filters or column masks.
Written by AI from the articles below · updated Nov 9, 10:34 AM ET
Check the sources:
Article timeline
The articles in this story. Times are ET.
- Databricks BlogOfficialDatabricks releases UCDataset in Beta to stream Unity Catalog tables to GPUs
AIDatabricks releases UCDataset in Beta on AI Runtime, a PyTorch data loader that streams Unity Catalog tables directly to every GPU in a training job without exports or copies. On a simulated 300 GB recommendation workload across 8 NVIDIA H100 GPUs, Databricks reports an average 10x training-time improvement over reading an exported Parquet copy. The Beta reads Delta tables and cannot yet read tables with row filters or column masks.
Heat Trend
Not enough continuous observations to show a trend yet.