Databricks releases UCDataset in Beta to stream Unity Catalog tables to GPUs
AIDatabricks releases UCDataset in Beta on AI Runtime, a PyTorch data loader that streams Unity Catalog tables directly to every GPU in a training job without exports or copies. On a simulated 300 GB recommendation workload across 8 NVIDIA H100 GPUs, Databricks reports an average 10x training-time improvement over reading an exported Parquet copy.
Source details
Databricks Blog · databricks.comPublished · added here
Original title: Accelerate GPU training on Unity Catalog tables with UCDataset