TPUs have different bottlenecks from GPUs: few cores so you don’t need yo think about parallelism as much, large VMEM so easier to fuse kernels, but vector units have lower throughput so anything non matmul require much more care
Tri Dao compares TPU and GPU bottlenecks for kernel design
AISummary
Tri Dao says TPUs have few cores, so parallelism matters less, and large VMEM makes kernel fusion easier. He adds that TPU vector units have lower throughput, so non-matmul operations need much more care.
Post on XView on X
Tri DaoVerified on X
@tri_dao
Releasing FLAT: Flash Linear Attention on TPUs 🚀 Pallas training kernels for Linear Attention, Mamba-2 and GLA. On one TPU v6e, forward + backward is up to 3x faster than FLA on an H100 for GLA, and 1.5x faster for Mamba-2. 🧵
Source: Tri Dao · x.comPublished · added here