FlashAttention-3 Released: Achieves Unprecedented Speed and Precision with Advanced Hardware Utilization and Low-Precision Computing

FlashAttention-3 aims to improve the performance of attention mechanisms in Transformers by minimizing memory reads and writes, optimizing computation and data movement, and leveraging low-precision computations. It achieves 1.5 to 2 times faster speed than FlashAttention-2 with FP16 and reaches up to 740 TFLOPS, 75% of the theoretical maximum FLOPs on H100 GPUs. With FP8, it achieves close to 1.2 PFLOPS, reducing error by 2.6 times compared to baseline FP8 attention.

FlashAttention-3 utilizes modern GPU capabilities by exploiting the asynchrony of Tensor Cores and TMA to overlap computation and data movement, interleaving block-wise matrix multiplication and softmax operations, and leveraging incoherent processing for FP8 low-precision computations3. This allows for efficient data transfer and processing, maximizing computational throughput and accuracy while reducing memory footprint and operational costs for large-scale AI deployments.