PyTorch 2.13.0: FlexAttention Comes to Apple Silicon

PyTorch 2.13.0 shipped in July 2026 with FlexAttention support on Apple Silicon, a deterministic backward path for FlexAttention on CUDA, and an updated FSDP2 distributed-training layer — changes that sit underneath most transformer workloads, not on top of them.

FlexAttention reaches Apple Silicon through MPS

The release adds FlexAttention support on Apple Silicon through the MPS backend. Attention is the core computation behind transformer-based large language models, multimodal systems, and agents, so a new execution backend for it is not a cosmetic addition.

Teams that develop on Mac hardware but deploy to NVIDIA infrastructure now have a local path to run FlexAttention during development. That is not the same as a correctness guarantee: a model that passes on CUDA is not automatically confirmed to match on MPS without an explicit cross-backend regression test comparing the two outputs directly.

A deterministic CUDA backward pass, and why it is a testing issue

PyTorch 2.13.0 also introduces a deterministic backward path for FlexAttention on CUDA. Reproducibility is usually treated as a research-only concern, but it has a direct testing consequence: if the same training step produces different results across identical runs with the same seed, assertions inside an automated test suite stop being reliable.

The execution stack got another layer

A PyTorch 2.13.0 workload now passes through more stages before it reaches silicon: the model layer, autograd and the graph, TorchInductor, the Triton/CuTeDSL kernel layer, and finally CUDA or MPS. Each added layer is a place a regression can hide, which is why the release notes push developers toward asking which part of a pipeline changes behavior, rather than whether the upgrade installs cleanly.

Correctness and performance are separate checks on that stack, and passing one says nothing about the other. As an illustrative example (not measured PyTorch benchmark data): two backends can both pass a correctness test while one runs at double the latency of the other — a functional pass that hides a real performance regression.

What changed for distributed training

The release also updates FSDP2, the layer distributed workloads use to coordinate a model across multiple GPUs over the network through torchcomms. Paired with the attention and compilation changes, the real upgrade question for a team is not whether import torch still succeeds, but which specific parts of an existing pipeline could now behave differently on the new stack.

Before you upgrade

• Run a cross-backend regression test — compare CPU/CUDA output against MPS output on your actual model — before moving a Mac-based dev environment onto MPS.
• If your test suite assumes bit-identical outputs across runs, check whether the new deterministic CUDA backward path changes any existing tolerance thresholds.
• Validate FSDP2-based multi-GPU jobs separately from single-GPU correctness checks — treat it as a distributed-training upgrade, not a drop-in replacement.