Taming Outlier Tokens in Diffusion Transformers

Apple Machine LearningAugust 5, 2026en

Apple Machine Learning

AI Global Wire

We study outlier tokens in Diffusion Transformers (DiTs) for image generation. Prior work has shown that Vision Transformers (ViTs) can produce a small number of high-norm tokens that attract disproportionate attention while carrying limited local information, but their role in generative models remains underexplored. We show that this phenomenon appears in both the encoder and denoiser of modern Representation Autoencoder (RAE)-DiT pipelines: pretrained ViT encoders can produce outlier representations, and DiTs themselves can develop internal outlier tokens, especially in intermediate layers…

This is a short summary published by AI Global Wire. The full article is owned and hosted by Apple Machine Learning — open it there to read it in full.

• Verktyg
• Forskning
• Bild