Spiral RoPE: Rotate Your Rotary Positional Embeddings in the 2D Plane
Published in International Conference on Machine Learning (ICML), 2026
Spiral RoPE extends the standard axial 2D Rotary Position Embedding in vision transformers beyond horizontal and vertical directions. Embedding channels are partitioned into groups associated with uniformly distributed directions, and each group is rotated according to the projection of the patch position onto its direction. Spiral RoPE consistently improves performance across classification, segmentation, and generation tasks.
Authors: Haoyu Liu, Sucheng Ren, Tingyu Zhu, Peng Wang, Cihang Xie, Alan Yuille, Zeyu Zheng, Feng Wang