Provable Repair of Vision Transformers

Nawas, Stephanie and Zhe, Tao and Thakur, Aditya V.
International Symposium on AI Verification (SAIV), 2024

Vision Transformers have emerged as state-of-the-art image recognition tools, but may still exhibit incorrect behavior. Incorrect image recognition can have disastrous consequences in safety-critical real-world applications such as self-driving automobiles. In this paper, we present Provable Repair of Vision Transformers (PRoViT), a provable repair approach that guarantees the correct classification of images in a repair set for a given Vision Transformer without modifying its architecture. PRoViT avoids negatively affecting correctly classified images (drawdown) by minimizing the changes made to the Vision Transformer’s parameters and original output. We observe that for Vision Transformers, unlike for other architectures such as ResNet or VGG, editing just the parameters in the last layer achieves correctness guarantees and very low drawdown. We introduce a novel method for editing these last-layer parameters that enables PRoViT to efficiently repair state-of-the-art Vision Transformers for thousands of images, far exceeding the capabilities of prior provable repair approaches.

@inproceedings{SAIV2024,
  author = {Nawas, Stephanie and Zhe, Tao and Thakur, Aditya V.},
  title = {Provable Repair of Vision Transformers},
  year = {2024},
  publisher = {Springer, Cham},
  url = {https://doi.org/10.1007/978-3-031-65112-0_8},
  doi = {10.1007/978-3-031-65112-0_8},
  booktitle = {International Symposium on AI Verification {(SAIV)}}
}