Abstract

Vision Transformer models, such as ViT, Swin Transformer, and Transformer-in-Transformer, have recently gained significant traction in computer vision tasks due to their ability to capture the global relation between features which leads to superior performance. However, they are compute-heavy and difficult to deploy in resource-constrained edge devices. Existing hardware accelerators, including those for the closely-related BERT transformer models, do not target highly resource-constrained environments. In this paper, we address this gap and propose ViTA - a configurable hardware accelerator for inference of vision transformer models, targeting resource-constrained edge computing devices and avoiding repeated off-chip memory accesses. We employ a head-level pipeline and inter-layer MLP optimizations, and can support several commonly used vision transformer models with changes solely in our control …

Date

May 21, 2023

Authors

Shashank Nag, Gourav Datta, Souvik Kundu, Nitin Chandrachoodan, Peter A Beerel

Conference

2023 IEEE International Symposium on Circuits and Systems (ISCAS)

Pages

1-5

Publisher

IEEE