@inproceedings{zhao-etal-2022-fine,
    title = "Fine- and Coarse-Granularity Hybrid Self-Attention for Efficient {BERT}",
    author = "Zhao, Jing  and
      Wang, Yifan  and
      Bao, Junwei  and
      Wu, Youzheng  and
      He, Xiaodong",
    editor = "Muresan, Smaranda  and
      Nakov, Preslav  and
      Villavicencio, Aline",
    booktitle = "Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = may,
    year = "2022",
    address = "Dublin, Ireland",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-emnlp/2022.acl-long.330/",
    doi = "10.18653/v1/2022.acl-long.330",
    pages = "4811--4820",
    abstract = "Transformer-based pre-trained models, such as BERT, have shown extraordinary success in achieving state-of-the-art results in many natural language processing applications. However, deploying these models can be prohibitively costly, as the standard self-attention mechanism of the Transformer suffers from quadratic computational cost in the input sequence length. To confront this, we propose FCA, a fine- and coarse-granularity hybrid self-attention that reduces the computation cost through progressively shortening the computational sequence length in self-attention. Specifically, FCA conducts an attention-based scoring strategy to determine the informativeness of tokens at each layer. Then, the informative tokens serve as the fine-granularity computing units in self-attention and the uninformative tokens are replaced with one or several clusters as the coarse-granularity computing units in self-attention. Experiments on the standard GLUE benchmark show that BERT with FCA achieves 2x reduction in FLOPs over original BERT with {\ensuremath{<}}1{\%} loss in accuracy. We show that FCA offers a significantly better trade-off between accuracy and FLOPs compared to prior methods."
}