@inproceedings{peng-etal-2022-rethinking,
    title = "Rethinking Positional Encoding in Tree Transformer for Code Representation",
    author = "Peng, Han  and
      Li, Ge  and
      Zhao, Yunfei  and
      Jin, Zhi",
    editor = "Goldberg, Yoav  and
      Kozareva, Zornitsa  and
      Zhang, Yue",
    booktitle = "Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing",
    month = dec,
    year = "2022",
    address = "Abu Dhabi, United Arab Emirates",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/fix-sig-urls/2022.emnlp-main.210/",
    doi = "10.18653/v1/2022.emnlp-main.210",
    pages = "3204--3214",
    abstract = "Transformers are now widely used in code representation, and several recent works further develop tree Transformers to capture the syntactic structure in source code. Specifically, novel tree positional encodings have been proposed to incorporate inductive bias into Transformer.In this work, we propose a novel tree Transformer encoding node positions based on our new description method for tree structures.Technically, local and global soft bias shown in previous works is both introduced as positional encodings of our Transformer model.Our model finally outperforms strong baselines on code summarization and completion tasks across two languages, demonstrating our model{'}s effectiveness.Besides, extensive experiments and ablation study shows that combining both local and global paradigms is still helpful in improving model performance. We release our code at \url{https://github.com/AwdHanPeng/TreeTransformer}."
}