@inproceedings{hu-etal-2022-fast,
    title = "Fast-{R}2{D}2: A Pretrained Recursive Neural Network based on Pruned {CKY} for Grammar Induction and Text Representation",
    author = "Hu, Xiang  and
      Mi, Haitao  and
      Li, Liang  and
      de Melo, Gerard",
    editor = "Goldberg, Yoav  and
      Kozareva, Zornitsa  and
      Zhang, Yue",
    booktitle = "Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing",
    month = dec,
    year = "2022",
    address = "Abu Dhabi, United Arab Emirates",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/add-emnlp-2024-awards/2022.emnlp-main.181/",
    doi = "10.18653/v1/2022.emnlp-main.181",
    pages = "2809--2821",
    abstract = "Chart-based models have shown great potential in unsupervised grammar induction, running recursively and hierarchically, but requiring O(n{\textthreesuperior}) time-complexity. The Recursive Transformer based on Differentiable Trees (R2D2) makes it possible to scale to large language model pretraining even with a complex tree encoder, by introducing a heuristic pruning method.However, its rule-based pruning process suffers from local optima and slow inference. In this paper, we propose a unified R2D2 method that overcomes these issues. We use a top-down unsupervised parser as a model-guided pruning method, which also enables parallel encoding during inference. Our parser casts parsing as a split point scoring task by first scoring all split points for a given sentence and then using the highest-scoring one to recursively split a span into two parts. The reverse order of the splits is considered as the order of pruning in the encoder. We optimize the unsupervised parser by minimizing the Kullback{--}Leibler distance between tree probabilities from the parser and the R2D2 model.Our experiments show that our Fast-R2D2 significantly improves the grammar induction quality and achieves competitive results in downstream tasks."
}