@inproceedings{wang-etal-2026-arkrepobench,
    title = "{A}rk{R}epo{B}ench: A Repository-Level Code Completion Benchmark for {H}armony{OS} Development",
    author = "Wang, Yanlin  and
      Zhang, Bowen  and
      Wang, Yanli  and
      Guo, Daya  and
      Zhuo, Terry Yue  and
      Chen, Jiachi  and
      Liu, Mingwei  and
      Zhang, Xingong  and
      Zheng, Zibin",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {ACL} 2026",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.findings-acl.969/",
    pages = "19409--19429",
    ISBN = "979-8-89176-395-1",
    abstract = "ArkTS is the primary programming language for Huawei{'}s HarmonyOS ecosystem, which has expanded across smartphones, tablets, and IoT devices. While large language models have demonstrated strong code generation capabilities for mainstream languages, their performance on ArkTS remains largely unexplored. To address this gap, we introduce ArkRepoBench, the first repository-level code completion benchmark for ArkTS to our knowledge, 7,519 samples from 20 official HarmonyOS repositories. The benchmark covers multiple difficulty levels and further categorizes completion instances into Single-File, Cross-File Independent, and Cross-File Dependent settings based on dependency analysis, distinguishing the mere presence of cross-file context from its actual necessity. Our experiments show that: (1) ArkTS completion consistently underperforms mainstream languages under our experimental settings, suggesting language-specific challenges associated with this emerging language; (2) open-source 7B models achieve performance comparable to close-source models; (3) cross-file context is a double-edged sword, with sparse retrieval(Jaccard) outperforming dense methods on ArkTS; and (4) HarmonyOS API documentation consistently improves performance, suggesting the benefits of domain-specific enhancements in low-resource settings."
}