

\section{Shared Task Submission}
We selected a number of systems for final submission to the shared task, based on our evaluation results. We use the ChatGPT system with a chunk size of 20, the \textsc{mBART} system, and several of the augmented character-level neural systems. We aim to select a diverse set of augmented systems, so we select the $\textsc{Copy}_{ext}$, $\textsc{Copy}_{tr} + \textsc{Copy}_{ext}$, and $\textsc{Copy}_{ext} + \textsc{Per}$ systems for the LSTM model and the $\textsc{Copy}_{ext}$, $\textsc{Copy}_{ext} + \textsc{Trans}$, and $\textsc{Copy}_{ext} + \textsc{Per} + \textsc{Con}$ systems for the pointer-generator model.

We train final models using the training data and specified synthetic dataset. We perform hyperparameter search and select the optimal model architecture for each language and model, which we report in appendix~\ref{sec:hparams-space}. We train models for 1000 epochs, selecting the best model according to validation accuracy.

We report results from the covered test set in \autoref{tab:test-results}. Disappointingly, we observe significant performance discrepancies from our dev set results, with only the ChatGPT-based system maintaining similar scores. We propose three possible factors that could have caused this.

First, all of the datasets involved are quite small, and it is possible that through random variability, the test set was meaningfully different in distribution from the evaluation set. Neural models can be vulnerable to distributional shift, particularly when training data is scarce \citep{linzen-2020-accelerate}, which may explain why the non-neural baseline model fared better.

We briefly investigate whether this is the case by examining the types of linguistic changes in each data split. Specifically, for each desired linguistic change in the evaluation and test datasets (which might include multiple changes from a single example), we compute the number of times that change occurs in the training dataset, and average over all changes. This gives us a rough estimate of how common the linguistic changes are in the model's training data.



We report these results in \autoref{tab:change-frequencies}. We find that for Bribri and Guaran\'{i}, the distribution is very similar between the dev and test sets, while for Maya, the test set contains changes that are far more rare (-23.6 points) on average. As Maya was the language where we observed the greatest discrepancy in performance, this could be a contributing factor, and represents an important consideration for neural models.

The other potential contributing factor is that due to the small datasets and difficult nature of the task, the performance of our models was highly variable. For augmentation strategies such as synonym replacement, the base assumption that synonyms are even present in a dataset of this size might not be accurate. During training, we often observed dev accuracy curves that swung wildly, sometimes jumping up or down by 10 points in a single epoch. Furthermore, since we performed a large number of experiments and selected our final models using the same evaluation set, we may have unintentionally overfit to the specific evaluation set and chosen systems that did not generalize well to the new data. In the future, this could be avoided by using many-fold cross-validation to select models rather than a single dev set.\footnote{We considered this, but it was ultimately too resource-intensive for the number of experiments we wished to run.} 



Finally, we saw significant performance benefits to including sentence copying in \autoref{fig:copy-strategies}, and we employed this in all of our submitted character-level systems. However, this strategy is most beneficial when it includes the sentences and lemmas that appear in the data being evaluated. It is possible that our external corpora happened to contain more overlap with the dev set examples than those in the test set, which could significantly impact performance. We suspect the strategy of retraining including the test inputs as synthetic examples could alleviate this.



Overall, these results serve as a cautionary example of the risks of selecting final systems based on limited evaluation metrics in extremely low-resource scenarios.

\input{tables/changeFreq}