\input{figures/augStrategies}
\input{figures/augVariousSources}
\input{figures/strategyCombos}

\section{Results and Discussion}
\label{sec:results}



\subsection{Evaluation}
We report results on the evaluation split provided for the shared task. Models are evaluated with per-sentence accuracy, \textsc{BLEU} score \citep{papineni2002bleu}, and \textsc{chrF} score \citep{popovic-2015-chrf}.

\subsection{Models} We compare the various architectures described in \autoref{sec:system} and report results in \autoref{tab:arch-results}.


\paragraph{Character-level neural models.} Our character-level models strongly outperform the baselines on Maya, are competitve on Bribri, and underperform on Guaran\'{i}. Within the character-level architectures, the LSTM models perform best in nearly all cases. For the smaller datasets (which have roughly 200-300 training examples), the standard LSTM model achieves the best performance, while on Maya ($\sim$ 600 examples) the pointer-generator LSTM outperforms. This may indicate that the pointer-generator model needs a certain amount of training data to effectively utilize the pointing mechanism and outperform a standard LSTM, and only the Maya dataset meets that threshold.

For Guaran\'{i}, all of the sequence-to-sequence models perform very poorly. Qualitative analysis of the results shows that the models struggle to repeat back valid sentences in the language at all.


\paragraph{Pretrained multilingual models.} mBART achieves our second best performance on Maya (second to pointer-generator LSTM), and the results for Guaran\'{i} and Bribri are also competitive with those of ChatGPT models. Unlike the character-level models, mBART tokenizes the source into subwords; hinting at the possible advantages of using subwords and the information they could carry from the model being pretrained on other languages. 


\paragraph{Large language models.} The ChatGPT-based approach achieves competitive performance, providing evidence that the model is able to capture some patterns correctly through in-context learning. The approach outperforms all other models on Guaran\'{i} (the language with the least training data), demonstrating that the LLM is able to leverage its vast training knowledge as a strong prior on the task at hand, and to make robust generalizations from the available data.

We observe minimal differences based on the chunk size, except for Maya where the smaller chunk size performs significantly better. The system using smart retrieval (SR) is able to achieve close performance for Guaran\'{i} and Maya, but underperforms on Bribri; SR is potentially a viable way to reduce prompt size and thereby cost.

LLMs offer a promising approach to building NLP systems for under-resourced languages, particular when using in-context learning for rare languages, as here. However, the high cost of inference, lack of control (due to the closed-source nature of the models), and privacy concerns are major considerations for practical usage in an endangered language context. 

\subsection{Data augmentation}
Based on the results of the previous section, we select the LSTM and pointer-generator LSTM for our experiments with various augmentation strategies. Noting that the three shared task metrics do not always align in their assessment of best-performing model, we primarily focus on chrF, as accuracy and BLEU score tend to have high variability.\footnote{We observe these metrics jump wildly during training. Furthermore, having even a single incorrect output character can affect the accuracy and BLEU metrics significantly.}







We present results for models trained using each of the data augmentation strategies in \autoref{fig:strategies}. The copying strategies tend to be the strongest, followed by the stem perturbation strategy. The other strategies show mixed results, and in some cases underperform the baseline.

% \begin{figure*}
%     \centering
%     \includegraphics{assets/bribri-copy-strategies.pdf}
%     \includegraphics{assets/guarani-copy-strategies.pdf}
%     \caption{Results for strategies incorporating sentence copying using various sources. \textsc{Copy_tr} uses only the training data, \text}
%     \label{fig:enter-label}
% \end{figure*}

\input{tables/resultsSubmission}

\paragraph{Sentence copying.} We focus on a number of variations and combinations of the copy strategy and report results in \autoref{fig:copy-strategies}, finding that all of our strategies generally improve over the baseline. Unsurprisingly, the models trained 
on data including the source sentences of the evaluation set outperformed those without by an average of 14.46 chrF points.
%While this was not a valid strategy for the shared task, if it is feasible to use t
This strategy, in which the model is retrained before running inference and the target outputs are neither required nor exposed, provides clear benefits in this highly low-resource scenario. 

The $\textsc{Copy}_{ext}$ strategies show mixed results, sometimes matching or outperforming the $\textsc{Copy}_{all}$ strategies (as in Bribri) but sometimes underperforming (as in Maya, LSTM). Combining strategies shows mixed results, and we suspect that after a certain number of synthetic examples, the utility of this strategy declines.


\paragraph{Combined strategies.}

Finally, we experiment with combinations of augmentation strategies, directly concatenating the synthetic datasets, with results in \autoref{fig:combined-strategies}. We observe mixed results---for Guaran\'{i} and Maya, none of the combined strategies show significant improvements over individual strategies, and in some cases performance degrades somewhat. We do see improvements in Bribri with the combined $\textsc{Copy}_{all} + \textsc{Per}$ strategy and the $\textsc{Copy}_{all} + \textsc{Per} + \textsc{Con}$ strategy over any of the individual strategies. Broadly, we find that synthetic data of this sort can only help up to a certain amount, and creating more synthetic data does not necessarily continue to improve performance.

