
\section{Models}
\label{sec:system}
We explore a number of models, including small sequence-to-sequence models, pretrained multilingual models, and large language models. 
For most models, input for a given instance consists of the source sentence plus the expected set of linguistic changes (e.g. PERSON:1\_PL in Table~\ref{tab:example}).



\subsection{Character-level neural models}
\label{sec:archs}
We compare several different small character-level sequence-to-sequence models, using the \texttt{Yoyodyne} library for implementation.\footnote{\url{https://github.com/CUNY-CL/yoyodyne}}

\paragraph{LSTM.} We use a standard encoder-decoder LSTM with cross-attention. LSTMs have proven effective at inflection tasks \citep{cotterell-etal-2018-conll}, outperforming transformers under certain conditions \citep{wu-etal-2021-applying}. The expected linguistic changes are concatenated with the source sentence.

\paragraph{Transformer.} \citet{wu-etal-2021-applying} also finds that in many cases, the transformer can outperform recurrent networks at character-level tasks. Thus, we also compare with an encoder-decoder transformer. Linguistic changes are treated as in the LSTM.

\paragraph{Pointer-generator.} For tasks such as summarization (and the current task!) where the output sequences may share many tokens with the input sequence, the pointer-generator mechanism \citep{see-etal-2017-get} has proven effective. The mechanism is a modification of an encoder-decoder architecture that introduces a pointing mechanism, where the model can copy a token from the input sequence rather than generating a novel token. Unlike the prior models, linguistic changes are encoded and attended to separately, so that they cannot be ``pointed to'' by the pointer-generator mechanism. We explore both LSTMs and Transformers with pointer-generator mechanisms.  \\


\noindent We performed a hyperparameter search to determine the optimal hyperparameters for both the attentive-LSTM and pointer generator.  The results of our search, our final hyperparameters, are given in \autoref{tab:hyperparameters}.  The full hyperparameter space we explored is reported in \autoref{sec:hparams-space}. We train all models on a NVIDIA A100 GPU, with Adam optimization, a linear scheduler, a learning rate of 0.001, and a dropout of 0.2. 
% \noindent All models use the \texttt{Yoyodyne} default hyperparameters, which originate from the optimal configuration found in \citet{wiemerslage-etal-2024-quantifying}. Models use a linear feature encoder, one encoder and decoder layer, an embedding size of 128, and a hidden size of 512.
We also explored using a larger architecture with the parameters described in \citet{yang-etal-2022-generalizing}, however, we find these models nearly always underperform by a wide margin.\footnote{Results are given in \autoref{sec:larger-arch}. We observe that the larger models tend to overfit the training data, with much higher validation loss than their smaller counterparts.}



\subsection{Pretrained multilingual models}
Transfer learning is a common strategy used to overcome limited data in lower-resource languages. To this end, we utilize mBART \cite{liu-etal-2020-multilingual-denoising}, which has shown a promising capability of generalization in the case of unseen languages \cite{liu-etal-2021-continual}. The desired linguistic change is appended to the source sentence, separated by the model separation token.

\subsection{Large language models}
Large language models (LLMs) generally struggle on rare, low-resource languages that are not well-represented in their training corpora \citep{robinson-etal-2023-chatgpt, ahuja-etal-2023-mega}. However, LLMs may be able to achieve better performance on these languages through \textbf{in-context learning} (also known as \textit{few-shot prompting}), where a small number of examples for a novel task are provided in the prompt at inference time \citep{NEURIPS2020_1457c0d6}. With ever-increasing context lengths, LLMs have even been able to learn completely novel languages using comprehensive linguistic resources provided in the context \citep{tanzer2024a}.

We utilize the ChatGPT API and the GPT-4 model to study in-context learning for our sentence transformation task \citep{openai2024gpt4}. Since the provided training splits are very small, we provide the entire training set as context in our prompts. We also experiment with attempting to provide a more focused, relevant context, by filtering training examples to only those that have a linguistic change in common with the test sentences.

We utilize the \texttt{gpt-4-0125-preview} model, with temperature of 0 and a fixed random seed of 430. Full details about our prompting strategy are provided in \autoref{sec:chatgpt-app}. As making an API call for every unique test example is fairly expensive, we prompt the model to make predictions on chunks consisting of multiple examples. We experiment with chunks of 20 and 80 examples. 
