To evaluate our approach, we split the dataset into a training and evaluation split, containing 80,911 and 58,299 clinic description $\leftrightarrow$ masterlist pairs, respectively. For fine-tuning, we used only the training split. For evaluation, we split the evaluation set into gallery and probe sets in a 4:1 ratio, in a setup similar to 5-fold cross-validation, where gallery entries form the vector store data. Each fold is stratified based on the masterlist entries, such that each fold contains approximately the same distribution of masterlist entries. Specifically, for each masterlist entry, we distribute its associated clinic descriptions evenly across all folds -- for example if 5 clinic descriptions map to the same masterlist entry, each fold will contain exactly 1 such mapping.

\input{tables/bm25-misses.tex}

\input{tables/hybrid}
\input{tables/masterlist}

\noindent \textbf{Evaluation Metrics.}
Our primary evaluation metric is Accuracy@k, which measures whether a ground-truth masterlist description is in the first k returned results for a query clinic description. Our target is to optimize for Acc@1, but we also include the results for Acc@3, Acc@5 and Acc@100. In a real-life use of such an system will involve suggesting top-3 or top-5 most similar masterlist entries, and Acc@3 and Acc@5 provides insight into the usefulness of our system. We also include Acc@100, as a low value indicates a problem with the chosen search technique, but usually it indicates the presence of incorrect annotations. In all our results, we show the mean and standard deviation across 5 folds.

\input{tables/historical-data}

\subsection{Comparison between different types of search indexes}
In Table \ref{tab:hybrid}, we show a comparison between dense, sparse, and hybrid approaches. For dense embeddings, we used a fine-tuned mE5 \cite{wangMultilingualE5Text2024} model. The results show that the fine-tuned dense model consistently outperforms both sparse and hybrid search methods. When searching only masterlist entries, the dense approach achieves 26.2\% higher Acc@1 than the sparse approach. When using both masterlist and associated mappings, the dense approach obtains a 17.2\% Acc@1 margin. The sparse approach also shows poor performance for Acc@100, indicating that a bag-of-word approach is not appropriate for this task, and semantic understanding is needed. Hybrid search fails to outperform dense search as it is limited by the poor performance of sparse search. In Table \ref{tab:bm25-misses} we show selected examples of clinic descriptions where sparse embeddings fail to capture variations in text descriptions.

\subsection{Fine-tuning with metric learning}
In Table \ref{tab:masterlist} we compare the performance of three dense embedding models: mE5-large \citep{wangMultilingualE5Text2024}, RoBERT-large \citep{masala-etal-2020-robert}, and BioClinicalBERT \citep{alsentzer2019clincalbert}. We obtained that mE5 has higher off-the-shelf retrieval accuracy compared to RoBERT and BioClinicalBERT. This advantage stems from mE5's design as a sentence-transformer model specifically trained to evaluate similarity between sentences or descriptions, whereas RoBERT and BioClinicalBERT is adapted for sentence embedding through a pooling operation over token embeddings.

Sparse search initially outperforms both RoBERT and BioClinicalBERT. However, after fine-tuning, all dense embeddings surpass sparse embeddings in performance metrics, with mE5 maintaining its position as the highest-performing model.

Table \ref{tab:historical} illustrates the impact of incorporating both masterlist and associated mappings in search processes. The inclusion of reduces the performance difference between E5 and the other two models. While the relative ranking of models remains consistent, E5 achieves the highest performance with an Acc@1 of 85.2\% and an Acc@5 of 95\%. Notably, Acc@1 metric may under-represent actual performance. Manual inspection of misclassified results reveals many plausible matches. This discrepancy occurs due to the presence of duplicate entries within the masterlist itself—entries with slightly different formulations that reference identical medical procedures. The markedly higher Acc@3 metric, which captures whether the ground-truth result appears within the first three recommendations, supports this observation. Although duplicated masterlist results present a methodological challenge for evaluation, they do not compromise practical application. The real-world accuracy exceeds the reported metrics, as demonstrated in the next section.

%\input{tables/combined}

\subsection{Doctor evaluation}\label{doctor-evaluation}
\input{tables/doctor-results}
Our medical procedure mapping system was used to map new unmapped procedures. We evaluated on new procedure descriptions from 10 clinics, comprising 12,836 unique descriptions. After mapping the procedures using a fine-tuned mE5 models trained on all available data, doctors validated each pair to determine if the masterlist assignment was correct. As shown in Table \ref{tab:doctor-results}, the model achieves a real-world Acc@1 of 94.7\%. The 98.5\% Acc@2 indicates that doctors considered either the first or second recommendation correct, while for only 1\% of entries, doctors assigned a different description than the ones recommended.

Another notable aspect is the speed of the mappings process. While manually mapping the 12,836 descriptions would take more than 60 hours, using our retrieval system reduces this to only 3 minutes, resulting in an 1200$\times$ speedup. 

