SubmissionNumber#=%=#17 FinalPaperTitle#=%=#Fine-tuning LLMs to Extract Epilepsy Seizure Frequency Data from Health Records ShortPaperTitle#=%=# NumberOfPages#=%=#12 CopyrightSigned#=%=#Ben Holgate JobTitle#==# Organization#==#King's College London, Strand, London, WC2R 2LS, United Kingdom Abstract#==#We developed a new methodology of extracting the frequency of a patient's epilepsy seizures from unstructured, free-text outpatient clinic letters by: first, devising a singular unit of measurement for seizure frequency; and second, fine-tuning a generative Large Language Model (LLM) on our bespoke annotated dataset. We measured frequency by the number of seizures per month: one seizure or more requires an integer; and less than one a decimal. This approach enables us to track whether a patient's seizures are improving or not over time. We found fine-tuning improves the F1 score of our best-performing LLM, Ministral-8B-Instruct-2410, by around three times compared to an untrained model. We also found Ministral demonstrated an impressive ability for mathematical reasoning. Author{1}{Firstname}#=%=#Ben Author{1}{Lastname}#=%=#Holgate Author{1}{Username}#=%=#bholgate Author{1}{Email}#=%=#benjamin.holgate@kcl.ac.uk Author{1}{Affiliation}#=%=#King's College London Author{2}{Firstname}#=%=#Joe Author{2}{Lastname}#=%=#Davies Author{2}{Email}#=%=#joe.m.davies@kcl.ac.uk Author{2}{Affiliation}#=%=#King's College London Author{3}{Firstname}#=%=#Shichao Author{3}{Lastname}#=%=#Fang Author{3}{Email}#=%=#shichao.1.fang@kcl.ac.uk Author{3}{Affiliation}#=%=#King's College London Author{4}{Firstname}#=%=#Joel S. Author{4}{Lastname}#=%=#Winston Author{4}{Email}#=%=#joel.winston@kcl.ac.uk Author{4}{Affiliation}#=%=#King's College London Author{5}{Firstname}#=%=#James T. Author{5}{Lastname}#=%=#Teo Author{5}{Email}#=%=#jamesteo@nhs.net Author{5}{Affiliation}#=%=#King's College London Author{6}{Firstname}#=%=#Mark P. Author{6}{Lastname}#=%=#Richardson Author{6}{Email}#=%=#mark.richardson@kcl.ac.uk Author{6}{Affiliation}#=%=#King's College London ========== èéáğö