\documentclass[11pt]{article}

% Change "review" to "final" to generate the final (sometimes called camera-ready) version.
% Change to "preprint" to generate a non-anonymous version with page numbers.
\usepackage[final]{acl}

% Standard package includes
\usepackage{times}
\usepackage{latexsym}

% For proper rendering and hyphenation of words containing Latin characters (including in bib files)
\usepackage[T1]{fontenc}
% For Vietnamese characters
% \usepackage[T5]{fontenc}
% See https://www.latex-project.org/help/documentation/encguide.pdf for other character sets

% This assumes your files are encoded as UTF8
\usepackage[utf8]{inputenc}

% This is not strictly necessary, and may be commented out,
% but it will improve the layout of the manuscript,
% and will typically save some space.
\usepackage{microtype}

% This is also not strictly necessary, and may be commented out.
% However, it will improve the aesthetics of text in
% the typewriter font.
\usepackage{inconsolata}

%Including images in your LaTeX document requires adding
%additional package(s)
\usepackage{graphicx}

% If the title and author information does not fit in the area allocated, uncomment the following
%
%\setlength\titlebox{<dim>}
%
% and set <dim> to something 5cm or larger.


% Hinzugefügt von Robin
\usepackage{hyperref}
\usepackage[normalem]{ulem}
\usepackage{array}
\usepackage{booktabs}
\usepackage{amsmath}
\usepackage{pdfpages}

% neues float listing
\usepackage{float}
\newfloat{listingRobin}{htbp}{lop}
\floatname{listingRobin}{Listing}



\title{Generative Information Extraction from Biographical Sources}

% Author information can be set in various styles:
% For several authors from the same institution:
% \author{Author 1 \and ... \and Author n \\
%         Address line \\ ... \\ Address line}
% if the names do not fit well on one line use
%         Author 1 \\ {\bf Author 2} \\ ... \\ {\bf Author n} \\
% For authors from different institutions:
% \author{Author 1 \\ Address line \\  ... \\ Address line
%         \And  ... \And
%         Author n \\ Address line \\ ... \\ Address line}
% To start a separate ``row'' of authors use \AND, as in
% \author{Author 1 \\ Address line \\  ... \\ Address line
%         \AND
%         Author 2 \\ Address line \\ ... \\ Address line \And
%         Author 3 \\ Address line \\ ... \\ Address line}

\author{Robin Winkle \\
  University of Potsdam \\
  \texttt{robin\_winkle@outlook.com} \\\And
  Jörn Kreutel \\
  Berliner Hochschule für Technik \\
  \texttt{jkreutel@bht-berlin.de} \\\And
  Manfred Stede \\
  University of Potsdam \\
  \texttt{stede@uni-potsdam.de}
  \\}

%\author{
%  \textbf{First Author\textsuperscript{1}},
%  \textbf{Second Author\textsuperscript{1,2}},
%  \textbf{Third T. Author\textsuperscript{1}},
%  \textbf{Fourth Author\textsuperscript{1}},
%\\
%  \textbf{Fifth Author\textsuperscript{1,2}},
%  \textbf{Sixth Author\textsuperscript{1}},
%  \textbf{Seventh Author\textsuperscript{1}},
%  \textbf{Eighth Author \textsuperscript{1,2,3,4}},
%\\
%  \textbf{Ninth Author\textsuperscript{1}},
%  \textbf{Tenth Author\textsuperscript{1}},
%  \textbf{Eleventh E. Author\textsuperscript{1,2,3,4,5}},
%  \textbf{Twelfth Author\textsuperscript{1}},
%\\
%  \textbf{Thirteenth Author\textsuperscript{3}},
%  \textbf{Fourteenth F. Author\textsuperscript{2,4}},
%  \textbf{Fifteenth Author\textsuperscript{1}},
%  \textbf{Sixteenth Author\textsuperscript{1}},
%\\
%  \textbf{Seventeenth S. Author\textsuperscript{4,5}},
%  \textbf{Eighteenth Author\textsuperscript{3,4}},
%  \textbf{Nineteenth N. Author\textsuperscript{2,5}},
%  \textbf{Twentieth Author\textsuperscript{1}}
%\\
%\\
%  \textsuperscript{1}Affiliation 1,
%  \textsuperscript{2}Affiliation 2,
%  \textsuperscript{3}Affiliation 3,
%  \textsuperscript{4}Affiliation 4,
%  \textsuperscript{5}Affiliation 5
%\\
%  \small{
%    \textbf{Correspondence:} \href{mailto:email@domain}{email@domain}
%  }
%}

\newcommand{\jkcmt}[1]{\noindent(\textcolor{red}{\textbf{JK:} \emph{#1})}}
\renewcommand{\jkcmt}[1]{}

\newcommand{\scare}[1]{`#1'}

\newcommand{\flfddr}[0]{\textsc{FLFDDR}}

\begin{document}
\maketitle



\begin{abstract}
%When encyclopedias of literary writers provide information about author biographies, they select attributes they consider important, and they phrase the information in their favorite styles. We are dealing with a large set of literary encyclopedias that cover authors from the former German Democratic Republic (GDR), and our ultimate aim is to fuse all information into a single structured database. In this paper, we tackle the core information extraction problem and study to what extent LLMs can assist in mapping individual pieces of information from various encyclopedias to a single format. Specifically, we compare Qwen-2.5-32B and Llama-3-70B-Instruct, test a variety of prompting strategies, and propose a new evaluation function for assessing performance on this task, which may generalize to similar information extraction settings.

Biographical sources, such as literature encyclopedias, encode knowledge about historical figures in textual form. In this paper, we address the task of consolidating structured biographical information about authors from the former German Democratic Republic into a unified database. To this end, we present a generalizable Information Extraction (IE) system based on LLM prompting. Specifically, we compare two midsized open-source models, Qwen-2.5-32B and Llama-3-70B-Instruct, investigate a range of Prompt Engineering (PE) strategies, and propose a semantic similarity-based evaluation metric for open-ended IE. Our experiments on an unpublished annotated subset of biographical texts deliver moderate precision and variable recall, highlighting both the potential and current limitations of generative IE in the Digital Humanities.



\end{abstract}


%\section{Old Introduction}

%Our proposal is situated in the context of a project that aims to create a bio-biographical data set on approximately 3.400 authors of literary works which have been identified as forming the literary field \cite{bourdieu_feld} of the former German Democratic Republic (GDR). The project aims to compile all available published and archival sources regarding those authors,  explicitly highlight conflicting information from different sources, and shall This data set shall facilitate both qualitative and quantitative research on GDR literature. Central to making this research effort feasible is the automatic extraction of structured information from the sources. 

%The data set underlying this proposal was collected during a pilot study on a subset of about 10\% of these authors. Data collection was done manually by junior domain experts and was supported by a customised user interface for data entry. However, for extending the data set towards a wider coverage of the literary field it appears not only sensible, but also necessary, considering the available human resources, to aim at a semi automated process. Data extraction shall be done automatically and the extraction results shall be verified and, if necessary, modified by the human experts. 

%As a pilot study, biographical data for the students of a literary writing program, who represent about 10\% of all GDR authors, have been collected from more than 2000 articles from dictionaries, among other sources, and have been linked with an existing bibliography on GDR literature \cite{biban}. This data collection was supported by a graphical user interface for data entry, but the actual extraction of biographical data from the sources and the consolidation of information from various sources on a given author was done manually by junior domain experts. However, for extending the data set towards a wider coverage of the literary field it appears not only sensible, but also necessary, considering the available human resources, to aim at a semi automated process. Data extraction shall be done automatically and the extraction results shall be verified and, if necessary, modified by the human experts. 

%Our contribution will report on first experiments that employ generative AI for data extraction on the basis of the project's biographical data model... 

%Contributions of our paper:
%\begin{itemize}
   % \item first experiments on running LLMs to extract information from biographical domain
   % \item experiments with prompt tuning and variants (chain of thought vs. others)
   % \item generalizable evaluation metrics, configurable for alternative data and domains
%\end{itemize}


\section{Introduction}

Information Extraction (IE) plays an instrumental role in the Digital Humanities and related fields, where the systematic transformation of unstructured or semi-structured texts into machine-readable data is a prerequisite for large-scale cultural and historical analysis. 

Our paper is situated in the context of the research project \emph{Forschungsplattform Literarisches Feld DDR} (\flfddr) which aims to create a bio-bibliographical database that covers approximately 3,400 authors of literary works which have been identified as forming the literary field \citep{bourdieu_feld} of the former German Democratic Republic (GDR). By compiling all available sources regarding those authors --explicitly highlighting conflicting information-- the project seeks to enable qualitative and quantitative research on a complete literary field.\footnote{The list of all authors and comments on their compilation are available here: \url{https://ddr-literatur.de/daily-research/korpus-autorinnen-in-der-ddr/}} As of today, data for a subset of those authors, who participated in a study program on literary writing and who sum up to about 10\% of the complete corpus,\footnote{See: \emph{Forschungsplattform Literarisches Feld DDR: Autor*innen, Werke, Netzwerke. Pilotprojekt: Die Student*innen des Instituts für Literatur "Johannes R. Becher" Leipzig}, funded by Deutsche Forschungsgemeinschaft (DFG), project number 419244741, 2019-2023.} have been successfully collected, using a customized user interface for manual data entry. To extend the database, a semi-automated process will be required, which motivates the focus of this paper on the extraction of biographical information as an instance of the broader IE task.
 
Recent advances in Natural Language Processing (NLP) have enabled increasingly efficient IE methods, reducing the need for task- or domain-specific parsing rules \citep{Harvesting-of-Biographical-Data--Plum-etal-2019}. In particular,  transformer-based Large Language Models (LLMs) have been used for IE in a generative, prompt-based setting, which we term \textit{Generative IE} (GenIE). While these models are typically optimized for dialogue rather than structured data production \citep{Are-LLMs-good-at-structured-output--Liu-etal-2024}, they have shown strong generalization capabilities across diverse NLP tasks, which has driven their adoption in IE.


\begin{figure*}[t]
    \centering 
        \includegraphics[width=0.6\linewidth]{figures/Workflow overview (simpler).jpg}%
    \caption{High-level representation of the extraction workflow, including the multi-step (top) vs. single-step approach (bottom).}
    \label{fig:Workflow-overview}
\end{figure*}


In this work, we present a generalizable end-to-end GenIE framework, designed to obtain structured information from a set of natural language texts. Figure \ref{fig:Workflow-overview} provides a high-level representation of the extraction workflow. The input documents (on the left) are supplied to an LLM alongside customizable prompts that specify the format of the structured output (bottom right) and guide the extraction process\footnote{The top right represents an alternative, sequential prompting approach explained in Section \ref{sec:Experiments}.}. The framework supports systematic assessment of Prompt Engineering (PE) strategies and retains source text fragments for all extractions to enable retroactive verifiability and human-in-the-loop validation.

We assess the framework on an annotated subset of the biographical material curated within the \flfddr\ project. Using two moderately sized open-source LLMs, we analyze the impact of different prompting strategies and discuss overall trends and challenges. As an additional contribution, we propose and apply a semantic similarity–based evaluation approach tailored specifically to open IE settings, addressing the limitations of exact-match metrics and restricted GenIE. Our results highlight both the potential and the current limitations of GenIE systems, contributing empirical evidence to ongoing discussions on the responsible and effective use of LLMs in Digital Humanities.



% =====================================
\section{Background and Related Work}

IE is the task of obtaining structured information from unstructured or semi-structured texts written in natural language \citep{IE-Distilling-structured-from-unstructured--McCallum-2005}. During recent decades, computational approaches to IE have enabled the creation of analytically accessible databases for diverse data types, including biographical data \citep{Harvesting-of-Biographical-Data--Plum-etal-2019}. Within the literary domain, encyclopedias and handbooks, such as those contained in our data set, are among the primary sources for biographical data, listing and contextualizing important life events of literary authors, including a person's place of birth, membership in an institution or the publication of specific works. Structured representations of this data may draw on event-centric data models, which link multiple temporally and contextually connected biographical facts to a unique event within a person's life narrative \citep{BioCRM-Data-Model--Tuominen-etal-2017}.\footnote{The concrete event types and attributes employed in the \flfddr\ project are motivated by the project's rather wide scope aiming at providing a generic foundation for detailed analyses linking life and work of authors.
%Where most of the event types are generic, like \emph{Birth}, \emph{Death}, \emph{Education}, \emph{Employment} or \emph{Membership}, some types are more closely related to the historical context of GDR history, e.g. \emph{MilitaryService}, \emph{ForcedStay} or \emph{Surveillance}. 
See table \ref{tab:permissible-field-values-extraction-scope} for a full list of event types and attributes from the \flfddr\ event data model which have been taken into consideration for the  pilot study presented here. For more information on the project's data model, see \citep{flfddr_werkstattbericht}.} Extracting such structured representations from text poses challenges for traditional pipeline-based IE systems, which depend on explicitly modeling intermediate subtasks such as Text Segmentation or Named Entity Recognition (NER). 

We address this challenge through a generalizable framework for large-scale IE using pre-trained Large Language Models (LLMs) and Prompt Engineering (PE). While related work has sometimes referred to this approach as "LLM-driven IE", we adopt the term "Generative Information Extraction (GenIE)", originally introduced by \citet{GenIE--Josifoski-etal-2022}, since it better reflects the use of generative sequence modeling for structured output. GenIE inherently integrates multiple IE subtasks into an end-to-end sequence generation task \citep{PromptNER-Prompt-Locating-and-Typing-for-NER--Shen-etal-2023}, bypassing the need to explicitly obtain intermediate extraction results.  However, it also faces some well-documented challenges associated with generative transformer models in general, such as hallucinations \citep{hallucinations-in-llms--lei-etal-2025}, biases \citep{Adressing-Biases-in-Gen-AI--Wei-etal-2025}, and output variability \citep{Are-LLMs-good-at-structured-output--Liu-etal-2024}. 

%Our framework is designed to require minimal effort, employing out-of-the-box LLMs without task-specific fine-tuning.


Related work on GenIE systems has focused particularly on medicine \citep{LLMs-accelarate-annotation-for-medical-IE--Google-Research-2023, Collaborative-LLMs-for-data-extraction-in-living-systematic-reviews--Khan-etal-2025, Improving-LLMs-for-clinical-NER-via-PE}. \citet{Scalable-IE-from-electronic-health-records-using-LLMs--Gu-etal-2025} evaluate the "out-of-the-box" capabilities of chat-optimised models by prompting them to extract medical determinants from text-based health records. Their hypothesis --that general-purpose LLMs can be adopted for IE without task-specific fine-tuning-- aligns with the methodology presented in this work. The results of their study suggest that even moderately-sized open-source models substantially outperform naive pattern-matching and can deliver strong performance in IE-related tasks in various domains, particularly when effective PE is applied. 

Recent studies have also reported encouraging results with GenIE systems in other domains, including law \citep{RE-in-Legal-Domain-Based-on-LLMs--Li-Yi-2024}, finance \citep{Extracting-Financial-Data-from-Unstructured-Leveraging-LLMs--Li-etal-2025, LLMs-for-Financial-and-Investment-Management--Kong-etal-2024}, and scientific text \citep{Structured-IE-from-Scientific-Text-with-LLMs--Dagdelen-etal-2024}. \citet{Extract-accurate-data-PE--Polak-2024} achieve up to 90.8\% precision and 87.7\% recall in a study on the use of \textit{ChatGPT} for data extraction in a materials science context. However, their formulation evaluates the IE task under a highly constrained matching scheme: extracted outputs are compared against ground truth labels using strict domain-specific equivalence criteria. In contrast, our work targets fully open-ended generative extraction without predefined value sets, and is designed to operate with open-source models and without the need for strict equivalence definitions.

The implementation of our framework is in part informed by these studies. However, to the best of our knowledge, there have not been any systematic attempts to extract structured data from biographical text sources using GenIE systems.








% ======================================
\section{Data} \label{sec:data}

\begin{figure}
    \centering
    \includegraphics[width=\linewidth]{figures/Fred Rodrian Beispiel.png}
    \caption{Source text about author Fred Rodrian from \emph{Kürschners Deutscher
Literatur-Kalender Nekrolog 1971-1998} lexicon.}
    \label{fig:fred-rodrian}
\end{figure}

\begin{figure}
    \centering
    \includegraphics[width=\linewidth]{figures/Christine Ernst Beispiel.png}
    \caption{Source text about author Christine Ernst from \emph{Wikipedia}}
    \label{fig:christine-ernst}
\end{figure}


This study draws on biographical texts compiled in the interdisciplinary \flfddr project. The complete corpus contains documents from about 600 different German language sources, including journals, newspapers, online materials, interviews, and around 2,300 articles from 150 literature encyclopedias and handbooks \cite{flfddr_werkstattbericht}. 

Some instances in the source data were digitized using OCR, which may have introduced a slight degree of character-level noise. Other forms of noise may arise from a small amount of inconsistencies between source texts and evaluation data, for example, in the case of author Christine Ernst who was known as Christine Lindner before changing her name. As illustrated by Figures \ref{fig:fred-rodrian}-\ref{fig:christine-ernst}, the input documents feature a considerable amount of domain-specific abbreviations (e.g. "Lit.büros" for \textit{Literaturbüros}\footnote{Translation: "Literary Association"}) as well as metadata strings without direct informational value (e.g. "GND: 139456139").  Excluding extreme outliers, the texts contain approximately 1,170 characters on average, with substantial variation between sources. 

In our experiments, we use a subset of 188 articles from 72 sources manually annotated by five trained student research assistants under the supervision of literary scholars. Table \ref{tab:source-lexicons-counts} in the appendix shows the most common text sources in the source corpus and the evaluation subset. The subset primarily features literature encyclopedia\footnote{German: \textit{Literaturlexika}} articles, a curated and quality-controlled text type containing highly condensed biographical information. As a result, the annotated documents are mostly compact semi-structured entries using a telegraphic style, such as in the example shown in Figure \ref{fig:fred-rodrian}. This style is characterized by minimal phrasing with infrequent use of function words and connectors. More narrative texts, such as the one represented by Figure \ref{fig:christine-ernst}, are present but less frequent in the evaluation set.



We process the annotations to fit the format described in Section \ref{sec:output-format}. After cleaning, deduplication, and consistency checks, the final evaluation set contains documents labelled with 6,787 information units. The most prevalent event types extracted by annotators capture information relating to authors' education and professional trajectories, besides singular events like birth and death. Attributes such as \textit{begin}, \textit{end}, and \textit{place} that occur across multiple event types are among the most frequent.  Since the annotations emerged out of the project’s data collection workflow, they exhibit occasional gaps in coverage and mismatches between surface forms and normalized attribute values, a point we return to in Section \ref{sec:error-analysis}.  


% ======================================
\section{Methods} \label{sec: methods}

This section describes the methodological design of our experiments, including the selected output representation, language models, prompting strategies, and a customized evaluation setup.

\subsection{Output Format} \label{sec:output-format}

Data models based on the Resource Description Framework (RDF) for biographical data like the one proposed by \citet{BioCRM-Data-Model--Tuominen-etal-2017} commonly use knowledge triples that rely on abstract references such as Uniform Resource Identifiers (URI) and entity placeholders to form a graph-like representation. This can present challenges from an LLM generation perspective, since these models are trained for fluent text generation rather than for consistent handling of abstract identifiers. For example, an author may be represented by a URI such as \texttt{ex:author\_123}, which must be reused consistently across events, a requirement that generative models may fail to satisfy reliably.


\begin{table}[h]
  \centering
  \begin{tabular}{ll}
    \hline
    \textbf{Field $f$}  & \textbf{Field value $x_f$} \\
    \hline
    \textsc{Subject}   & Rodrian, Fred \\
    \textsc{Event type}& birth \\
    \textsc{Attribute} & place \\
    \textsc{Value}     & Berlin \\
    \textsc{Source fragment}    & geb. am 14.7.1926 im \\
                        &  Berliner Osten \\
    \hline
  \end{tabular}
  \caption{\label{tab:ie-example}
    Structured information unit associated with text fragment relating to the
birth of Fred Rodrian.}
\end{table}

Therefore, we adopt a flattened JSON output format with natural language values that is better aligned with the generative strengths of LLMs. A tabular representation of this format is given in Table \ref{tab:ie-example}. This format allows for an arbitrary number of tuples, each consisting of a field $f$ and the appropriate field value $x_f$. In our experiments, each extracted information unit is configured to contain the following fields: (1) a \textsc{subject} (the person concerned), (2) an \textsc{event type} (e.g., \textit{birth}, \textit{education}), (3) an \textsc{attribute} specifying the event further (e.g., \textit{place, institution}), (4) the extracted \textsc{value} encoded as a string, number, or date, and finally, (5) a \textsc{source fragment} that grounds the information unit in the original text. Here, the term \textsc{attribute} denotes a property of an \textsc{event type} in the underlying event data model, e.g., the "place" attribute of a "birth" event.  While the \textsc{value} is unconstrained, \textsc{event types} and \textsc{attributes} are selected from predefined lists (see Table \ref{tab:permissible-field-values-extraction-scope}) in our experiments to constrain the extraction scope and improve output consistency.

\begin{table*}[t]
\centering
\makebox[\textwidth][c]{%
    \resizebox{0.75\textwidth}{!}{%
        \begin{tabular}{|l|p{12cm}|}
        \hline
        \textbf{Event Types} & \textbf{Attributes} \\
        \hline
        Acquaintance & acquaintanceType | begin | durationOfEvent | end | participants \\
        Birth & place | begin \\
        Citizenship & begin | end | institution \\
        ConfessionAffiliation & begin | confession | institution \\
        Death & begin | deathType | participants | place \\
        Education & begin | educationType | educationalContentsOfEducation | end | institution | place \\
        Employment & begin | end | institution | place | professionsOfEmployment \\
        Flat & begin | durationOfEvent | end | migratedFromPlace | place \\
        ForcedStay & begin | durationOfEvent | end | forcedStayType | institution | place | politicallyMotivated \\
        Funding & begin | durationOfEvent | end | fundingInstitution | place \\
        Granting & awardTypeOfGranting | begin \\
        Journey & begin | destinations | durationOfEvent | end | participants | place \\
        Membership & begin | durationOfEvent | end | institution | participants | place \\
        MilitaryService & begin | durationOfEvent | end | institution | participatedWarOfService | place | professionsOfEmployment \\
        NamedEvent & begin | durationOfEvent | end \\
        NoEmployment & begin | durationOfEvent | end \\
        Origin & socialClasses \\
        Orphancy & begin \\
        Parenthood & begin | childGender | childName \\
        Surveillance & begin | durationOfEvent | end | place | surveillingInstitution \\
        \hline
        \end{tabular}
    }%
}
\caption{Permissible values for extraction fields "Event type" and "Attribute".}
\label{tab:permissible-field-values-extraction-scope}
\end{table*}




\subsection{LLM Selection} \label{sec:llm-selection}

To avoid reproducibility issues and access limitations of commercial LLMs, only open-source models were considered. The \textit{Hugging Face} Open-Source Language Model Leaderboard\footnote{ \url{https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard}, archived, last accessed in June 2025}, a well-established source with extensive model coverage and diverse benchmark evaluations, serves as the starting point for a multi-stage selection process balancing performance, openness, computational cost, and practical feasibility. Models were ranked according to three evaluation suites relevant to structured information extraction: IFEval (format adherence), MuSR (language understanding), and MMLU-Pro (domain knowledge and reasoning). Models are further evaluated by affordability and environmental footprint. To complement the leaderboard-based shortlist, we examined reports from related IE work to highlight models proven effective in structured extraction.

From this process, two models offering a strong balance of quality and efficiency were selected: \textit{Qwen-2.5-32B}\footnote{\url{https://huggingface.co/Qwen/Qwen2.5-32B}} \citep{QWEN2-Technical-Report--Alibaba-2024} and \textit{Llama-3-70B-Instruct}\footnote{\url{https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct}} \citep{The-Llama-3-Herd-of-Models--MetaAI-2024}. A brief heuristic check on a small sample of biographical texts confirmed that both models produce stable, well-formatted outputs suitable for the GenIE task at hand.

\subsection{Prompt Engineering}



Given its central role in guiding the model’s reasoning and extraction decisions, prompt design is handled through a modular setup that automatically assembles prompts by integrating different functional components such as concise instructions, brief descriptions of the input context, examples illustrating the output format, or extraction scope constraints. A condensed version of the specific prompt used in our experiments is provided in Table \ref{tab: prompt}. By switching components in or out, the system can be adapted to test various extraction strategies while keeping prompts stable and manageable. 
%Appendix \ref{sec:appendix} provides an in-depth analysis of an input-output pair, including an example prompt.


The prompting module also incorporates a set of advanced Prompt Engineering (PE) strategies designed to influence how the model reasons through the extraction task. While general PE aims to optimize prompts to be clear, task-specific and concise, advanced PE targets the model’s internal reasoning behavior or attempts to refine the answer space by providing specially crafted output examples.

In this study, three such strategies are implemented. (i) Few-shot prompting \citep{Many-shot-ICl--Agarwal-etal-2024} is used to guide the model by analogy, providing concise examples that demonstrate the expected output format and reduce ambiguity in tasks requiring consistent formatting. (ii) Chain-of-Thought (CoT) prompting \citep{CoT-elicits-reasoning-in-LLMs--Google-brain-team-2022} is included as an optional component to encourage explicit intermediate reasoning. And finally, (iii) self-refinement \citep{Self-Refine-Iterative-refinement-with-self-feedback--Madaan-etal-2023} is integrated as a lightweight revision step, allowing the model to critique and correct its own output to improve completeness and adherence to format requirements.

These techniques represent widely adopted model-agnostic methods that can be integrated without external retrieval systems or task-specific fine-tuning. We systematically assess the contribution of these methods to extraction quality through empirical experiments, cf. Section \ref{sec:Experiments}. 


\subsection{Evaluation Metric} \label{sec:evaluation}

For open IE tasks, traditional accuracy is ill-defined since the space of true negatives (TN) is difficult to define or possibly unbounded: if a text contains only sparse information about a subject, then an IE system would have to generate a large number of TN (e.g. date of birth: "Unknown", date of death: "Unknown", and so forth) to  receive a high evaluation score.  Furthermore, strict surface-form matching can be brittle since valid extraction values are not drawn from a closed set in open IE (e.g. if a text cites a person's main occupation as "author", "writer" should also count as a valid extraction value). This contrasts with closed IE, where the task is framed as a multi-choice classification problem \citep[p. 4]{Scalable-IE-from-electronic-health-records-using-LLMs--Gu-etal-2025} or the TN space is well-bounded at fragment level \citep[p. 9]{Extract-accurate-data-PE--Polak-2024}. 


To robustly compare extractions against human annotations, we adopt a type-aware semantic similarity function $ \sigma : X \times X \rightarrow [0,1] $ that operates at the level of an extraction field $f$ and determines the similarity between this field's value $x_f$ (cf. Section \ref{sec:output-format}) and its counterpart in the annotation set. If $x_f$ is a string literal, cosine similarity is computed using suitable embedding-based representations. This is complemented by specialized similarity scoring functions for cases where $x_f$ is a date or numerical value. 

To estimate the overall alignment score $\phi(e, g)$ between a given candidate unit $e \in \mathcal{E}$ and a gold unit $g \in \mathcal{G}$, let \( F \) denote the set of extraction fields required to meet the minimum similarity threshold $\theta_{f}$ for $f \in F$, and let $w_f$ denote the assigned weight of $f$. Let $x_{f,e}$ and $x_{f,g}$ be the values of $f$ in $e$ and $g$, respectively. The alignment score is then given by Equation \ref{eq: gated_similarity_function} if for all $f \in F$: $\sigma_f(x_{f,e}, x_{f,g}) \geq \theta_{f}$, otherwise it is equal to zero. 

\begin{equation}
\phi(e, g) =
\frac{\sum_{f \in F} w_f \cdot \sigma_f(x_{f,e}, x_{f,g})}{\sum_{f \in F} w_f}
\label{eq: gated_similarity_function}
\end{equation}


We use $\phi$ to obtain the best precision-oriented alignment $\mu_P(e) = \max_{g \in \mathcal{G}} \phi(e,g)$ and compute soft precision for a set of extracted information units $\mathcal{E}$ as:

    \begin{equation}
    \text{Precision*} =
\frac{1}{|\mathcal{E}|}
\sum_{e \in \mathcal{E}} \mu_P(e)
    \label{eq:precision-aggregated}
    \end{equation}


We then reverse the comparison to obtain the best recall-oriented alignment $\mu_R(g) = \max_{e \in \mathcal{E}} \phi(e,g)$ and then calculate soft recall for a set of gold information units $\mathcal{G}$ as:

    \begin{equation}
    \text{Recall*} = \min \Bigg(
    \frac{1}{|\mathcal{G}|} \sum_{g \in \mathcal{G}} \mu_R(g), \ 
    \frac{|\mathcal{E}|}{|\mathcal{G}|} \Bigg)
    \label{eq:recall-aggregated}
    \end{equation}

The upper bound ensures that recall never exceeds the maximum achievable standard recall score, which is equal to the number of candidate elements divided by the number of gold elements, assuming perfect precision. 

To summarize, the metric compares the set of extracted units $\mathcal{E}$ against the set of human-annotated units $\mathcal{G}$ by finding, for each element in one set, the closest semantic match in the other set. High scores indicate strong alignment, while low scores signal spurious or unrecalled units. These scores are then aggregated across the entire set to compute overall precision, recall, and F1. Unlike prior work which relies on closed label spaces and employs strict matching, this metric is robust to surface form variability and allows GenIE systems to generate any extraction value that is semantically equivalent. Our evaluation approach is designed to be generalizable and can be adapted to different data sets and domains, since for each field $f$ contained in an extraction unit for a given domain, the similarity function $\sigma_f$, the minimum similarity threshold $\theta_f$ and the weight $w_f$ are configurable. 

% ======================================
\section{Experiments} \label{sec:Experiments}

We evaluate the proposed framework by comparing several experimental variants to a \textsc{base} configuration. These variants, summarized in Table \ref{tab:exp-configs}, systematically isolate the effect of individual factors, such as model size, number of few-shot examples, multi-step extraction and reasoning strategies.

\begin{table}[t]
\centering
\makebox[\columnwidth][c]{%
    \resizebox{\linewidth}{!}{%
        \begin{tabular}{|>{\raggedright\arraybackslash}p{2.7cm}||p{2.0cm}|>{\raggedright\arraybackslash}p{2.0cm}|p{2.0cm}|p{2.0cm}|p{1.8cm}|}
        \hline
        \textbf{Configuration}  & \textbf{Model}             &\textbf{Heuristically optimized Prompt}& \textbf{Subtask Handling}  & \textbf{Reasoning}                           & \textbf{Number of examples}          \\
        \hline
        \textsc{base}             & Qwen-2.5 (32B)&Yes& simultaneous                                     & default                & 2\\ 
        \hline

        \hline

        \textsc{exmp+} &  &&              &         &         13\\
        \hline
        
        \textsc{larger-m}&          Llama-3 (70B)&&              & &         \\
        \hline
        

        \textsc{larger-m-exmp+} &  Llama-3 (70B)&& & & 13\\

        \hline

        \textsc{multi}&           && sequential&         &        \\
        \hline

        \textsc{CoT} &  &&              &         CoT&         13\\

        \hline
        

        
        \end{tabular}%
    }%
}
\caption{Experimental configurations with deviations from the default setup highlighted. Blank entries inherit the default.}
\label{tab:exp-configs}
\end{table}

The \textsc{base} configuration employs the standard \textit{Qwen-2.5} model with a single-step prompting setup, default reasoning, and two in-context examples. \textsc{larger-m} replaces the baseline model with the larger LLM (cf. Section \ref{sec:llm-selection}) while keeping all other settings fixed. \textsc{larger-m exmp+} extends this setting by increasing the number of few-shot examples. Analogously, \textsc{exmp+} augments \textsc{base} by including more output examples without changing model size.

Additionally, \textsc{multi} implements a multi-step IE pipeline (see Figure \ref{fig:Workflow-overview}, top row), in which the model is sequentially prompted to (i) segment input texts, (ii) extract structured information from segments, and then (iii) perform a single self-refinement pass aimed at improving completeness and strict format adherence. The \textsc{CoT} variant activates explicit CoT instructions, encouraging step-by-step reasoning in the model response.

The performance of the system in the different configurations is assessed using the semantic similarity-based open extraction metric described in Section \ref{sec:evaluation}, computed against the manually annotated reference data mentioned in Section \ref{sec:data}. For our experiments, we set $\theta=1$ for all extraction fields other than \textsc{value}, effectively enforcing exact matching for these fields since their extraction is constrained by predefined lists (cf. Section \ref{sec:output-format}). For string comparison we use an embedding model fine-tuned for cosine similarity computation in German and English\footnote{\url{https://huggingface.co/T-Systems-onsite/cross-en-de-roberta-sentence-transformer}.}.



% ====================================
\section{Results}


\begin{table*}[t]
\centering
\resizebox{\textwidth}{!}{
\begin{tabular}{l>{\centering\arraybackslash}p{0.11\linewidth}%
                >{\centering\arraybackslash}p{0.10\linewidth}%
                >{\centering\arraybackslash}p{0.11\linewidth}%
                >{\centering\arraybackslash}p{0.15\linewidth}%
                >{\centering\arraybackslash}p{0.15\linewidth}%
                >{\centering\arraybackslash}p{0.11\linewidth}%
                >{\centering\arraybackslash}p{0.08\linewidth}}
\toprule
\textbf{Metric/Experiment} & \textbf{\textsc{multi}} & \textbf{\textsc{cot}} & \textbf{\textsc{exmp+}} & \textbf{\textsc{larger-m}} & \textbf{\textsc{larger-m-exmp+}} & \textbf{\textsc{base}} \\
\midrule
F1 & 0.45 & 0.61 & 0.63 & 0.52 & 0.68 & 0.55\\
Precision & 0.71 & 0.61 & 0.68 & 0.68 & 0.62 & 0.71 \\
Recall & 0.33 & 0.62 & 0.59 & 0.43 & 0.75 & 0.45 \\
\% False positives & 12.43 & 20.03 & 12.89 & 15.70 & 17.59 & 13.74 \\
\% False negatives & 48.78 & 5.93 & 7.74 & 31.71 & 6.53 & 10.07 \\
\midrule
Extraction density & 0.49 & 0.99 & 0.88 & 0.69 & 1.80 & 0.93 \\
%Empty field rate & 10.61 & 1.71 & 0.31 & 5.58 & 16.35 & 1.77 \\
\% Top fields overlap & 62.2 & 60.0 & 64.4 & 55.6 & 62.2 & 62.2 \\
\% Date values & 37.5 & 32.1 & 28.3 & 40.2 & 37.4 & 35.8 \\
Fragment length & 34.92 & 41.44 & 46.51 & 57.58 & 28.72 & 48.74 \\
\bottomrule
\end{tabular}
}


\caption{Evaluation results from different experiments, showing average scores across all input documents.}



\label{tab:Results}
\end{table*}


We report the empirical results of our experiments and analyse the behavior of the proposed GenIE framework for the configurations described in Section \ref{sec:Experiments}. First, we present a quantitative assessment using the evaluation metric described in Section \ref{sec:evaluation} complemented by several supplementary measures, then we conduct a qualitative error analysis that contextualizes observed performance patterns.

\subsection{Performance}


Table \ref{tab:Results} summarizes performance at the document level across all configurations. Overall precision was relatively stable (0.61–0.71), while recall varied substantially (0.33–0.75), resulting in F1 scores between 0.45 and 0.68. The strongest configuration, \textsc{larger-m exmp+}, achieved the highest F1 (0.68) driven by a large recall gain (0.75), at the cost of reduced precision (0.62). 

Extraction density, i.e. the number of units generated per 100 input characters, shows a clear positive correlation with recall and F1, implying that configurations producing more extraction units consistently achieve higher coverage, despite introducing additional false positives (FP). \textsc{larger-m exmp+} exhibits the highest extraction density (1.80 units per 100 characters) paired with one of the lowest false negative (FN) rates (6.53\%), supporting the intuition that under-generation is more detrimental than moderate over-generation.

Our experiments show that including more examples in the prompt (i.e., few-shot prompting) produced the most substantial performance gains, particularly for the larger model (+0.33 recall from \textsc{larger-m} to \textsc{larger-m exmp+}). In contrast, CoT prompting yields only marginal F1 improvements, primarily by trading precision for recall, and multi-step execution with self-refinement degrades performance due to severely reduced extraction density. The direct model comparison reveals that simply using the larger model without prompt optimisation did not improve performance.

Beyond core metrics, the distribution of extraction fields in the system's output largely aligned with the evaluation set, displaying an overall overlap of 60-64\%. In both sets, \textit{Education}, \textit{Birth}, and \textit{Employment} events were among the most prevalent event types. However, \textit{Granting} events (i.e., when an author was awarded a prize) were extracted less frequently by the GenIE framework, while \textit{ForcedStay} and \textit{MilitaryService} type events gained relative importance. We note that the system output exhibits a systematic bias towards the extraction of date values (e.g., \texttt{13-01-1970}). The percentage of extracted information units containing these values was consistently higher, ranging from 28.3\% to 40.2\% across configurations, compared to only 9.9\% in the gold annotations.

Overall, the results highlight extraction density and example-based prompting as the main drivers of performance, while CoT prompting, multi-step task decomposition, and model scaling in isolation provided limited benefit.


\subsection{Error Analysis and Discussion} \label{sec:error-analysis}

Manual inspection of system outputs\footnote{An in-depth analysis of an example output can be viewed on \href{https://github.com/git-the-language-nerd/Example-Analysis---GenIE-from-Biographical-Sources-Long-Paper-/blob/da257a0039e720d82732a3e9a15c572bcff257c8/}{github.com}} reveals that human annotations were frequently incomplete. As a result, many factually correct system extractions absent from the gold data, were classified as FPs. Precision scores therefore need to be interpreted with caution, as they partly reflect annotation coverage rather than true extraction quality. For example, in one document, the system correctly extracted information on the social background of author Adolf Görtz (\textit{son of a factory worker}), but no corresponding annotation exists in the evaluation set. Such cases are penalized as FPs despite being grounded in the text, highlighting an inherent limitation of annotation-based evaluation for open-ended IE.

Beside these "false FPs", a common pattern concerned surface-level variation between extracted values and gold annotations. Here, the evaluation metric proved rather robust since it was specifically designed to account for this type of variation (cf. Section \ref{sec:evaluation}). For example, in the case of one article that mentioned an author's employment in "Children's and young adult literature", the GenIE system extracted this value literally while human annotators simply noted "Children's books" in the equivalent field. In these cases, the embedding-based metric assigns partial credit based on semantic similarity, yielding high but sub-perfect scores. While this systematically pulls evaluation scores away from full precision, it reflects a fairer assessment than exact-match evaluation, which would fail to credit semantic similarity entirely.

Less frequently, errors arose from metric failures. For instance, the historically equivalent place names (e.g., \textit{Chemnitz} vs. \textit{Karl-Marx-Stadt}) were treated as dissimilar in one case. This type of error reduces scores despite factual correctness and highlights that evaluation performance is partly constrained by the robustness of the embedding model. Future work could mitigate such cases by incorporating external norm data or taxonomies for entities such as places and institutions.

The manual analysis further reveals that truly invalid extractions, e.g. units with uninterpretable strings, were rare and were assigned zero scores by the evaluation pipeline. Similarly infrequent were extractions with missing values, which typically reflected underspecified information in the source text. Finally, a small number of FNs received non-zero scores due to alignment with semantically related but incorrect gold fields. Although suboptimal, analysis suggests that these cases had a limited practical and numerical impact on overall performance.


\section{Conclusion}

This study presents, to the best of our knowledge, the first systematic investigation of large-scale IE from biographical texts driven by midsized general-purpose LLMs. The results indicate that the proposed GenIE framework has great potential for effective application in this domain, enabling the extraction of structured biographical information with minimal preparation effort required. 

Unlike many similar studies, we adopt an open IE setting that allows for unrestricted output generation. To tailor to this task formulation, we propose a semantic similarity–based evaluation metric. Experiments were conducted on a curated, unpublished dataset of German biographical encyclopedia entries, with domain expert annotations serving as gold references. 

We observe that the precision of the IE framework remained relatively stable (approximately 0.61–0.71), while recall varied substantially (approximately 0.33–0.75). While limited precision is partly attributable to incomplete annotation coverage and is therefore less concerning, the absence of high recall remains a notable limitation. At the same time, recall and F1 show a strong positive correlation with extraction density, suggesting that under-generation is more detrimental than moderate over-generation in this setting.

Manual analysis further confirms that many apparent FPs reflect missing gold annotations rather than factual errors, and that the evaluation metric largely behaves as intended. The substantial overlap in extraction field distributions between system output and annotations provides an encouraging signal of structural alignment, despite a systematic bias toward date value extractions. 

Controlled ablation experiments reveal that PE strategies offered limited performance gains. Clear prompt formulation and few-shot prompting yield the most consistent improvements, whereas advanced reasoning strategies such as CoT provided little benefit.

Overall, while the output quality of our framework is encouraging, further investigation will be required to fully assess the capabilities of such systems. With appropriate refinement and optimization, the proposed framework may serve as a template for practical deployment. Future research may extend it to new domains, improve open GenIE evaluation methods, and explore a wider range of models.


%Outlook: merging/comparing extraction results from various texts for a single author \\






\section*{Limitations and Ethical Considerations}

The dataset used in this study is focused on a specific historical period, namely authors from the GDR and the evaluation subset primarily features one text type, i.e. encyclopedia entries. Results may not generalize to other domains or (low-resource) languages. Due to resource constraints, experiments used only two open-source LLMs and modest inference parameters. Proprietary models or extensive parameter tuning might yield better performance. This study does not distinguish or quantify different types of hallucinations, focusing instead on the overall occurrence of extraction errors. Evaluation methods for open-ended IE remain imperfect and future research should explore more flexible metrics. Future studies using our proposed metric must ensure the robustness of the core embedding model. Furthermore, the evaluation metric is not yet sensitive to multiple occurrences of events of the same type, as it matches extraction units against all events of the detected type in the annotations for a given source text. 

Ethically, LLM use entails significant environmental costs, potential bias, and limited interpretability. Biographical data on living individuals should be extracted only with caution, as it can enable the generation of detailed personal profiles \citep{Profile-Generation-from-web-sources-an-IE-system--Ranjan-etal-2022}. While such information may serve legitimate purposes, e.g. in law enforcement, it may also carry risks of misuse for malicious purposes.



\section*{Acknowledgements}

%The authors thank the University of Potsdam for providing access to  computational resources. 
We are deeply grateful to the \flfddr\ project team for creating the corpus that made this study possible. We would also like to thank the reviewers for their valuable and detailed feedback.

% Bibliography entries for the entire Anthology, followed by custom entries
%\bibliography{anthology,custom}
% Custom bibliography entries only

% JK: wenn die Formatänderung nicht zulässig ist, müsste small entfernt werden...
{
% evtl. Formatierung ändern? footnotesize würde uns Platz verschaffen... aber auch noch  nicht genug, wie ich gerade sehe...
%\footnotesize
\bibliography{custom}
}

\pagebreak
\onecolumn

\appendix

\section{Appendix}
\label{sec:appendix}

\begin{table}[h]
\centering
\makebox[\textwidth][c]{%
\resizebox{1.00\linewidth}{!}{%
\begin{tabular}{lrlr}
\toprule
\multicolumn{2}{c}{\textbf{Entire dataset}} & \multicolumn{2}{c}{\textbf{Evaluation data}} \\
\cmidrule(lr){1-2} \cmidrule(lr){3-4}
Source & \#Docs & Source & \#Docs \\
\midrule
Deutsche Biographie & 141&
Deutsche Biographie& 34\\
 Deutsches Literatur-Lexikon& 116&  Deutsches Literatur-Lexikon& 21\\
Wikipedia& 67& Schriftsteller der DDR (Boettcher)& 13\\
Schriftsteller der DDR (Boettcher)& 45&  KuerschnerNekrologe1971-1998& 6\\

 Kuerschner 2012& 26& Chemnitzer Autoren&6\\
 \bottomrule
\end{tabular}%
}}
\caption{Top sources in the entire data set vs. evaluation subset.}
\label{tab:source-lexicons-counts}
\end{table}






\begin{table*}[h]
\centering
\makebox[\textwidth][c]{%
    \resizebox{1\textwidth}{!}{%
        \begin{tabular}{|l|p{15cm}|}
        \hline
        \textbf{Purpose} & \textbf{Prompt part} \\
        \hline
        define system role & {You are an information extraction assistant. Output must be valid JSON string with the keys 'SUBJECT', 'EVENT\_TYPE', 'ATTRIBUTE', 'VALUE' and the correct values corresponding to these keys.} \\ \hline
        
        instructions & {From the input, extract structured information units that each represent a single biographical event and have these fields: 'SUBJECT', 'EVENT\_TYPE', 'ATTRIBUTE', 'VALUE'"} \\ \hline
        
        avoid inference & {Only extract information present in the INPUT. Do not add new information or any information from your training data.} \\ \hline
        
        restrict scope (fields) & {Consider only the following EVENT\_TYPEs and ATTRIBUTES, skip all others:  [...]} \\ \hline
        
        restrict scope (subject) & {Only structured information on the following target person is relevant: [...]} \\ \hline
        
        source fidelity & {Keep the original wording of the short text fragment where the extracted information was found. Do not change the original wording}\\ \hline

        output format & {In your response, do not add any explanation or comments, instead return only the structured data.} \\ \hline
        
        completeness & {It is important that you exract as much information as possible!!} \\ \hline
        
        Few shot instructions & {The following is an example for the correct OUTPUT you should generate for a given INPUT. INPUT:} \\ \hline

        example input & {INPUT: Görtz, Adolf Erzählung (Kinder- und Jugendliteratur) 31.5. 1920, Köln. 1958-61 Studium am Institut für Lit. „Johannes R. Becher“. Ehrenurk. im Preisausschreiben anläßlich der IV. Weltfestspiele der Jugend und Studenten (1953); 3. Preis im Preisausschreiben des Ministeriums für Kultur (Kinder- und Jugendliteratur) (1953); Lobende Anerkennung im gleichen Preisausschreiben (1954), Leipzig C 1, Jacobstraße 14. Schriftsteller der Deutschen Demokratischen Republik. Bearbeiter des biographischen Teils: Joachim Ret, Egon Sartorius. Bearbeiter des bibliographischen Teils: Helmut Donner, Hans Heininger. Leipzig 1961.} \\ \hline

        example output & \begin{tabular}{p{0.95\linewidth}}
fragment\_text:``Görtz, Adolf Erzählung (Kinder- und Jugendliteratur)''\\
extracted\_info:[{SUBJECT:Adolf Görtz, \\
EVENT\_TYPE:Employment, \\
ATTRIBUTE:professionsOfEmployment, \\
VALUE:Kinderliteratur, Jugendliteratur}]\\
\\
fragment\_text:``31.5. 1920, Köln''\\
extracted\_info:[{SUBJECT:Adolf Görtz, \\
EVENT\_TYPE:Birth, \\
ATTRIBUTE:begin, \\
VALUE:1920-05-31}, \\
{SUBJECT:Adolf Görtz, \\
EVENT\_TYPE:Birth, \\
ATTRIBUTE:place, \\
VALUE:Köln}]\\
\\
fragment\_text:``1958--61 Studium am Institut für Lit. ``Johannes R. Becher'' ''\\
extracted\_info:[{SUBJECT:Adolf Görtz, \\
EVENT\_TYPE:Education, \\
ATTRIBUTE:institution, \\
VALUE:Literaturinstitut Johannes R. Becher}, \\
{SUBJECT:Adolf Görtz, \\
EVENT\_TYPE:Education, \\
ATTRIBUTE:begin, \\
VALUE:1958}, \\
{SUBJECT:Adolf Görtz, \\
EVENT\_TYPE:Education, \\
ATTRIBUTE:end, \\
VALUE:1961}]\\
\\
fragment\_text:``Ehrenurk. im Preisausschreiben anläßlich der IV. Weltfestspiele der Jugend und Studenten (1953)''\\
extracted\_info:[{SUBJECT:Adolf Görtz, \\
EVENT\_TYPE:

[...]
}
\end{tabular}

        
        
        \\ \hline
        \end{tabular}
    }%
}
\caption{Prompt used in the experiments (shortened and re-formatted for readability).}
\label{tab: prompt}
\end{table*}











%\label{app: Example Output 1}

%The following pages document and analyze an example output. Note that this example was produced in a preliminary test run, i.e. not using the revised and optimized setup.

%pagebreak

%\includepdf[pages=-, fitpaper=true]{figures/Example 1 commented - art00143 (ohne post-proc).pdf}

\end{document}
