%
% File acl2020.tex
%
%% Based on the style files for ACL 2020, which were
%% Based on the style files for ACL 2018, NAACL 2018/19, which were
%% Based on the style files for ACL-2015, with some improvements
%%  taken from the NAACL-2016 style
%% Based on the style files for ACL-2014, which were, in turn,
%% based on ACL-2013, ACL-2012, ACL-2011, ACL-2010, ACL-IJCNLP-2009,
%% EACL-2009, IJCNLP-2008...
%% Based on the style files for EACL 2006 by 
%%e.agirre@ehu.es or Sergi.Balari@uab.es
%% and that of ACL 08 by Joakim Nivre and Noah Smith

\documentclass[11pt,a4paper]{article}
\usepackage[hyperref]{acl2020}
\usepackage{times}
\usepackage{latexsym}
\usepackage{listings}
\usepackage{graphicx}
\usepackage{subfig}
\renewcommand{\UrlFont}{\ttfamily\small}

% This is not strictly necessary, and may be commented out,
% but it will improve the layout of the manuscript,
% and will typically save some space.
\usepackage{microtype}

\usepackage{color}
\newcommand{\hong}[1]{{\color{red}Hong: #1}}
\newcommand{\fei}[1]{{\color{blue}Fei: #1}}
\newcommand{\yonghao}[1]{{\color{green}Yonghao: #1}}
\usepackage{soul}
\aclfinalcopy % Uncomment this line for the final submission
%\def\aclpaperid{***} %  Enter the acl Paper ID here

%\setlength\titlebox{5cm}
% You can expand the titlebox if you need extra space
% to show all the authors. Please do not make the titlebox
% smaller than 5cm (the original size); we will check this
% in the camera-ready version and ask you to change it back.

\newcommand\BibTeX{B\textsc{ib}\TeX}

\title{BENTO: A Visual Platform for Building Clinical NLP Pipelines \\Based on CodaLab}



\author{Yonghao Jin, Fei Li \and Hong Yu\\
  Department of Computer Science, University of Massachusetts \\ 
  Lowell, MA, USA \\
  }
%   \texttt{yonghao.jyh1@gmail.com} \\
%   \And
%   Fei Li \\
%   Department of Computer Science\\ 
%   University of Massachusetts\\ 
%   Lowell, MA, USA \\
%   \texttt{foxlf823@gmail.com} \\
%   \AND
%   Hong Yu \\
%   Department of Computer Science\\
%   University of Massachusetts Lowell\\
%   Lowell, MA, USA \\
%   \texttt{hong\_yu@uml.edu} \\
% }


\date{}

\begin{document}
\maketitle
\begin{abstract}

%CodaLab\footnote{\href{https://codalab.org/}{codalab.org}} is a widely-used platform for conducting and managing computational research. 
CodaLab\footnote{\href{https://codalab.org/}{codalab.org}} is an open-source web-based platform for collaborative computational research. Although CodaLab has gained popularity in the research community, its interface has limited support for creating reusable tools that can be easily applied to new datasets and composed into pipelines.
% , i.e. composing . Unlike single-model-based applications such as image processing, pipeline is important for NLP applications.
% \hong{you may want to rephrase the context --- all I want to say is that you need to highlight "pipeline" and WYSIWYG (what you see is what you get)}
% \yonghao{edited}
In clinical domain, natural language processing (NLP) on medical notes generally involves multiple steps, like tokenization, named entity recognition, etc. Since these steps require different tools which are usually scattered in different publications, it is not easy for researchers to use them to process their own datasets. In this paper, we present \textit{BENTO}, a workflow management platform with a graphic user interface (GUI) that is built on top of CodaLab, to facilitate the process of building clinical NLP pipelines. BENTO comes with a number of clinical NLP tools that have been pre-trained using medical notes and expert annotations and can be readily used for various clinical NLP tasks. It also allows researchers and developers to create their custom tools (e.g., pre-trained NLP models) and use them in a controlled and reproducible way. In addition, the GUI interface enables researchers with limited computer background to compose tools into NLP pipelines and then apply the pipelines on their own datasets in a ``what you see is what you get'' (WYSIWYG) way. Although BENTO is designed for clinical NLP applications, the underlying architecture is flexible to be tailored to any other domains.

% as a use case, we have integrated  into BENTO. These NLP models have been pre-trained using medical notes and expert annotations, which can be used for various clinical NLP tasks, such as de-identification and adverse drug event extraction. 

\end{abstract}

\begin{figure}[t]
\includegraphics[width=0.48\textwidth]{images/architecture.png}
\caption{The architecture of BENTO. The BENTO back end 
% \hong{you mean "front end", not "back end"?}
% \yonghao{The front end is the web interface, here is the back end that stores tools}
stores the description files of various tools (e.g., pre-trained NLP models), processes static contents of the application and handles compilation of the user-defined pipelines. The CodaLab back end stores the datasets (bundles) and executes computational jobs. The two back end servers are brought behind a single domain name using a reverse proxy server.}
\label{fig:architecture}
\centering
\end{figure}

\section{Introduction}
With the machine learning research going deep, computational models are becoming increasingly large with intensive hyper-parameters tuning, making the research difficult to reproduce. To tackle this problem, researchers 
% \hong{should you say researchers or microsoft? By the way are they for NLP only? I thought that it could be used for any software development, including imaging, etc? on the second thought, it may be easier for you to just say software, ...}
% \yonghao{i think researchers is fine, since it is an open source project. I have changed NLP to machine learning.}
have developed CodaLab as an open-source platform for researchers and software developers. However, CodaLab has limited support for reusable tools that can be easily applied to different datasets and be composed into computational pipelines. 

% Processing EHRs with NLP techniques is a crucial subject since electronic health records (EHRs) contains a great number of narrative notes and they should be pre-processed before utilized by downstream healthcare applications \citep{xia2013modeling, yu2014classification, rochefort2015accuracy}.
Building pipelines is essential for the research of certain domains. Take the medical informatics research as an example, a complete NLP analysis on medical notes often involves multiple steps like tokenization, de-identification \citep{dernoncourt2017identification, liu2017identification}, entity recognition \citep{Li2018ade, xu2017bidirectional, jagannatha2016structured} and normalization \citep{Li2019norm, li2017cnn, cho2017method}, relation extraction \citep{Li2018ade, he2019classifying}, etc. Since these steps require different tools and these tools are usually scattered in different publications, it is far from trivial to leverage these tools on new datasets even though the authors have released the source code. 
Therefore, we developed a user-friendly workflow management platform, BiomEdical Nlp TOolkits (\textit{BENTO}), to facilitate the process of building and applying of clinical NLP pipelines.


The architecture of BENTO is illustrated in Figure \ref{fig:architecture}. BENTO has three main components. The web interface is supported by two back ends brought together by a reverse-proxy server. The CodaLab back end stores the datasets and executes computational jobs. The BENTO back end serves tool information and transforms user-defined pipelines to CodaLab commands.

The advantages of such architecture are two-fold. First, it is flexible to use CodaLab as the back end for adding custom tools (e.g., pre-trained NLP models) and processing data in a controlled and reproducible way. All the tools are containerized with Docker\footnote{\href{https://www.docker.com/}{docker.com}}, which makes the platform to keep a unified interface to manage the models and not need to maintain different operating environment for different models. Second, the web interface makes it easier for users to construct NLP pipelines through editing flowcharts and then apply the pipelines to their data. The web-based architecture also makes the platform widely accessible without complex installation and configuration.
% \hong{the next few sentences need to be revised. as they seem to come out from nowhere. You could say something like this to start: We chose those clinical NLP tools due to their importance in EHR-based research. 
% the above is good. I think that you need to summerize it clearly to incorporate into the Abstract, and earlier on Introduction.  Don't wait too late.}
% \yonghao{the original sentences are moved up}

In this paper, we also show the examples of using BENTO to integrate several clinical NLP applications such as hypoglycemia detection \citep{Jin2019hypoglecimia} and adverse drug event extraction \citep{Li2018ade}, and build pipelines based on these tools. BENTO helps build NLP pipelines, which is a promising system to accelerate the medical informatics research.



\section{Related Work}
Galaxy \citep{afgan2018galaxy} is a similar computational platform that is focused in bioinformatics and computational biology, whose interface inspires the design of ours.
%%HOng: this sentence needs to rewrite: It heavily inspires our design for the user interface. 
The main restriction of the Galaxy platform is that users can only access the tools managed by administrators and cannot define their own tools. 
In linguistic research community, other related platforms include lingvis.io \citep{el2019lingvis}, which is focused on integrating NLP operations with visualizations , and Argo \citep{rak2012argo}, a web-based text mining workbench based on the UIMA framework.
Stanford CoreNLP \citep{manning2014stanford} provides a commonly used NLP tool set. On the library level, NLTK \citep{hardeniya2016natural} is a popular Python library that integrates multiple widely used NLP tools. OpenNLP \citep{morton2005opennlp} is a Java library that provides machine learning based toolkits for NLP tasks. FudanNLP \citep{qiu2013fudannlp} is a Java based library which integrates the machine learning models and datasets for Chinese NLP. 

In the medical domain, NILE \citep{yu2013nile} is a Java package which includes rule based NLP methods for information extraction from medical notes. 
Apache cTAKES \citep{apache2018clinical} and CLAMP \citep{soysal2018clamp} are two clinical NLP systems with pipeline-based architecture in the UIMA framework. 
Both systems have a graphical user interface, allowing users to build pipelines from build-in UIMA components.
However, the UIMA framework has a steep learning curve. It is also not widely used in the machine-learning-based NLP research. 
Furthermore, most NLP applications are often released as command line programs. Therefore, it is hard to extend applications that use the UIMA framework with new models. In contrast,  
tools on our BENTO platform are based on command line programs and users can easily define their own tools with little restriction.
% \hong{You need to say more about cTAKES, as it is also a pipeline-based}
% \hong{what about the CLAMP tool?  It is pipeline based, and it also has a user interface.  you need to compare the differences https://clamp.uth.edu/tutorial.php; https://academic.oup.com/jamia/article/25/3/331/4657212}
% \yonghao{updated}


\section{System Description}

BENTO mainly comprises three parts: a front-end web application, a BENTO back end server and a CodaLab back end. As shown in Figure \ref{fig:architecture}, BENTO has a web-based user interface, from which 
%the user interface is the BENTO web application. The BENTO web application provides the user interface which can be accessed by a browser. 
users can upload data, edit tools, submit jobs and perform various other operations. The BENTO back end is a web server that is mainly used for storing the tools, including the user-defined ones, so they can be accessed in different sessions. The CodaLab back end is used for execution of each computational job. When a tool is being executed, BENTO will generate a series of CodaLab commands based on the tool information and the input bundles. The outputs of the tool are the run bundles generated from those commands
which can be passed on to the down-stream tools and inspected by the users on the CodaLab interface.


\begin{figure*}[!ht]
    \centering
    \includegraphics[width=\textwidth]{images/interface.png}\par
    \caption{BENTO Web Interface. The interface can be roughly divided into three parts from left to right: tool panel, canvas panel and worksheet panel. The tool panel lists the current available tools organized in a tree view. The canvas panel contains the flowchart of the current pipeline. Every node represents a tool or dataset and each connection indicates the data flow in the pipeline.\footnotemark[3] This figure shows an example of the pipeline for entity and relation extraction. The worksheet panel displays the content of the CodaLab worksheet such as bundles and their UUIDs.}
    \label{fig:interface}
    
\end{figure*}


\subsection{Web Interface}
As shown in Figure \ref{fig:interface}, the user interface of our platform is a web application 
that can be roughly divided into three panels from left to right: tool panel, canvas panel and worksheet panel. The tool panel lists the current available tools on the platform organized in a hierarchical file system structure along with the meta information. Users can edit the \emph{User Tools} folder using the buttons listed on the top menu bar. To run a tool, users can simply drag it to the canvas panel to the right and a tool node will appear on the canvas. A node, shown in the workflow Figure \ref{fig:interface}, contains several input and output ports, corresponding to the inputs and outputs of the tool.

\footnotetext[3]{For simplicity, pre-processing steps like tokenization is built-in in each tool.}

% \begin{figure}[h]
%     \centering
%     \includegraphics[width=0.3\textwidth]{images/toolnode.png}
%     \caption{Tool Node}
%     \label{fig:toolnode}
% \end{figure}

Tool nodes can be linked together to form a pipeline and the connections represent the data flow during execution (Figure \ref{fig:interface}). A connection starts from an output port and ends in an input port. An input port accepts only a single connection while an output port can initiate one or multiple connections. Users can edit the tool by clicking the \emph{Editor button}
(\includegraphics[height=\fontcharht\font`\B]{images/button.png}) on the top right corner 
and the node will be toggled to an editor interface (Figure \ref{fig:toolnodeeditor}). The editor contains the expression of the tool (Section \ref{sec:bento_back_end}), which can be modified by the users. The rightmost part is the worksheet panel that displays the content of the current selected worksheet. Worksheets are editable markup documents provided by CodaLab.
Dragging a bundle entry from the worksheet panel to the canvas will create a data node.
A data node is similar to the tool node except that it does not have any input port which naturally represents a data entity in a computational pipeline.


\begin{figure}[h]
    \centering
    \includegraphics[width=0.48\textwidth]{images/tooleditor.png}
    \caption{The CodaLang expression for the tool NER in Figure \ref{fig:interface}. The expression can be roughly split into three sections indicated by the dashed squares. The first section declares the arguments of this tool. As seen, the tool takes three bundles as inputs: \emph{config}, \emph{input} and \emph{pretrained\_model}. The second section declares a constant \emph{code} which is initialized with an existing bundle. The third section is a string template for generating the CodaLab command. }
    \label{fig:toolnodeeditor}
\end{figure}

\subsection{CodaLab Back End}
An important design goal of BENTO is flexibility. Users should be able to easily define their own tools on the BENTO platform and customize existing tools at the command line level. For this reason, we use CodaLab as the back end for tool execution on the BENTO platform. CodaLab is a cloud-based platform designed for running computational experiments in data-oriented research. In CodaLab, researchers can easily set up a reproducible environment and run arbitrary command line by specifying a docker image and bundle dependencies. In CodaLab, bundles are immutable objects that hold the content of datasets. The output files produced by that command will be saved into a new bundle and can be further passed to down-stream experiments.

All datasets in BENTO are stored as CodaLab bundles. The tools and pipelines will be compiled into CodaLab commands. Users could submit commands to the CodaLab back end via the web interface. Such design makes the computational results of the BENTO platform reproducible through CodaLab. Since CodaLab will record dependency information in run bundles, it is also easy to recreate the pipeline on our platform from existing result bundles. Using CodaLab as the back end also mitigates the engineering challenges such as job scheduling and data management.

\subsection{BENTO Back End}
\label{sec:bento_back_end}

The BENTO back end is for storing tools and generating CodaLab commands from the pipeline graphs.

\subsubsection{CodaLang: A Tool Configuration Language}
The tools in BENTO are described via our custom language called \emph{CodaLang}\footnotemark[4]
% \footnote{A thorough introduction can be found at https://github.com/jyh1/codalang}.
It acts as an intermediate layer between the web interface and CodaLab. It has a succinct syntax for specifying the interfaces of a tool, i.e. the inputs and outputs. It also provides a string template mechanism for creating CodaLab commands from input arguments. For example, the CodaLang expression for the node NER in Figure \ref{fig:interface} is shown in Figure \ref{fig:toolnodeeditor}. 

\footnotetext[4]{A thorough introduction can be found at https://github.com/jyh1/codalang .}

The configuration is composed of three sections which are highlighted with dotted squares. The first section declares the arguments of the tool, corresponding to the three input ports of the node. The second section creates a constant variable \emph{code} which is assigned an existing bundle. The third section is a string template for generating the CodaLab command. It includes execution options (e.g., \textit{request-docker-image}) and tool bash commands. The template variables are circled by the squares in the same color with their declarations. Once the values of the tool arguments are determined, a CodaLab command can be easily generated based on the command template. The run bundle created by the command will be used as results and can be passed on to down-stream tools in the pipeline. Through CodaLang, users can easily modify existing tools or create their own tools. The tool configuration can also be automatically generated from the dependency information of a bundle.


\begin{figure}[t]
    \centering
    \includegraphics[width=0.48\textwidth]{images/codalabcommand.png}
    \caption{
    The CodaLab commands generated from the pipeline in Figure \ref{fig:interface}. Two CodaLab commands are generated based on two steps in the pipeline, 
    namely NER and relation extraction. The bundle dependency information is highlighted in orange and the shell commands are colorized in red. The results in the first step are saved in the variable \textit{bundle\_0} (circled in blue squares), which is used as a bundle dependency in the command of the second step.}
    \label{fig:codalabcommand}
\end{figure}

\subsubsection{Pipeline Execution}

We have described how BENTO transforms a single tool to a CodaLab command. In this section, we will describe how BENTO transforms a tool pipeline into multiple CodaLab commands. In a tool pipeline, tools are connected together to form a directed acyclic graph. During execution, tools are transformed to CodaLab commands according to their topological order in the graph. Take the pipeline in Figure \ref{fig:interface} as an example, its corresponding CodaLab commands shown in Figure \ref{fig:codalabcommand}.

As shown in Figure \ref{fig:codalabcommand}, the bundle dependency information is highlighted in orange and the shell commands are colorized in red. The two CodaLab commands correspond to the two tool nodes in the pipeline of Figure \ref{fig:interface}. The first command is generated from the tool NER based on its tool configuration in Figure \ref{fig:toolnodeeditor}. The results of this command are saved in the variable \textit{bundle\_0}, which will be employed as a bundle dependency in the command of the tool for relation extraction. The web interface takes the responsibility of submitting the generated commands to CodaLab. When the pipeline begins to run, the worksheet panel will display the information of the newly created run bundles.


\section{Tools Integrated in BENTO}
In this section, we list the tools that have already been integrated to our platform, including:

\begin{itemize}
\item Hypoglycemic Event Detection \citep{Jin2019hypoglecimia}: Hypoglycemic events are common and potentially dangerous conditions among patients being treated for diabetes. This tool can be used to automatically detect hypoglycemic events from EHR notes.
\item Clinical Entity Recognition \citep{Li2018ade}: This tool has been built to recognize 9 types of clinical entities such as medications, indications and adverse drug events (ADEs).
\item Clinical Relation Extraction \citep{Li2018ade}: This tool is able to extract 7 types of relations between clinical entities such as medications and their durations, dosages and frequencies.
\item Disease Name Normalization \citep{Li2019norm}: This tool can be used to normalize disease names to some controlled vocabularies such SNOMED\footnotemark[5]
% \footnote{https://www.snomed.org} 
and MEDIC \citep{davis2012medic}.
\item De-identification: This tool is able to recognize 18 types of protected health information that needs to be removed to de-identify patient notes. We employed BERT \citep{ devlin-etal-2019-bert} to build a de-identification model whose performance is comparable with the state-of-the-art system \citep{dernoncourt2017identification}.
\end{itemize}

\footnotetext[5]{https://www.snomed.org}

We provide examples and instructions to use these tools on the \href{https://bio-nlp.org/bentodemo/}{demo page} of our platform. For convenience, these tools all take plain text files as inputs and have the pre-processing and tokenization components built-in. In the future, we will integrate stand-alone components dedicated for pre-processing and tokenization to BENTO which can be shared by different application tools. We also plan to incorporate more NLP tools developed by our group\citep{rumeng2017hybrid,rawat2019naranjo,lalor2019improving,zheng2018assessing}. 

\section{Conclusion}
In this paper, we have described the design of the workflow management platform BENTO. To the best of our knowledge, BENTO represents the first web-based workflow management platform for NLP research. Using BENTO, researchers can make use of existing tools or define their own tools. Computational pipelines can be configured through a web-based user-interface and then automatically executed on CodaLab. BENTO includes a number of clinical NLP tools to facilitate the process of EHR notes. A demo of our platform is available at \href{https://bio-nlp.org/bentodemo/}{bio-nlp.org/bentodemo/}.

\bibliographystyle{acl_natbib}
\bibliography{acl2020}

\end{document}
