\documentclass[10pt]{article}

\usepackage{fullpage}
\usepackage{setspace}
\usepackage{parskip}
\usepackage{titlesec}
\usepackage[section]{placeins}
\usepackage{xcolor}
\usepackage{breakcites}
\usepackage{lineno}
\usepackage{hyphenat}





\PassOptionsToPackage{hyphens}{url}
\usepackage[colorlinks = true,
            linkcolor = blue,
            urlcolor  = blue,
            citecolor = blue,
            anchorcolor = blue]{hyperref}
\usepackage{etoolbox}
\makeatletter
\patchcmd\@combinedblfloats{\box\@outputbox}{\unvbox\@outputbox}{}{%
  \errmessage{\noexpand\@combinedblfloats could not be patched}%
}%
\makeatother


\usepackage[round]{natbib}
\let\cite\citep




\renewenvironment{abstract}
  {{\bfseries\noindent{\abstractname}\par\nobreak}\footnotesize}
  {\bigskip}

\titlespacing{\section}{0pt}{*3}{*1}
\titlespacing{\subsection}{0pt}{*2}{*0.5}
\titlespacing{\subsubsection}{0pt}{*1.5}{0pt}


\usepackage{authblk}


\usepackage{graphicx}
\usepackage[space]{grffile}
\usepackage{latexsym}
\usepackage{textcomp}
\usepackage{longtable}
\usepackage{tabulary}
\usepackage{booktabs,array,multirow}
\usepackage{amsfonts,amsmath,amssymb}
\providecommand\citet{\cite}
\providecommand\citep{\cite}
\providecommand\citealt{\cite}
% You can conditionalize code for latexml or normal latex using this.
\newif\iflatexml\latexmlfalse
\providecommand{\tightlist}{\setlength{\itemsep}{0pt}\setlength{\parskip}{0pt}}%

\AtBeginDocument{\DeclareGraphicsExtensions{.pdf,.PDF,.eps,.EPS,.png,.PNG,.tif,.TIF,.jpg,.JPG,.jpeg,.JPEG}}

\usepackage[utf8]{inputenc}
\usepackage[ngerman,english]{babel}










\begin{document}

\title{Natural language processing applied to incident reports in radiation
oncology--determination of completeness and classification}



\author[1]{Hui Wang}%
\author[1]{John Kildea}%
\author[1]{felixkply}%
\author[1]{Felix Mathew}%
\affil[1]{Affiliation not available}%


\vspace{-1em}



  
  \date{May 05, 2024}


\begingroup
\let\center\flushleft
\let\endcenter\endflushleft
\maketitle
\endgroup





\selectlanguage{english}
\begin{abstract}
\textbf{Purpose:~}Incident learning in radiotherapy may improve the
quality of care by reducing the recurrence of incidents. Incident
reports are often descriptions in free-text format,~which makes it
difficult to extract information efficiently and determine if a report
is complete. Natural language processing (NLP) is a proven technique in
extracting information from clinical texts and classifying safety
reports, but the utility of NLP in classifying radiotherapy incident
reports is untested. This project illustrates an NLP approach to
classifying radiotherapy incident reports by the process step of
incident occurrence, and determining if a report is complete.

\textbf{Methods:} A~preliminary analysis that consisted of tagging of
parts-of-speech (POS)~ and annotation with the Unified Medical Language
System (UMLS) was conducted on 519 reports. The summary statistics of
the preliminary analysis was used to compare fictitious reports deemed
complete and reports of real incidents. Multinomial classification by
the process step of incident occurrence was transformed into multiple
binomial classifications by one-hot encoding. The reports were
subsequently cleaned and vectorized into a bag of words, on which the
naive binomial bayesian classifiers were trained and tested.
Receiver-operating-characteristic (ROC) curves were then plotted to
evaluate each binomial classifier.

\textbf{Results:}~The number of occurrence of several POS were
statistically different in the fictitious and real reports, and the
performance of the binary classifiers varied from poor to excellent with
an average area under the curve (AUC) of 0.85.

\textbf{Conclusion:} Determining if a report is complete requires domain
knowledge outside the scope of statistical NLP. However, NLP can
classify incidents by process step with fair performance, and
demonstrates the potential to classify incidents by other data elements
of the~Canadian National System for Incident Reporting - Radiation
Treatment~(NSIR-RT).%
\end{abstract}%



\sloppy


\section*{Introduction}

{\label{460459}}

Radiotherapy is the use of ionizing radiation to treat cancer, and is
used in almost two thirds of cancer treatments~\cite{Kaur_2011,Montgomery2016}. The
procedure is multi-step and highly complex, and involves many different
staff groups. The complexity of the procedure arises from the wide range
of cancers treated, technologies used, and expertise required. Although
errors only occur approximately once per 600
treatments~\cite{Ford2012,Mans2010}, the consequence to the patient can be
significant.

\par\null

Incident learning consists of identifying, reporting, and investigating
actual and potential incidents, and installing protocols to prevent the
recurrence of incidents. Since radiotherapy incidents may cause
irreversible and severe damage to the patient, incident learning must
include precursors and near misses rather than consisting exclusively of
trial-and-error~at great cost to the
patients~\href{https://www.authorea.com/users/222819/articles/277226-using-natural-language-processing-to-classify-incident-reports-in-radiation-oncology\#Milosevic2016}{(Milosevic
2016a)}. The advantages of incident learning from both actual and
potential incidents is paralleled in many industrial environments where
errors are of catastrophic consequence.
Figure~{\ref{919594}}~\cite{Jehring_1951} illustrates
the fact that in industrial environments, for every fatal incident,
there are 30 minor events and 300 near misses~\cite{Jehring_1951}. Root
cause analysis of the minor events and near misses has been shown to
yield valuable lessons that prevent serious
incidents~\cite{Cooke_2007}.~ For example, incident learning has
reduced the risk of fatal accidents by 73\% in 10 years, and is a basic
feature of nuclear operations~\cite{Ford2012}.~

\par\null\selectlanguage{english}
\begin{figure}[h!]
\begin{center}
\includegraphics[width=0.70\columnwidth]{figures/Screen-Shot-2017-12-29-at-10-34-24/Screen-Shot-2017-12-29-at-10-34-24}
\caption{{An illustration of the relative occurrence of major accidents, minor
injuries, and near misses in industrial environments.~
{\label{919594}}%
}}
\end{center}
\end{figure}

The~benefits of recursive self-improvement by incident learning has
inspired collaborative incident learning initiatives within institutions
as well as across national and international
communities~\href{https://www.authorea.com/users/222819/articles/277226-using-natural-language-processing-to-classify-incident-reports-in-radiation-oncology\#Donaldson2007}{(Donaldson
2007)}. In Canada, the National System for Incident Reporting in
Radiation Treatment (NSIR-RT) taxonomy is implemented as an online
reporting and analysis system. The taxonomy comprises 33 data elements
in 6 categories: impact, discovery, patient, details, treatment
delivery, and investigation. The taxonomy has generated a high level of
agreement within the Canadian community, and is aligned with the
American and European radiation treatment incident
classifications~\cite{Milosevic_2016}.

In January 2016, the Department of Radiation Oncology of Cedar Cancer
Centre of the McGill University Health Centre (MUHC) in Montreal, Canada
deployed the Safety and Incident Learning System (SaILS) that includes
incident reporting, investigations, tracking, and data
visualization.~The MUHC version of~SaILS is compatible with NSIR-RT
taxonomy, and has processed~519 incidents from January 2016 to September
2017. Reporting an incident requires~mainly~a description and a
one-sentence descriptor of the incident. For each incident,
investigators then retrospectively and manually classified the incidents
into one of the menu choices for each data element in the NSIR-RT
taxonomy~\cite{Montgomery2017}. For example, the process step of incident
occurrence is one of the data elements of the NSIR-RT taxonomy, and it
has the menu choices: patient assessment and consultation, imaging for
radiotherapy planning, treatment planning, pre-treatment review and
verification, treatment delivery, on-treatment quality management, and
other.~The reporting interface of SaILS is shown in
Figure~{\ref{716773}}.\selectlanguage{english}
\begin{figure}[h!]
\begin{center}
\includegraphics[width=0.70\columnwidth]{figures/Screen-Shot-2017-12-28-at-21-52-25/Screen-Shot-2017-12-28-at-21-52-25}
\caption{{{\label{716773}}
The SaILS interfaces for incident reporting and investigation.~
{\label{716773}}%
}}
\end{center}
\end{figure}

The SaILS incident reports are similar to the safety reports of various
industrial environments in that they are in an unstructured free-text
format, which makes it difficult to determine if a report has described
the incident adequately. The lack of detail may present challenges to
subsequent investigations, diminishing the benefits of incident
learning. Another inherent challenge posed by incident learning from
free-text reports is the quantity of incident reports may exceed
investigatory resources. For example, aviation and railway companies
must process hundreds of incident reports each month~\cite{gulijk2016,Tanguy2016}.

\par\null

Natural language processing (NLP) is an emerging technique for analyzing
large quantities of texts from a global point of
view~\cite{Nerbonne_2012}. For example, using NLP to classify aviation
safety reports have permitted investigators to reserve their time for
the most unique or concerning incidents~\cite{Tanguy2016}. Furthermore,
NLP has also been effective in extracting information pertaining to
disease and disease progression, and even classifying tumour status from
radiology reports~\cite{Cheng_2009}.~Nonetheless, the utility of using
NLP in classifying radiotherapy incident reports and determining if
these reports are complete is as yet untested.~

\section*{Objective}

{\label{700693}}

The objective of this study is to assess if NLP can determine the
completeness of a radiotherapy incident report, and automate the
classification of incident reports by the process step of incident
occurrence, which is one of the data elements of NSIR-RT taxonomy.

\par\null

\section*{Materials and Methods}

{\label{930916}}

\subsection*{Preliminary Analysis}

{\label{752515}}

The preliminary analysis was conducted to reveal trends in the
linguistic data of the reports. First, each word in each report were
tagged with its part-of-speech. Then, each medical terminal was tagged
with its semantic type in the Unified Medical Language System ontology.
The number of times each part-of-speech and each semantic type appears
in a report was counted. From the counts, summary statistics that
included the mean, standard deviation, minimum, maximum, and the 25th,
50\%, and 75\% percentiles, were calculated.

\par\null

\subsubsection*{Data Structure}

{\label{392345}}

The pandas library~\cite{mckinneypandas} is the most popular data analysis
tool in Python. Its data structure, the Dataframe, is similar to an
Excel sheet. Throughout this project, the Dataframe is used to store the
results, including those of basic linguistic analyses like word count
and sentence count, and the occurrence of parts-of-speech and medical
terminologies in each incident description and descriptor. An
illustration of the Dataframe is illustrated in Figure 3, and the
schematic overview of the preliminary analysis is included in that of
completeness analysis, illustrated in
Fig~{\ref{209119}}.

\par\null\selectlanguage{english}
\begin{figure}[h!]
\begin{center}
\includegraphics[width=0.70\columnwidth]{figures/Screen-Shot-2018-01-02-at-08-08-42/Screen-Shot-2018-01-02-at-08-08-42}
\caption{{A Pandas Dataframe containing 5 fictitious radiotherapy incident reports
{\label{579133}}%
}}
\end{center}
\end{figure}

\par\null

\subsubsection*{POS Tagging}

{\label{961076}}

A part-of-speech (POS) Tagger assigns a POS, such as noun or verb, to
each word. POS Tagging in the preliminary analysis is completed with the
Natural Language Toolkit (NLTK)~\cite{Loper:2002:NNL:1118108.1118117}, a Python library
that offers a comprehensive set
of~\href{https://pythonprogramming.net/natural-language-toolkit-nltk-part-speech-tagging/}{35
fine grained POS tags}, such as singular proper noun and plural proper
noun.~

\par\null

NLTK methods are only able to be used on document objects referred to as
corpuses, which must be created from plain text files. As a result, the
description and descriptor of each incident were converted into text
files, and then into corpuses through a
\href{http://www.nltk.org/api/nltk.corpus.reader.html\#nltk.corpus.reader.plaintext.PlaintextCorpusReader}{PlainTextCorpusReader
constructor}. Then, for each POS tag in NLTK's tag set, a new column was
added to the Dataframe, and the number of occurrence of the POS in the
corpus was stored in the column.

\par\null

\subsubsection*{UMLS Annotation}

{\label{766612}}

The Unified Medical Language System (UMLS) is a comprehensive thesaurus
and ontology of medical concepts~\cite{Bodenreider_2004}. UMLS has enabled
the development of NLP tools that behave as they understand the language
of biomedicine and health~\cite{md2009}. For example, Apache
cTakes~\cite{Savova_2010}, used in the preliminary analysis, is a java
program able to annotate medical terminologies recognized by the UMLS.
In the preliminary analysis, cTAKES tagged each medical terminology with
one of the following UMLS semantic types:~diseases/disorders,
signs/symptoms, anatomical sites, procedures, or medications.

\par\null

Since Apache cTAKES is a standalone Java program that offers access from
the command line, the descriptions and the descriptors of the incidents
in text files were batch processed by a bash script. cTAKES outputs a
metadata file, so getter methods were programmed to extract the semantic
type annotations from the metadata. Then, for each semantic type, a new
column was added to the dataset, and the number of occurrence of each
semantic type in the text file was stored into the column.

\par\null

\subsubsection*{Summary Statistics}

{\label{657778}}

In addition to the number of occurrence of each POS, the fractional POS
compositions were calculated by dividing the number of occurrence by the
word count. Summary statistics that describes the central tendencies,
dispersion, and the shape of a dataset's distribution were then
generated for each column that represents the occurrence of POS and
semantic type in the Dataframe. Specifically, for each Dataframe column,
the mean, standard deviation, minimum, maximum, and the 25th, 50\%, and
75\% percentiles, were calculated to reveal overall data trends.~

\par\null

\subsection*{Completeness Analysis}

{\label{212735}}

20 fictitious incident reports were generated as the ``gold standard''
of incident reports during the creation of NSIR-RT~\cite{Liszewski_2015}.
During the completeness analysis, these 20 fictitious reports were
regarded as supreme examples of completeness, and were then subjected to
the preliminary analysis described above. The datasets of real and
fictitious reports, including the summary statistics of the preliminary
analysis, were then concatenated. A column for a new binary variable
that indicates if an incident is fictitious or not is added alongside
the columns in the summary statistics. Then, the Pearson correlation
coefficients were calculated between the ``fictitiousness'' column and
each column in the summary statistics. For each variable that achieved a
Pearson correlation coefficient less than 0.05, a comparative histogram
of the variable overlaying the real and fictitious data was plotted. The
variables with the most different distributions between the real and
fictitious incidents were then considered as classifiers for
completeness. The Pearson correlations were calculated with the open
source Python library, SciPy version 0.19.1~\cite{python}. The
schematic overview of the completeness analysis in illustrated in
Figure~{\ref{209119}}.~\selectlanguage{english}
\begin{figure}[h!]
\begin{center}
\includegraphics[width=0.70\columnwidth]{figures/Screen-Shot-2017-12-29-at-04-04-33/Screen-Shot-2017-12-29-at-04-04-33}
\caption{{Schematic overview of the completeness analysis. Data objects are
represented by rectangular components, and data manipulations by yellow
chevrons. Abbreviations: POS: part-of-speech; UMLS: Unified Medical
Language System
{\label{209119}}%
}}
\end{center}
\end{figure}

\subsection*{Incident Classification}

{\label{171564}}

Incident classification consisted of tokenizing the reports, cleaning
the data, machine learning, and evaluation. Even though there was some
overlap between the preliminary analysis and the data preparation steps
of tokenizing and data cleaning, completing these steps during the
preliminary analysis would interfere with the POS tagging and UMLS
annotation. Thus, these steps are detailed in separate sections under
incident classification separately from the preliminary analysis. The
interferences are also further explained in these separate sections.

\par\null

In addition, data preparation are completed in spaCy \cite{honnibal-johnson:2015:EMNLP}
rather than NLTK. spaCy~is a~Python NLP alternative of NLTK capable of
piping custom tokenization and data cleaning functions, and machine
learning classifiers powered by neural networks. In addition to the
basic NLP functions included in NLTK, spaCy offers API support which
allows integration into the SaILS workflow. Data preparation for
classification was done with spaCy for spaCy's potential for integrating
the classifier into the SaILS workflow using the API. However, this is
reserved for future work due to time constraint. The data preparation
described below could have also been completed by NLTK with similar
results. A schematic overview of incident classification is illustrated
in Figure~{\ref{296325}}.\selectlanguage{english}
\begin{figure}[h!]
\begin{center}
\includegraphics[width=0.70\columnwidth]{figures/Incident-Classification/Incident-Classification}
\caption{{Schematic overview of incident classification. Data objects are
represented by rectangular components, and data manipulation by yellow
chevrons.
{\label{296325}}%
}}
\end{center}
\end{figure}

\subsubsection*{Tokenize}

{\label{287913}}

The default tokenizer in both NLTK and spaCy would split ``50gy/10fx''
into a single token. However, splitting it into ``gy'' and ``fx'' is
advantageous as another string, i.e. the ``gy'' and ``fx'' in
``50gy/5fx'' can then inform the classifier along with those in
``50gy/10fx''. Other examples that require a custom tokenizer are
``ptv40'', and ``lost(forgotten)'', which are both split into single
tokens by the default tokenizer.

\par\null

Thus, a split function was created to replace that of the default
tokenizer. The split function lowercased the reports, and removed all
punctuation and numbers, and then tokenized the reports. The custom
tokenizer then replaced the default tokenizer in the pipeline. Each
report was then processed into a list of tokens.

\par\null

\subsubsection*{Clean Data}

{\label{687327}}

Data cleaning consisted of autocorrecting words and removing common
words of little value such as ``the'' and ``an'' from the reports. They
are common practice in NLP, and would improve the classifiers by
improving the information content of the dataset. This was not done in
the preliminary analysis because autocorrect often changes medical words
such as ``cbct'' into a words that autocorrect recognizes, thus
interfering with the UMLS tagging. This, however, would not inhibit
classification as the all of the occurrences of a medical terminology
all map to the same autocorrection. In addition, removing stop words was
not included in the preliminary analysis because this would interfere
with the calculations of the fractional POS compositions. Thus, the data
cleaning in the incident classification was conducted separately from
the preliminary analysis.

\par\null

In data cleaning, Autocorrect 0.3.0~\cite{jonasmccallum}, a Python 3
spelling corrector, was first added to the pipeline as spelling errors
were common in the reports. An example of spelling errors is ``wothout''
which would not be removed by the subsequent stop word removal.~

\par\null

Stop words are words that occur extremely frequently, and therefore
contain little value for NLP. Examples of stop words are ``without'',
``at'', ``the'', ``which'', 'is". A function that removes a token if it
is in spaCy's set of stop words is also added to the pipeline.

\par\null

\subsubsection*{Machine Learning}

{\label{758831}}

The reports were then converted into numerical data in a~bag of words by
a count vectorizer. The bag-of-words model treats linguistic data as
literally a bag of words, ignoring grammar and word order, but retaining
the multiplicity, the number of times each word appears in a
report~\cite{Zhang_2010}. The process of the count vectorizer
converting linguistic data into a bag of words is illustrated in
Figure~{\ref{491366}} \cite{dataquest}. First, the
set of unique tokens from all the incidents serve as the column names of
a zero matrix, and the incidents serve the row names of the matrix. For
each occurrence of a word in an incident, the cell that corresponds to
the incident and the word increments by 1.~\selectlanguage{english}
\begin{figure}[h!]
\begin{center}
\includegraphics[width=0.56\columnwidth]{figures/Screen-Shot-2017-12-29-at-07-28-58/Screen-Shot-2017-12-29-at-07-28-58}
\caption{{A illustration of the bag-of-words model.
{\label{491366}}%
}}
\end{center}
\end{figure}

One-hot encoding converted the multinomial classification of incidents
by process step into multiple binary classifications by each menu choice
of the process step. Figure~{\ref{852010}}
\cite{classifier} is an illustration of one-hot encoding. A new column
is created for each menu choice of the process step, and the cell that
corresponds to a given incident and menu choice is 1 if the incident
occurred during the process step of the menu choice, and 0 otherwise.
This was not done for post-treatment completion as there was only one
incident report of this menu choice in the given dataset, and this is
not sufficient for training a classifier.\selectlanguage{english}
\begin{figure}[h!]
\begin{center}
\includegraphics[width=0.70\columnwidth]{figures/OHE-Image/OHE-Image}
\caption{{An illustration of one-hot-encoding.
{\label{852010}}%
}}
\end{center}
\end{figure}

The Python library for machine learning, sklearn~\cite{scikit-learn}, was
then used to split the data into the training and testing sets, and to
carry out the subsequent machine learning steps. A naive bayesian
classifier algorithm~\cite{Friedman:1997:BNC:274158.274161} was selected because the columns
in the bag of words are assumed to be independent of each other. The
naive bayesian classifier was trained on the training set, and tested on
the testing set.

\par\null

\subsubsection*{Evaluate}

{\label{155383}}

The output of the binary classifier is the probability from 0 to 1 of an
incident belonging to themenu choice of the process step. Different
thresholds can be set from 0 to 1 for different true positive and false
positive rates. A receiver-operating-characteristic (ROC) curve plots
the false positive rate on the x-axis and the true positive rate on the
y-axis for different thresholds. The area under the curve (AUC) is a
conventional method of comparing the performance of binary classifiers.
The more the ROC curve hugs the upper left corner, the higher the the
AUC, and the better the performance of the classifier.~ROC curves were
plotted for each binary classifier, and the associated AUCs were
calculated using sklearn.

\par\null

To prevent overfitting the classifier to words with low occurrence in
all of the dataset, only words with an occurrence greater than an
arbitrary threshold are included in the bag of words, and the columns
headed by the other words are removed from the dataset used to train and
test the classifiers. The grid search method of sklearn performs an
exhaustive search of the threshold possibilities, and outputs the
threshold and the associated classifier that offer the highest AUC.

\par\null

\subsubsection*{Other Models}

{\label{954490}}

In this section, attempts to represent the reports as numerical data
with other models~ other than the bag-of-words model are presented.
These models were explored before the objective of classifying incident
reports by process step was defined. As a result, the models were
compared by their performance in classifying the incident reports by if
the incident was a near miss, a menu choice of event type, an
arbitrarily chosen data element of the NSIR-RT.

\par\null

In these other models, the reports were not vectorized into a
bag-of-words count vector. Instead, the POS and medical terminology
occurrence columns from the preliminary analysis served as the numerical
data for the the classifier. A new column for the a variable that is 1
if an incident is a near miss, and 0 otherwise is added to the
Dataframe. Pearson correlation coefficients between the ``near miss''
column and each of the columns from the preliminary analysis were
calculated. The occurrence of medical terminology were excluded from the
further consideration for weak correlation. The three alternative
datasets used to train the classifier were the POS with the strongest
correlation, the 3 most strongly correlated POS, and all POS. The
trained classifier were then evaluated by the methods described above,
and their performance was compared to that of the bag-of-words model.

\par\null

\section*{Results}

{\label{153883}}

\subsection*{Preliminary Analysis}

{\label{987458}}

Of the 519 incidents, 325 were actual incidents, 103 were near misses,
and 91 were reportable circumstances.~The length of the reports
demonstrates high variance. The average word count of the incident
description is 24.83\selectlanguage{ngerman}±20.73 words. The corresponding histogram indicates
that the average word count of the incident is left-skewed. The average
word count of the incident descriptors is 5.13±3.98 words, that of the
average sentence count of the incident description is 3.61±2.65
sentences, and that of the incident descriptor is 1.26±0.55 sentences.
In the incident descriptions, the average of the noun and verb
compositions were 0.42±0.21 nouns and 0.17±0.10 verbs, respectively, and
both the associated histograms demonstrated bimodal distribution. The
summary statistics of the preliminary analysis are shown in
Table~{\ref{247116}}, and the histograms of the word
count, noun composition, and verb composition are shown in
Figures~{\ref{123287}}-{\ref{315712}}.~\selectlanguage{english}
\begin{table}[h!]
\centering
\normalsize\begin{tabulary}{1.0\textwidth}{CCCCCCC}
Descriptive statistics of the preliminary analysis &  &  &  &  &  &  \\
 & Incident description word count & Descriptor word count & Incident  description sentence count & Descriptor sentence count & Incident description verb composition & Incident description noun composition \\
Mean & 24.83 & 5.13 & 3.61 & 1.26 & 0.17 & 0.42 \\
STD & 20.72 & 3.98 & 2.65 & 0.55 & 0.1 & 0.21 \\
min & 3 & 1 & 1 & 1 & 0 & 0 \\
0.25 & 12 & 2 & 2 & 1 & 0.12 & 0.29 \\
0.5 & 19 & 4 & 3 & 1 & 0.18 & 0.36 \\
0.75 & 31 & 7 & 4 & 1 & 0.24 & 0.47 \\
Max & 242 & 28 & 24 & 5 & 0.6 & 1 \\
\end{tabulary}
\caption{{Summary~ statistics of the preliminary analysis;~
{\label{247116}}
Abbreviations: STD: standard deviation; min: minimum value; 25\%: 25\%
percentile%
}}
\end{table}\selectlanguage{english}
\begin{figure}[h!]
\begin{center}
\includegraphics[width=0.42\columnwidth]{figures/download-(4)1/download-(4)1}
\caption{{Histogram of the of incident description by word count
{\label{123287}}%
}}
\end{center}
\end{figure}\selectlanguage{english}
\begin{figure}[h!]
\begin{center}
\includegraphics[width=0.42\columnwidth]{figures/download-(5)1/download-(5)1}
\caption{{Histogram of the of incident description by noun composition
{\label{325814}}%
}}
\end{center}
\end{figure}\selectlanguage{english}
\begin{figure}[h!]
\begin{center}
\includegraphics[width=0.42\columnwidth]{figures/download-(6)1/download-(6)1}
\caption{{Histogram of the of incident description by verb composition
{\label{315712}}%
}}
\end{center}
\end{figure}

The fractional POS compositions of incident descriptions grouped by the
process step is shown in Table~{\ref{928258}}, and the
medical terminology compositions of the incident descriptions are shown
in Table {\ref{670961}}. Notably, patient
assessment/consultation achieved the highest word count of 42.48 words,
and on-treatment quality management achieved the lowest word count of
19.78 words. In addition,~ the patient assessment/consultation was the
highest in the composition of POS grouped as ``other.''~

\par\null

As expected, the procedure composition was the highest in the process
steps associated with planning and delivery, and lower in those
associated with review and verification. All of the process step groups
were low in symptom, disease, and anatomy compositions.

\par\null\selectlanguage{english}
\begin{table}[h!]
\centering
\normalsize\begin{tabulary}{1.0\textwidth}{CCCCCCCCC}
Parts-of-speech compositions of incidents descriptions grouped by process step &  &  &  &  &  &  &  &  \\
Process step & Word count  & Verb composition & Noun composition & Preposition composition & Pronoun composition & Adverb composition & Adjective composition & Other composition \\
Patient assessment/consultation & 42.38 & 0.21 & 0.35 & 0.16 & 0.04 & 0.03 & 0.03 & 0.21 \\
Imaging for radiotherapy planning & 29.26 & 0.2 & 0.38 & 0.15 & 0.03 & 0.05 & 0.06 & 0.13 \\
Treatment planning & 26.82 & 0.2 & 0.38 & 0.17 & 0.01 & 0.04 & 0.06 & 0.15 \\
Pre-treatment review and verification & 24.39 & 0.19 & 0.4 & 0.15 & 0.02 & 0.05 & 0.06 & 0.15 \\
Treatment delivery & 27.65 & 0.18 & 0.38 & 0.16 & 0.03 & 0.05 & 0.06 & 0.16 \\
On-treatment quality management & 19.78 & 0.18 & 0.38 & 0.14 & 0.01 & 0.06 & 0.06 & 0.18 \\
Other & 26.11 & 0.22 & 0.41 & 0.18 & 0.06 & 0.04 & 0.03 & 0.11 \\
\end{tabulary}
\caption{{Fractional parts-of-speech compositions of incident descriptions grouped
by process step
{\label{928258}}%
}}
\end{table}\selectlanguage{english}
\begin{table}[h!]
\centering
\normalsize\begin{tabulary}{1.0\textwidth}{CCCCCC}
Medical terminology compositions of incidents descriptions grouped by process step &  &  &  &  &  \\
Process step & Medical composition & Procedure composition & Symptom composition & Disease composition & Anatomy composition \\
Patient assessment/consultation & 0 & 0.06 & 0 & 0.02 & 0 \\
Imaging for radiotherapy planning & 0.01 & 0.05 & 0.01 & 0.01 & 0.03 \\
Treatment planning & 0 & 0.01 & 0.01 & 0.03 & 0 \\
Pre-treatment review and verification & 0 & 0.03 & 0.01 & 0.02 & 0.01 \\
Treatment delivery & 0.01 & 0.04 & 0.01 & 0.01 & 0.01 \\
On-treatment quality management & 0.01 & 0.02 & 0 & 0.01 & 0.02 \\
Other & 0 & 0.07 & 0.01 & 0.03 & 0 \\
\end{tabulary}
\caption{{Medical terminology compositions of incident descriptions grouped by
process step
{\label{670961}}%
}}
\end{table}\subsection*{Completeness Analysis}

{\label{917861}}

24 out of 35 POS occurrences achieved a Pearson correlation coefficient
with fictitiousness. The Pearson correlation coefficients of the 24 POS
are shown in Table~{\ref{766182}}. The corresponding
comparative histograms plotted are too numerous to show here. However,
almost all of them are characterized by nearly complete overlaps. The 4
histograms with the most different distributions between the real and
fictitious reports belong to the proper nouns, verbs, adverbs, and
prepositions, shown in
Figures~{\ref{393447}}-{\ref{153993}}.~\selectlanguage{english}
\begin{table}[h!]
\centering
\normalsize\begin{tabulary}{1.0\textwidth}{CC}
Pearson correlations between POS occurrences and fictitiousness &  \\
Word class & p value \\
Coordinating conjunction & 0.01 \\
Cardinal digit & -0.01 \\
Existential there & -0.02 \\
Foreign word & -0.01 \\
Preposition/subordinating conjunction & 0.04 \\
Comparative adjective & -0.02 \\
Superlative adjective & -0.02 \\
Modal & 0.03 \\
Noun plural & 0.04 \\
Proper noun singular & -0.04 \\
Predeterminer & -0.01 \\
Possessive ending & -0.01 \\
Personal pronoun & 0 \\
Possessive pronoun & 0.03 \\
Adverb & 0.01 \\
Comparative adverb & -0.01 \\
Superlative adverb & -0.01 \\
Particle & -0.04 \\
Interjection & -0.01 \\
Verb base form & -0.03 \\
Verb 3rd person singular present & 0.04 \\
wh-pronoun & -0.01 \\
wh-adverb & 0 \\
\end{tabulary}
\caption{{Pearson correlation coefficients between parts-of-speech occurrences and
fictitiousness
{\label{766182}}%
}}
\end{table}\selectlanguage{english}
\begin{figure}[h!]
\begin{center}
\includegraphics[width=0.42\columnwidth]{figures/download-(1)/download-(1)}
\caption{{Comparative histogram between real and fictitious reports by proper
nouns
{\label{393447}}%
}}
\end{center}
\end{figure}\selectlanguage{english}
\begin{figure}[h!]
\begin{center}
\includegraphics[width=0.42\columnwidth]{figures/download-(3)/download-(3)}
\caption{{Comparative histogram between real and fictitious reports by verbs
{\label{856106}}%
}}
\end{center}
\end{figure}\selectlanguage{english}
\begin{figure}[h!]
\begin{center}
\includegraphics[width=0.42\columnwidth]{figures/download-(2)/download-(2)}
\caption{{Comparative histogram between real and fictitious reports by adverbs
{\label{312483}}%
}}
\end{center}
\end{figure}\selectlanguage{english}
\begin{figure}[h!]
\begin{center}
\includegraphics[width=0.42\columnwidth]{figures/download/download}
\caption{{Comparative histogram between real and fictitious reports by
propositions
{\label{153993}}%
}}
\end{center}
\end{figure}

\subsection*{Incident Classification}

{\label{843658}}

The number of nouns correlated most strongly with ``near miss'', with
the next three being the the number of words, determiners, and
adjectives. Common examples of determiners are ``the,'' this,"
``these,'' ``my,'' ``his,'' which," ``whatever,'' etc. The classifier
for ``near miss'' that was trained exclusively on the number of nouns
achieved an AUC of 0.67; that trained on the top 4 most strongly
correlated POS achieved an AUC of 0.70; and that trained on all POS also
achieved an AUC of 0.70. The bag of words model achieved the highest AUC
of 0.82. The ROC curves of the alternative attempts are shown in
Figure~{\ref{812672}}. Thus, the bag-of-words model was
chosen to classify incident reports by process step.\selectlanguage{english}
\begin{figure}[h!]
\begin{center}
\includegraphics[width=0.42\columnwidth]{figures/Screen-Shot-2017-12-28-at-18-30-29/Screen-Shot-2017-12-28-at-18-30-29}
\caption{{ROC curves of other models used to classify ``near miss''~
{\label{812672}}%
}}
\end{center}
\end{figure}

The classifiers achieved very high AUCs of 0.99, 0.99, and 0.98 in
classifying imaging for radiotherapy planning (n=81), patient
assessment/consultation (n=66), and other (n=15), respectively; a high
AUC of 0.86 in classifying treatment planning (n=68); a medium AUC of
0.78 in classifying on-treatment quality management (n=9), and low AUCs
of 0.69 and 0.65 in classifying~treatment delivery (n=96) and
pre-treatment review and verification (n=33).~The ROC curves of the
classifiers are shown in Figure~{\ref{435189}}.\selectlanguage{english}
\begin{figure}[h!]
\begin{center}
\includegraphics[width=0.70\columnwidth]{figures/Screen-Shot-2017-12-28-at-17-33-48/Screen-Shot-2017-12-28-at-17-33-48}
\caption{{ROC curves of the classifiers of process step of incident occurrence
{\label{435189}}%
}}
\end{center}
\end{figure}

\section*{Discussion}

{\label{723307}}

The number of actual incidents was three times greater than that of near
misses and reportable circumstances. If the ratio of major incidents to
minor incidents to near misses shown in
Figure~{\ref{919594}} remotely holds true, then many
near misses and reportable circumstances are unreported. It is possible
that such potential incidents are regular and have been perceived as
acceptable inefficiencies and risks for valid reasons.~

\par\null

The high variance in the word counts of incident descriptions was
partially explained by the variance in the word counts of incident
descriptions grouped by process step . The average word count of patient
assessment or consultation step was almost a third to two times greater
than that of other process steps. Incidents in other process steps often
can be attributed to a single technical error, and both the problem and
the correction in these incidents may be more straightforward. On the
other hand, incidents in patient assessment/consultation most often
involve scheduling, for which the causes are more varied and complex.

\par\null

The medical terminology compositions indicate that nearly all of the
process step groups contained low compositions of terminologies
pertaining to medication, symptom, disease, and anatomy, and the
composition of terminology pertaining to procedure is 5-7 times greater
than that of other terminology compositions. This indicates a high focus
on the procedural safety by the practitioners. A future direction for
the project is allowing patients to fill out incident reports, which may
contain a higher composition of terminologies pertaining to anatomy and
symptoms.~This in turn can potentially augment the focus on patient
comfort in incident learning, and inspire a~patient-centric approach to
improving the quality of care.

\par\null

The completeness analysis revealed a high difference in word counts of
the real and fictitious reports, which can explain much of the
differences in the number of POS. The near complete overlap in the
comparative histograms of the real and fictitious reports may suggest
that the context which is lost in reducing words into the number of the
POS is an essential component of determining completeness. A review of
the reports showed that an extremely short incident report can
adequately describe the incident to a reader with domain knowledge, and
lead to acceptable classification and investigation.

\par\null

Imposing criteria that each report must meet to ensure completeness was
also considered. However, such measures would not permit the omission of
information implicit to participants with domain knowledge. Thus, this
may exasperate participants, and diminish participation.~On the other
hand, integrating domain knowledge and context into NLP techniques to
determine completeness is outside of the scope of this project.

\par\null

The bag-of-words model outperformed the POS-occurrence models because
the a bag of words retains the semantics of the reports. In addition,
the bag-of-words model may increase in its superiority to the other
models as the size of the dataset grows. This is because an increased
number of entries may further strengthen the relevance of some words to
the process step and weaken that of the words that contribute to
overfitting. On the other hand, the increased dataset of POS occurrence
will regress to the respective means of POS occurrence of each process
step, which in itself is inadequate as a classifier, as revealed in the
completeness analysis.

\par\null

The classifiers for imaging for radiotherapy planning (n=81) and patient
assessment or consultation (n=66) are excellent as they achieved AUCs of
0.99, and there were sufficient incidents of these process steps in the
dataset for testing. The~near perfect AUC of the menu choice of
``other'' (n=15) can probably be attributed to the low number incidents
in the menu choice, leading to insufficient training and testing. The
classifier for treatment delivery (n=68) was good as it achieved an AUC
for 0.86. The performance of the classifiers for on-treatment quality
management (n=9, AUC=0.78) and pre-treatment review and verification
(n=33, AUC=0.65) likely suffered from the low number of incidents in
these process steps. Finally, the classifier for treatment delivery was
poor (n=96, AUC=0.69), in spite of the fact that the classifier had the
largest training set as the highest number of incidents occurred in this
process step. This may be attributed to the fact that the incidents in
treatment delivery are highly varied. It follows that this classifier,
as well as the others, can significantly improve from an increased
dataset. In the case of aviation safety reports, 600 aviation safety
reports were generated each month~\cite{Tanguy2016}, providing an
enormous and comprehensive training set. The classifiers of this project
were trained on only two years of incident reports, and will likely be
improved by a larger dataset.

\subsection*{}

{\label{171891}}

Alternative algorithms are also worthwhile to explore to improve the
performance. The algorithm used in the aviation safety reports was the
Support Vector Machine algorithm \cite{Tanguy2016}. In addition,~spaCy
offers a multinomial text classifier powered by neural networks. The
classifier object is exportable, and APIs can be easily created for
integrating the classifier into the SaILS pipeline. Another algorithm
alternative is the multinomial naive bayesian classifier by sklearn,
which also offers API support. It is worthwhile to compare the
performance of these alternative algorithms before deploying the
classifier.

\par\null

Since the bag-of-words model retains the semantics of the reports, it
can in theory be applied to other data elements of the NSIR-RT taxonomy.
Automating the classification of incident reports by these data elements
is another future direction for the project.

\par\null

\section*{Conclusion}

{\label{539324}}

Determining if a report is complete requires domain knowledge outside
the scope of this project. However, NLP demonstrates potential for
classifying radiotherapy incident reports by process step as well as
other data elements in NSIR-RT taxonomy.

\selectlanguage{english}
\FloatBarrier
\bibliographystyle{plainnat}
\bibliography{bibliography/converted_to_latex.bib%
}

\end{document}

