\documentclass{statsoc}

\usepackage{geometry}
\geometry{papersize={7.444in,9.681in},total={4.8in,6.8in}}

\usepackage{setspace}
\singlespacing

\usepackage{graphicx}
\usepackage[space]{grffile}
\usepackage{latexsym}
\usepackage{textcomp}
\usepackage{longtable}
\usepackage{tabulary}
\usepackage{booktabs,array,multirow}
\usepackage{amsfonts,amsmath,amssymb}
\providecommand\citet{\cite}
\providecommand\citep{\cite}
\providecommand\citealt{\cite}
\usepackage{url}
\usepackage{hyperref}
\hypersetup{colorlinks=false,pdfborder={0 0 0}}
\usepackage{etoolbox}
\makeatletter
\patchcmd\@combinedblfloats{\box\@outputbox}{\unvbox\@outputbox}{}{%
  \errmessage{\noexpand\@combinedblfloats could not be patched}%
}%
\makeatother
% You can conditionalize code for latexml or normal latex using this.
\newif\iflatexml\latexmlfalse
\AtBeginDocument{\DeclareGraphicsExtensions{.pdf,.PDF,.eps,.EPS,.png,.PNG,.tif,.TIF,.jpg,.JPG,.jpeg,.JPEG}}


\usepackage[english]{babel}



\iflatexml
\documentclass[12pt,letterpaper]{article}
\fi

% \usepackage[final]{microtype} % Not (yet) supported by LaTeXML.
% ODD fontenc listed in packages supported by LaTeXML
% \usepackage[T1]{fontenc} % Not (yet) supported by LaTeXML.

\usepackage[utf8]{inputenc}
% ODD lmodern listed in packages supported by LaTeXML
% \usepackage[]{lmodern} % Not (yet) supported by LaTeXML.
% Already in Authorea header \usepackage[]{natbib}
% Already in Authorea header \usepackage[]{amsmath}
% Already in Authorea header \usepackage[]{amsfonts}
% Already in Authorea header \usepackage[]{amssymb}
% DECASTRO xfrac not listed in packages supported by LaTeXML
% \usepackage[]{xfrac}

% Already in Authorea header \usepackage[]{graphicx}
% ODD ae listed in packages supported by LaTeXML
% \usepackage[]{ae,aecompl} % Not (yet) supported by LaTeXML.
% \usepackage[]{ae,aecompl} % Not (yet) supported by LaTeXML.

% \usepackage[document]{ragged2e} % Not (yet) supported by LaTeXML.

% \usepackage[]{floatrow} % Not (yet) supported by LaTeXML.

% 07/06/2017 comment out because it screws up output to PDF with JRSS template
% \usepackage[justification=RaggedRight,singlelinecheck=no]{caption}

% Already in Authorea header \usepackage[]{xcolor}
% ODD colortbl listed in packages supported by LaTeXML
% \usepackage[]{colortbl} % Not (yet) supported by LaTeXML.

% Not in Authorea header but skip since rotated tables moved to separate files \usepackage[]{rotating}
% Already in Authorea header \usepackage[]{multirow}
% Already in Authorea header \usepackage[]{booktabs}

\usepackage[section]{placeins}
\newcommand{\gt}{>}
\newcommand{\lt}{<}




\title[Cumulative ROC curves for discriminating three ...]{Cumulative ROC curves for discriminating three or more ordinal outcomes
with cutpoints on a shared continuous measurement scale}

\author[B. Rey deCastro ]{B. Rey deCastro}
\address{Centers for Disease Control \& Prevention}




\begin{document}

\selectlanguage{english}
\begin{abstract}
A procedure is described for extending classic receiver operator
characteristic (ROC) curve analysis to discriminate three or more
ordinal outcomes on a shared continuous measurement scale. The procedure
comprises a two-stage, semiparametric approach combining conventional
cumulative logit regression with a cumulative extension of ROC curve
analysis. Performance with a three-level (ternary) ordinal outcome was
evaluated under simulation, which included comparison of several
criteria for selecting cutpoints to discriminate outcome levels: the
Youden Index, Matthews Correlation Coefficient, Total Accuracy, and
Markedness. Cutpoints computed from maximum likelihood regression
parameters were also evaluated. The procedure performs as expected under
a variety of simulated conditions: unbalanced data, proportional odds
and non-proportional odds, and areas under the ROC curve (AUCs) from
0.70 to 0.95. Total Accuracy selected cutpoints with the least absolute
percent-bias among the criteria compared, while the bias of parametric
cutpoints was less than for Total Accuracy cutpoints and often
negligible. The procedure was also applied to publicly available data
related to computer imaging and biomarker exposure science, yielding
good to excellent AUCs, as well as cutpoints with sensitivities and
specificities of commensurate quality. Cumulative ROC curve analysis is
readily implemented with available software, and the author provides the
URL for his implementation for ternary ordinal outcomes in the SAS
statistical software application.

Key Words: bioassay, cumulative logit regression, clinical chemistry,
diagnostic test, Fieller's Method, machine learning, multinomial
outcomes, ordinal outcomes, receiver operating characteristic curve

Disclaimers:~The findings and conclusions in this report are those of
the author and do not necessarily represent the official position of the
Centers for Disease Control and Prevention. The author declares that he
has no actual or potential competing financial interests.%
\end{abstract}%






\section*{Introduction}

{\label{425421}}

Classic receiver operator characteristic (ROC) curve analysis addresses
the relation of continuous measurements to binary outcomes
\cite{Agresti2014}, including identification of a cutpoint or threshold
on the continuous measurement scale discriminating the outcome levels.
From its origins in signal detection theory \cite{Egan1966} and
application in early radio detection and ranging systems, the technique
has been used in fields as diverse as clinical chemistry
\cite{Hanley2005}, radiology~\cite{Pepe2004}, psychology
\cite{Swets_1973}, and machine learning \cite{Hernandez-Orallo2012,Majnik2013,Prati2011a}.

Extension beyond binary outcomes would be desirable for the increased
scope of applications. One readily implemented approach is to group
multinomial outcome levels into binomial levels and run classic ROC
curve analysis, but this loses information and biases test accuracy
\cite{Obuchowski2004a}. There have been other, more sophisticated proposals
spanning a range of theoretical approaches~\cite{Nakas2010,Nakas2013,Inacio2011,Li2012,Kijewski1989,Mossman1999,Dreiseitl2000,Edwards2004}, but the
complexity of these noteworthy proposals has limited their application.
Additional methods have been implemented and some enjoy broad
use~\cite{Lopez-Raton2014,Chang2017,Budczies2012a,Camp2004}, yet theoretical justification may be sparse.

This paper proposes a two-stage, semiparametric approach combining
conventional cumulative logit regression with a cumulative extension of
ROC curve analysis to discriminate ordinal outcome levels. The
performance of this approach is evaluated under simulation, with
comparison of several criteria used with classic ROC curves to select
cutpoints. Results from these criteria are compared to cutpoints
computed from maximum likelihood estimates (MLEs) of the regression
parameters. The procedure is also demonstrated with publicly available
data.

\section*{Formulation}

{\label{821826}}

The classic empirical ROC curve is computed by comparing a binary
outcome \(Y\)with a continuous measure \(X\)
where each observed level of \(X\) is evaluated as a
candidate cutpoint discriminating observed~\(Y=1\ \text{(positive) from}\ Y=2\ \text{(negative)}\).
Observations exceeding the candidate cutpoint are classified positive
with respect to the continuous measurement, while those less than or
equal to the cutpoint are classified negative. As in a
\(2\times2\) contingency table, the count of correct
classifications among positive outcomes comprises the true positives
(\(TP\)) and among negative outcomes the true negatives
(\(TN\)). The count of incorrect classifications among
negative outcomes comprises the false positives (\(FP\)) and
among positive outcomes the false negatives (\(FN\)). These
counts are used to compute: sensitivity, which is the probability that
an observation with a positive outcome is correctly classified by a
continuous measurement above a candidate cutpoint~\(\left(\text{sensitivity}=\text{TP/[TP+FN]}\right)\);
and specificity, which is the probability that an observed negative
outcome is correctly classified by a continuous measurement at or below
a candidate cutpoint~\(\left(\text{specificity}=\text{TN/[TN+FP]}\right)\). Thence, coordinates for the
empirical ROC curve are computed where the abscissa is
\(1-\text{specificity} (=\text{false positive rate; FPR})\)~and the ordinate is \(\text{sensitivity} (=\text{true positive rate; TPR})\). The best
cutpoint \(X^{*}\)~given the data may be identified from te
ROC curve coordinates with a criterion that maximizes
\(\text{TPR}\) and minimizes \(\text{FPR}\). Cross-referencing
the identified ROC curve coordinate with its observed continuous
measurement yields the cutpoint distinguishing the binary outcomes. A
variety of cutpoint criteria are available, such as the Youden Index,
Matthews Correlation Coefficient, and Total Accuracy \cite{Youden1950,Matthews1975,metz1978basic}.
In addition, the ability of the continuous measurement to discriminate
between outcome levels, which is equivalent to the strength of the
association between the two, may be represented by the area under the
ROC curve (AUC; also known as the c-statistic), which is the probability
that an observation with a positive outcome will have a higher
continuous measurement than an observation with a negative outcome. ~~

Since the ROC curve describes the relationship between a binary outcome
and continuous predictor, it is directly related to logistic regression
\cite{Lloyd2002,Qin2003,Krzanowski2011}. For the binary outcome where \(Y=2\) is
the reference outcome level, let \(\pi_1 = \text{Pr} \left[ Y=1 \right]\). The univariate
logistic model with continuous predictor \(X\)~and linear
parameters \(\{\alpha,\beta\}\)~is:

\begin{equation} \label{eq:1}
    \text{logit}\left( \pi_{1} \right) = \log\left( \frac{\pi_{1}}{1 - \pi_{1}} \right)  = \alpha + \beta X
\end{equation}

Let~\(\hat{\pi}_{1i}\) be the probability that
\(Y=1\)~predicted by the regression model at the
\(i\text{th}\) observation \(X_i\) for
\(i=1, \dots , N\)~where \(N\) is the number of
observations. Analogous to the approach above, each predicted
probability may serve as a candidate cutpoint discriminating
\(Y=1\ \text{from}\ Y=2\). Coordinates comprising the ROC curve may then be
computed, except they are based on counts on the probability scale
monotonically transformed by the regression model from the original
continuous scale. As above, the best probability
cutpoint~\(\hat{\pi}^*\)~may be selected with a suitable optimality
criterion. The cutpoint on the scale of the continuous predictor
\(X^*\)~can be recovered by cross-referencing
\(\hat{\pi}^*\)~with its corresponding observed measurement. Whether
using the scale of the continuous predictor or the predicted
probability, the resulting ROC curves are identical because the curve is
rank-based and invariant to monotonic transformations of the continuous
predictor \cite{Agresti2014}. The first stage of the proposed approach
exploits this invariance through a generalization of the logistic model
embodied by the cumulative logit model.

\subsection*{Stage 1: Cumulative Logit
Regression}

{\label{374916}}

The cumulative logit regression model predicts probabilities for an
ordinal outcome \(Y=j\) for \(j=1, \dots , J\) levels, where
for demonstration \(J=3\)~and the reference outcome level is
\(Y=3\). Let \(\pi_j = \text{Pr}\left[ Y \le j \right]\), then with continuous
predictor \(X\) and linear parameters \(\{ \alpha_j,\beta_j \}\)
the cumulative logit regression model is:

\begin{equation} \label{eq:2}
\text{logit}\left( \pi_{j} \right) = \log\left( \frac{\pi_{j}}{1 - \pi_{j}} \right) = \alpha_j + \beta_j X,\ \text{for} \ j = 1, \dots , J - 1
\end{equation}

For each~\(j \text{th}\) outcome level up to \(J-1\), a
cumulative logit is estimated with its own regression parameters
\(\{ \hat{\alpha}_j, \hat{\beta}_j \}\). The formulation in Eq.
{\ref{eq:2}}~is known as non-proportional odds, where
the log-odds \(\hat{\beta}_j\) differs between outcome levels.
Simplification is possible in a proportional odds formulation where
constant log-odds \(\beta_j=\beta\)~is assumed among outcome levels
\cite{mccullagh1989generalized}.

Let~\(\hat{\pi}_{ij} = \hat{\text{Pr}} \left[ Y_{ij} \leq j|X_i \right]\), then in terms of the parameter
estimates~\(\{ \hat{\alpha}_j,\hat{\beta}_j \} \) the predicted cumulative probability for
the~\(j\text{th}\) outcome level at the~\(i \text{th}\)
observation is:

\begin{equation}\label{eq:3}
\hat{\pi}_{ij} = \frac{\exp\left( \hat{\alpha}_j + \hat{\beta}_j X_i \right) }{1 + \exp\left( \hat{\alpha}_j + \hat{\beta}_j X_i \right)} , \quad \text{for} \ j = 1, \dots , J - 1
\end{equation}

and the predicted individual probability is the difference of adjacent
cumulative probabilities:

\begin{equation}\label{eq:4}
\frac{\exp\left( \hat{\alpha}_j + \hat{\beta}_j X_i \right) }{1 + \exp\left( \hat{\alpha}_j + \hat{\beta}_j X_i \right)}-\frac{\exp\left( \hat{\alpha}_{j-1} + \hat{\beta}_{j-1} X_i \right) }{1 + \exp\left( \hat{\alpha}_{j-1} + \hat{\beta}_{j-1} X_i \right)}
\end{equation}

For \(J=2\)~ this model reduces to the logistic model, but
the cumulative logit model is similarly able to transform the continuous
predictor to the predicted probability scale, except that each outcome
level gets a predicted probability function.

We have so far recalled that ROC curves are invariant to monotonic
transformation of the continuous measurement, including transformation
by a logistic regression model to the predicted probability scale. In
addition, we have reviewed the cumulative logit model and its
transformation of a single continuous predictor to a series of separate
predicted probabilities for each level of the ordinal outcome. These
probabilities are comprehensive and mutually exclusive with respect to
the outcome, and are suitable for computing a series of ``cumulative''
ROC curves.

\subsection*{Stage 2: Cumulative ROC
Curves}

{\label{803714}}

Calculation of the ROC curve on the predicted probability scale can be
readily extended to count \(TP\), \(TN\),
\(FP\), and \(FN\)~for each cumulative logit,
resulting in \(J-1\)~cumulative ROC curves. For the
cumulative logit associated with the~\(j\text{th}\)~outcome, let
\(p_{jk}\)~be the \(k\text{th}\)~candidate cumulative
probability cutpoint from among the \(\hat{\pi}_{ij}\), then one may
count~\(TP_{jk}\), \(TN_{jk}\), \(FP_{jk}\), and
\(FN_{jk}\)~with the indicator function~\(I\left(\cdot\right)\) by
comparing the outcome \(Y_i\) with \(\hat{\pi}_{ij}\) vs.
\(p_{jk}\) for \(i=1,\dots,N \);~\(j=1,\dots, J-1\);
and~\(k=1,\dots, N\):~

\begin{equation}\label{eq:5}
TP_{jk} = \sum_i I \left( \hat{\pi}_{ij} \gt p_{jk} \ \ \text{AND} \ \ Y_i \leq j \right)
\end{equation}



\begin{equation}\label{eq:6}
TN_{jk} = \sum_i I\left( \hat{\pi}_{ij} \le p_{jk} \ \ \text{AND} \ \ Y_i \gt j\right)
\end{equation}



\begin{equation}\label{eq:7}
FP_{jk} = \sum_i I \left( \hat{\pi}_{ij} \gt p_{jk} \ \ \text{AND} \ \ Y_i \gt j \right)
\end{equation}



\begin{equation}\label{eq:8}
FN_{jk} = \sum_i I\left( \hat{\pi}_{ij} \le p_{jk} \ \ \text{AND} \ \ Y_i\le j \right)
\end{equation}

From these counts, the coordinates \(\left(FPR_{jk},TPR_{jk}\right)\)~for the
\(j\text{th}\)~cumulative \(ROC_j\)~curve can be computed,
where~\(FPR_{jk}=1-\left(TN_{jk}/\left[TN_{jk}+FP_{jk}\right]\right)\)~and \(TPR_{jk}=\left(TP_{jk}/\left[TP_{jk}+FN_{jk}\right]\right)\). Continuing with the
case of the ternary ordinal outcome, \(p_{1k}\) is the
\(k \text{th}\)~candidate cutpoint from the first cumulative logit,
and \(p_{2k}\) from the second, so that the cumulative
\(ROC_1\)~curve discriminates between \(Y=1\ \text{vs.}\ Y=2\ \text{or}\ 3\), and
the cumulative \(ROC_2\)~curve discriminates between
\(Y=1\ \text{or}\ 2\ \text{vs.}\ Y=3\). Analogous to the binary case, an optimal
probability cutpoint for the \(j\text{th}\) outcome level
\(\hat{\pi}_j^*\) may be selected from the cumulative
\(ROC_j\) curve using an optimality criterion. The cutpoint on
the scale of the continuous predictor \(X_j^*\)~is recovered
by cross-referencing \(\hat{\pi}_j^*\) with its corresponding observed
measurement.

Alternatively, ROC curve analysis may be forgone altogether by computing
the \(J-1\) cutpoints from the MLE cumulative logit
regression parameters, where \(X_j^*=-\left( \hat{\alpha}_{j-1}/\hat{\beta}_{j-1} \right)\).

\section*{Simulations}

{\label{803846}}

Cumulative ROC curve analysis for a ternary ordinal outcome was
evaluated under conditions simulating AUCs = 0.70, 0.75, 0.85, 0.90, and
0.95. Cutpoints for the continuous predictor were set
at~\(X^*_2 = -5\) and \(X^*_3= 5\)~by designating
\(\alpha_{j-1}\) and \(\beta_{j-1}\) based on the
relationship~\(X_j^*=-\left( \alpha_{j-1}/\beta_{j-1} \right)\). Random variates of the continuous
predictor were obtained from a normal distribution~\(X_i \sim N\left(0,\sigma^2=100 \right)\)
truncated at the 10th and 90th percentiles. Truncation improved the
chances of obtaining random variates that would successfully converge to
a maximum likelihood solution for the regression model. Random variates
of the ternary outcome~\(Y_i\) were then obtained from a
multinomial distribution defined by probabilities computed from
Eq.~{\ref{eq:4}}~with~\(\alpha_{j-1}\),~\(\beta_{j-1}\),
and random variates~\(X_i\). For the proportional odds
condition with~\(\text{AUC}_1=\text{AUC}_2=0.90\), parameters were designated
at~\(\alpha_1=-1.70\),~\(\alpha_2=1.70\), and~\(\beta=0.34\) .
For the first non-proportional odds condition (referred to as the NPO1
condition) with~\(\text{AUC}_1=0.75\) and~\(\text{AUC}_2=0.85\), parameters
were designated
at~\(\alpha_1=-0.75\),~\(\beta_1=0.15\)~and~\(\alpha_2=1.25\),~\(\beta_2=0.25\);
and for the second (NPO2 condition)
with~\(\text{AUC}_1=0.70\)~and~\(\text{AUC}_2=0.95\), parameters
were~\(\alpha_1=-0.70\),~\(\beta_1=0.14\)~and~\(\alpha_2=4.70\),~\(\beta_2=0.94\).
For each condition, 10,000 datasets were simulated with nested sample
sizes~\(n=75,150,\text{and}\ 300\)~unequally allocated among the outcome levels.
A cumulative logit regression model was fit to each dataset and
cumulative ROC curves computed. Simulations were run with the FREQ,
LOGISTIC, and SURVEYSELECT subroutines of the SAS software application,
version 9.4~\cite{SASInstitute2010}.

Several cutpoint selection criteria were evaluated for their ability to
correctly identify designated cutpoints from cumulative ROC curves: the
Youden Index \textbf{} (also known as Informedness \textbf{} and
\(\Delta P^{\prime}\)), Matthews Correlation Coefficient, Total Accuracy,
and Markedness~\(\left( \Delta P \right)\) \cite{6221710,Powers2011}. These criteria
and their ranges are presented in Table S1. Each criterion embodies
certain merits, but all achieve their respective optimal level at the
ROC curve coordinate where the criterion is at its observed maximum.
Cutpoints were also computed directly from MLE cumulative logit
regression parameters.

Supplemental Tables S2 -- S3 confirm that distributions realized during
the proportional odds and NPO1 simulations were approximately centered
at the levels designated above
for~\(\alpha_{j-1}\),~\(\beta_{j-1}\), and~\(AUC\).
Supplemental Table S4 shows, however, that for the NPO2 condition the
medians of the realized distributions
for~\(\alpha_{j-1}\)~and~\(\beta_{j-1}\) differed by as much as 29
percent from their designated values, even as the realized AUCs were
centered on their designated values. Tables S5 -- S7 display the median,
2.5th and 97.5th percentiles, and the percent-bias of cutpoints selected
by each criterion across sample sizes~\(n\). Percent-biases
are the median of percent-differences between realized cutpoints
(selected and parametric) and designated cutpoints. The ROC curve-based
cutpoint selection criteria exhibited a range of biases. Among both
proportional (Table S5) and non-proportional (Tables S6 and S7) odds
conditions, absolute values of the percent-biases ranged from 2.8 --
144.2 percent. Total Accuracy demonstrated the best performance with
biases ranging from 2.8 -- 11.7 percent, while the other criteria
performed considerably worse.~

Forgoing ROC curve analysis and calculating cutpoints from the MLE
regression parameters yielded small, often negligible absolute
percent-biases \(\left( \lt 2.3 \ \text{percent} \right)\)~for both proportional and
non-proportional odds conditions. In addition, across all sample sizes,
parametric cutpoints consistently out-performed ROC curve-based cutpoint
selection criteria.

Notably, for the NPO2 condition (Supplemental Table S7), the differences
in the realized cumulative logit parameters did not entail discrepancies
in realized cutpoints compared to the other simulation conditions,
whether the cutpoints were selected by criteria or calculated
parametrically.

Absolute percent-bias for Total Accuracy cutpoints usually diminished
with increasing sample size from \(n = 75\) to
\(300\). The one exception was for the upper cutpoint
\(\left( X=5.00 \right)\) in the NPO2 simulation condition, where absolute
percent-bias worsened slightly at~\(n = 150\). In addition, the
absolute percent-bias of lower Total Accuracy cutpoints were usually
greater than for upper cutpoints, except for \( n = 150\)~and
\(300\) of the NPO2 condition. For parametric cutpoints,
absolute percent-bias was negligible at all sample sizes, except in the
NPO2 condition, where parametric cutpoint bias was the lowest within the
condition (0.1 -- 2.3 percent), but slightly greater compared to other
conditions (0.0 -- 0.3 percent).

\section*{Real-World Data}

{\label{287568}}

Two publicly available datasets with ternary ordinal outcomes were
analyzed with the cumulative ROC curve approach where~cutpoints~were
selected with the Total Accuracy criterion and computed parametrically.
Panel (\emph{a}) in
Figures~{\ref{550411}}~--~{\ref{370806}}~displays
histograms for each dataset overlaid with Total Accuracy cutpoints,
while panel (\emph{b}) shows the cutpoints on their respective
cumulative ROC curves. Tables {\ref{Table1}} --
{\ref{Table2}}~present the Total Accuracy and
parametric cutpoints, as well as their sensitivities, specificities, and
AUCs. Confidence intervals for parametric cutpoints were calculated
with~Fieller's~Method~\cite{Fieller1944} and for AUCs with Wald's
Method~\cite{Wald1939}. Cumulative logit regression models and
cumulative ROC curves were computed with the FREQ and LOGISTIC
subroutines of the SAS software application, version
9.4~\cite{SASInstitute2010}.

\subsection*{Cork Stopper Quality}

{\label{185409}}

The data comprise measurements of material defects appearing in digital
images of cork stoppers~\cite{Campilho1985,DeSa2001}. An automated image
processing system optically scanned cork defects and quantified several
characteristics, including the number, area, and perimeter of the
defects. Fifty cork stoppers were quantified in each of three quality
levels~\(\left(N=150\right)\), where~\(Y=1\ \text{(poor)}\),~\(2\ \text{(normal)}\),
and~\(3\ \text{(superior)}\). In the Stage 1 cumulative logit model, cork
stopper quality was predicted by the total number of pixels with
defects. The score test for proportional odds (\emph{p}-value = 0.31)
supported a proportional odds configuration for the model. The parameter
estimates are:~\(\hat{\alpha}_1=-13.64\),~\(\hat{\alpha}_2=-7.05\),
and~\(\hat{\beta}_{area}=-0.036\).

The ability of defect area to discriminate cork stopper quality is
excellent, with \(\text{AUCs} \gt 0.97\) for both cumulative ROC curves (Table
~{\ref{Table1}}). Total Accuracy identified~cutpoints
where the total number of pixels with defects were 205 (distinguishing
poor or normal quality vs. superior) and 369 (poor vs. normal,
superior), and both had excellent sensitivities and specificities
\(\gt 0.93\). Parametrically computed cutpoints were at 194.8
{[}95\%CI: 177.1, 213.7{]} and 376.8 {[}352.9, 403.9{]}~pixels, with
sensitivities and specificities comparable to those of the Total
Accuracy cutpoints, although specificity for the lower cutpoint and
sensitivity for the upper cutpoint were somewhat attenuated.

\subsection*{Tobacco Smoke Exposure}

{\label{419412}}

Human exposure to chemicals can be estimated from measurements of trace
compounds in samples of human urine. Some of these compounds, known as
biomarkers, are associated with exposure to tobacco smoke, which may
arise either from direct inhalation while smoking,~or from indirect
inhalation of tobacco smoke present in the environment (i.e.,
second-hand tobacco smoke; SHS). One such biomarker is a
tobacco-specific N-nitrosamine known as ~NNAL
(4-{[}methylnitrosamino{]}-1-{[}3-pyridyl{]}-1-butanol; CAS No.
76014-81-8), which is present in both mainstream tobacco smoke and
smokeless tobacco products. NNAL was measured in urine from a
representative sample of the United States civilian
population~\(\ge 12\) years old~\(\left(N=16,900 \right)\)~obtained
during the 2007 -- 2012 cycles of the National Health and Nutrition
Examination Survey (NHANES;~\citealt{CDCNationalCenterforHealthStatisticsa}). Subjects reported being
in one of three ordinal exposure categories: non-exposed subjects who
neither used tobacco products nor were exposed to
SHS~\(\left(Y=1; n_1=12,372 \right)\); SHS-exposed subjects who did not smoke
tobacco~\(\left(Y=2; n_2=927 \right)\); and tobacco smokers~\(\left(Y=3; n_3=3,691 \right)\).
Subjects were excluded from this analysis if they reported using
smokeless tobacco in order to eliminate this potential source of NNAL.
The natural log of urinary NNAL concentration predicted exposure levels
in the Stage 1 cumulative logit model. The score test (\emph{p}-value
\textless{}0.001) supported a non-proportional odds configuration for
the model. The parameter estimates
are:~\(\hat{\alpha}_1=-4.60\),~\(\hat{\alpha}_2=-4.08\),~\(\hat{\beta}_{ln \left(NNAL \right),1}=-1.13\),
and~\(\hat{\beta}_{ln \left(NNAL \right),2}=-1.25\).

The ability of~\(ln(NNAL)\) to discriminate ternary tobacco smoke
exposure levels is excellent with~\(\text{AUCs} \gt 0.95\) for both
cumulative ROC curves (Table {\ref{Table2}}). Total
Accuracy identified cutpoints at~\(ln(NNAL)\) concentrations of
-4.09 (non-exposed vs. SHS-exposed, smokers) and -3.38 ng/mL
(non-exposed, SHS-exposed vs. smokers). Since the non-proportional odds
configuration permits each cumulative ROC curve to differ in
discriminatory power, the cumulative ROC curve associated with the lower
cutpoint had an AUC of 0.9515 {[}95\%CI: 0.9476, 0.9554{]}, while the
upper cutpoint's curve had an AUC that was slightly, but significantly
better at 0.9639 {[}0.9603, 0.9674{]}. Parametric cutpoints are at -4.05
{[}95\%CI: -4.11, -4.00{]} and -3.26 {[}-3.32, -3.21{]} ng/mL. Total
Accuracy's upper cutpoint was below the parametric upper cutpoint's
lower 95 percent confidence limit, but it is unclear which is
preferable. The Total Accuracy upper cutpoint had excellent sensitivity
(0.9597) and good specificity (0.8067), but this was reversed for the
parametric upper cutpoint, which had good sensitivity (0.8092) and
excellent specificity (0.9585). Another basis for comparison is the
Total Accuracy criterion, which can be calculated for parametric
cutpoints from
their~\(TP_j\),~\(TN_j\),~\(FP_j\),
and~\(FN_j\). This, too, failed to be conclusive since the
criterion for the Total Accuracy vs. parametric upper cutpoints were
hardly different at 0.9163 vs. 0.9161.

\section*{Discussion}

{\label{201607}}

The cumulative logit model subsumes multinomial ordinal outcome levels
within a single model, yet each outcome level gets its own cumulative
logit, so that predicted individual probabilities for each level (Eq.
{\ref{eq:4}}) are mutually exclusive, comprehensive
over the outcome levels, and sum to unity for each observation of the
continuous measurement. Another appeal of the cumulative logit model is
that its predicted probabilities (Eq.s~{\ref{eq:3}} and
{\ref{eq:4}}) change in direct proportion to the
continuous measurement across all outcome levels. Even more, the
ordinality of the outcome ensures that cutpoints separate successive
pairs of adjacent outcome levels.

Assuming proportional odds constrains the log-odds of the continuous
predictor to be constant for all levels of the ordinal outcome. This
imposes statistical equivalence on the AUCs of the cumulative ROC
curves, so that the ROC curves will appear approximately overlapped. In
contrast, when the log-odds of the predictor are non-proportional, which
represents varying strength in the continuous predictor's association at
each outcome level, the AUCs of the cumulative ROC curves will differ
and the curves will appear nested. Notably, the rank-order of the AUCs
(and hence the order of nesting) is independent of the order of the
ordinal outcome levels. This flexibility may be especially desirable in
certain settings, such as in a clinical trial where a medication is
associated with greater potency at the worst level of the health outcome
compared to less acute levels.

Evaluated under simulation, cumulative ROC curve analysis performed as
expected for a variety of conditions, but with the qualification that if
ROC curve-based cutpoint criteria are to be used, results from simulated
unbalanced data indicate that Total Accuracy yields minimally biased
cutpoints compared to the Youden Index, Matthews Correlation
Coefficient, and Markedness. Alternatively, parametric cutpoints have
the advantage of being maximum likelihood and consequently had absolute
percent-biases that were less than Total Accuracy's and were often
negligible.

Analysis of real-world data demonstrated that cumulative ROC curve
analysis yields reasonable results. Continuous measurements in both
datasets displayed varying degrees of overlap among the ternary outcome
levels. The tobacco smoke exposure data were relatively large, but also
strikingly unbalanced across the outcome levels, especially at the
intermediate outcome level. The intermediate SHS-exposed category was
small (5.5 percent) and the distribution was skewed to the extreme
exposure levels (72.8 percent non-exposed vs. 21.7 percent smokers).
Notwithstanding, the cumulative ROC curve approach identified cutpoints
with good to excellent sensitivity and specificity.~

The cumulative ROC curve approach readily generalizes to more than three
outcome levels through specification of the cumulative logit model.
Nonetheless, discriminating discrete outcome levels postulates that the
continuous measurement is associated with an~\emph{a priori} number of
latent and ordinal classes. If the cumulative logit model in Stage 1
specifies an outcome with~\(J \gt 2\) ordinal levels,
determination of cutpoints may be difficult if the outcome is actually
binomial or is otherwise different than assumed. The magnitude of this
difficulty may be revealed in exploratory data analysis, by poor model
fit, and by cutpoints with poor sensitivity and specificity. For the
tobacco smoke exposure data, although prior assumption of an
intermediate outcome level (i.e., secondhand smoke-exposed) was
plausible, there was cause for doubt since this level was observed
infrequently. In addition, the infrequency of the SHS-exposed outcome
level contrasts with the simulated data, where use of a normal
distribution as a source of random variates for the continuous predictor
leads to more frequent intermediate outcomes (\textasciitilde{}40
percent) compared to the lower-most and uppermost outcomes
(\textasciitilde{}20 percent each). These considerations
notwithstanding,~the natural log of urinary NNAL was an excellent
discriminator of the three tobacco smoke exposure levels.

The proposed approach admits alternative formulations of the Stage 1
model, where other multinomial models may be implemented through
substitution of the cumulative logit link function. Alternative models
for ordinal outcomes, such as adjacent categories and continuation ratio
(including complementary log-log and Cox proportional hazards), and
nominal outcomes (with the generalized logit) all predict probabilities
entirely suitable for subsequent calculation of cumulative ROC curves.
Conceptual interpretation of these alternative link functions, however,
~necessarily varies, sometimes substantially, ~and may therefore be less
directly interpretable than the cumulative logit. Exploring the
performance and utility of these alternative link functions may
nonetheless be fruitful.

Cumulative ROC curve analysis appears to be efficacious for a univariate
continuous predictor, and the regression framework may be extended with
the addition of covariates to the Stage 1 cumulative logit model. This
can be expected to enhance discriminatory power by accounting for other
influential or potentially confounding influences \cite{Tosteson1988}. In
any particular case, however, it may not be clear whether additional
covariates will adversely affect the overall concavity of the cumulative
ROC curves, thereby hindering selection of cutpoints. Stratification by
potentially confounding factors may be helpful in resolving these
difficulties.

One challenge posed by the cumulative logit model is its sample size
demands, which arise from the potentially numerous parameters that must
be estimated. In the proportional odds configuration, the univariate
cumulative logit model has~\(J-1\) intercepts plus one slope
for the continuous predictor, but this nearly doubles in the
non-proportional odds configuration, which has~\(2 \times \left(J-1\right)\)
regression parameters.

Confidence intervals for parametric cutpoints were computed from
\(t_{df=2,1-2} \times s_{X^*}\) where \(s_{X^*}\)~was estimated with Fieller's
Method \cite{Fieller1944}.~Since the parametric cutpoint is the ratio of
two model parameters, the Delta Method and Fieller's are both applicable
for computing the variance \cite{Cox1990a}, but Fieller's Method is
favored since it tends to provide better coverage despite potential
asymmetry of the confidence interval \cite{Zerbe1978,Hirschberg2010a}. In addition,
\citet*{Hirschberg2010a} recommend Fieller's Method when the computed ratios
are positive while the correlation between the numerator and denominator
are negative.

\section*{Conclusions}

{\label{523357}}

The cumulative ROC curve method comprises a straightforward combination
of cumulative logit regression with ROC curve analysis, and is readily
implemented with available statistical software. Cutpoint selection
criteria from classic ROC curve analysis are still applicable, as well
as established performance measures, such~as sensitivity, specificity,
and AUC. Cumulative ROC curve analysis performed as expected under
simulation and with real-world data for a variety of conditions,
including balanced and unbalanced data, proportional and
non-proportional odds assumptions for the cumulative logit model, and
AUCs associated with fair, good, and excellent performance (AUC = 0.70
-- 0.95). Of the ROC curve-based cutpoint criteria, Total Accuracy was
the least biased in simulation compared to the Youden Index, Mathews
Correlation Coefficient, and Markedness. Calculation of cutpoints from
cumulative logit regression parameters, which forgoes evaluation of
cumulative ROC curves, demonstrated minimal bias, owing to parameter
estimation with maximum likelihood methods. Since parametric cutpoints
are calculated from the ratio of MLE model parameters, variance was
calculated with Fieller's Method. Cumulative ROC curve analysis and
parametric cutpoints for ternary ordinal outcomes~\(\left(J=3\right)\)
were implemented with version 9.4 of the SAS statistical software, and
the author's programs are freely available for download without warranty
or guarantee from the author's GitHub site (bitly URL).

\par\null

\pagebreak

\section*{Figure Captions}

{\label{793017}}

Figure 1: Cork Stopper Quality (N = 150). (\emph{a}) Percent
distribution of total defective area {[}px{]} by cork stopper quality
levels, overlaid with cutpoints (purple dashed lines) selected with the
Total Accuracy criterion from cumulative ROC curves. (\emph{b})
Cumulative ROC curves (focused on upper-left quadrant) indicating
coordinates for Total Accuracy cutpoints: poor vs. normal, superior
quality (green) and poor, normal vs. superior (orange).

\par\null

Figure 2: Tobacco Smoke Exposure, (N = 16,900). (\emph{a}) Percent
distribution of the natural log of urinary NNAL {[}ng/mL{]} by tobacco
smoke exposure level, overlaid with cutpoints (purple dashed lines)
selected with the Total Accuracy criterion from cumulative ROC curves.
(\emph{b}) Cumulative ROC curves (focused on upper-left quadrant)
indicating coordinates for Total Accuracy cutpoints: none vs.
second-hand smoke (SHS), smoker (green) and none, SHS vs. smoker
(orange).

\pagebreak

\section*{Tables}

{\label{691119}}\par\null\selectlanguage{english}
\begin{table}[htbp]
\caption{{Cork stopper quality (N = 150). Cutpoints of total defective area [px] discriminating cork stopper quality levels. Cutpoints were selected from cumulative ROC curves with Total Accuracy criterion and computed parametrically. Proportional odds assumed among quality levels.}}
\label{Table1}
    \begin{tabular}{rccc}
    \toprule
          &       & Sn: Sensitivity & AUC \\
    Outcome Levels & Cutpoint & Sp: Specificity & [95CI] \\
    \midrule
    \multicolumn{4}{c}{\textit{Total Accuracy}} \\
    \midrule
    Poor vs.  & \multirow{2}[1]{*}{369} & Sn: 0.9400 & 0.9847 \\
    Normal, Superior &       & Sp: 0.9700 & [0.9655, 1.0000] \\
          &       &       &  \\
    Poor, Normal vs.  & \multirow{2}[1]{*}{205} & Sn: 0.9300 & 0.9732 \\
    Superior &       & Sp: 0.9400 & [0.9501, 0.9963] \\
    \midrule
          &       &       &  \\
    \multicolumn{4}{c}{\textit{Parametric [95CI]}} \\
    \midrule
    Poor vs.  & 376.8 & Sn: 0.8800 & \multirow{2}[1]{*}{--} \\
    Normal, Superior & [352.9, 403.9] & Sp: 0.9800 &  \\
          &       &       &  \\
    Poor, Normal vs.  & 194.8 & Sn: 0.9300 & \multirow{2}[1]{*}{--} \\
    Superior & [177.1, 213.7] & Sp: 0.9000 &  \\
    \bottomrule
    \end{tabular}%
\end{table}



\pagebreak\selectlanguage{english}
\begin{table}[htbp]
\caption{{Tobacco smoke exposure (N = 16,990). Cutpoints of the natural log of urinary NNAL [ng/ml] discriminating tobacco smoke exposure levels. Cutpoints were selected from cumulative ROC curves with Total Accuracy criterion and computed parametrically. Non-proportional odds assumed among exposure levels.}}
\label{Table2}
    \begin{tabular}{rccc}
    \toprule
          &       & Sn: Sensitivity & AUC \\
    Outcome Levels & Cutpoint & Sp: Specificity & [95CI] \\
    \midrule
    \multicolumn{4}{c}{\textit{Total Accuracy}} \\
    \midrule
    Non-Exposed vs.  & \multirow{2}[1]{*}{-4.092} & Sn: 0.9597 & 0.9515 \\
    SHS, Combusted &       & Sp: 0.8067 & [0.9476, 0.9554] \\
          &       &       &  \\
    Non-Exposed, SHS vs.  & \multirow{2}[1]{*}{-3.381} & Sn: 0.9630 & 0.9639 \\
    Combusted &       & Sp: 0.8613 & [0.9603, 0.9674] \\
    \midrule
          &       &       &  \\
    \multicolumn{4}{c}{\textit{Parametric [95CI]}} \\
    \midrule
    Non-Exposed vs.  & -4.053 & Sn: 0.8039 & \multirow{2}[1]{*}{--} \\
    SHS, Combusted & [-4.108, -3.998] & Sp: 0.9602 &  \\
          &       &       &  \\
    Non-Exposed, SHS vs.  & -3.264 & Sn: 0.8487 & \multirow{2}[1]{*}{--} \\
    Combusted & [-3.319, -3.208] & Sp: 0.9661 &  \\
    \bottomrule
    \end{tabular}%
\end{table}



\pagebreak

\section*{Figures}

{\label{924423}}\par\null\selectlanguage{english}
\begin{figure}[h!]
\begin{center}
\includegraphics[width=1.00\columnwidth]{figures/fig1-corkCombo-17-5May05/fig1-corkCombo-PNAS-17-6Jun27}
\caption{{Cork Stopper Quality (N = 150). (\emph{a}) Percent distribution of total
defective area {[}px{]} by cork stopper quality levels, overlaid with
cutpoints (purple dashed lines) selected with the Total Accuracy
criterion from cumulative ROC curves. (\emph{b}) Cumulative ROC curves
(focused on upper-left quadrant) indicating coordinates for Total
Accuracy cutpoints: poor vs. normal, superior quality (green) and poor,
normal vs. superior (orange).
{\label{550411}}%
}}
\end{center}
\end{figure}



\pagebreak\selectlanguage{english}
\begin{figure}[h!]
\begin{center}
\includegraphics[width=1.00\columnwidth]{figures/fig3-nnalCombo-17-5May05/fig2-nnalCombo-PNAS-17-6Jun29}
\caption{{Tobacco Smoke Exposure, (N = 16,900). (\emph{a}) Percent distribution of
the natural log of urinary NNAL {[}ng/mL{]} by tobacco smoke exposure
level, overlaid with cutpoints (purple dashed lines) selected with the
Total Accuracy criterion from cumulative ROC curves. (\emph{b})
Cumulative ROC curves (focused on upper-left quadrant) indicating
coordinates for Total Accuracy cutpoints: none vs. second-hand smoke
(SHS), smoker (green) and none, SHS vs. smoker (orange).
{\label{370806}}%
}}
\end{center}
\end{figure}



\pagebreak

\selectlanguage{english}
\FloatBarrier
\bibliographystyle{Chicago}
\bibliography{bibliography/converted_to_latex.bib%
}

\end{document}

