Simulations
Cumulative ROC curve analysis for a ternary ordinal outcome was evaluated under conditions simulating AUCs = 0.70, 0.75, 0.85, 0.90, and 0.95. Cutpoints for the continuous predictor were set at \(X^*_2 = -5\) and \(X^*_3= 5\) by designating \(\alpha_{j-1}\) and \(\beta_{j-1}\) based on the relationship \(X_j^*=-\left( \alpha_{j-1}/\beta_{j-1} \right)\). Random variates of the continuous predictor were obtained from a normal distribution \(X_i \sim N\left(0,\sigma^2=100 \right)\) truncated at the 10th and 90th percentiles. Truncation improved the chances of obtaining random variates that would successfully converge to a maximum likelihood solution for the regression model. Random variates of the ternary outcome \(Y_i\) were then obtained from a multinomial distribution defined by probabilities computed from Eq. \ref{eq:4} with \(\alpha_{j-1}\), \(\beta_{j-1}\), and random variates \(X_i\). For the proportional odds condition with \(\text{AUC}_1=\text{AUC}_2=0.90\), parameters were designated at \(\alpha_1=-1.70\), \(\alpha_2=1.70\), and \(\beta=0.34\) . For the first non-proportional odds condition (referred to as the NPO1 condition) with \(\text{AUC}_1=0.75\) and \(\text{AUC}_2=0.85\), parameters were designated at \(\alpha_1=-0.75\), \(\beta_1=0.15\) and \(\alpha_2=1.25\), \(\beta_2=0.25\); and for the second (NPO2 condition) with \(\text{AUC}_1=0.70\) and \(\text{AUC}_2=0.95\), parameters were \(\alpha_1=-0.70\), \(\beta_1=0.14\) and \(\alpha_2=4.70\), \(\beta_2=0.94\). For each condition, 10,000 datasets were simulated with nested sample sizes \(n=75,150,\text{and}\ 300\) unequally allocated among the outcome levels. A cumulative logit regression model was fit to each dataset and cumulative ROC curves computed.
Simulations were run with the FREQ, LOGISTIC, and SURVEYSELECT subroutines of the SAS software application, version 9.4 \cite{SASInstitute2010}.
Several cutpoint selection criteria were evaluated for their ability to correctly identify designated cutpoints from cumulative ROC curves: the Youden Index (also known as Informedness and \(\Delta P^{\prime}\)), Matthews Correlation Coefficient, Total Accuracy, and Markedness \(\left( \Delta P \right)\) \cite{6221710,Powers2011}.
These criteria and their ranges are presented in Table S1. Each criterion embodies certain merits, but all achieve their respective optimal level at the ROC curve coordinate where the criterion is at its observed maximum. Cutpoints were also computed directly from MLE cumulative logit regression parameters.