Please note : the data presented in all course material for the statistical module are
generated by computers to demonstrate the methodologies, and should not be confused with
actual clinical information
Introduction
Binary Tests
Pretest and Posttest Probability
Receiver Operator Characteristics
The statistics of prediction consists of two major domains.
The first domain is to develop a prediction model, so that one or more observations (tests) can be used to predict an event of interest (outcome). This is often technically complex, and currently a rapidly changing and advancing field of numerical research under the general category of artificial intelligence. This domain is not be covered in this module
The second domain is to evaluate the quality of a particular prediction model, and create an algorithm to use it in the clinical setting. This is the domain that will be covered in this module, and the programs used are in
StatPgm 5a. Tests : Binary Tests and
StatPgm 5b. Tests : Receiver Operator Characteristics (ROC)
The model
The model is to use a Test to predict an outcome
 The Outcome is an event of interest, and in a test is what we set out to diagnose or predict
 An outcome can be a measurement. e.g. how tall will the person be when he grows up
 An outcome can be a set of alternatives. e.g. Is the patient suffering from urinary infection, appendicitis, or food poisoning
 An outcome can be binary, no/yes, false/true, negative/positive. e.g. Will this delivery require a Caesarean Section
 The Test is an observation we use to predict an outcome
 A test can be a measurement. e.g. the mother's height to predict a need for Caesarean Section, the oestrogen level to predict ovulation
 A test can be binary, no/yes, false/true, negative/positive. e.g. vaginal bleeding to predict Placenta Previa, abnormal nuchal translucency to predict chromosomal abnormality in a baby.
 The Prediction is any process, logical, mathematical, statistical, that links the test to the outcome
 Using a measurement test to predict a measurement outcome uses the regression analysis, which is covered in
Contents_2b. Correlation and Regression, and will not be further covered here
 Using a binary test to predict a measurement outcome uses the 95% confidence interval of difference between two groups,
which is covered in Contents_3. Comparing Two Groups, and will not be
further covered here
 Binary Test to predict binary outcome. The most commonly used model, will be discussed in detail in this page
 Measurement test to predict binary outcome, the Receiver Operator Characteristics, will be discussed in detail in this page
Terms commonly used in evaluating tests
 Outcome is the event of interest we wish to predict.
 Outcome Positive (OPos) is when the event eventuates.
 Outcome Negative (ONeg) is when the event fails to eventuates.
 Test is the observation that we use to predict an Outcome.
 Test Positive (TPos) in a binary test is when the test result corresponds with what we use to predict Outcome Positive
 Test Negative (TNeg) in a binary test is when the test result does not correspond with what we use to predict Outcome Positive
 Reference Data is a selected dataset containing appropriate numbers of OPos and ONeg, which we will use to evaluate
the quality of prediction. The number of cases in this set are divided into the following
 True Positives (TP) is the number of cases that are Test Positive and Outcome Positive
 False Positives (FP) is the number of cases that are Test Positive but Outcome Negative
 False Negatives (FN) is the number of cases that are Test Negative but Outcome Positive
 True Negatives (TN) is the number of cases that are Test Negative and Outcome Negative
From these the following quality parameters can be calculated
 True Positive Rate (TPR) is the proportion of Outcome Positives that are also Test Positive. TPR = TP / (TP+FN).
The term Sensitivity is the same as True Positive Rate
 True Negative Rate (TNR) is the proportion of Outcome Negatives that are also Test Negative. TNR = TN / (TN+FP).
The term Specificity is the same as True Negative Rate
 False Positive Rate (FPR) is the proportion of Outcome Negatives that are Test Positive. FPR = FP / (FP+TN) = 1TNR.
 False Negative Rate (FNR) is the proportion of Outcome Positives that are Test Negative. FNR = FN / (FN+TP) = 1TPR.
 Youden's Index (YI) is a summary indicator of overall quality of prediction, assuming TPR and FPR to be of
equal importance. YI = (TPR+TNR)/2
 Likelihood Ratio for Test Positive (LR+) is the ratio of True and False Positive Rates, LR+ = TPR/FPR,
and represents the ratio of probabilities between Outcome Positive and Outcome Negative, when the test is Test Positive
 Likelihood Ratio for Test Negative (LR) is the ratio of False and True Negative Rates, LR = FNR/TNR,
and represents the ratio of probabilities between Outcome Positive and Outcome Negative, when the test is Test Negative
Introduction
This panel supports the program 5a i. in StatPgm 5a. Tests : Binary Tests.
Reference Dataset 1
The default example data from program 5a i is used to evaluate the quality of antepartum haemorrhage as a test to predict Placenta Previa as an Outcome. The reference dataset consists of 50 pregnancies, with the following data and analysis
 There are 25 cases with Placenta Praevia (outcome positive)
 12 of the 25 had antepartum haemorrhage (test positive) True Positive (TP) = 12
 13 of the 25 did not have antepartum haemorrhage (test negative) False Negative (FN) = 13
 There are 25 cases with no Placenta Praevia (outcome negative)
 5 of the 25 had antepartum haemorrhage (test positive) False Positive (FP) = 5
 20 of the 25 did not have antepartum haemorrhage (test negative) True Negative (TN) = 20
 From these observation, the quality parameters are calculated
 The True Positive Rate TPR = TP / (TP + FN) = 12 / (12+13) = 12 / 25 = 0.48. In other words, 48% of those with placenta
praevia had an antepartum haemorrhage
 The True Negative Rate TNR = TN / (TN + FP) = 20 / (20+5) = 20 / 25 = 0.8. In other words, 80% of those with no placenta
praevia did not have an antepartum haemorrhage
 The False Positive Rate FPR = FP / (FP + TN) = 5 / (5+20) = 5 / 25 = 0.20. In other words, 20% of those with no placenta
praevia had an antepartum haemorrhage
 The False Negative Rate FNR = FN / (FN + TP) = 13 / (13+12) = 13 / 25 = 0.52. In other words, 50% of those with placenta
praevia did not have an antepartum haemorrhage
 The Youden Index YI = (TPR+TNR)/2 = (0.48+0.8)/2 = 1.28/2 = 0.64
 From TPR, TNR, FNR, and FPR, the Likelihood Ratios are calculated
 The Likelihood Ratio Test Positive LR+ = TPR / FPR = 0.48 / 0.20 = 2.4. In other words, when antepartum haemorrhage is
observed, Placenta Praevia is 2.8 times as likely as no Placenta Praevia
 The Likelihood Ratio Test Negative LR = FNR / TNR = 0.52 / 0.8 = 0.65. In other words, in the
absence of antepartum haemorrhage, Placenta Praevia is 0.65 times as likely as no Placenta Praevia
Reference Dataset 2
From another hospital, we obtained another dataset which used transverse lie near term as a Test to predict Placenta Praevia as an outcome
 There are 50 cases with Placenta Praevia (outcome positive)
 28 of the 50 had transverse lie (test positive) True Positive (TP) = 28
 22 of the 50 did not have transverse lie (test negative) False Negative (FN) = 22
 There are 50 cases with no Placenta Praevia (outcome negative)
 5 of the 50 had transverse lie (test positive) False Positive (FP) = 5
 45 of the 50 did not have transverse lie (test negative) True Negative (TN) = 45
 From these observation, the quality parameters are calculated
 The True Positive Rate TPR = TP / (TP + FN) = 28 / (28+22) = 28 / 50 = 0.56. In other words, 56% of those with placenta
praevia had transverse lie
 The True Negative Rate TNR = TN / (TN + FP) = 45 / (45+5) = 45 / 50 = 0.9. In other words, 90% of those with no placenta
praevia had no transverse lie
 The False Positive Rate FPR = FP / (FP + TN) = 5 / (5+45) = 5 / 50 = 0.1. In other words, 10% of those with no placenta
praevia had transverse lie
 The False Negative Rate FNR = FN / (FN + TP) = 22 / (22+28) = 22 / 50 = 0.44. In other words, 44% of those with placenta
praevia had no transverse lie
 The Youden Index YI = (TPR+TNR)/2 = (0.56+0.9)/2 = 1.46/2 = 0.73
 From TPR, TNR, FNR, and FPR, the Likelihood Ratios are calculated
 The Likelihood Ratio Test Positive LR+ = TPR / FPR = 0.56 / 0.1 = 5.6. In other words, when transverse lie is observed,
Placenta Praevia is 5.6 times as likely than no Placenta Praevia
 The Likelihood Ratio Test Negative LR = FNR / TNR = 0.44 / 0.9 = 0.4889. In other words, in the
absence of transverse lie, Placenta Praevia is 0.49 times as likely than no Placenta Praevia
Introduction
The Likelihood Ratios, as calculated in the previous section, is used to estimate the probability (risk, chance of) of the outcome of interest in future situations. The mathematics involved is called Bayesian probability, where
Posttest Probability = function(Pretest Probability, Likelihood Ratio)
The calculation of Posttest Probability is performed by the program StatPgm 5a ii. Bayesian Conversion of Pretest to Posttest Probability Using Likelihood Ratio, from StatPgm 5a. Tests : Binary Tests, and consists of the following steps
 A Pretest Probability is defined. This is the probability of Placenta Previa before the test result is known, and is based on prior knowledge, epidemiological data, or simply a guess.
 The Pretest Odd is calculated. Pretest odd = Pretest probability / (1  Pretest Probability)
 The Posttest Odd is calculated using the Likelihood Ratio. Posttest Odd = Pretest Odd * Likelihood Ratio
 The Posttest Probability is calculated. Posttest Probability = Posttest Odd / (1 + Posttest Odd)
If there is more than one test available, the Posttest Probability of the previous test becomes the Pretest Probability of the subsequent test
Example 1. Using Antepartum Haemorrhage as the test
From the data we obtained in the previous panel, we established the Likelihood Ratio for test positive LR+ = 2.4 and for
test negative LR = 0.65
From our experience, we know that about 1% of our mothers will have Placenta Praevia. We can
therefore start with a pretest probability of 1% (0.01)
The calculations are as follows.
 The Pretest Odd = Pretest Probability / (1Pretest Probability) = 0.01 / 0.99 = 0.0101
 With Antepartum Haemorrhage, the Likelihood Ratio LR+ = 2.4
 Posttest Odd = Pretest Odd * LR+ = 0.0101 * 2.4 = 0.024
 Posttest Probability = Posttest Odd / (1 + Posttest Odd) = 0.024 / (1.024) = 0.024 (2.4%)
 the presence of antepartum haemorrhage changes the probability of Placenta Praevia from 1% to 2.4%
 With no Antepartum Haemorrhage, the Likelihood Ratio LR = 0.65
 Posttest Odd = Pretest Odd * LR = 0.0101 * 0.65 = 0.0066
 Posttest Probability = Posttest Odd / (1 + Posttest Odd) = 0.0066 / (1.0066) = 0.0065 (0.7%)
 the absence of antepartum haemorrhage changes the probability of Placenta Praevia from 1% to 0.7%
Example 2. Use Transverse Lie as the test
From the data we obtained in the previous panel, we established the Likelihood Ratio for test positive LR+ = 5.6 and for
test negative LR = 0.49
From our experience, we know that about 1% of our mothers will have Placenta Praevia. We can
therefore start with a pretest probability of 1% (0.01)
The calculations are as follows.
The Pretest Odd = Pretest Probability / (1Pretest Probability) = 0.01 / 0.99 = 0.0101
 With Transverse lie, the Likelihood Ratio LR+ = 5.6
 Posttest Odd = Pretest Odd * LR+ = 0.0101 * 5.6 = 0.0566
 Posttest Probability = Posttest Odd / (1 + Posttest Odd) = 0.0566 / (1.0566) = 0.0535 (5.4%)
 the presence of transverse lie changes the probability of Placenta Praevia from 1% to 5.4%
 With no Transverse Lie, the Likelihood Ratio LR = 0.49
 Posttest Odd = Pretest Odd * LR = 0.0101 * 0.49 = 0.0049
 Posttest Probability = Posttest Odd / (1 + Posttest Odd) = 0.0049 / (1.0049) = 0.0049 (0.5%)
 the absence of transverse lie changes the probability of Placenta Praevia from 1% to 0.5%
Example 3. When Antepartum Haemorrhage and Transverse Lie are both present
Students should as exercises, estimate probability of Placenta Praevia using these data, in all combinations of
antepartum haemorrhage and Transverse Lie, and when the two are known in different orders.
Introduction
Many predictive tests are not binary, they are stepwise or continuous measurements. Examples of these are pulse rate to predict postoperative blood loss, blood pressure to predict eclampsia, induction/labour interval to predict need for Caesarean Section, age of the patient to predict success in IVF, and many others.
Usually, in these tests, changing test values are related to changing probability of the outcome, and the mathematical relationship
between test values and outcome is called the Receiver Operator Characteristics (ROC). The reason for the name is because the mathematics was first describe to relate the strength of signals from RADAR receivers and the actual arrival of enemy aircrafts during the Second World War.
The calculations and production of ROC plots are carried out by the computer program StatPgm 5b. Tests : Receiver Operator Characteristics (ROC) and students only have to be able to interpret the results or make minor changes to the graphics. The following description are therefore intended to help the student understand ROC, and students are not required to perform the calculations themselves.
The default example data in StatPgm 5b. Tests : Receiver Operator Characteristics (ROC) will be used for this discussion.
StatPgm 5b i. Create Table for ROC Analysis from 2 Columns of Data is an utility where raw data can be converted into a table that can be used for the ROC analysis proper. The example consists of a two column table with each row representing a case, column 1 the test measurement, and column 2 either + for positive outcome or  for negative outcome.
In the default example, the data has 50 rows (women admitted to the labour ward). Column 1 is the maternal height in cms, and column 2 is + for Caesarean Section and  for vaginal delivery.
The result is a table of 3 columns, which is used for the ROC analysis proper
 Column 1 contains the measurements, in order of magnitude. In this example, they are maternal height in cms
 Column 2 are the number of positive cases with the value. In this example the number of Caesarean Sections with that height
 Column 3 are the number of negative cases with the value. In this example the number of vaginal deliveries with that height
StatPgm 5b ii. Receiver Operator Characteristics (ROC) Analysis
The table created by 5b.i can be inserted into the text box and used for ROC analysis. Two buttons represents the two options for analysis
 High Test Values related to positive outcome is used if Outcome Positive is associated with higher test values
 Low Test Values related to positive outcome is used if Outcome Positive is associated with lower test values
As we are testing whether shorter women are more likely to required Caesarean Section, the "Low Test Values related to positive outcome" should be chosen.
Theoretical Discussions
The example data, the maternal height of 25 women who delivered by Caesarean Section and 25 who delivered vaginally are plotted as shown to the right. Those delivered by Caesarean Section are on the left, and those delivered vaginally on the right.
It can be seen that those with shorter maternal height are more likely to be delivered by Caesarean section, but considerable overlap exists.
To start the discussion, let us arbitrarily decide a cut off value of 154cms as short stature, a test to predict Caesarean Section (horizontal line).
 Of the 25 cases delivered by Caesarean Section (left) 12 were less than 154 cms in height, and 13 were 154cms or more. This makes
 True Positive TP = 12
 False Negative FN = 13
 True Positive Rate TPR = 12 / (12 + 13) = 12 / 25 = 0.48 or 48%
 Of the 25 cases delivered vaginally (right), 5 were less than 154 cms, and 20 were 154cms or more. This makes
 False Positive FP = 5
 True Negative TN = 20
 False Positive Rate FPR = 5 / (5 + 20) = 5 / 25 = 0.2 or 20%
If we decrease the height value for cut offs, there will be fewer true and false positive cases, and more true and false negative cases.
If we increase the height value for cut offs, there will be more true and false positive cases and fewer true and false negative cases.
The consequence of changing the cut off would be the alteration of the True and False Positive Rate, and the two Likelihood Ratios.
The relationship between the True and False Positive Rate, as the cut off values change, forms the Receiver Operator Characteristics, as shown in the plot to the right.
Area under the ROC
The ROC graphic exists within a square defined by the True and False Positive Rates, each have values of 0 to 1. The total possible area
under the ROC curve is therefore 1.
If the test has no predictive value, then the True and False Positive Rates will increase of decrease at the same rate. The line is then the diagonal, and the
area under the ROC curve is 0.5
In a perfect test, where the values from outcome positive and outcome negative do not overlap, then the True Positive Rate will remain at the value of 1,
while the False Positive Rate decreases from 1 to 0. The ROC line would then hug the top left borders, and the area under the ROC is 1.
In most cases, the area is between 0.5 and 1, and the test of statistical significance is whether the area under the ROC differs from the null value of 0.5
Using our example data, Area under ROC (θ)=0.7776, 95% CI =0.6491 to 0.9061. As the 95% confidence interval does not overlap the null value of 0.5, we can conclude that maternal height is a significant predictor for Caesarean Section.
The wider use of ROC
Value  FPR  TPR  TNR  YI  TPR/TNR  TNR/TPR  LR+  LR 
162  0.96  1  0.04  0.04  25.00  0.04   
161.5  0.92  1  0.08  0.08  12.50  0.08   
161  0.84  1  0.16  0.16  6.25  0.16   
160  0.84  0.96  0.16  0.12  6.00  0.17  1.14  0.25 
159  0.76  0.96  0.24  0.2  4.00  0.25  1.26  0.17 
158.5  0.68  0.92  0.32  0.24  2.88  0.35  1.35  0.25 
158  0.48  0.92  0.52  0.44  1.77  0.57  1.92  0.15 
157.5  0.48  0.88  0.52  0.4  1.69  0.59  1.83  0.23 
157  0.4  0.88  0.6  0.48  1.47  0.68  2.20  0.20 
156.5  0.36  0.8  0.64  0.44  1.25  0.80  2.22  0.31 
156  0.36  0.68  0.64  0.32  1.06  0.94  1.89  0.50 
155.5  0.24  0.6  0.76  0.36  0.79  1.27  2.50  0.53 
155  0.2  0.6  0.8  0.4  0.75  1.33  3.00  0.50 
154  0.2  0.48  0.8  0.28  0.60  1.67  2.40  0.65 
153.5  0.12  0.4  0.88  0.28  0.45  2.20  3.33  0.68 
153  0.04  0.36  0.96  0.32  0.38  2.67  9.00  0.67 
152.5  0.04  0.28  0.96  0.24  0.29  3.43  7.00  0.75 
152  0  0.28  1  0.28  0.28  3.57   
151.5  0  0.24  1  0.24  0.24  4.17   
151  0  0.2  1  0.2  0.20  5.00   
150.5  0  0.16  1  0.16  0.16  6.25   
150  0  0.12  1  0.12  0.12  8.33   
149  0  0.08  1  0.08  0.08  12.50   
147.5  0  0  1  0  0.00    
The ROC describes the predictive characteristics of a test over its whole range, rather than a single point. It therefore provides more information to help the clinician. All the parameters at each test value are therefore displayed. A subset of that table, used to determine cutoff values, is shown in the table to the right.
 The Youden Index (YI), a measure of overall accuracy of the test, YI = (TPR+TNR)/2. Using a cut off value where the Youden
Index is maximum produces the fewest overall false predictions.
In our example, this is when maternal height=157cms and the Youden
Index=0.48, the FPR=0.4, and TPR=0.88
 The ratio of True Positive Rate / True Negative Rate (TPR/TNR), a measure of effective inclusion of True Positive cases.
This is used to determine the cut off values used in a test for screening or early alert purposes.
Commonly a TPR/TNR
ratio of 2 to 3 is chosen for this purpose, and in our example, this is at the level where maternal height=158.5cms and the
ratio TPR/TNR=2.88, the FPR=0.68, and TPR=0.92.
An example of using this for early alert can be a rule in the labour
ward that all women admitted with a height less than 158.5cms should be reviewed by qualified and experienced senior staff
soon after admission to formalize a plan of management during labour.
 The ratio of True Negative Rate / True Positive Rate (TNR/TPR), a measure of effective exclusion of false Positive cases.
This is used to determine the cut off values used in a test for clinical intervention, especially if the intervention incurs
high costs or involves high risks.
Commonly a TNR/TPR ratio of 2 to 3 is chosen for this purpose, and in our example, this is at
the level where maternal height=153cms and the TNR/TPR ratio=2.67, the FPR=0.04 and TPR=0.36.
An example of using this
action cutoff can be a rule that all women admitted with a height less than 153cms should be immediately reviewed by qualified
and experienced senior staff, and considered for elective Caesarean Section.
