This page explains meta-analysis of predictions that use binary Tests (Test Positive T+ and Test Negative T-) to predict
binary Outcomes (Outcome Positive O+ and Outcome Negative O-), as calculated in the
Meta-analysis for Predictions Program Page
.
In predictive tests, a high True Positive Rate (TPR) is often attained at the expense of having also a high False Positive
Rate (FPR). Published results of predictive tests therefore vary accordingly, some with high TPR and high FPR, others low
TPR and low FPR. Meta-analysis is therefore important in integrating multiple published results to obtain an overview
of relationships between a particular Test with a particular Outcome.
The model proposed by Moses and others (see reference) uses the concept of the
Receiver Operator Characteristics (ROC), plotting the Sensitivity and Specificity of
each study on the ROC chart, and develop statistical methods to fit a ROC curve over the data.
From each study, True Positive Rate (TPR) and False Positive Rate (FPR) are calculated as follows.
- Data entry uses the numbers of True Positives (TP), False Positives (FP), False Negatives (FN), and True Negatives (TN)
- If there is any zero (0) values in the data, or if the second button "Do Meta-analysis Adding 0.5 to All Values in Data" is
clicked, 0.5 is added to all values in the data. The reason for doing so is to ensure that there is no zero value in the
data, which will crash the program with a division by zero error. Even when there is actually no zero value in the data,
user may still choose to use this adjustment so that the results are comparable, as this adjustment is suggested by Moses
et. al. (see reference), and thus the default in many algorithms.
- True Positive Rate TPR = TP / (TP+FN), False Positive Rate FPR = FP / (FP+TN). The table with data, with or
without 0.5 adjustment, and the calculated FPR and TPR, is then presented.
Curve Fitting TPR from FPR
As data points near the extremes are unstable, and for prediction purposes, the data that counts are those with
TPR>=0.5 and FPR<=0.5, only these data points are used in curve fitting. The procedures are as follows.
- Using the data from each study (i)
- Logit(FPR_{i}) u_{i} = Log((FPR_{i} / (1-FPR_{i})))
- Logit(TPR_{i}) v_{i} = Log((TPR_{i} / (1-TPR_{i})))
- Log Odds Ratio Y_{i} = u_{i}-v_{i}
- Standard Error of Log Odds Ratio X_{i} = u_{i}+v_{i}
A standard linear regression analysis is then carried out with X_{i} as independent variable x and Y_{i} as
dependent variable y, so that y = a + bx. In this formula
- b represents changes to the Log Odds Ratio related to changes in Standard Error, a bias caused by different
sample sizes in the data. If b significantly deviate from null (0), then the Log Odds Ratio (y) is unstable, and the results of
curve fitting difficult to interpret. If b does not significantly deviate from null, then the results of curve fitting can
be taken to be free from bias and confidently interpreted.
- a represents the Log Odds Ratio when x, Standard Error = 0. Approximately, it represents mean Log Odds Ratio anyway if
b is statistically not significantly different from null.
- The curve fitting results between FPR and TPR are as follows
- Mean value TPR = 1 / (1 + exp(-a / ( 1 - b)) * exp(log((1 - FPR) / FPR) * ((1 + b) / (1 - b))))
- Given that b is supposed to be null (0), the term ((1 + b) / (1 - b)) is closed or equal to 1, and is therefore optional.
This term is included in the program in the Meta-analysis for Predictions Program Page
, so the results may be marginally
different from algorithms calculated without this term.
- For 95% confidence interval, a = a±1.96SE_{y}, where SE_{y} is the Standard Error of y
As the resulting curve is a mathematically created entity, it is inappropriate to label it as a Receiver Operator Characteristic
(ROC). The overall effect size, the result of the meta-analysis is represented by Q*, which is a point on the fitted curve
where TPR=TNR(1-FPR). The formula also allows an estimate of the Standard Error and therefore the 95% confidence interval of
Q*, in terms of the difference in TPR at Q*
TP | FP | FN | TN |
23 | 15 | 0 | 12 |
18 | 10 | 0 | 6 |
16 | 4 | 3 | 14 |
10 | 6 | 7 | 7 |
15 | 5 | 6 | 15 |
13 | 7 | 2 | 24 |
15 | 2 | 5 | 14 |
7 | 2 | 3 | 24 |
20 | 8 | 2 | 10 |
15 | 4 | 2 | 11 |
25 | 7 | 5 | 9 |
This panel explains the results obtained from the example data used in the
Meta-analysis for Predictions Program Page
,
as shown in the table to the right. Please note the data is computer generated to demonstrate the procedure, and they do
not represent any real observations.
TP | FP | FN | TN | FPR | TPR |
23.5 | 15.5 | 0.5 | 12.5 | 0.5536 | 0.9792 |
18.5 | 10.5 | 0.5 | 6.5 | 0.6176 | 0.9737 |
16.5 | 4.5 | 3.5 | 14.5 | 0.2368 | 0.825 |
10.5 | 6.5 | 7.5 | 7.5 | 0.4643 | 0.5833 |
15.5 | 5.5 | 6.5 | 15.5 | 0.2619 | 0.7045 |
13.5 | 7.5 | 2.5 | 24.5 | 0.2344 | 0.8438 |
15.5 | 2.5 | 5.5 | 14.5 | 0.1471 | 0.7381 |
7.5 | 2.5 | 3.5 | 24.5 | 0.0926 | 0.6818 |
20.5 | 8.5 | 2.5 | 10.5 | 0.4474 | 0.8913 |
15.5 | 4.5 | 2.5 | 11.5 | 0.2813 | 0.8611 |
25.5 | 7.5 | 5.5 | 9.5 | 0.4412 | 0.8226 |
0.5 is added to all items to avoid a zero value, which will cause the program to crash. The actual data used is therefore
as shown in the table to the left. Included also are the False and True Positive Rates (FPR, TPR) calculated.
The studies where FPR<=0.5 and TPR>=0.5, 9 out of the 11 studies, are then used for curve fitting to produce the Summary
ROC, with the following results.
The Mean Log Odds Ratio, calculated without regression
- Mean = 2.2958
- Standard Deviation = 0.8113
- Standard Error = 0.2868
Linear regression y=(2.3519) + (-0.1838)x
- a = 2.3519, Regressed Mean Log Odds Ratio
- b = 0.2868, t = -0.6359 p = 0.6394, not statistically significant
- As b is not statistically significant, we can rule out bias caused by Standard Errors.
- Please note that, without bias, the actual and regressed mean Log Odds Ratio are very similar
FPR | TPR |
| Mean | 95%CI | Q* | Q*CI | Data |
0.01 | 0.2348 | 0.1603 | 0.3304 | | | |
0.02 | 0.3326 | 0.2366 | 0.4448 | | | |
0.03 | 0.3989 | 0.2922 | 0.5162 | | | |
0.04 | 0.4491 | 0.3364 | 0.5672 | | | |
0.05 | 0.4892 | 0.3733 | 0.6062 | | | |
0.06 | 0.5224 | 0.4049 | 0.6375 | | | |
0.07 | 0.5506 | 0.4325 | 0.6633 | | | |
0.08 | 0.5751 | 0.4571 | 0.6852 | | | |
0.09 | 0.5967 | 0.4792 | 0.704 | | | |
0.1 | 0.6158 | 0.4992 | 0.7204 | | | |
0.11 | 0.633 | 0.5176 | 0.735 | | | |
0.12 | 0.6486 | 0.5345 | 0.748 | | | |
0.13 | 0.6629 | 0.5501 | 0.7597 | | | |
0.14 | 0.6759 | 0.5647 | 0.7703 | | | |
0.15 | 0.688 | 0.5783 | 0.78 | | | |
0.16 | 0.6992 | 0.5911 | 0.7889 | | | |
0.17 | 0.7096 | 0.6031 | 0.7971 | | | |
0.18 | 0.7194 | 0.6145 | 0.8047 | | | |
0.19 | 0.7285 | 0.6253 | 0.8118 | | | |
0.2 | 0.7371 | 0.6355 | 0.8184 | | | |
0.21 | 0.7452 | 0.6453 | 0.8246 | | | |
0.22 | 0.7529 | 0.6546 | 0.8305 | | | |
0.23 | 0.7602 | 0.6635 | 0.836 | | | |
0.24 | 0.7671 | 0.672 | 0.8412 | | | |
0.25 | 0.7737 | 0.6801 | 0.8461 | | | |
0.26 | 0.78 | 0.688 | 0.8507 | | | |
0.27 | 0.786 | 0.6955 | 0.8552 | | | |
0.28 | 0.7918 | 0.7028 | 0.8594 | | | |
0.29 | 0.7973 | 0.7098 | 0.8635 | | | |
0.3 | 0.8026 | 0.7166 | 0.8673 | | | |
0.31 | 0.8077 | 0.7232 | 0.871 | | | |
0.32 | 0.8126 | 0.7295 | 0.8746 | | | |
0.33 | 0.8173 | 0.7357 | 0.878 | | | |
0.34 | 0.8219 | 0.7416 | 0.8812 | | | |
0.35 | 0.8263 | 0.7474 | 0.8844 | | | |
0.36 | 0.8306 | 0.7531 | 0.8874 | | | |
0.37 | 0.8348 | 0.7586 | 0.8904 | | | |
0.38 | 0.8388 | 0.7639 | 0.8932 | | | |
0.39 | 0.8427 | 0.7691 | 0.896 | | | |
0.4 | 0.8465 | 0.7742 | 0.8986 | | | |
0.41 | 0.8501 | 0.7792 | 0.9012 | | | |
0.42 | 0.8537 | 0.784 | 0.9037 | | | |
0.43 | 0.8572 | 0.7888 | 0.9061 | | | |
0.44 | 0.8606 | 0.7934 | 0.9085 | | | |
0.45 | 0.8639 | 0.7979 | 0.9108 | | | |
0.46 | 0.8672 | 0.8024 | 0.913 | | | |
0.47 | 0.8703 | 0.8067 | 0.9152 | | | |
0.48 | 0.8734 | 0.811 | 0.9173 | | | |
0.49 | 0.8764 | 0.8152 | 0.9194 | | | |
0.5 | 0.8794 | 0.8193 | 0.9214 | | | |
0.51 | 0.8823 | 0.8234 | 0.9234 | | | |
0.52 | 0.8851 | 0.8274 | 0.9253 | | | |
0.53 | 0.8879 | 0.8313 | 0.9272 | | | |
0.54 | 0.8906 | 0.8351 | 0.929 | | | |
0.55 | 0.8933 | 0.8389 | 0.9309 | | | |
0.56 | 0.8959 | 0.8427 | 0.9326 | | | |
0.57 | 0.8985 | 0.8463 | 0.9344 | | | |
0.58 | 0.9011 | 0.85 | 0.9361 | | | |
0.59 | 0.9036 | 0.8536 | 0.9378 | | | |
0.6 | 0.906 | 0.8571 | 0.9394 | | | |
0.61 | 0.9085 | 0.8606 | 0.941 | | | |
0.62 | 0.9109 | 0.8641 | 0.9426 | | | |
0.63 | 0.9132 | 0.8675 | 0.9442 | | | |
0.64 | 0.9156 | 0.8709 | 0.9457 | | | |
0.65 | 0.9179 | 0.8742 | 0.9473 | | | |
0.66 | 0.9201 | 0.8775 | 0.9488 | | | |
0.67 | 0.9224 | 0.8808 | 0.9503 | | | |
0.68 | 0.9246 | 0.8841 | 0.9517 | | | |
0.69 | 0.9268 | 0.8873 | 0.9532 | | | |
0.7 | 0.929 | 0.8905 | 0.9546 | | | |
0.71 | 0.9311 | 0.8937 | 0.956 | | | |
0.72 | 0.9333 | 0.8969 | 0.9574 | | | |
0.73 | 0.9354 | 0.9 | 0.9588 | | | |
0.74 | 0.9375 | 0.9032 | 0.9602 | | | |
0.75 | 0.9396 | 0.9063 | 0.9615 | | | |
0.76 | 0.9417 | 0.9094 | 0.9629 | | | |
0.77 | 0.9437 | 0.9125 | 0.9642 | | | |
0.78 | 0.9458 | 0.9156 | 0.9656 | | | |
0.79 | 0.9479 | 0.9187 | 0.9669 | | | |
0.8 | 0.9499 | 0.9218 | 0.9682 | | | |
0.81 | 0.952 | 0.9249 | 0.9696 | | | |
0.82 | 0.954 | 0.9281 | 0.9709 | | | |
0.83 | 0.9561 | 0.9312 | 0.9722 | | | |
0.84 | 0.9581 | 0.9343 | 0.9735 | | | |
0.85 | 0.9602 | 0.9375 | 0.9749 | | | |
0.86 | 0.9622 | 0.9407 | 0.9762 | | | |
0.87 | 0.9643 | 0.9439 | 0.9775 | | | |
0.88 | 0.9664 | 0.9471 | 0.9789 | | | |
0.89 | 0.9686 | 0.9504 | 0.9802 | | | |
0.9 | 0.9707 | 0.9538 | 0.9816 | | | |
0.91 | 0.9729 | 0.9572 | 0.983 | | | |
0.92 | 0.9752 | 0.9607 | 0.9844 | | | |
0.93 | 0.9775 | 0.9643 | 0.9859 | | | |
0.94 | 0.9798 | 0.968 | 0.9874 | | | |
0.95 | 0.9823 | 0.9719 | 0.9889 | | | |
0.96 | 0.9849 | 0.9759 | 0.9906 | | | |
0.97 | 0.9877 | 0.9803 | 0.9923 | | | |
0.98 | 0.9907 | 0.9852 | 0.9942 | | | |
0.99 | 0.9943 | 0.9908 | 0.9964 | | | |
0.2358 | | | | 0.7642 | | |
0.2358 | | | | | 0.7136 | |
0.2358 | | | | | 0.8149 | |
0.5536 | | | | | | 0.9792 |
0.6176 | | | | | | 0.9737 |
0.2368 | | | | | | 0.825 |
0.4643 | | | | | | 0.5833 |
0.2619 | | | | | | 0.7045 |
0.2344 | | | | | | 0.8438 |
0.1471 | | | | | | 0.7381 |
0.0926 | | | | | | 0.6818 |
0.4474 | | | | | | 0.8913 |
0.2813 | | | | | | 0.8611 |
0.4412 | | | | | | 0.8226 |
Curve Fit formula
- TPR = 1 / (1 + exp(-a / ( 1 - b)) * exp(log((1 - FPR) / FPR) * ((1 + b) / (1 - b))))
- TPR = 1 / (1 + exp(-2.3519 / ( 1 - -0.1838)) * exp(log((1 - FPR) / FPR) * ((1 + -0.1838) / (1 - -0.1838))))
- TPR = 1 / (1 + 0.1371 * exp(log((1 - FPR) / FPR) * 0.6895))
- For the 95% confidence interval, a is replaced with a±SE_{y} (2.3519±0.2868),
0.1371 replaced with 0.2205 and 0.0853
Main results
- Q* is the value where True Positive and Negative rates are equal (TPR=TNR, TPR=1-FPR)
- In this example TPR=0.7642, and FPR=1-0.7642=0.2358
- The Standard Error of Q* SEQ* = 0.0258 95%CI = 0.7136 to 0.8149.
- In other words, at FPR=0.2358, 95% CI of TPR is 0.7136 to 0.8149
Plotting Coordinates, as shown in the table to the right. The program calculates the coordinates for lines and data points
in a table in such a format
as to allow the whole table to be imported into Excel to produce the Summary ROC Plot. The table includes the False Positive
Rate (x) and the True Positive Rate (y) coordinates for the Summary ROC curve and its 95% Confidence Interval. These are
tabulated with 0.01 intervals
The table continues with the coordinate for Q* and its 95% confidence intervals
Finally, the coordinates for all the data points from the input data.
The program then creates a bitmap of all the data and results, using the MacroPlot algorithm, as shown to the left.
The bitmap contains the following elements
- The plot has False Positive Rate (FPR) as the x axis, and True Positive Rate (TPR) as the y axis
- The diagonal line marks the right lower triangle as the null value of 0.5
- The upper left quarter square marks the area where significant relationship between Test and Outcome interacts. The data
points within this area are used to carry out the curve fitting for the Summary ROC
- The solid round dots representing the coordinates of the input data
- The solid diamond and vertical line representing the Effect Q* and its 95% confidence interval
- The 3 curve lines representing the 3 Summary ROC lines, mean, and 95% confidence interval.
The Area under the Summary ROC (θ), mean and its 95% Confidence Intervals
Moses et.al. (see reference) provide no algorithm for estimating the area. In fact the authors suggest that calculating
θ may be inappropriate, given different sample size and variances from the different studies, and that estimating the
Summary curve uses only a subset of the input data (those with TPR>=0.5 and FPR<0.5).
The program from Meta-analysis for Predictions Program Page
nevertheless calculated the areas under the 3 estimated
Summary ROCs (mean and the 95% Confidence Intervals) by summing up the area under each of the 100 slices with 0.1 intervals
of the FPR. The algorithm used is the same as that in the Area Under Coordinates Program Page
. In this example,
θ has a mean of 0.8297, and 95% CI of 0.7677 to 0.8787
Moses LE, Shapiro D, and Littenberg B(1993) Combining independent studies of a
diagnostic test into a Summary ROC Curve: Data analysis approaches and some
additional considerations. Statistics in Medicine 12:p1293-1316.