Our reanalysis showed that 99.6% of all possible subsets yield lower AUCs than the reported 1.0 (see Figure 1). A bit surprising to us, appropriate performance metrics were already available in their own code: average cross-validation AUC (0.65) and locked model AUC (0.73).

👇 7/13

To demonstrate this, we repeated the train-test split 1000 times, keeping all other analysis steps identical to those used by the original authors. Average AUC dropped to 0.74 (accuracy to 0.68), revealing that the reported AUC is not a robust measure of the model's performance (Figure 2).

👇 10/13

Our reanalysis showed that 99.6% of all possible subsets yield lower AUCs than the reported 1.0 (see Figure 1). A bit surprising to us, appropriate performance metrics were already available in their own code: average cross-validation AUC (0.65) and locked model AUC (0.73).

👇 7/13