Machine learning (ML) algorithms developed to predict in-hospital mortality after acute MI offered more meaningful gains in model calibration than in accuracy, researchers found.
Parsing through data on 29 variables from the American College of Cardiology (ACC) Chest Pain-MI Registry, extreme gradient descent boosting (XGBoost) and meta-classifier models offered no substantive improvement in discrimination compared with standard logistic regression modeling (C-statistics 0.90 for both vs 0.89), reported Harlan Krumholz, MD, SM, of Yale School of Medicine, and colleagues.
However, the two ML models showed nearly perfect agreement between observed and predicted risk across the risk spectrum. Of the people deemed moderate-to-high risk in logistic regression, 27% were more accurately reclassified as low risk by the XGBoost model and 25% by the meta-classifier model — both more consistent with the observed event rates.
“These findings suggest that ML models are not associated with substantially better prediction of risk of death after acute MI but may offer greater resolution of risk, which can better clarify the individual risk for adverse outcomes,” Krumholz’s group reported in a paper published online in JAMA Cardiology.
“However, these improvements are unlikely to be clinically meaningful, and it’s unclear whether they would be sufficient to justify the corresponding loss of interpretability,” commented JAMA Cardiology statistics editor Michael Pencina, PhD, of Duke Clinical Research Institute in Durham, North Carolina, and colleagues in an editorial.
Krumholz’s group maintained that the XGBoost ML model is interpretable “because it represents a collection of decision trees, thereby ensuring transparency in its application that specifically addresses the concerns with black-box ML models.”
Not all ML models performed well either: A neural network algorithm fared worse than logistic regression based on the same inputs in the study.
“This is not the first time that ML has offered only modest or no improvement of traditional regression models in clinical medicine. For every study in which it yielded superior results, there are others in which the gains disappoint. Indeed, the balance of evidence suggests that for typical clinical prediction tasks, wherein predictions are based on a modest number of clinical variables, ML algorithms are on par with logistic regression,” according to Pencina’s group.
As a rule of thumb, the editorialists said, the value of ML increases with greater complexity of the underlying data.
Study authors identified all U.S. acute MI hospitalizations in 2011-2016 from the ACC registry. There were 755,402 patients included (mean age 65 years, 65.5% men).
Predictions of in-hospital mortality were based on patient comorbidities, medical history, presentation characteristics, and initial laboratory values.
Krumholz and colleagues acknowledged that the registry used for the study lacked important information such as duration of comorbidities and control of chronic diseases. Furthermore, the machine learning models were not externally validated in a population outside the registry.
As illustrated by the study, “the generalized linear model is powerful, and only rarely is there a price — a substantial loss of performance — for choosing it. When developing a prediction model, we should choose the simplest tool that will do the job. By honing our intuitions about the likely value added by ML, we can maximize our efforts and sacrifice the simplicity and interpretability of the linear model only when necessary,” Pencina and colleagues concluded.
The study was supported in part by the ACC Foundation and an NIH grant.
Krumholz disclosed receiving personal fees from UnitedHealth, IBM Watson Health, Element Science, Aetna, Facebook, the Siegfried and Jensen Law Firm, the Arnold and Porter Law Firm, the Martin/Baughman Law Firm, and the National Center for Cardiovascular Diseases in Beijing; being a founder of Hugo Health and Refactor Health; and receiving grants and/or contracts from CMS, Medtronic, the FDA, Johnson & Johnson, and the Shenzhen Center for Health Information.
Pencina reported institutional grants from Sanofi/Regeneron and Amgen as well as personal fees from Boehringer Ingelheim.
social experiment by Livio Acerbo #greengroundit #thisisnotapost #thisisart