Examination of two widely used health datasets reveals poor data provenance and potential reliability issues, raising concerns about clinical predictive models built from them and increasing calls for more rigorous research standards.
Research: Evidence of unreliable data and insufficient data provenance in clinical predictive model research and clinical practice. Image credit: Andrey_Popov/Shutterstock.com
New research published in BMC Medicine We conducted an exploratory analysis to examine the data and reporting quality of two large public datasets on stroke and diabetes that are widely used in clinical prediction models.
High research churn raises concerns about data quality
By 2024, researchers have published an estimated 250,000 clinical prediction models to help clinicians diagnose diseases, estimate prognosis, and guide treatment decisions. Because these models can directly impact patient care, their reliability depends on both robust analytical methods and high-quality underlying data.
To improve transparency, the Transparency Reporting of Multivariable Predictive Models for Individual Prognosis or Diagnosis (TRIPOD) guidelines were introduced in 2015 to provide a framework for reporting predictive model studies. The 2024 TRIPOD+AI update expands these recommendations to include both traditional regression and machine learning models, and places greater emphasis on documenting metadata that records where the data comes from, how it was collected, and whether it can be trusted and reused.
The increasing availability of large regularly collected health datasets is accelerating the development of clinical predictive models. But it has also encouraged what researchers describe as “rapid churn” — rapid, routine work that prioritizes publication volume over meaningful scientific progress. According to the authors, this approach increases the risk of false discoveries and can waste valuable research resources.
Concerns about data quality have already led some publishers and journals to tighten editorial policies following misuse of widely used datasets such as the Global Burden of Disease Database and the National Health and Nutrition Examination Survey Database. Other examples, such as untestable cancer cell lines that ultimately led to paper retractions, further highlight the impact of poor data provenance.
Initiatives such as the Findable, Accessible, Interoperable and Reusable (FAIR) principles encourage improved data management, but their implementation remains inconsistent. Similarly, repositories such as Kaggle make datasets widely accessible but do not require users to provide comprehensive provenance information. The authors argue that without stronger standards to verify data provenance, unreliable datasets will continue to circulate in the scientific literature, potentially undermining evidence-based medicine.
Studies that assessed provenance using the TRIPOD+AI standard
Two publicly available health datasets with likely low data origins were selected for their high download volume and relevance to clinical predictive modeling research. One dataset focuses on stroke and the other on diabetes, both accessed from Kaggle on August 27, 2025. The current study aimed to use these datasets to uncover data provenance issues in clinical prediction models.
Each dataset was evaluated using nine TRIPOD+AI data provenance items, and exploratory analyzes such as checking simulated data, unexpected correlations between variables, anomalous distributions, and duplicate rows were conducted to assess reliability. Kaggle’s public discussion regarding the origin of the data was also reviewed and concerns were raised with Kaggle.
We searched Google Scholar to identify peer-reviewed articles that used these datasets for model development or validation and screened them for full text inclusion. Exclusions include non-peer-reviewed works and non-English papers. The authors noted that this search strategy may have underestimated the use of these datasets, as studies that did not include direct Kaggle links were not collected.
Discrepancies in the report were documented, as well as confirmation of disclosure of the origin of the dataset and review of ethical approval and clinical use potential statements. Policy adoption was investigated through Altmetric and Overton. Using OpenAlex, author affiliations by country were analyzed and research volume over time was plotted.
Questionable datasets supporting over 125 studies
An evaluation of two widely used Kaggle health datasets for clinical predictive modeling research revealed significant concerns regarding data provenance and reliability. Of the 653 research efforts identified, 125 published papers used these datasets to develop or validate clinical prediction models.
Evaluation using the 9-item TRIPOD+AI revealed significant deficiencies in both datasets. No information was provided about when, where, why, or how the data were collected, and reliability could not be determined. IIndependently verified. Both datasets failed all nine items of the TRIPOD+AI data provenance assessment.
The stroke dataset included 5,110 cases, which had irregular patient IDs, improbable blood glucose and age distributions, and unrealistically little missing data. Similarly, the diabetes dataset consisted of 100,000 cases with repetitive and unnatural values, artificial correlations, and many duplicate entries. Taken together, these findings indicate that both datasets are likely to be synthetic, fabricated, or unreliable, and therefore unsuitable for research or clinical applications.
The 125 articles included were from 32 countries. However, coverage of ethical approval was rare, and most articles lacked sufficient information about the origin of the data. Only a minority explained their data sources, and the majority did not meet basic transparency standards.
Nevertheless, these datasets were widely cited and frequently used to make clinical care recommendations. Of the 125 studies, 3 models showed evidence of potential for practical use, 1 was cited in a medical device patent, and 86 review articles referenced these models.
Some papers described actual or potential use in clinical practice, and 11 studies developed web- or app-based prediction tools with graphical user interfaces, two of which were publicly accessible. None of the studies are referenced in policy documents.
The number of publications using these datasets continues to grow, despite continuing concerns about the quality and reliability of the underlying data.
Improve data set transparency to protect patient care
This study highlights the urgent need to address the use of unreliable data in clinical predictive model research. Reliable data and transparent methods are essential to ensure reliable clinical decisions and protect patient care. The authors recommend actions by journals, publishers, data repositories, researchers, and clinicians to improve standards and promote responsible research practices.
The authors also emphasized that this study only investigated two publicly available Kaggle datasets, and it remains unclear how widespread similar data provenance issues are in other datasets and repositories.
Click here to download your PDF copy.
