AI helps researchers investigate how thousands of medical studies share code
The researchers wanted to understand how often clinical prediction studies share code and whether shared code is usable. Finding this information was difficult because papers describe code availability in inconsistent ways. Links and relevant statements could appear in many places and formats, rather than in a standard table. They used a large language model to screen nearly 4,000 papers, identify statements about code availability, and extract repository links. Software retrieved those repositories and assessed features such as documentation, software requirements, and reuse terms. Two researchers independently checked 500 papers and a separate sample of repositories, then compared their judgments with the automated results. The LLMs helped process scattered, unstructured text at scale, but they were not treated as infallible. Agreement was stronger for concrete checks, such as whether a license file existed, than for judging documentation quality. The team therefore emphasized broad patterns, not every individual count. The findings are helping inform TRIPOD-Code, a reporting guideline intended to clarify what researchers should share.
How did the researchers use large language models to investigate code sharing in medical studies?
The researchers wanted to understand how often clinical prediction studies share code and whether shared code is usable. Finding this information was difficult because papers describe code availability in inconsistent ways. Links and relevant statements could appear in many places and formats, rather than in a standard table.
They used a large language model to screen nearly 4,000 papers, identify statements about code availability, and extract repository links. Software retrieved those repositories and assessed features such as documentation, software requirements, and reuse terms. Two researchers independently checked 500 papers and a separate sample of repositories, then compared their judgments with the automated results.
The LLMs helped process scattered, unstructured text at scale, but they were not treated as infallible. Agreement was stronger for concrete checks, such as whether a license file existed, than for judging documentation quality. The team therefore emphasized broad patterns, not every individual count. The findings are helping inform TRIPOD-Code, a reporting guideline intended to clarify what researchers should share.
How many studies did they review, and what share of them reported sharing code?
The review covered almost 4,000 studies that developed, updated, or validated clinical prediction models. These studies cited TRIPOD or TRIPOD+AI, and their full texts were available through PubMed Central’s open-access collection. The scale let the team examine code-sharing practices across a large body of research rather than relying on a few examples.
Only about 1 in 8 studies reported sharing code. The researchers found those reports by using a large language model to identify statements about code availability and extract links. They then retrieved linked repositories and assessed whether they included useful documentation, software requirements, and terms explaining reuse.
Sharing was not consistent across journals or countries. Among journals, the proportion of studies sharing code ranged from zero to more than 70 percent. Some journals with low sharing rates had written code-sharing policies, indicating that policies alone may not be enough to encourage sharing. Because automated classification can make mistakes, the authors emphasized overall patterns rather than treating every count as exact.
What makes a code repository well documented and reusable?
A well-documented, reusable code repository gives other researchers enough information to understand what the code does and run it to reproduce the paper’s results. That matters because a bare link or a pile of files may not reveal how the analysis was carried out. Sharing code is only a first step if others cannot interpret or use it.
The team assessed repositories for completeness of documentation, software requirements, and terms of reuse. In practice, this means explaining the code’s purpose and providing guidance that helps another person run it; software requirements tell users what the code depends on, while reuse terms clarify how it may be used. The researchers found repositories ranging from extremely well-organized and documented to those offering almost no guidance.
These standards are not yet consistently met. The study found that public code did not always come with the supporting information needed for reuse, and judgments about documentation were harder to make consistently than concrete checks such as whether a license file existed. TRIPOD-Code is being developed to clarify what researchers should share.
What can happen when researchers cannot inspect or run the code behind a medical study?
A study’s data and code can show what researchers actually did: how they selected patients, handled missing values, and evaluated results. Without access to these materials, others have fewer ways to examine the methods behind a reported finding. The article emphasizes that methods sections cannot capture every choice in full detail.
For example, researchers might be unable to check how a clinical prediction model was built or tested if its code is unavailable or impossible to run. The study describes clinical prediction models as tools for estimating patient outcomes, such as disease risk, and notes that they inform decisions about patient care. The article does not report a specific case where unavailable code caused a wrong clinical decision; the concern is reduced scrutiny and difficulty building on the work.
The review found that code sharing remains uncommon, and public code does not always include adequate guidance for reuse. Better-organized repositories can help others understand and reproduce reported results. TRIPOD-Code is being developed to clarify what researchers should share, supporting closer examination of methods.
Why does it matter how clinical prediction models are built and tested before they inform patient care?
Clinical prediction models estimate patient outcomes, such as the risk of developing a disease. Because these models can inform decisions about patient care, it matters how researchers built and evaluated them. Others need to be able to examine the methods and understand what the model’s results rest on.
Building a model involves choices about preparing data, developing the model, and evaluating its performance. A study’s data and code can help answer practical questions: how patients were selected, how missing values were handled, and how results were evaluated. These details help researchers scrutinize a model rather than relying only on a short description in a paper.
The review found that code sharing remains uncommon: only about 1 in 8 studies reported sharing code. Even when code was public, repositories varied in documentation and supporting information. The findings are informing TRIPOD-Code, a reporting guideline being developed to clarify what researchers should share. The goal is to make model development and evaluation easier for others to inspect and build on.
When patient records cannot be shared publicly, how can researchers still help others reuse their code?
Patient records may not always be suitable for public release, but code can still be shared with information that helps others understand and try it. The article’s final example is that some researchers provided synthetic datasets to assist others. It does not give further details about those datasets, so their precise role in the examples is not specified.
As a general, established approach, synthetic data can imitate the structure of real data without being the original patient records. Paired with clear documentation and software requirements, such data can help another researcher check how code works and learn how to run it. This is general context beyond the article’s brief mention, not a reported finding about the exact synthetic datasets reviewed.
The review found that public code does not consistently come with the supporting information needed for reuse. Repositories varied widely in their documentation, and the team assessed software requirements and terms of reuse too. TRIPOD-Code is being developed to clarify what researchers should share, which may help make code useful even when underlying patient records cannot be made public.
What are “researcher degrees of freedom,” and how can choices in preparing data or evaluating a model affect a study’s results?
“Researcher degrees of freedom” refers to the choices researchers make while conducting a study. In the article, Tom Pollard uses the term for decisions about preparing data, developing models, and evaluating performance. These decisions matter because different choices can affect the results, while a paper’s methods section can capture only so much detail.
In a medical study, for example, researchers must decide how patients are selected for inclusion, how missing values are handled, and how results are evaluated. The article presents these as questions that access to data and code can help answer. Examining those materials lets other researchers inspect the methods behind the reported findings instead of seeing only a brief account.
The review found that code sharing is uncommon and that public code does not always include enough documentation or supporting information for reuse. This limits how readily others can examine the choices behind a model. The team’s findings are informing TRIPOD-Code, a guideline being developed to clarify what researchers should share and make those choices easier to scrutinize.
This brief was written by AI from the original reporting and checked by other models. Names, figures and quotes come from the source; read it for full context.
Read more in the JupiteX app
Pulse is free. New stories every 4 hours, each one broken into the questions that explain it.
Or read more news on the web