Improving reproducibility in systems biology
BioModels designs an eight-point scorecard to improve mathematical model reproducibility in the life sciences
Summary
- The reproducibility crisis is a growing concern in the life sciences
- After analysing hundreds of mathematical models, EMBL-EBI researchers found that half were not directly reproducible due to incorrect or missing information in the manuscript
- To address the issue, they propose an eight‐point reproducibility scorecard for modellers, reviewers and journal editors
24 March, Cambridge – Reproducibility is the idea that the same results can be achieved when an experiment is repeated by another scientist. This is essential for ensuring that research is credible and reliable.
A 2016 Nature survey showed that over 70% of participants failed to reproduce others' experiments and over 50% failed to reproduce their own results. Following up on the study, the BioModels team at EMBL-EBI systematically analysed 455 mathematical models. Strikingly, they found that almost half couldn’t be reproduced using the information provided in the manuscript.
“These findings were particularly worrying given that mathematical models are considered to be less susceptible to the reproducibility crisis, because they don’t carry any of the risks or uncertainties of experimental methods,” explains Rahuman Sheriff, BioModels Project Leader.
Some of the reasons these models couldn’t be reproduced included missing parameters values, missing initial conditions and inconsistency in model structure.
What is BioModels?
BioModels is one of the largest public open‐source databases of quantitative mathematical models, where the models are manually curated and semantically enriched. In March 2021, BioModels had made 1000 curated models available to the scientific community.
A simple checklist
To address the issue, the BioModels team has come up with an eight-point scorecard that modellers, reviewers and journals can use when publishing or reviewing a model. The scorecard was published in the journal Molecular Biology Systems.
“We hope that this scorecard will help modellers include all the necessary information in the manuscript to improve reproducibility,” continues Sheriff. “We would love to get feedback from the community on how to improve the scorecard, thus making mathematical models in the life sciences more reliable” says Henning Hermjakob, the Head of Molecular Systems at EMBL-EBI.
The scorecard consists of eight questions. For each “yes”, the model gets one point. The BioModels team strongly recommends that models should score at least four points before publication.
Reproducibility scorecard
- Are the mathematical expressions described in the manuscript/supplementary material?
- Are the parameters and entity initial levels listed (as a table) in the manuscript/supplementary material?
- Are simulation conditions including software/programming environment, algorithm, changes in parameters/concentration/states and any data normalisation described under each simulation figure or attached as a supplementary material?
- Are the model code(s) for the mathematical expression and simulation shared publicly?
- Are the model codes available in standard formats such as SBML, COMBINE archive, SED‐ML and are syntactically validated?
- Are the model codes deposited in a relevant open model database?
- Are the model codes well documented to unambiguously identify model entities/variables? (with additional annotation of reactions, mathematical expressions, events, conditions, etc. when relevant.). Are the models in standard formats such as SBML and COMBINE Archive are semantically enriched, i.e. annotated with controlled vocabularies such as Gene Ontology and ChEBI and database resources such as Gene Ontologies?
- Are the numerical results shared publicly along with the model codes?
“Our community has been working for over 10 years to improve reusability of simulation studies, and to reduce time and effort building and verifying complex computational biology models,” says Dagmar Waltemath, the Vice-chair of the COmputational MOdeling in BIology NEtwork (COMBINE) community. “Our focus is on providing standard formats and interoperability solutions together with the right semantic information. However, we haven’t seen as much increase in model reuse as we had wished for. I was excited to read about the scorecard, because it tackles the problem from a different angle - giving feedback to scientists on how “good” a model is in terms of reusability. I hope this will motivate people to better follow the standards and recommendations we develop.”
BioModels is inviting comments from the modelling community on their new eight-point scorecard. Please email biomodels-cura@ebi.ac.uk if you have any questions or suggestions.
Source article
TIWARI, K., et al. (2021). Reproducibility in systems biology modelling. Molecular Systems Biology. Published online 23 02; DOI: 10.15252/msb.20209982
Funding
This work has been supported by EMBL core funding, the Innovative Medicines Initiative 2 Joint Undertaking under grant agreement no. 116030, and BBSRC BB/N019482/1 and BB/N019474/1 (MultiMod).
