University of Maryland Libraries Logo

Systematic Review

  • Library Help
  • What is a Systematic Review (SR)?

Steps of a Systematic Review

  • Framing a Research Question
  • Developing a Search Strategy
  • Searching the Literature
  • Managing the Process
  • Meta-analysis
  • Publishing your Systematic Review

Forms and templates

Logos of MS Word and MS Excel

Image: David Parmenter's Shop

  • PICO Template
  • Inclusion/Exclusion Criteria
  • Database Search Log
  • Review Matrix
  • Cochrane Tool for Assessing Risk of Bias in Included Studies

   • PRISMA Flow Diagram  - Record the numbers of retrieved references and included/excluded studies. You can use the Create Flow Diagram tool to automate the process.

   •  PRISMA Checklist - Checklist of items to include when reporting a systematic review or meta-analysis

PRISMA 2020 and PRISMA-S: Common Questions on Tracking Records and the Flow Diagram

  • PROSPERO Template
  • Manuscript Template
  • Steps of SR (text)
  • Steps of SR (visual)
  • Steps of SR (PIECES)

Adapted from  A Guide to Conducting Systematic Reviews: Steps in a Systematic Review by Cornell University Library

Source: Cochrane Consumers and Communications  (infographics are free to use and licensed under Creative Commons )

Check the following visual resources titled " What Are Systematic Reviews?"

  • Video  with closed captions available
  • Animated Storyboard
  • << Previous: What is a Systematic Review (SR)?
  • Next: Framing a Research Question >>
  • Last Updated: Jan 26, 2024 4:35 PM
  • URL: https://lib.guides.umd.edu/SR

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

  • Knowledge Base

Methodology

  • Systematic Review | Definition, Example, & Guide

Systematic Review | Definition, Example & Guide

Published on June 15, 2022 by Shaun Turney . Revised on November 20, 2023.

A systematic review is a type of review that uses repeatable methods to find, select, and synthesize all available evidence. It answers a clearly formulated research question and explicitly states the methods used to arrive at the answer.

They answered the question “What is the effectiveness of probiotics in reducing eczema symptoms and improving quality of life in patients with eczema?”

In this context, a probiotic is a health product that contains live microorganisms and is taken by mouth. Eczema is a common skin condition that causes red, itchy skin.

Table of contents

What is a systematic review, systematic review vs. meta-analysis, systematic review vs. literature review, systematic review vs. scoping review, when to conduct a systematic review, pros and cons of systematic reviews, step-by-step example of a systematic review, other interesting articles, frequently asked questions about systematic reviews.

A review is an overview of the research that’s already been completed on a topic.

What makes a systematic review different from other types of reviews is that the research methods are designed to reduce bias . The methods are repeatable, and the approach is formal and systematic:

  • Formulate a research question
  • Develop a protocol
  • Search for all relevant studies
  • Apply the selection criteria
  • Extract the data
  • Synthesize the data
  • Write and publish a report

Although multiple sets of guidelines exist, the Cochrane Handbook for Systematic Reviews is among the most widely used. It provides detailed guidelines on how to complete each step of the systematic review process.

Systematic reviews are most commonly used in medical and public health research, but they can also be found in other disciplines.

Systematic reviews typically answer their research question by synthesizing all available evidence and evaluating the quality of the evidence. Synthesizing means bringing together different information to tell a single, cohesive story. The synthesis can be narrative ( qualitative ), quantitative , or both.

Here's why students love Scribbr's proofreading services

Discover proofreading & editing

Systematic reviews often quantitatively synthesize the evidence using a meta-analysis . A meta-analysis is a statistical analysis, not a type of review.

A meta-analysis is a technique to synthesize results from multiple studies. It’s a statistical analysis that combines the results of two or more studies, usually to estimate an effect size .

A literature review is a type of review that uses a less systematic and formal approach than a systematic review. Typically, an expert in a topic will qualitatively summarize and evaluate previous work, without using a formal, explicit method.

Although literature reviews are often less time-consuming and can be insightful or helpful, they have a higher risk of bias and are less transparent than systematic reviews.

Similar to a systematic review, a scoping review is a type of review that tries to minimize bias by using transparent and repeatable methods.

However, a scoping review isn’t a type of systematic review. The most important difference is the goal: rather than answering a specific question, a scoping review explores a topic. The researcher tries to identify the main concepts, theories, and evidence, as well as gaps in the current research.

Sometimes scoping reviews are an exploratory preparation step for a systematic review, and sometimes they are a standalone project.

Prevent plagiarism. Run a free check.

A systematic review is a good choice of review if you want to answer a question about the effectiveness of an intervention , such as a medical treatment.

To conduct a systematic review, you’ll need the following:

  • A precise question , usually about the effectiveness of an intervention. The question needs to be about a topic that’s previously been studied by multiple researchers. If there’s no previous research, there’s nothing to review.
  • If you’re doing a systematic review on your own (e.g., for a research paper or thesis ), you should take appropriate measures to ensure the validity and reliability of your research.
  • Access to databases and journal archives. Often, your educational institution provides you with access.
  • Time. A professional systematic review is a time-consuming process: it will take the lead author about six months of full-time work. If you’re a student, you should narrow the scope of your systematic review and stick to a tight schedule.
  • Bibliographic, word-processing, spreadsheet, and statistical software . For example, you could use EndNote, Microsoft Word, Excel, and SPSS.

A systematic review has many pros .

  • They minimize research bias by considering all available evidence and evaluating each study for bias.
  • Their methods are transparent , so they can be scrutinized by others.
  • They’re thorough : they summarize all available evidence.
  • They can be replicated and updated by others.

Systematic reviews also have a few cons .

  • They’re time-consuming .
  • They’re narrow in scope : they only answer the precise research question.

The 7 steps for conducting a systematic review are explained with an example.

Step 1: Formulate a research question

Formulating the research question is probably the most important step of a systematic review. A clear research question will:

  • Allow you to more effectively communicate your research to other researchers and practitioners
  • Guide your decisions as you plan and conduct your systematic review

A good research question for a systematic review has four components, which you can remember with the acronym PICO :

  • Population(s) or problem(s)
  • Intervention(s)
  • Comparison(s)

You can rearrange these four components to write your research question:

  • What is the effectiveness of I versus C for O in P ?

Sometimes, you may want to include a fifth component, the type of study design . In this case, the acronym is PICOT .

  • Type of study design(s)
  • The population of patients with eczema
  • The intervention of probiotics
  • In comparison to no treatment, placebo , or non-probiotic treatment
  • The outcome of changes in participant-, parent-, and doctor-rated symptoms of eczema and quality of life
  • Randomized control trials, a type of study design

Their research question was:

  • What is the effectiveness of probiotics versus no treatment, a placebo, or a non-probiotic treatment for reducing eczema symptoms and improving quality of life in patients with eczema?

Step 2: Develop a protocol

A protocol is a document that contains your research plan for the systematic review. This is an important step because having a plan allows you to work more efficiently and reduces bias.

Your protocol should include the following components:

  • Background information : Provide the context of the research question, including why it’s important.
  • Research objective (s) : Rephrase your research question as an objective.
  • Selection criteria: State how you’ll decide which studies to include or exclude from your review.
  • Search strategy: Discuss your plan for finding studies.
  • Analysis: Explain what information you’ll collect from the studies and how you’ll synthesize the data.

If you’re a professional seeking to publish your review, it’s a good idea to bring together an advisory committee . This is a group of about six people who have experience in the topic you’re researching. They can help you make decisions about your protocol.

It’s highly recommended to register your protocol. Registering your protocol means submitting it to a database such as PROSPERO or ClinicalTrials.gov .

Step 3: Search for all relevant studies

Searching for relevant studies is the most time-consuming step of a systematic review.

To reduce bias, it’s important to search for relevant studies very thoroughly. Your strategy will depend on your field and your research question, but sources generally fall into these four categories:

  • Databases: Search multiple databases of peer-reviewed literature, such as PubMed or Scopus . Think carefully about how to phrase your search terms and include multiple synonyms of each word. Use Boolean operators if relevant.
  • Handsearching: In addition to searching the primary sources using databases, you’ll also need to search manually. One strategy is to scan relevant journals or conference proceedings. Another strategy is to scan the reference lists of relevant studies.
  • Gray literature: Gray literature includes documents produced by governments, universities, and other institutions that aren’t published by traditional publishers. Graduate student theses are an important type of gray literature, which you can search using the Networked Digital Library of Theses and Dissertations (NDLTD) . In medicine, clinical trial registries are another important type of gray literature.
  • Experts: Contact experts in the field to ask if they have unpublished studies that should be included in your review.

At this stage of your review, you won’t read the articles yet. Simply save any potentially relevant citations using bibliographic software, such as Scribbr’s APA or MLA Generator .

  • Databases: EMBASE, PsycINFO, AMED, LILACS, and ISI Web of Science
  • Handsearch: Conference proceedings and reference lists of articles
  • Gray literature: The Cochrane Library, the metaRegister of Controlled Trials, and the Ongoing Skin Trials Register
  • Experts: Authors of unpublished registered trials, pharmaceutical companies, and manufacturers of probiotics

Step 4: Apply the selection criteria

Applying the selection criteria is a three-person job. Two of you will independently read the studies and decide which to include in your review based on the selection criteria you established in your protocol . The third person’s job is to break any ties.

To increase inter-rater reliability , ensure that everyone thoroughly understands the selection criteria before you begin.

If you’re writing a systematic review as a student for an assignment, you might not have a team. In this case, you’ll have to apply the selection criteria on your own; you can mention this as a limitation in your paper’s discussion.

You should apply the selection criteria in two phases:

  • Based on the titles and abstracts : Decide whether each article potentially meets the selection criteria based on the information provided in the abstracts.
  • Based on the full texts: Download the articles that weren’t excluded during the first phase. If an article isn’t available online or through your library, you may need to contact the authors to ask for a copy. Read the articles and decide which articles meet the selection criteria.

It’s very important to keep a meticulous record of why you included or excluded each article. When the selection process is complete, you can summarize what you did using a PRISMA flow diagram .

Next, Boyle and colleagues found the full texts for each of the remaining studies. Boyle and Tang read through the articles to decide if any more studies needed to be excluded based on the selection criteria.

When Boyle and Tang disagreed about whether a study should be excluded, they discussed it with Varigos until the three researchers came to an agreement.

Step 5: Extract the data

Extracting the data means collecting information from the selected studies in a systematic way. There are two types of information you need to collect from each study:

  • Information about the study’s methods and results . The exact information will depend on your research question, but it might include the year, study design , sample size, context, research findings , and conclusions. If any data are missing, you’ll need to contact the study’s authors.
  • Your judgment of the quality of the evidence, including risk of bias .

You should collect this information using forms. You can find sample forms in The Registry of Methods and Tools for Evidence-Informed Decision Making and the Grading of Recommendations, Assessment, Development and Evaluations Working Group .

Extracting the data is also a three-person job. Two people should do this step independently, and the third person will resolve any disagreements.

They also collected data about possible sources of bias, such as how the study participants were randomized into the control and treatment groups.

Step 6: Synthesize the data

Synthesizing the data means bringing together the information you collected into a single, cohesive story. There are two main approaches to synthesizing the data:

  • Narrative ( qualitative ): Summarize the information in words. You’ll need to discuss the studies and assess their overall quality.
  • Quantitative : Use statistical methods to summarize and compare data from different studies. The most common quantitative approach is a meta-analysis , which allows you to combine results from multiple studies into a summary result.

Generally, you should use both approaches together whenever possible. If you don’t have enough data, or the data from different studies aren’t comparable, then you can take just a narrative approach. However, you should justify why a quantitative approach wasn’t possible.

Boyle and colleagues also divided the studies into subgroups, such as studies about babies, children, and adults, and analyzed the effect sizes within each group.

Step 7: Write and publish a report

The purpose of writing a systematic review article is to share the answer to your research question and explain how you arrived at this answer.

Your article should include the following sections:

  • Abstract : A summary of the review
  • Introduction : Including the rationale and objectives
  • Methods : Including the selection criteria, search method, data extraction method, and synthesis method
  • Results : Including results of the search and selection process, study characteristics, risk of bias in the studies, and synthesis results
  • Discussion : Including interpretation of the results and limitations of the review
  • Conclusion : The answer to your research question and implications for practice, policy, or research

To verify that your report includes everything it needs, you can use the PRISMA checklist .

Once your report is written, you can publish it in a systematic review database, such as the Cochrane Database of Systematic Reviews , and/or in a peer-reviewed journal.

In their report, Boyle and colleagues concluded that probiotics cannot be recommended for reducing eczema symptoms or improving quality of life in patients with eczema. Note Generative AI tools like ChatGPT can be useful at various stages of the writing and research process and can help you to write your systematic review. However, we strongly advise against trying to pass AI-generated text off as your own work.

If you want to know more about statistics , methodology , or research bias , make sure to check out some of our other articles with explanations and examples.

  • Student’s  t -distribution
  • Normal distribution
  • Null and Alternative Hypotheses
  • Chi square tests
  • Confidence interval
  • Quartiles & Quantiles
  • Cluster sampling
  • Stratified sampling
  • Data cleansing
  • Reproducibility vs Replicability
  • Peer review
  • Prospective cohort study

Research bias

  • Implicit bias
  • Cognitive bias
  • Placebo effect
  • Hawthorne effect
  • Hindsight bias
  • Affect heuristic
  • Social desirability bias

A literature review is a survey of scholarly sources (such as books, journal articles, and theses) related to a specific topic or research question .

It is often written as part of a thesis, dissertation , or research paper , in order to situate your work in relation to existing knowledge.

A literature review is a survey of credible sources on a topic, often used in dissertations , theses, and research papers . Literature reviews give an overview of knowledge on a subject, helping you identify relevant theories and methods, as well as gaps in existing research. Literature reviews are set up similarly to other  academic texts , with an introduction , a main body, and a conclusion .

An  annotated bibliography is a list of  source references that has a short description (called an annotation ) for each of the sources. It is often assigned as part of the research process for a  paper .  

A systematic review is secondary research because it uses existing research. You don’t collect new data yourself.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

Turney, S. (2023, November 20). Systematic Review | Definition, Example & Guide. Scribbr. Retrieved February 15, 2024, from https://www.scribbr.com/methodology/systematic-review/

Is this article helpful?

Shaun Turney

Shaun Turney

Other students also liked, how to write a literature review | guide, examples, & templates, how to write a research proposal | examples & templates, what is critical thinking | definition & examples, what is your plagiarism score.

systematic review step by step guide

Systematic Reviews

  • Introduction

Step by Step: Systematic Review Handbooks

Basic steps.

  • 1. Planning a Review
  • 2. Defining Your Question & Criteria
  • 3. Standards & Protocols
  • 4. Search Terms & Strategies
  • 5. Locating Published Research
  • 6. Locating Grey Literature
  • 7. Managing & Documenting Results
  • 8. Selecting & Appraising Studies
  • 9. Extracting Data
  • 10. Writing a Systematic Review
  • Tools & Software
  • Guides & Tutorials
  • Accessing Resources
  • Research Assistance

Templates and checklists are provided throughout this guide to support the many tasks required to complete a review.

  • Systematic Review Workbook: Medical Sciences Library, Texas A&M University A detailed guide and template for developing, conducting, & reporting reviews.

A systematic review uses specific procedures to locate, evaluate and synthesize the results of relevant research to address your research question. Procedures are explicitly defined in advance, in order to ensure transparency, reproducibility, and minimize bias. The below handbooks provide step by step guidance for conducting a systematic review. 

You may be required to follow a specific review standard or protocol, or you may use handbooks for more general guidance. This guide frequently references the Cochrane Handbook .

  • Cochrane Handbook for Systematic Reviews of Interventions This handbook provides detailed guidance on conducting a Cochrane systematic review, however is a valuable resource for all authors,
  • Systematic Reviews of Health Promotion and Public Health Interventions (Cochrane Collaboration) Guidelines specific to advising and supporting systematic reviews of health promotion and public health interventions within the Cochrane Collaboration.
  • Systematic Review Study Guide Concise 16 page step-by-step guide from Monash University Library.

Planning a Review & Research Question Development 

(Guide tabs 1-3)

  • Assemble a team 
  • Develop a specific question: Frameworks like  PICO and FINER can help guide this process
  • Specify inclusion/exclusion criteria
  • Choose a standard (for exp.,  IOM , MECIR )
  • Develop a protocol
  • Consider registering your protocol ( PROSPERO ) 

Searching for Literature

(Guide tabs 4-6)

  • Construct individual search strategies for  Cochrane Summaries , PubMed and any other relevant databases ( PsycINFO , ClinicalTrials.gov, ICTRP , etc.)
  • Run your searches in all databases
  • Search grey literature (association websites, conference proceedings, theses and dissertations, contact researchers in the field) 

Managing & Documenting Search Results 

(Guide tab 7 )

  • Export all citations to a citation manager (see Tools & Software )
  • De-duplicate records, keeping track of original numbers found, duplicates, and numbers after de-duplicating

Screening & Selecting Studies​

(Guide tab 8)

  • Screen citations by title/abstract using inclusion/exclusion criteria 
  • Obtain full-text of possibly relevant citations 
  • Screen citations a second time by full-text. Record reasons for each excluded study

Extracting Data

(Guide tab 9)

  • Extract findings and data from studies 
  • Evaluate the quality and risk of bias of studies
  • Tabulate characteristics of included studies

Writing your Review

(Guide tab 10)

  • Synthesize findings and data; summarize findings 
  • Write article (refer to reporting protocols such as the PRISMA checklist)
  • Update searches if it's been more than 6 months and add relevant studies 
  • Submit to journal

Adapted from MD Anderson Cancer Center

  • << Previous: Introduction
  • Next: 1. Planning a Review >>
  • Last Updated: Oct 26, 2023 1:41 PM
  • URL: https://libguides.ucmerced.edu/systematic-reviews

University of California, Merced

UCI Libraries Mobile Site

  • Langson Library
  • Science Library
  • Grunigen Medical Library
  • Law Library
  • Connect From Off-Campus
  • Accessibility
  • Gateway Study Center

Libaries home page

Email this link

Systematic reviews & evidence synthesis methods.

  • Schedule a Consultation / Meet our Team
  • What is Evidence Synthesis?
  • Types of Evidence Synthesis
  • Evidence Synthesis Across Disciplines
  • Finding and Appraising Existing Systematic Reviews
  • 0. Develop a Protocol
  • 1. Draft your Research Question
  • 2. Select Databases
  • 3. Select Grey Literature Sources
  • 4. Write a Search Strategy
  • 5. Register a Protocol
  • 6. Translate Search Strategies
  • 7. Citation Management
  • 8. Article Screening
  • 9. Risk of Bias Assessment
  • 10. Data Extraction
  • 11. Synthesize, Map, or Describe the Results
  • Open Access Evidence Synthesis Resources

Requirements for the Systematic Review Process

Systematic reviews are a huge endeavor, so here are a few requirements if you are thinking of employing this methodology:

  • Systematic reviews require time . 12-24 months is usual from conception to submission.
  • Systematic reviews require a team . Four (4) or more team members are recommended. A principal investigator, a second investigator, a librarian, and someone well-versed in statistics forms the basic team. Ideally the team might have another investigator and someone to coordinate all the moving pieces. Smaller teams are possible, three is the realistic minimum . Two investigators each wearing more than one hat and one librarian. Sometimes an investigator has the time and energy to coordinate. Occasionally one of the investigators is also a statistical guru.
  • * An exception to this rule is an "empty review," which retrieves zero studies that meet the inclusion criteria. Empty reviews are relatively uncommon, but may be used to demonstrate a need for future research in an area. However, an empty review may instead indicate that the research question was defined too narrowly. 

Why do a systematic review? A well done systematic review is a major contribution to the literature. But the requirements in time and effort are massive. Cochrane estimates one year from conception to completion. This does not including time for review, revision and publication. You need to assemble a team and they need to commit for the duration.

A good place to start is with a consultation with a librarian. Visit the " Schedule a Consultation " page to learn why.

  • << Previous: Finding and Appraising Existing Systematic Reviews
  • Next: 0. Develop a Protocol >>
  • Last Updated: Feb 14, 2024 4:20 PM
  • URL: https://guides.lib.uci.edu/evidence-synthesis

Off-campus? Please use the Software VPN and choose the group UCIFull to access licensed content. For more information, please Click here

Software VPN is not available for guests, so they may not have access to some content when connecting from off-campus.

  • UNC Libraries
  • HSL Academic Process
  • Systematic Reviews
  • Step 8: Write the Review

Systematic Reviews: Step 8: Write the Review

Created by health science librarians.

HSL Logo

  • Step 1: Complete Pre-Review Tasks
  • Step 2: Develop a Protocol
  • Step 3: Conduct Literature Searches
  • Step 4: Manage Citations
  • Step 5: Screen Citations
  • Step 6: Assess Quality of Included Studies
  • Step 7: Extract Data from Included Studies

About Step 8: Write the Review

Write your review, report your review with prisma, review sections, plain language summaries for systematic reviews, writing the review- webinars.

  • Writing the Review FAQs

  Check our FAQ's

   Email us

  Chat with us (during business hours)

   Call (919) 962-0800

   Make an appointment with a librarian

  Request a systematic or scoping review consultation

Search the FAQs

In Step 8, you will write an article or a paper about your systematic review.  It will likely have five sections: introduction, methods, results, discussion, and conclusion.  You will: 

  • Review the reporting standards you will use, such as PRISMA. 
  • Gather your completed data tables and PRISMA chart. 
  • Write the Introduction to the topic and your study, Methods of your research, Results of your research, and Discussion of your results.
  • Write an Abstract describing your study and a Conclusion summarizing your paper. 
  • Cite the studies included in your systematic review and any other articles you may have used in your paper. 
  • If you wish to publish your work, choose a target journal for your article.

The PRISMA Checklist will help you report the details of your systematic review. Your paper will also include a PRISMA chart that is an image of your research process. 

Click an item below to see how it applies to Step 8: Write the Review.

Reporting your review with PRISMA

To write your review, you will need the data from your PRISMA flow diagram .  Review the PRISMA checklist to see which items you should report in your methods section.

Managing your review with Covidence

When you screen in Covidence, it will record the numbers you need for your PRISMA flow diagram from duplicate removal through inclusion of studies.  You may need to add additional information, such as the number of references from each database, citations you find through grey literature or other searching methods, or the number of studies found in your previous work if you are updating a systematic review.

How a librarian can help with Step 8

A librarian can advise you on the process of organizing and writing up your systematic review, including: 

  • Applying the PRISMA reporting templates and the level of detail to include for each element
  • How to report a systematic review search strategy and your review methodology in the completed review
  • How to use prior published reviews to guide you in organizing your manuscript 

Reporting standards & guidelines

Be sure to reference reporting standards when writing your review. This helps ensure that you communicate essential components of your methods, results, and conclusions. There are a number of tools that can be used to ensure compliance with reporting guidelines. A few review-writing resources are listed below.

  • Cochrane Handbook - Chapter 15: Interpreting results and drawing conclusions
  • JBI Manual for Evidence Synthesis - Chapter 12.3 The systematic review
  • PRISMA 2020 (Preferred Reporting Items for Systematic Reviews and Meta-Analysis) The aim of the PRISMA Statement is to help authors improve the reporting of systematic reviews and meta-analyses.

Tools for writing your review

  • RevMan (Cochrane Training)
  • Methods Wizard (Systematic Review Accelerator) The Methods Wizard is part of the Systematic Review Accelerator created by Bond University and the Institute for Evidence-Based Healthcare.
  • UNC HSL Systematic Review Manuscript Template Systematic review manuscript template(.doc) adapted from the PRISMA 2020 checklist. This document provides authors with template for writing about their systematic review. Each table contains a PRISMA checklist item that should be written about in that section, the matching PRISMA Item number, and a box where authors can indicate if an item has been completed. Once text has been added, delete any remaining instructions and the PRISMA checklist tables from the end of each section.
  • The PRISMA 2020 statement: an updated guideline for reporting systematic reviews The PRISMA 2020 statement replaces the 2009 statement and includes new reporting guidance that reflects advances in methods to identify, select, appraise, and synthesise studies.
  • PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews This document is intended to enhance the use, understanding and dissemination of the PRISMA 2020 Statement. Through examples and explanations, the meaning and rationale for each checklist item are presented.

The PRISMA checklist

The Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) is a 27-item checklist used to improve transparency in systematic reviews. These items cover all aspects of the manuscript, including title, abstract, introduction, methods, results, discussion, and funding. The PRISMA checklist can be downloaded in PDF or Word files.

  • PRISMA 2020 Checklists Download the 2020 PRISMA Checklists in Word or PDF formats or download the expanded checklist (PDF).

The PRISMA flow diagram

The PRISMA Flow Diagram visually depicts the flow of studies through each phase of the review process. The PRISMA Flow Diagram can be downloaded in Word files.

  • PRISMA 2020 Flow Diagrams The flow diagram depicts the flow of information through the different phases of a systematic review. It maps out the number of records identified, included and excluded, and the reasons for exclusions. Different templates are available depending on the type of review (new or updated) and sources used to identify studies.

Documenting grey literature and/or hand searches

If you have also searched additional sources, such as professional organization websites, cited or citing references, etc., document your grey literature search using the flow diagram template version 1 PRISMA 2020 flow diagram for new systematic reviews which included searches of databases, registers and other sources or the version 2 PRISMA 2020 flow diagram for updated systematic reviews which included searches of databases, registers and other sources . 

Complete the boxes documenting your database searches,  Identification of studies via databases and registers, according to the PRISMA flow diagram instructions.  Complete the boxes documenting your grey literature and/or hand searches on the right side of the template, Identification of studies via other methods, using the steps below.

Need help completing the PRISMA flow diagram?

There are different PRISMA flow diagram templates for new and updated reviews, as well as different templates for reviews with and without grey literature searches. Be sure you download the correct template to match your review methods, then follow the steps below for each portion of the diagram you have available.

Click to view the step-by-step explanation of the PRISMA flow diagram

Step 1: Preparation Download the flow diagram template version 1 PRISMA 2020 flow diagram for new systematic reviews which included searches of databases and registers only or the version 2 PRISMA 2020 flow diagram for updated systematic reviews which included searches of databases and registers only . 

Click to view the step-by-step explanation of the grey literature & hand searching portion of the PRISMA flow diagram

Step 1: Preparation Download the flow diagram template version 1 PRISMA 2020 flow diagram for new systematic reviews which included searches of databases, registers and other sources or the version 2 PRISMA 2020 flow diagram for updated systematic reviews which included searches of databases, registers and other sources . 

Click to view the step-by-step explanation of review update portion of the PRISMA flow diagram

Step 1: Preparation Download the flow diagram template version 2 PRISMA 2020 flow diagram for updated systematic reviews which included searches of databases and registers only or the version 2 PRISMA 2020 flow diagram for updated systematic reviews which included searches of databases, registers and other sources . 

For more information about updating your systematic review, see the box Updating Your Review? on the Step 3: Conduct Literature Searches page of the guide.

Sections of a Scientific Manuscript

Scientific articles often follow the IMRaD format: Introduction, Methods, Results, and Discussion.  You will also need a title and an abstract to summarize your research.

You can read more about scientific writing through the library guides below.

  • Structure of Scholarly Articles & Peer Review • Explains the standard parts of a medical research article • Compares scholarly journals, professional trade journals, and magazines • Explains peer review and how to find peer reviewed articles and journals
  • Writing in the Health Sciences (For Students and Instructors)
  • Citing & Writing Tools & Guides Includes links to guides for popular citation managers such as EndNote, Sciwheel, Zotero; copyright basics; APA & AMA Style guides; Plagiarism & Citing Sources; Citing & Writing: How to Write Scientific Papers

Sections of a Systematic Review Manuscript

Systematic reviews follow the same structure as original research articles, but you will need to report on your search instead of on details like the participants or sampling. Sections of your manuscript are shown as bold headings in the PRISMA checklist.

Refer to the PRISMA checklist for more information.

Consider including a Plain Language Summary (PLS) when you publish your systematic review. Like an abstract, a PLS gives an overview of your study, but is specifically written and formatted to be easy for non-experts to understand. 

Tips for writing a PLS:

  • Use clear headings e.g. "why did we do this study?"; "what did we do?"; "what did we find?"
  • Use active voice e.g. "we searched for articles in 5 databases instead of "5 databases were searched"
  • Consider need-to-know vs. nice-to-know: what is most important for readers to understand about your study? Be sure to provide the most important points without misrepresenting your study or misleading the reader. 
  • Keep it short: Many journals recommend keeping your plain language summary less than 250 words. 
  • Check journal guidelines: Your journal may have specific guidelines about the format of your plain language summary and when you can publish it. Look at journal guidelines before submitting your article. 

Learn more about Plain Language Summaries: 

  • Rosenberg, A., Baróniková, S., & Feighery, L. (2021). Open Pharma recommendations for plain language summaries of peer-reviewed medical journal publications. Current Medical Research and Opinion, 37(11), 2015–2016.  https://doi.org/10.1080/03007995.2021.1971185
  • Lobban, D., Gardner, J., & Matheis, R. (2021). Plain language summaries of publications of company-sponsored medical research: what key questions do we need to address? Current Medical Research and Opinion, 1–12. https://doi.org/10.1080/03007995.2021.1997221
  • Cochrane Community. (2022, March 21). Updated template and guidance for writing Plain Language Summaries in Cochrane Reviews now available. https://community.cochrane.org/news/updated-template-and-guidance-writing-plain-language-summaries-cochrane-reviews-now-available
  • You can also look at our Health Literacy LibGuide:  https://guides.lib.unc.edu/healthliteracy 

How to Approach Writing a Background Section

What Makes a Good Discussion Section

Writing Up Risk of Bias

Developing Your Implications for Research Section

  • << Previous: Step 7: Extract Data from Included Studies
  • Next: FAQs >>
  • Last Updated: Feb 8, 2024 9:22 AM
  • URL: https://guides.lib.unc.edu/systematic-reviews

Search & Find

  • E-Research by Discipline
  • More Search & Find

Places & Spaces

  • Places to Study
  • Book a Study Room
  • Printers, Scanners, & Computers
  • More Places & Spaces
  • Borrowing & Circulation
  • Request a Title for Purchase
  • Schedule Instruction Session
  • More Services

Support & Guides

  • Course Reserves
  • Research Guides
  • Citing & Writing
  • More Support & Guides
  • Mission Statement
  • Diversity Statement
  • Staff Directory
  • Job Opportunities
  • Give to the Libraries
  • News & Exhibits
  • Reckoning Initiative
  • More About Us

UNC University Libraries Logo

  • Search This Site
  • Privacy Policy
  • Accessibility
  • Give Us Your Feedback
  • 208 Raleigh Street CB #3916
  • Chapel Hill, NC 27515-8890
  • 919-962-1053

Book cover

Handbook of Research Methods in Health Social Sciences pp 805–826 Cite as

Conducting a Systematic Review: A Practical Guide

  • Freya MacMillan 2 ,
  • Kate A. McBride 3 ,
  • Emma S. George 4 &
  • Genevieve Z. Steiner 5  
  • Reference work entry
  • First Online: 13 January 2019

2181 Accesses

1 Citations

It can be challenging to conduct a systematic review with limited experience and skills in undertaking such a task. This chapter provides a practical guide to undertaking a systematic review, providing step-by-step instructions to guide the individual through the process from start to finish. The chapter begins with defining what a systematic review is, reviewing its various components, turning a research question into a search strategy, developing a systematic review protocol, followed by searching for relevant literature and managing citations. Next, the chapter focuses on documenting the characteristics of included studies and summarizing findings, extracting data, methods for assessing risk of bias and considering heterogeneity, and undertaking meta-analyses. Last, the chapter explores creating a narrative and interpreting findings. Practical tips and examples from existing literature are utilized throughout the chapter to assist readers in their learning. By the end of this chapter, the reader will have the knowledge to conduct their own systematic review.

  • Systematic review
  • Search strategy
  • Risk of bias
  • Heterogeneity
  • Meta-analysis
  • Forest plot
  • Funnel plot
  • Meta-synthesis

This is a preview of subscription content, log in via an institution .

Buying options

  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
  • Available as EPUB and PDF
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Barbour RS. Checklists for improving rigour in qualitative research: a case of the tail wagging the dog? BMJ. 2001;322(7294):1115–7.

Article   Google Scholar  

Butler A, Hall H, Copnell B. A guide to writing a qualitative systematic review protocol to enhance evidence-based practice in nursing and health care. Worldviews Evid-Based Nurs. 2016;13(3):241–9.

Cook DJ, Mulrow CD, Haynes RB. Systematic reviews: synthesis of best evidence for clinical decisions. Ann Intern Med. 1997;126(5):376–80.

Dixon-Woods M, Bonas S, Booth A, Jones DR, Miller T, Sutton AJ, … Young B. How can systematic reviews incorporate qualitative research? A critical perspective. Qual Res. 2006;6(1):27–44. https://doi.org/10.1177/1468794106058867 .

Greenhalgh T. How to read a paper: the basics of evidence-based medicine. 4th ed. Chichester/Hoboken: Wiley-Blackwell; 2010.

Google Scholar  

Hannes K, Lockwood C, Pearson A. A comparative analysis of three online appraisal instruments’ ability to assess validity in qualitative research. Qual Health Res. 2010;20(12):1736–43. https://doi.org/10.1177/1049732310378656 .

Higgins JPT, Green S. Cochrane handbook for systematic reviews of interventions (Version 5.1.0 [updated March 2011]). The Cochrane Collaboration; 2011.  http://handbook-5-1.cochrane.org/

Higgins JPT, Altman DG, Gøtzsche PC, Jüni P, Moher D, Oxman AD, … Sterne JAC. The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. BMJ. 2011;343. https://doi.org/10.1136/bmj.d5928 .

Hillier S, Grimmer-Somers K, Merlin T, Middleton P, Salisbury J, Tooher R, Weston A. FORM: an Australian method for formulating and grading recommendations in evidence-based clinical guidelines. BMC Med Res Methodol. 2011;11:23. https://doi.org/10.1186/1471-2288-11-23 .

Humphreys DK, Panter J, Ogilvie D. Questioning the application of risk of bias tools in appraising evidence from natural experimental studies: critical reflections on Benton et al., IJBNPA 2016. Int J Behav Nutr Phys Act. 2017; 14 (1):49. https://doi.org/10.1186/s12966-017-0500-4 .

King R, Hooper B, Wood W. Using bibliographic software to appraise and code data in educational systematic review research. Med Teach. 2011;33(9):719–23. https://doi.org/10.3109/0142159x.2011.558138 .

Koelemay MJ, Vermeulen H. Quick guide to systematic reviews and meta-analysis. Eur J Vasc Endovasc Surg. 2016;51(2):309. https://doi.org/10.1016/j.ejvs.2015.11.010 .

Lucas PJ, Baird J, Arai L, Law C, Roberts HM. Worked examples of alternative methods for the synthesis of qualitative and quantitative research in systematic reviews. BMC Med Res Methodol. 2007;7:4–4. https://doi.org/10.1186/1471-2288-7-4 .

MacMillan F, Kirk A, Mutrie N, Matthews L, Robertson K, Saunders DH. A systematic review of physical activity and sedentary behavior intervention studies in youth with type 1 diabetes: study characteristics, intervention design, and efficacy. Pediatr Diabetes. 2014;15(3):175–89. https://doi.org/10.1111/pedi.12060 .

MacMillan F, Karamacoska D, El Masri A, McBride KA, Steiner GZ, Cook A, … George ES. A systematic review of health promotion intervention studies in the police force: study characteristics, intervention design and impacts on health. Occup Environ Med. 2017. https://doi.org/10.1136/oemed-2017-104430 .

Matthews L, Kirk A, MacMillan F, Mutrie N. Can physical activity interventions for adults with type 2 diabetes be translated into practice settings? A systematic review using the RE-AIM framework. Transl Behav Med. 2014;4(1):60–78. https://doi.org/10.1007/s13142-013-0235-y .

Moher D, Schulz KF, Altman DG. The CONSORT statement: revised recommendations for improving the quality of reports of parallel group randomized trials. BMC Med Res Methodol. 2001;1:2. https://doi.org/10.1186/1471-2288-1-2 .

Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. 2009;6(7):e1000097. https://doi.org/10.1371/journal.pmed.1000097 .

Moher D, Shamseer L, Clarke M, Ghersi D, Liberati A, Petticrew M, et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Syst Rev. 2015;4:1. https://doi.org/10.1186/2046-4053-4-1 .

Mulrow CD, Cook DJ, Davidoff F. Systematic reviews: critical links in the great chain of evidence. Ann Intern Med. 1997;126(5):389–91.

Peters MDJ. Managing and coding references for systematic reviews and scoping reviews in EndNote. Med Ref Serv Q. 2017;36(1):19–31. https://doi.org/10.1080/02763869.2017.1259891 .

Steiner GZ, Mathersul DC, MacMillan F, Camfield DA, Klupp NL, Seto SW, … Chang DH. A systematic review of intervention studies examining nutritional and herbal therapies for mild cognitive impairment and dementia using neuroimaging methods: study characteristics and intervention efficacy. Evid Based Complement Alternat Med. 2017;2017:21. https://doi.org/10.1155/2017/6083629 .

Sterne JA, Hernán MA, Reeves BC, Savović J, Berkman ND, Viswanathan M, … Higgins JP. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ. 2016;355. https://doi.org/10.1136/bmj.i4919 .

Tong A, Sainsbury P, Craig J. Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Health Care. 2007;19(6):349–57. https://doi.org/10.1093/intqhc/mzm042 .

Tong A, Palmer S, Craig JC, Strippoli GFM. A guide to reading and using systematic reviews of qualitative research. Nephrol Dial Transplant. 2016;31(6):897–903. https://doi.org/10.1093/ndt/gfu354 .

Uman LS. Systematic reviews and meta-analyses. J Can Acad Child Adolesc Psychiatry. 2011;20(1):57–9.

Download references

Author information

Authors and affiliations.

School of Science and Health and Translational Health Research Institute (THRI), Western Sydney University, Penrith, NSW, Australia

Freya MacMillan

School of Medicine and Translational Health Research Institute, Western Sydney University, Sydney, NSW, Australia

Kate A. McBride

School of Science and Health, Western Sydney University, Sydney, NSW, Australia

Emma S. George

NICM and Translational Health Research Institute (THRI), Western Sydney University, Penrith, NSW, Australia

Genevieve Z. Steiner

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Freya MacMillan .

Editor information

Editors and affiliations.

School of Science and Health, Western Sydney University, Penrith, NSW, Australia

Pranee Liamputtong

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Singapore Pte Ltd.

About this entry

Cite this entry.

MacMillan, F., McBride, K.A., George, E.S., Steiner, G.Z. (2019). Conducting a Systematic Review: A Practical Guide. In: Liamputtong, P. (eds) Handbook of Research Methods in Health Social Sciences. Springer, Singapore. https://doi.org/10.1007/978-981-10-5251-4_113

Download citation

DOI : https://doi.org/10.1007/978-981-10-5251-4_113

Published : 13 January 2019

Publisher Name : Springer, Singapore

Print ISBN : 978-981-10-5250-7

Online ISBN : 978-981-10-5251-4

eBook Packages : Social Sciences Reference Module Humanities and Social Sciences Reference Module Business, Economics and Social Sciences

Share this entry

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

  • Publish with us

Policies and ethics

  • Find a journal
  • Track your research
  • Open access
  • Published: 01 August 2019

A step by step guide for conducting a systematic review and meta-analysis with simulation data

  • Gehad Mohamed Tawfik 1 , 2 ,
  • Kadek Agus Surya Dila 2 , 3 ,
  • Muawia Yousif Fadlelmola Mohamed 2 , 4 ,
  • Dao Ngoc Hien Tam 2 , 5 ,
  • Nguyen Dang Kien 2 , 6 ,
  • Ali Mahmoud Ahmed 2 , 7 &
  • Nguyen Tien Huy 8 , 9 , 10  

Tropical Medicine and Health volume  47 , Article number:  46 ( 2019 ) Cite this article

1.71m Accesses

252 Citations

95 Altmetric

Metrics details

The massive abundance of studies relating to tropical medicine and health has increased strikingly over the last few decades. In the field of tropical medicine and health, a well-conducted systematic review and meta-analysis (SR/MA) is considered a feasible solution for keeping clinicians abreast of current evidence-based medicine. Understanding of SR/MA steps is of paramount importance for its conduction. It is not easy to be done as there are obstacles that could face the researcher. To solve those hindrances, this methodology study aimed to provide a step-by-step approach mainly for beginners and junior researchers, in the field of tropical medicine and other health care fields, on how to properly conduct a SR/MA, in which all the steps here depicts our experience and expertise combined with the already well-known and accepted international guidance.

We suggest that all steps of SR/MA should be done independently by 2–3 reviewers’ discussion, to ensure data quality and accuracy.

SR/MA steps include the development of research question, forming criteria, search strategy, searching databases, protocol registration, title, abstract, full-text screening, manual searching, extracting data, quality assessment, data checking, statistical analysis, double data checking, and manuscript writing.

Introduction

The amount of studies published in the biomedical literature, especially tropical medicine and health, has increased strikingly over the last few decades. This massive abundance of literature makes clinical medicine increasingly complex, and knowledge from various researches is often needed to inform a particular clinical decision. However, available studies are often heterogeneous with regard to their design, operational quality, and subjects under study and may handle the research question in a different way, which adds to the complexity of evidence and conclusion synthesis [ 1 ].

Systematic review and meta-analyses (SR/MAs) have a high level of evidence as represented by the evidence-based pyramid. Therefore, a well-conducted SR/MA is considered a feasible solution in keeping health clinicians ahead regarding contemporary evidence-based medicine.

Differing from a systematic review, unsystematic narrative review tends to be descriptive, in which the authors select frequently articles based on their point of view which leads to its poor quality. A systematic review, on the other hand, is defined as a review using a systematic method to summarize evidence on questions with a detailed and comprehensive plan of study. Furthermore, despite the increasing guidelines for effectively conducting a systematic review, we found that basic steps often start from framing question, then identifying relevant work which consists of criteria development and search for articles, appraise the quality of included studies, summarize the evidence, and interpret the results [ 2 , 3 ]. However, those simple steps are not easy to be reached in reality. There are many troubles that a researcher could be struggled with which has no detailed indication.

Conducting a SR/MA in tropical medicine and health may be difficult especially for young researchers; therefore, understanding of its essential steps is crucial. It is not easy to be done as there are obstacles that could face the researcher. To solve those hindrances, we recommend a flow diagram (Fig. 1 ) which illustrates a detailed and step-by-step the stages for SR/MA studies. This methodology study aimed to provide a step-by-step approach mainly for beginners and junior researchers, in the field of tropical medicine and other health care fields, on how to properly and succinctly conduct a SR/MA; all the steps here depicts our experience and expertise combined with the already well known and accepted international guidance.

figure 1

Detailed flow diagram guideline for systematic review and meta-analysis steps. Note : Star icon refers to “2–3 reviewers screen independently”

Methods and results

Detailed steps for conducting any systematic review and meta-analysis.

We searched the methods reported in published SR/MA in tropical medicine and other healthcare fields besides the published guidelines like Cochrane guidelines {Higgins, 2011 #7} [ 4 ] to collect the best low-bias method for each step of SR/MA conduction steps. Furthermore, we used guidelines that we apply in studies for all SR/MA steps. We combined these methods in order to conclude and conduct a detailed flow diagram that shows the SR/MA steps how being conducted.

Any SR/MA must follow the widely accepted Preferred Reporting Items for Systematic Review and Meta-analysis statement (PRISMA checklist 2009) (Additional file 5 : Table S1) [ 5 ].

We proposed our methods according to a valid explanatory simulation example choosing the topic of “evaluating safety of Ebola vaccine,” as it is known that Ebola is a very rare tropical disease but fatal. All the explained methods feature the standards followed internationally, with our compiled experience in the conduct of SR beside it, which we think proved some validity. This is a SR under conduct by a couple of researchers teaming in a research group, moreover, as the outbreak of Ebola which took place (2013–2016) in Africa resulted in a significant mortality and morbidity. Furthermore, since there are many published and ongoing trials assessing the safety of Ebola vaccines, we thought this would provide a great opportunity to tackle this hotly debated issue. Moreover, Ebola started to fire again and new fatal outbreak appeared in the Democratic Republic of Congo since August 2018, which caused infection to more than 1000 people according to the World Health Organization, and 629 people have been killed till now. Hence, it is considered the second worst Ebola outbreak, after the first one in West Africa in 2014 , which infected more than 26,000 and killed about 11,300 people along outbreak course.

Research question and objectives

Like other study designs, the research question of SR/MA should be feasible, interesting, novel, ethical, and relevant. Therefore, a clear, logical, and well-defined research question should be formulated. Usually, two common tools are used: PICO or SPIDER. PICO (Population, Intervention, Comparison, Outcome) is used mostly in quantitative evidence synthesis. Authors demonstrated that PICO holds more sensitivity than the more specific SPIDER approach [ 6 ]. SPIDER (Sample, Phenomenon of Interest, Design, Evaluation, Research type) was proposed as a method for qualitative and mixed methods search.

We here recommend a combined approach of using either one or both the SPIDER and PICO tools to retrieve a comprehensive search depending on time and resources limitations. When we apply this to our assumed research topic, being of qualitative nature, the use of SPIDER approach is more valid.

PICO is usually used for systematic review and meta-analysis of clinical trial study. For the observational study (without intervention or comparator), in many tropical and epidemiological questions, it is usually enough to use P (Patient) and O (outcome) only to formulate a research question. We must indicate clearly the population (P), then intervention (I) or exposure. Next, it is necessary to compare (C) the indicated intervention with other interventions, i.e., placebo. Finally, we need to clarify which are our relevant outcomes.

To facilitate comprehension, we choose the Ebola virus disease (EVD) as an example. Currently, the vaccine for EVD is being developed and under phase I, II, and III clinical trials; we want to know whether this vaccine is safe and can induce sufficient immunogenicity to the subjects.

An example of a research question for SR/MA based on PICO for this issue is as follows: How is the safety and immunogenicity of Ebola vaccine in human? (P: healthy subjects (human), I: vaccination, C: placebo, O: safety or adverse effects)

Preliminary research and idea validation

We recommend a preliminary search to identify relevant articles, ensure the validity of the proposed idea, avoid duplication of previously addressed questions, and assure that we have enough articles for conducting its analysis. Moreover, themes should focus on relevant and important health-care issues, consider global needs and values, reflect the current science, and be consistent with the adopted review methods. Gaining familiarity with a deep understanding of the study field through relevant videos and discussions is of paramount importance for better retrieval of results. If we ignore this step, our study could be canceled whenever we find out a similar study published before. This means we are wasting our time to deal with a problem that has been tackled for a long time.

To do this, we can start by doing a simple search in PubMed or Google Scholar with search terms Ebola AND vaccine. While doing this step, we identify a systematic review and meta-analysis of determinant factors influencing antibody response from vaccination of Ebola vaccine in non-human primate and human [ 7 ], which is a relevant paper to read to get a deeper insight and identify gaps for better formulation of our research question or purpose. We can still conduct systematic review and meta-analysis of Ebola vaccine because we evaluate safety as a different outcome and different population (only human).

Inclusion and exclusion criteria

Eligibility criteria are based on the PICO approach, study design, and date. Exclusion criteria mostly are unrelated, duplicated, unavailable full texts, or abstract-only papers. These exclusions should be stated in advance to refrain the researcher from bias. The inclusion criteria would be articles with the target patients, investigated interventions, or the comparison between two studied interventions. Briefly, it would be articles which contain information answering our research question. But the most important is that it should be clear and sufficient information, including positive or negative, to answer the question.

For the topic we have chosen, we can make inclusion criteria: (1) any clinical trial evaluating the safety of Ebola vaccine and (2) no restriction regarding country, patient age, race, gender, publication language, and date. Exclusion criteria are as follows: (1) study of Ebola vaccine in non-human subjects or in vitro studies; (2) study with data not reliably extracted, duplicate, or overlapping data; (3) abstract-only papers as preceding papers, conference, editorial, and author response theses and books; (4) articles without available full text available; and (5) case reports, case series, and systematic review studies. The PRISMA flow diagram template that is used in SR/MA studies can be found in Fig. 2 .

figure 2

PRISMA flow diagram of studies’ screening and selection

Search strategy

A standard search strategy is used in PubMed, then later it is modified according to each specific database to get the best relevant results. The basic search strategy is built based on the research question formulation (i.e., PICO or PICOS). Search strategies are constructed to include free-text terms (e.g., in the title and abstract) and any appropriate subject indexing (e.g., MeSH) expected to retrieve eligible studies, with the help of an expert in the review topic field or an information specialist. Additionally, we advise not to use terms for the Outcomes as their inclusion might hinder the database being searched to retrieve eligible studies because the used outcome is not mentioned obviously in the articles.

The improvement of the search term is made while doing a trial search and looking for another relevant term within each concept from retrieved papers. To search for a clinical trial, we can use these descriptors in PubMed: “clinical trial”[Publication Type] OR “clinical trials as topic”[MeSH terms] OR “clinical trial”[All Fields]. After some rounds of trial and refinement of search term, we formulate the final search term for PubMed as follows: (ebola OR ebola virus OR ebola virus disease OR EVD) AND (vaccine OR vaccination OR vaccinated OR immunization) AND (“clinical trial”[Publication Type] OR “clinical trials as topic”[MeSH Terms] OR “clinical trial”[All Fields]). Because the study for this topic is limited, we do not include outcome term (safety and immunogenicity) in the search term to capture more studies.

Search databases, import all results to a library, and exporting to an excel sheet

According to the AMSTAR guidelines, at least two databases have to be searched in the SR/MA [ 8 ], but as you increase the number of searched databases, you get much yield and more accurate and comprehensive results. The ordering of the databases depends mostly on the review questions; being in a study of clinical trials, you will rely mostly on Cochrane, mRCTs, or International Clinical Trials Registry Platform (ICTRP). Here, we propose 12 databases (PubMed, Scopus, Web of Science, EMBASE, GHL, VHL, Cochrane, Google Scholar, Clinical trials.gov , mRCTs, POPLINE, and SIGLE), which help to cover almost all published articles in tropical medicine and other health-related fields. Among those databases, POPLINE focuses on reproductive health. Researchers should consider to choose relevant database according to the research topic. Some databases do not support the use of Boolean or quotation; otherwise, there are some databases that have special searching way. Therefore, we need to modify the initial search terms for each database to get appreciated results; therefore, manipulation guides for each online database searches are presented in Additional file 5 : Table S2. The detailed search strategy for each database is found in Additional file 5 : Table S3. The search term that we created in PubMed needs customization based on a specific characteristic of the database. An example for Google Scholar advanced search for our topic is as follows:

With all of the words: ebola virus

With at least one of the words: vaccine vaccination vaccinated immunization

Where my words occur: in the title of the article

With all of the words: EVD

Finally, all records are collected into one Endnote library in order to delete duplicates and then to it export into an excel sheet. Using remove duplicating function with two options is mandatory. All references which have (1) the same title and author, and published in the same year, and (2) the same title and author, and published in the same journal, would be deleted. References remaining after this step should be exported to an excel file with essential information for screening. These could be the authors’ names, publication year, journal, DOI, URL link, and abstract.

Protocol writing and registration

Protocol registration at an early stage guarantees transparency in the research process and protects from duplication problems. Besides, it is considered a documented proof of team plan of action, research question, eligibility criteria, intervention/exposure, quality assessment, and pre-analysis plan. It is recommended that researchers send it to the principal investigator (PI) to revise it, then upload it to registry sites. There are many registry sites available for SR/MA like those proposed by Cochrane and Campbell collaborations; however, we recommend registering the protocol into PROSPERO as it is easier. The layout of a protocol template, according to PROSPERO, can be found in Additional file 5 : File S1.

Title and abstract screening

Decisions to select retrieved articles for further assessment are based on eligibility criteria, to minimize the chance of including non-relevant articles. According to the Cochrane guidance, two reviewers are a must to do this step, but as for beginners and junior researchers, this might be tiresome; thus, we propose based on our experience that at least three reviewers should work independently to reduce the chance of error, particularly in teams with a large number of authors to add more scrutiny and ensure proper conduct. Mostly, the quality with three reviewers would be better than two, as two only would have different opinions from each other, so they cannot decide, while the third opinion is crucial. And here are some examples of systematic reviews which we conducted following the same strategy (by a different group of researchers in our research group) and published successfully, and they feature relevant ideas to tropical medicine and disease [ 9 , 10 , 11 ].

In this step, duplications will be removed manually whenever the reviewers find them out. When there is a doubt about an article decision, the team should be inclusive rather than exclusive, until the main leader or PI makes a decision after discussion and consensus. All excluded records should be given exclusion reasons.

Full text downloading and screening

Many search engines provide links for free to access full-text articles. In case not found, we can search in some research websites as ResearchGate, which offer an option of direct full-text request from authors. Additionally, exploring archives of wanted journals, or contacting PI to purchase it if available. Similarly, 2–3 reviewers work independently to decide about included full texts according to eligibility criteria, with reporting exclusion reasons of articles. In case any disagreement has occurred, the final decision has to be made by discussion.

Manual search

One has to exhaust all possibilities to reduce bias by performing an explicit hand-searching for retrieval of reports that may have been dropped from first search [ 12 ]. We apply five methods to make manual searching: searching references from included studies/reviews, contacting authors and experts, and looking at related articles/cited articles in PubMed and Google Scholar.

We describe here three consecutive methods to increase and refine the yield of manual searching: firstly, searching reference lists of included articles; secondly, performing what is known as citation tracking in which the reviewers track all the articles that cite each one of the included articles, and this might involve electronic searching of databases; and thirdly, similar to the citation tracking, we follow all “related to” or “similar” articles. Each of the abovementioned methods can be performed by 2–3 independent reviewers, and all the possible relevant article must undergo further scrutiny against the inclusion criteria, after following the same records yielded from electronic databases, i.e., title/abstract and full-text screening.

We propose an independent reviewing by assigning each member of the teams a “tag” and a distinct method, to compile all the results at the end for comparison of differences and discussion and to maximize the retrieval and minimize the bias. Similarly, the number of included articles has to be stated before addition to the overall included records.

Data extraction and quality assessment

This step entitles data collection from included full-texts in a structured extraction excel sheet, which is previously pilot-tested for extraction using some random studies. We recommend extracting both adjusted and non-adjusted data because it gives the most allowed confounding factor to be used in the analysis by pooling them later [ 13 ]. The process of extraction should be executed by 2–3 independent reviewers. Mostly, the sheet is classified into the study and patient characteristics, outcomes, and quality assessment (QA) tool.

Data presented in graphs should be extracted by software tools such as Web plot digitizer [ 14 ]. Most of the equations that can be used in extraction prior to analysis and estimation of standard deviation (SD) from other variables is found inside Additional file 5 : File S2 with their references as Hozo et al. [ 15 ], Xiang et al. [ 16 ], and Rijkom et al. [ 17 ]. A variety of tools are available for the QA, depending on the design: ROB-2 Cochrane tool for randomized controlled trials [ 18 ] which is presented as Additional file 1 : Figure S1 and Additional file 2 : Figure S2—from a previous published article data—[ 19 ], NIH tool for observational and cross-sectional studies [ 20 ], ROBINS-I tool for non-randomize trials [ 21 ], QUADAS-2 tool for diagnostic studies, QUIPS tool for prognostic studies, CARE tool for case reports, and ToxRtool for in vivo and in vitro studies. We recommend that 2–3 reviewers independently assess the quality of the studies and add to the data extraction form before the inclusion into the analysis to reduce the risk of bias. In the NIH tool for observational studies—cohort and cross-sectional—as in this EBOLA case, to evaluate the risk of bias, reviewers should rate each of the 14 items into dichotomous variables: yes, no, or not applicable. An overall score is calculated by adding all the items scores as yes equals one, while no and NA equals zero. A score will be given for every paper to classify them as poor, fair, or good conducted studies, where a score from 0–5 was considered poor, 6–9 as fair, and 10–14 as good.

In the EBOLA case example above, authors can extract the following information: name of authors, country of patients, year of publication, study design (case report, cohort study, or clinical trial or RCT), sample size, the infected point of time after EBOLA infection, follow-up interval after vaccination time, efficacy, safety, adverse effects after vaccinations, and QA sheet (Additional file 6 : Data S1).

Data checking

Due to the expected human error and bias, we recommend a data checking step, in which every included article is compared with its counterpart in an extraction sheet by evidence photos, to detect mistakes in data. We advise assigning articles to 2–3 independent reviewers, ideally not the ones who performed the extraction of those articles. When resources are limited, each reviewer is assigned a different article than the one he extracted in the previous stage.

Statistical analysis

Investigators use different methods for combining and summarizing findings of included studies. Before analysis, there is an important step called cleaning of data in the extraction sheet, where the analyst organizes extraction sheet data in a form that can be read by analytical software. The analysis consists of 2 types namely qualitative and quantitative analysis. Qualitative analysis mostly describes data in SR studies, while quantitative analysis consists of two main types: MA and network meta-analysis (NMA). Subgroup, sensitivity, cumulative analyses, and meta-regression are appropriate for testing whether the results are consistent or not and investigating the effect of certain confounders on the outcome and finding the best predictors. Publication bias should be assessed to investigate the presence of missing studies which can affect the summary.

To illustrate basic meta-analysis, we provide an imaginary data for the research question about Ebola vaccine safety (in terms of adverse events, 14 days after injection) and immunogenicity (Ebola virus antibodies rise in geometric mean titer, 6 months after injection). Assuming that from searching and data extraction, we decided to do an analysis to evaluate Ebola vaccine “A” safety and immunogenicity. Other Ebola vaccines were not meta-analyzed because of the limited number of studies (instead, it will be included for narrative review). The imaginary data for vaccine safety meta-analysis can be accessed in Additional file 7 : Data S2. To do the meta-analysis, we can use free software, such as RevMan [ 22 ] or R package meta [ 23 ]. In this example, we will use the R package meta. The tutorial of meta package can be accessed through “General Package for Meta-Analysis” tutorial pdf [ 23 ]. The R codes and its guidance for meta-analysis done can be found in Additional file 5 : File S3.

For the analysis, we assume that the study is heterogenous in nature; therefore, we choose a random effect model. We did an analysis on the safety of Ebola vaccine A. From the data table, we can see some adverse events occurring after intramuscular injection of vaccine A to the subject of the study. Suppose that we include six studies that fulfill our inclusion criteria. We can do a meta-analysis for each of the adverse events extracted from the studies, for example, arthralgia, from the results of random effect meta-analysis using the R meta package.

From the results shown in Additional file 3 : Figure S3, we can see that the odds ratio (OR) of arthralgia is 1.06 (0.79; 1.42), p value = 0.71, which means that there is no association between the intramuscular injection of Ebola vaccine A and arthralgia, as the OR is almost one, and besides, the P value is insignificant as it is > 0.05.

In the meta-analysis, we can also visualize the results in a forest plot. It is shown in Fig. 3 an example of a forest plot from the simulated analysis.

figure 3

Random effect model forest plot for comparison of vaccine A versus placebo

From the forest plot, we can see six studies (A to F) and their respective OR (95% CI). The green box represents the effect size (in this case, OR) of each study. The bigger the box means the study weighted more (i.e., bigger sample size). The blue diamond shape represents the pooled OR of the six studies. We can see the blue diamond cross the vertical line OR = 1, which indicates no significance for the association as the diamond almost equalized in both sides. We can confirm this also from the 95% confidence interval that includes one and the p value > 0.05.

For heterogeneity, we see that I 2 = 0%, which means no heterogeneity is detected; the study is relatively homogenous (it is rare in the real study). To evaluate publication bias related to the meta-analysis of adverse events of arthralgia, we can use the metabias function from the R meta package (Additional file 4 : Figure S4) and visualization using a funnel plot. The results of publication bias are demonstrated in Fig. 4 . We see that the p value associated with this test is 0.74, indicating symmetry of the funnel plot. We can confirm it by looking at the funnel plot.

figure 4

Publication bias funnel plot for comparison of vaccine A versus placebo

Looking at the funnel plot, the number of studies at the left and right side of the funnel plot is the same; therefore, the plot is symmetry, indicating no publication bias detected.

Sensitivity analysis is a procedure used to discover how different values of an independent variable will influence the significance of a particular dependent variable by removing one study from MA. If all included study p values are < 0.05, hence, removing any study will not change the significant association. It is only performed when there is a significant association, so if the p value of MA done is 0.7—more than one—the sensitivity analysis is not needed for this case study example. If there are 2 studies with p value > 0.05, removing any of the two studies will result in a loss of the significance.

Double data checking

For more assurance on the quality of results, the analyzed data should be rechecked from full-text data by evidence photos, to allow an obvious check for the PI of the study.

Manuscript writing, revision, and submission to a journal

Writing based on four scientific sections: introduction, methods, results, and discussion, mostly with a conclusion. Performing a characteristic table for study and patient characteristics is a mandatory step which can be found as a template in Additional file 5 : Table S3.

After finishing the manuscript writing, characteristics table, and PRISMA flow diagram, the team should send it to the PI to revise it well and reply to his comments and, finally, choose a suitable journal for the manuscript which fits with considerable impact factor and fitting field. We need to pay attention by reading the author guidelines of journals before submitting the manuscript.

The role of evidence-based medicine in biomedical research is rapidly growing. SR/MAs are also increasing in the medical literature. This paper has sought to provide a comprehensive approach to enable reviewers to produce high-quality SR/MAs. We hope that readers could gain general knowledge about how to conduct a SR/MA and have the confidence to perform one, although this kind of study requires complex steps compared to narrative reviews.

Having the basic steps for conduction of MA, there are many advanced steps that are applied for certain specific purposes. One of these steps is meta-regression which is performed to investigate the association of any confounder and the results of the MA. Furthermore, there are other types rather than the standard MA like NMA and MA. In NMA, we investigate the difference between several comparisons when there were not enough data to enable standard meta-analysis. It uses both direct and indirect comparisons to conclude what is the best between the competitors. On the other hand, mega MA or MA of patients tend to summarize the results of independent studies by using its individual subject data. As a more detailed analysis can be done, it is useful in conducting repeated measure analysis and time-to-event analysis. Moreover, it can perform analysis of variance and multiple regression analysis; however, it requires homogenous dataset and it is time-consuming in conduct [ 24 ].

Conclusions

Systematic review/meta-analysis steps include development of research question and its validation, forming criteria, search strategy, searching databases, importing all results to a library and exporting to an excel sheet, protocol writing and registration, title and abstract screening, full-text screening, manual searching, extracting data and assessing its quality, data checking, conducting statistical analysis, double data checking, manuscript writing, revising, and submitting to a journal.

Availability of data and materials

Not applicable.

Abbreviations

Network meta-analysis

Principal investigator

Population, Intervention, Comparison, Outcome

Preferred Reporting Items for Systematic Review and Meta-analysis statement

Quality assessment

Sample, Phenomenon of Interest, Design, Evaluation, Research type

Systematic review and meta-analyses

Bello A, Wiebe N, Garg A, Tonelli M. Evidence-based decision-making 2: systematic reviews and meta-analysis. Methods Mol Biol (Clifton, NJ). 2015;1281:397–416.

Article   Google Scholar  

Khan KS, Kunz R, Kleijnen J, Antes G. Five steps to conducting a systematic review. J R Soc Med. 2003;96(3):118–21.

Rys P, Wladysiuk M, Skrzekowska-Baran I, Malecki MT. Review articles, systematic reviews and meta-analyses: which can be trusted? Polskie Archiwum Medycyny Wewnetrznej. 2009;119(3):148–56.

PubMed   Google Scholar  

Higgins JPT, Green S. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. 2011.

Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ. 2009;339:b2535.

Methley AM, Campbell S, Chew-Graham C, McNally R, Cheraghi-Sohi S. PICO, PICOS and SPIDER: a comparison study of specificity and sensitivity in three search tools for qualitative systematic reviews. BMC Health Serv Res. 2014;14:579.

Gross L, Lhomme E, Pasin C, Richert L, Thiebaut R. Ebola vaccine development: systematic review of pre-clinical and clinical studies, and meta-analysis of determinants of antibody response variability after vaccination. Int J Infect Dis. 2018;74:83–96.

Article   CAS   Google Scholar  

Shea BJ, Reeves BC, Wells G, Thuku M, Hamel C, Moran J, ... Henry DA. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ. 2017;358:j4008.

Giang HTN, Banno K, Minh LHN, Trinh LT, Loc LT, Eltobgy A, et al. Dengue hemophagocytic syndrome: a systematic review and meta-analysis on epidemiology, clinical signs, outcomes, and risk factors. Rev Med Virol. 2018;28(6):e2005.

Morra ME, Altibi AMA, Iqtadar S, Minh LHN, Elawady SS, Hallab A, et al. Definitions for warning signs and signs of severe dengue according to the WHO 2009 classification: systematic review of literature. Rev Med Virol. 2018;28(4):e1979.

Morra ME, Van Thanh L, Kamel MG, Ghazy AA, Altibi AMA, Dat LM, et al. Clinical outcomes of current medical approaches for Middle East respiratory syndrome: a systematic review and meta-analysis. Rev Med Virol. 2018;28(3):e1977.

Vassar M, Atakpo P, Kash MJ. Manual search approaches used by systematic reviewers in dermatology. Journal of the Medical Library Association: JMLA. 2016;104(4):302.

Naunheim MR, Remenschneider AK, Scangas GA, Bunting GW, Deschler DG. The effect of initial tracheoesophageal voice prosthesis size on postoperative complications and voice outcomes. Ann Otol Rhinol Laryngol. 2016;125(6):478–84.

Rohatgi AJaiWa. Web Plot Digitizer. ht tp. 2014;2.

Hozo SP, Djulbegovic B, Hozo I. Estimating the mean and variance from the median, range, and the size of a sample. BMC Med Res Methodol. 2005;5(1):13.

Wan X, Wang W, Liu J, Tong T. Estimating the sample mean and standard deviation from the sample size, median, range and/or interquartile range. BMC Med Res Methodol. 2014;14(1):135.

Van Rijkom HM, Truin GJ, Van’t Hof MA. A meta-analysis of clinical studies on the caries-inhibiting effect of fluoride gel treatment. Carries Res. 1998;32(2):83–92.

Higgins JP, Altman DG, Gotzsche PC, Juni P, Moher D, Oxman AD, et al. The Cochrane Collaboration's tool for assessing risk of bias in randomised trials. BMJ. 2011;343:d5928.

Tawfik GM, Tieu TM, Ghozy S, Makram OM, Samuel P, Abdelaal A, et al. Speech efficacy, safety and factors affecting lifetime of voice prostheses in patients with laryngeal cancer: a systematic review and network meta-analysis of randomized controlled trials. J Clin Oncol. 2018;36(15_suppl):e18031-e.

Wannemuehler TJ, Lobo BC, Johnson JD, Deig CR, Ting JY, Gregory RL. Vibratory stimulus reduces in vitro biofilm formation on tracheoesophageal voice prostheses. Laryngoscope. 2016;126(12):2752–7.

Sterne JAC, Hernán MA, Reeves BC, Savović J, Berkman ND, Viswanathan M, et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ. 2016;355.

RevMan The Cochrane Collaboration %J Copenhagen TNCCTCC. Review Manager (RevMan). 5.0. 2008.

Schwarzer GJRn. meta: An R package for meta-analysis. 2007;7(3):40-45.

Google Scholar  

Simms LLH. Meta-analysis versus mega-analysis: is there a difference? Oral budesonide for the maintenance of remission in Crohn’s disease: Faculty of Graduate Studies, University of Western Ontario; 1998.

Download references

Acknowledgements

This study was conducted (in part) at the Joint Usage/Research Center on Tropical Disease, Institute of Tropical Medicine, Nagasaki University, Japan.

Author information

Authors and affiliations.

Faculty of Medicine, Ain Shams University, Cairo, Egypt

Gehad Mohamed Tawfik

Online research Club http://www.onlineresearchclub.org/

Gehad Mohamed Tawfik, Kadek Agus Surya Dila, Muawia Yousif Fadlelmola Mohamed, Dao Ngoc Hien Tam, Nguyen Dang Kien & Ali Mahmoud Ahmed

Pratama Giri Emas Hospital, Singaraja-Amlapura street, Giri Emas village, Sawan subdistrict, Singaraja City, Buleleng, Bali, 81171, Indonesia

Kadek Agus Surya Dila

Faculty of Medicine, University of Khartoum, Khartoum, Sudan

Muawia Yousif Fadlelmola Mohamed

Nanogen Pharmaceutical Biotechnology Joint Stock Company, Ho Chi Minh City, Vietnam

Dao Ngoc Hien Tam

Department of Obstetrics and Gynecology, Thai Binh University of Medicine and Pharmacy, Thai Binh, Vietnam

Nguyen Dang Kien

Faculty of Medicine, Al-Azhar University, Cairo, Egypt

Ali Mahmoud Ahmed

Evidence Based Medicine Research Group & Faculty of Applied Sciences, Ton Duc Thang University, Ho Chi Minh City, 70000, Vietnam

Nguyen Tien Huy

Faculty of Applied Sciences, Ton Duc Thang University, Ho Chi Minh City, 70000, Vietnam

Department of Clinical Product Development, Institute of Tropical Medicine (NEKKEN), Leading Graduate School Program, and Graduate School of Biomedical Sciences, Nagasaki University, 1-12-4 Sakamoto, Nagasaki, 852-8523, Japan

You can also search for this author in PubMed   Google Scholar

Contributions

NTH and GMT were responsible for the idea and its design. The figure was done by GMT. All authors contributed to the manuscript writing and approval of the final version.

Corresponding author

Correspondence to Nguyen Tien Huy .

Ethics declarations

Ethics approval and consent to participate, consent for publication, competing interests.

The authors declare that they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Additional files

Additional file 1:.

Figure S1. Risk of bias assessment graph of included randomized controlled trials. (TIF 20 kb)

Additional file 2:

Figure S2. Risk of bias assessment summary. (TIF 69 kb)

Additional file 3:

Figure S3. Arthralgia results of random effect meta-analysis using R meta package. (TIF 20 kb)

Additional file 4:

Figure S4. Arthralgia linear regression test of funnel plot asymmetry using R meta package. (TIF 13 kb)

Additional file 5:

Table S1. PRISMA 2009 Checklist. Table S2. Manipulation guides for online database searches. Table S3. Detailed search strategy for twelve database searches. Table S4. Baseline characteristics of the patients in the included studies. File S1. PROSPERO protocol template file. File S2. Extraction equations that can be used prior to analysis to get missed variables. File S3. R codes and its guidance for meta-analysis done for comparison between EBOLA vaccine A and placebo. (DOCX 49 kb)

Additional file 6:

Data S1. Extraction and quality assessment data sheets for EBOLA case example. (XLSX 1368 kb)

Additional file 7:

Data S2. Imaginary data for EBOLA case example. (XLSX 10 kb)

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Cite this article.

Tawfik, G.M., Dila, K.A.S., Mohamed, M.Y.F. et al. A step by step guide for conducting a systematic review and meta-analysis with simulation data. Trop Med Health 47 , 46 (2019). https://doi.org/10.1186/s41182-019-0165-6

Download citation

Received : 30 January 2019

Accepted : 24 May 2019

Published : 01 August 2019

DOI : https://doi.org/10.1186/s41182-019-0165-6

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Tropical Medicine and Health

ISSN: 1349-4147

  • Submission enquiries: Access here and click Contact Us
  • General enquiries: [email protected]

systematic review step by step guide

Banner

Conduct a systematic review: step-by-step guide: Home

  • What are systematic reviews?
  • How to conduct a systematic review?
  •  Literature search platforms
  • Books about systematic reviews

This page provides a step-by-step guide focusing on how to conduct a systematic review (Synthesis without a Meta-Analysis (SWiM) or Systematic Review with a Meta-Analysis). If you are looking for information focused on other types of reviews, such as rapid reviews, scoping reviews, umbrella reviews and living systematic reviews, please also visit the Main Types of Review  section. 

  • Step 1: Plan for resources & define your research
  • Step 2: Formulate an answerable research question
  • Step 3: Build search strategies
  • Step 4: Write your research protocol & register your review
  • Step 5: Conduct searches & manage references
  • Step 6: Critical appraisal & data extraction
  • Step 7: Report your systematic review

First, you will need to clearly define your research and plan ahead the resources you need :

A systematic review requires approximately 18 months from preparation.

A Research Team

A systematic review attempts to collect all existing evidence, by reviewing, analysing, validating and synthesising the evidence in a systematic way. It cannot be done by 1 or 2 people. Depending on your research type and the type of review you are going to do, your team would have (but is not limited to) project leaders, coordinators, subject experts, researchers, reviewers / validators, statisticians, data analysts and librarians.

 Databases 

It is recommended that your team search at least 2-3 databases.  Plan ahead for access to the databases most relevant to your research topic.

Citation Management Software

Besides word processing software, as the team will likely retrieve a large number of articles during the systematic review process, citation management software is needed for storing and sharing the citations and full-text articles effectively among teammates. This software can also help with de-duplication, in-text citations and generation of the final reference list. EndNote and RefWorks may be options. Check out our Referencing and Citation Management Guide for more details about training options.

 Protocol Templates, Checklists and Reporting Guidelines

Identify the appropriate checklists and guidelines on how to report systematic reviews for publication for the target organisation you plan to submit your review to. (See Step 5 to 7 for more details and toolboxes access.)

Library Resources and Services

identifying what material resources, software, academic databases, reference services (such as literature searching consultation and citation management tools training), and research guides (like this one) you can access via libraries will save your time.

What should be clearly defined at this stage:

  • your research objectives - what is the purpose of the review?
  • your research topic and scope - what is included (inclusion criteria) and what is excluded (exclusion criteria)? (not just the content but also the years of publication your research would cover)
  • The type of review your team are going to conduct  - quantitative / qualitative / mixed method? - this will depend on your topic and scope.

You may already have a clear scope of what your research is about and what is expected to be included and excluded in your project. However, a more promising way of starting a research project would be to not only have a topic, scope and a rough question in mind. Rather, your research team really needs an answerable research question to start with. Check out the tool box below for examples of  frameworks used to  formulate answerable research questions . 

Check out how to formulate a PICOT question in Ovid database

Check out this article to understand how SPIDER helps for synthesising qualitative evidence: Cooke, A., Smith, D.M. & Booth, A. (2012). Beyond PICO: the SPIDER tool for qualitative evidence synthesis. Qualitative Health Research . 22(10):1435-43. doi: 10.1177/1049732312452938.

Check out books about qualitative research methodology

Check out the following article to understand how to use ECLIPSE in health service / health policy research in detail:

Wildridge, V., & Bell, L. (2002). How CLIP became ECLIPSE: a mnemonic to assist in searching for health policy/management information.  Health Information & Libraries Journal ,  19 (2), 113-115.

Some frameworks are more easily applied to certain types of research than others. However, It is not the case that one framework has to be rigidly applied to a particular type of research without any flexibility. The examples given are just for guidance.  Which framework can be helpful for building your research question is highly dependant on your research objective(s) and the focus of your review.

Next, you will have to select the appropriate resources to search for the available evidence. This tab shows you some tips and tricks on how to  build your search strategies and how  conduct database searches.

Your answerable clinical question and PICO contains the key concepts of your research topic. Use those key concepts to start identifying related terms, synonyms (including variant spellings of medical terms) and related highly relevant concepts you would like to include in your search.

The following example shows you how to turn a PlCO and clinical question into a key concepts and related terms table:

Mediterranean diet

However, the key-concept and related-term table is not finalised at this stage. When you make use of the next tip to look up Subject Headings, you may find more related terms and concepts for your evidence-based research. Let's check out the next tip:  Use Subject Headings  to learn more.

Each bibliographic database has their own specific Subject Headings controlled vocabulary to index the key concepts in articles. How Subject Headings are searched and applied will vary depending on the database being searched. The following table shows you the main bibliographic databases with examples of different subject headings used for describing the same search term, for example: heart attack.

The videos below provides guidance on how to apply Subject Headings in different platforms:

To access the databases listed above, please visit our Databases page.

References:

Cochrane Training. (2019, May 14). Searching the Cochrane Library [Video]. Youtube. https://www.youtube.com/embed/HLD7w63rqB0?start=74

EBSCO. (2021, November 4). Using the CINAHL/MeSH Subject Headings Feature in EBSCOhost [Video]. EBSCO Connect. https://connect.ebsco.com/s/article/Using-the-CINAHL-MeSH-Headings-Feature-in-EBSCOhost-Tutorial?language=en_US

OvidWoltersKluwer. (2021, September 29). Mapping in Advanced Mode [Video]. YouTube. https://www.youtube.com/watch?v=3736KB9Udn

U.S. National Library of Medicine, (2020, November 30). PubMed Subject Search: How it works  [Video]. PubMed.gov, U.S. National Library of Medicine. https://www.nlm.nih.gov/oet/ed/pubmed/quicktours/topic_how_it_works/index.html?_gl=1*10vieks*_ga*OTc5Nzk1NjMwLjE2NDgxNzg5NzU.*_ga_7147EPK006*MTY0ODE4MjI2MS4yLjEuMTY0ODE4Mzk2OC4w*_ga_P1FPTH9PL4*MTY0ODE4MjI2MS4yLjEuMTY0ODE4Mzk2OC4w=

The "OR" and "AND" operator

. When applying the "OR" operator, it will broaden your search. When building your search strategy, it is recommended that to start with applying "OR" operator to connect between the key concepts with their related terms or synonyms first.

Given the convenient design of most of the main research database platforms, in reality, you are usually able to do all of the following within a platform:

(1) search key terms individually;, (2) explore related terms by mapping with subject headings  along the way; and, (3) apply the  boolean operators "or" and "and" by reviewing the search history, the following examples from 2 major database platforms (ovid platform and ebsco platform) demonstrate how you can search , map to subject headings and combine terms from your search history :, the following 2 examples of major database platform (i.e.: ovid platform and ebsco platform) demonstrating how you can refine your search strategy by applying search limits:,   when applying the above steps, it is important to note that building a good search strategy would not be a straightforward process. along the way, you will need to try out  different combinations and  review  the number of search results and quickly scan through article titles. then, if necessary,  adjusting  your keywords and combinations to make the search results more relevant to your research question., as explained above, subject headings and limits setting are  vary from database to database and search platform to search platform. therefore, the first developed search strategy upon one database via a specific search platform would need to be modified  to make it applicable for the others., need more help .

   Book a Training Session

  Request a Literature Search

After you are able to clearly define your research topic, objectives, scope, type of review and research question, you will need to write a protocol of your planned systematic review, which should provide the research methods that are   reproducible .

JBI. (2020,September 25). Pre-planning and protocol development for systematic reviews [video]. Youtube. https://youtu.be/K8Jasx0FLys

You may also read this  BioMedCentral blog  on why protocols are important.

The structural  requirements of a protocol can be vary depending on the publishers. Generally, the protocol should consist of the following content / sections: 

(also see the Toolbox at the bottom of this tab to see the structure requirements of protocol based on the publishing partners )

  • Title and Authors
  • Introduction
  • Review objectives
  • Research methods
  • Acknowledgements
  • Conflict of Interest
  • Funding / Ethical approval (if any)

Your research methods should at least include:

(1)  keywords / key terms / key concepts and selection criteria (inclusion and exclusion criteria) of literature as well as  databases and /or platforms  your team would use for searching. At this stage, you may not have a detailed  search strategy for each database you planned to use but you would still need to include a search strategy section. (The first example below shows you an example with a search strategy section and the second example paper below shows you a  search strategy section with a  detailed search strategy in Appendix 1.)

(2) how you will  validate   the findings of the included studies - What are the research design measurements that will minimise the bias of your research? (see the examples of systematic reviews with different risks of bias provided by KSE Evidence ). You will need validators (ideally, at least 2-3) to review the findings of the included studies so the systematic review can be produced with as  little bias as it can be. The validators are expected to have knowledge of the fields / be  experienced workers or, even better, experts of the relevant fields. The validators should be able to review the findings of the included studies independently .

(3) how is your team going to synthesis evidence / findings in a systematic way? This may be as a thematic approach and/ or a meta-ethnographic analysis for qualitative reviews, and meta-analysis for quantitative reviews or a mixed-method reviews etc.(see Step 5 Data Extraction section  for more details )

Examples of systematic review protocol:

  • Study protocol of a systematic review and qualitative evidence synthesis using two different approaches: Healthcare related needs and desires of older people with post-stroke aphasia

Pohontsch, N. J., Meyer, T., Eisenmann, Y., Metzendorf, M. I., Leve, V., & Lentsch, V. (2021). Study protocol of a systematic review and qualitative evidence synthesis using two different approaches: Healthcare related needs and desires of older people with post-stroke aphasia.  BMJ Open ,  11 (4), e039348.

  • Identifying COVID-19 and H1N1 vaccination hesitancy or refusal among health care providers across North America, the United Kingdom, Europe, and Australia: a scoping review protocol

Gallant, A. J., Steenbeek, A., & Curran, J. A. (2022). Identifying COVID-19 and H1N1 vaccination hesitancy or refusal among health care providers across North America, the United Kingdom, Europe, and Australia: a scoping review protocol.  JBI Evidence Synthesis ,  20 (1), 173-180.

  • The suggested structure of a Cochrane protocol for submission ; an example from the Cochrane Handbook for Systematic Reviews of Interventions (updated February 2022)

Methodological Expectations of Cochrane Intervention Reviews - The MECIR manual

PRISMA Protocol Guidance  including  PRISMA for systematic review protocols (PRISMA-P)  

PRISMA Flow Diagram generator   - click on " Create flow diagram" on the horizontal manual will bring you to the generator. If you are using R for meta-analysis. There is also a R package link provided by Evidence Synthesis Hackathon .

Template for Scoping Review Protocols  / JBI

Guide on Registering a review on PROSPERO (does not accept scoping reviews)

Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors).  Cochrane Handbook for Systematic Reviews of Interventions  version 6.3 (updated February 2022). Cochrane, 2022. Available from www.training.cochrane.org/handbook .

At this stage, your team can apply the set of search terms /  search strategies  that have been tested, modified, and then finalised , to conduct the search, review the search results, and well- document the review and selection process. Here is a step-by-step guide:

1. Conduct the search using your final  set of search terms / search strategies .  It is recommended that your team should do the search in at least two to three  publication databases. As explained under the Step 3 tab, each set of search terms / search strategies may not be identical from database to database. However, they should be similar in covering all key concepts and related terms as possible for each database.

2. To review the search results from each database, it is recommended to start with  scanning all articles where the  title seems relevant . Then import those citations and abstracts  into a citation management software (such as EndNote, Mendeley, Zotero or similar) for further review. ( Looking for EndNote training? Visit our Referencing and Citation Management Tools Guide for more details.)

3. As there is  overlap in the coverage between databases, and you may possibly retrieve a  large number of records from each database,  citation management software  can also help to store a large number of citations from each database and to de-duplicate records.  

4. To continue the review and selection process : Read the abstract for all articles and keep the articles that seem relevant. To further shortlist highly relevant research, retrieve and read the full-text  of articles of those with abstracts that seemed relevant. It needs to be at least   2 members of the SR team to review papers for inclusion and exclusion , to reduce bias in the process (learn more about critical appraisal from the next tab - Step 6) . The members will need to  review papers independently . Papers where there is disagreement on whether to include or exclude are then reconsidered.

5. Through out the process of review and selection, your team will need to well-document : (1) the initial   number   of articles your retrieved from each database  after de-duplication ; (2) the number of  articles, from each database, that ended up included and  excluded and (3) why some titles are  excluded from your paper (generally, the reasons of including or excluding an article should be matched with your  proposal ). All those numbers will need to be summarised in the PRISMA flow chart and included in your paper. The process will need to be reported under the Method section of your paper.

6. Review the reference lists within the remaining relevant articles in case there are a few more relevant articles listed that were not found through your database searches. If any additional articles are found, Include them in the final list of relevant articles and document   it.

In addition to citation management software there are free online platforms for team collaboration. These platforms usually allow reviewers to upload articles and provide functions such as labelling articles (with "included", "excluded" or "maybe") and enable reviewers to provide feedback or comments to help facilitate the selection process and keep the all reviewers of the research team on the same page. Again, the reviewers will need to review papers  independently . Papers where there is disagreement on whether to include or exclude are then reconsidered

Free Open Sources for systematic review team collaboration:

  • Colandr ​  (sign up required)
  • HAWC  (Health Assessment Workplace Collaborative)
  • LitStream  (from ICF International)
  • PICO Portal (free version available - registration requested)
  • Rayyan  (registration requested)
  • Systematic Review Data Repository (SRDR+)  (registration required) (from AHRQ, US Government)

  It is important to note that while there are a number of free open source tools available, you may be  restricted  to the tools / evidence synthesis software for completing this data extraction process by the requirements of publishing partners . Campbell Collaboration  is an example . Therefore, you should always check with the publisher / organisation you are going to submit your review with before you plan the screening, data extraction and analysis process of your review.

To appraise research articles, there are 2 overarching questions to keep in mind:

Critical appraisal.

Questions for assessing how serious the risk of bias is , and how precise the results are, will vary according to the  different research design aspects and statistical measurements  used in a research study. Unfold the following tabs to access to the list of questions and the Cochrane videos relevant to the different types of research studies.

The guidelines for reporting a systematic review that you should follow will vary depending on the publisher and the type of review. It is recommended that your team check the relevant publisher's website once your team has decided on where you intend to publish.

Example :  " Instructions for Authors" from JAMA  (scroll down to the row: "Meta-analysis" or the table "Clinical Review" to see what type of reviews and what reporting guidelines your team should follow).

The following toolbox includes a reporting checklist, a template for flow diagrams and some common reporting guidelines.

  • PRISMA 2020 Checklist for guiding the structure of your paper.
  • PRISMA Flow Diagram   which provides templates, depending on the type of review and databases used, for building the flowchart / diagram you may want to include in your paper.
  • Article about  PRISMA 2020 and PRISMA-S: common questions on tracking records and the flow diagram .

Rethlefsen, M. L., & Page, M. J. (2022). PRISMA 2020 and PRISMA-S: common questions on tracking records and the flow diagram.  Journal of the Medical Library Association: JMLA ,  110 (2), 253.

        Reporting guidelines apply for systematic review with a meta-analysis:

  • PRISMA Reporting Guidelines
  • PRISMA Flow Diagram generator  - click on " Create flow diagram" on the horizontal manual will bring you to the generator. If you are using R for meta-analysis, there is also a R package link provided by Evidence Synthesis Hackathon .
  • MOOSE (Meta-analysis of Observational Studies in Epidemiology) Reporting Guidelines

        Reporting guidelines apply for synthesis without meta-analysis (SWiM):

  • Synthesis without meta-analysis (SWiM): in systematic reviews: reporting guideline
  • EQUATOR (Enhancing the QUAlity and Transparency Of health Research) Reporting Guidelines
  • Last Updated: Jan 31, 2024 12:08 PM
  • URL: https://easternhealth.libguides.com/SR_step-by-step

© Eastern Health Library Service 2021-2023

University of Pittsburgh Library System

University of Pittsburgh Library System

  • Collections

Course & Subject Guides

A guide to systematic reviews and evidence synthesis service @ uls.

  • What is Evidence Synthesis?
  • How can the ULS assist you with your Evidence Synthesis?
  • What is a Systematic Review and where can you find them?

STEP 1. Identify your research question

Step 2. define inclusion and exclusion criteria, step 3. write a search strategy, step 4. register protocol, step 5. manage search results, step 6. select studies based on inclusion and exclusion criteria, step 7. extract data from included studies, step 7. assess quality of evidence in included studies, step 8. present results.

  • Are there any guidelines or standards for publishing reviews?
  • How can I get assistance with a review?

A well defined research question should address a gap in the current literature and is the essential starting point of your synthesis.

You can use the following frameworks to help construct your research question.

PICO for Quantitative Studies P       Population/Problem I         Intervention/Exposure C       Comparison O       Outcome Example: Is gabapentin (intervention), compared to placebo (comparison), effective in decreasing pain symptoms (outcome) in middle aged male amputees suffering phantom limb pain (population)?

PICo for Qualitative Studies P       Population/Problem I         Phenomenon of Interest  Co    Context Example: What are the experiences (phenomenon of interest) of caregivers providing home based care to patients with Alzheimer's disease (population) in Australia (context)?

SPICE S     Setting P   Perspective (for whom) I     Intervention/Exposure C   Comparison E   Evaluation Example: What are the benefits (evaluation) of a doula (intervention) for low income mothers (perspective) in the developed world (setting) compared to no support (comparison)?

SPIDER S     Sample PI   Phenomenon of Interest D     Design E     Evaluation R     Study Type Example: What are the experiences (evaluation) of wome n (sample) undergoing IVF treatment (phenomenon of interest) as assessed?

Design:  questionnaire or survey or interview

Study Type: qualitative or mixed method

The above was adapted from Cornell University A Guide to Evidence Synthesis: 1. Develop a Research Question https://guides.library.cornell.edu/evidence-synthesis/research-question

After finalizing your research question but before you start your search, you need to define your inclusion and exclusion criteria.  You must decide what contents an article MUST have before being included in the review.  You also must determine which attributes would exclude an article from the review.  

Common Inclusion/ Exclusion pictorial representation

Image from the University of Melbourne Libguide Systematic Reviews  https://unimelb.libguides.com/c.php?g=492361&p=3368110

Your search strategy must be exhaustive, encompasses multiple databases, include grey literature and be reproducible.  PRISMA guidelines state that the full search strategy for at least one major database should be reported in an appendix and published along with the review (  http://www.prisma-statement.org /).

The University Library System provides access to a wide range of databases which can be accessed by subject on the A-Z database list . Most databases have controlled vocabulary (a certain way words and phrases are indexed) which is unique to the database. This may require using different terms for different databases." Given the complexity of the many indexing languages and rules governing the various databases, we recommend that early in the process you make use of an experienced research librarian who can examine your search strategy and help you choose citation databases relevant to your review question ." ( Aromataris, Edoardo PhD; Riitano, Dagmara BHSC, BA Systematic Reviews: Constructing a Search Strategy and Searching for Evidence, AJN, American Journal of Nursing: May 2014 - Volume 114 - Issue 5 - p 49-56 doi: 10.1097/01.NAJ.0000446779.99522.f6 )

Grey literature is produced outside of traditional publishing and distribution norms.   This can included, among other things, white papers, government publications, working papers, preprints, unpublished trial data, and conference proceedings and abstracts. Grey literature can be found in some citation databases, as well as databases dedicated to grey literature.

Some databases dedicated to grey literature include:

  • Grey Literature Report A report from the NY Academy of Medicine of gray literature published between 1999 - 2016
  • Open Grey "Open access to 700.000 bibliographical references of grey literature (paper) produced in Europe and allows you to export records and locate the documents"
  • GreySource "A selection of web-based resources on grey literature"
  • Grey Matters Canadian Agency for Drugs and Technologies in Health provides a practical tool for searching health-related grey literature

Some sources for preprints include:

  • ASAPbio Accelerating Science Publication in Biology preprint server directory
  • OSF Preprints A searchable database of over 33 preprint repositories
  • OAD Disciplinary Repositories Open Access Directory list of preprint depositories by subject
  • OPENDOAR A "Global Directory of Open Access Repositories. You can search and browse through thousands of registered repositories based on a range of features"

An example of a complete and reproducible search strategy can be found in Appendix 1 of Petriwskyj, P. (2013). Family involvement in decision making for people with dementia in residential aged care: a systematic review of quantitative and qualitative evidence .  JBI Database of Systematic Reviews and Implementation Reports ,  11 (7), 131–282. https://doi.org/10.11124/jbisrir-2013-977

A protocol   lists the objectives, methods, and outcomes of primary interest of the systematic review.  Protocols promote transparency of methods and allows your peers to review how you will extract information to summarize the data. Registration of your protocol establishes your intent to conduct this review which may reduce the risk of others conducting similar reviews.

Here is an example of a published protocol 

Mengesha, M.M., Ajema, D., Teshome, A.  et al.  The association between diagnosis disclosure and adherence to antiretroviral therapy among adolescents living with HIV in sub-Saharan Africa: a protocol for systematic review and meta-analysis.  Syst Rev   9,  160 (2020). https://doi.org/10.1186/s13643-020-01420-8

Protocol Reporting Guidelines and Checklists

  • Methodological Expectations of Cochrane Intervention Reviews (MECIR) Standards for the conduct and reporting of new Cochrane Intervention Reviews, reporting of protocols and the planning, conduct and reporting of updates
  • PRISMA for systematic review protocols PRISMA 2020 is an expanded 27-item checklist intended to facilitate the preparation and reporting of a robust protocol for the systematic review
  • Cochrane Handbook 1.5 Protocol Development Lasserson TJ, Thomas J, Higgins JPT. Chapter 1: Starting a review. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.3 (updated February 2022). Cochrane, 2022. Available from www.training.cochrane.org/handbook.

Protocol Templates

  • PROSPERO Registration Form
  • Cochrane Qualitative Evidence Synthesis Template
  • Collaboration for Environmental Evidence (CEE) Systematic Map Protocol Template
  • Campbell Collaboration Template
  • Collaboration for Environmental Evidence (CEE) Systematic Review Protocol Template
  • Evidence Synthesis Protocol Template by Ghezzi-Kopel, K., & Porciello, J. found on Open Science Framework
  • Systematic Review Protocol Template by Sarah Vistintini (Maritime SPOR SUPPORT Unit (MSSU))
  • Open Science Framework Systematic Review Template Select Add New, and under step 2, select Generalized Systematic Review Registration
  • Template from Warwick University You can use this template available from Warwick University to create your protocol

Protocol Registries

  • Cochran Cochran protocols "contain information that defines the health problem and the intervention under investigation, how benefits and harms will be measured, and the type of appropriate study design"
  • Campbell Collaboration "The Campbell Collaboration promotes positive social and economic change through the production and use of systematic reviews and other evidence synthesis for evidence-based policy and practice"
  • PROSPERO "PROSPERO is an international database of prospectively registered systematic reviews in health and social care, welfare, public health, education, crime, justice, and international development, where there is a health related outcome"
  • Open Science Framework Register your protocol on this free open platform
  • Collaboration for Environmental Evidence (CEE) CEE seeks to promote and deliver evidence syntheses on issues of greatest concern to environmental policy and practice as a public service
  • JBI "JBI is a global organization promoting and supporting evidence-based decisions that improve health and health service delivery."
  • BioMed Central BioMed Central publishes a limited amount of protocols in Systematic Reviews

It is important to keep track of all search results from each database.  The use of a template is recommended to capture the following information:

  • Database Name
  • Date Searched
  • Keywords and Combination of Terms
  • Search History
  • Limiters (Language, Time Period, Publication Type)
  • Number of Results
  • Number of Duplicates

After running the search through a database, export the results to a citation manager. The method of export will depend on the database and the citation management tool used.  Once all results from all sources are uploaded into a citation manager, you will need to de-duplicate the result list.

Visit our Library Guide, " Introduction to Citation Management " for an introduction to citation management tools and links to upcoming citation management workshops workshops.

Start with the screening of title and abstract to determine if a reference is relevant to your review.  Obtain the full text of a reference if further screening is necessary.  At least two reviewers will be needed to make a final determination on inclusion.

                         

MECIR Box 4.6.c  Relevant expectations for conduct of intervention reviews , Cochran Handbook 

Mandatory IInclusion Data

There are systematic review tools available to help with the screening process:

  • Rayyan Web-based application to screen references and maintain systematic reviews. Rayyan also has a mobile app.
  • Abstrackr Developed at Brown University, "Abstrackr is a free online tool to help you upload and organize the results of a literature search for a systematic review.
  • CADIMA Web-based tool to manage your systematic review.
  • PICO Portal Uses machine learning and artificial intelligence to facilitate deduplication, identify non-RCT articles and highlight keywords.

Tools which have a subscription cost

  • Covidence This is the primary screening tool for Cochran Reviews. Covidence offers a FREE TRIAL for a single review containing 500 references or less. Single user and group pricing are available.
  • JBISUMARI The System for the Unified Management of the Assessment and Review of Information is used in Joanna Briggs Institute Reviews and is available as an individual subscription.
  • DistillerSR Commercially available systematic review software from Evidence Partners. DistillerSR offers various subscriptions, including a four month FREE subscription for students.

More tools for conducting systematic reviews can be found at the SR Toolbox

  • SR Toolbox The Systematic Review Toolbox is a community-driven, searchable, web-based catalogue of tools that support various tasks within the systematic review and wider evidence synthesis process.

The reviewers must read the full text of the articles which were selected for inclusion in the review.  The pertinent data must be extracted from each article.  A standardized data extraction form should be used. An example of a data extraction form can be found below.

  • Cochran Data and Extraction Assessment Form "This form can be used as a guide for developing your own data extraction form. Sections can be expanded and added, and irrelevant sections can be removed."
  • Brown, U. (2003). A Framework for Developing a Coding Scheme for Meta-Analysis. Western Journal of Nursing Research, 25(2), 205–222.

If your review will contain a meta-analysis you may want to code the data in order to automate the statistical analysis process.  Some systematic review software packages listed in step 6. can help you create coded data instruction forms. Instructions on designing a coded data extraction form can be found in the following article:

It is necessary to evaluate each study included in your review for bias.   Cochran defines bias as "a systematic error, or deviation from the truth, in results or inferences. Biases can operate in either direction: different biases can lead to underestimation or overestimation of the true intervention effect". ( Cochran Handbook 8.2.1 )

Bias is evaluated on a level of risk.  The risk of bias (RoB) can be demonstrated using a variety of tools:

  • RoB 2 "Version 2 of the Cochrane risk-of-bias tool for randomized trials (RoB 2) is the recommended tool to assess the risk of bias in randomized trials included in Cochrane Reviews. RoB 2 is structured into a fixed set of domains of bias, focussing on different aspects of trial design, conduct, and reporting. Within each domain, a series of questions ('signalling questions') aim to elicit information about features of the trial that are relevant to risk of bias. A proposed judgement about the risk of bias arising from each domain is generated by an algorithm, based on answers to the signalling questions. Judgement can be 'Low' or 'High' risk of bias, or can express 'Some concerns'."
  • ROBINS-I ROBINS-I tool (“Risk Of Bias In Non-randomized Studies - of Interventions”) is concerned with evaluating the risk of bias (RoB) in the results of NRSIs that compare the health effects of two or more interventions. The types of NRSIs that can be evaluated using this tool are quantitative studies estimating the effectiveness (harm or benefit) of an intervention, which did not use randomization to allocate units (individuals or clusters of individuals) to comparison groups."

More information and an analysis of RoB tools can be found in the article:

Ma, L. L., Wang, Y. Y., Yang, Z. H., Huang, D., Weng, H., & Zeng, X. T. (2020). Methodological quality (risk of bias) assessment tools for primary and secondary medical studies: what are they and which is better?.  Military Medical Research ,  7 (1), 7. https://doi.org/10.1186/s40779-020-00238-8

Some study quality assessment tools include

  • Critical Appraisal Skills Program Checklists Checklists from the Critical Appraisal Skills Program
  • Study Quality Assessment Tools Tools available from NIH National Heart, Lung and Blood Institute
  • Quality Assessment Tools Assessment tools from the NIH Office of Management
  • Critical Appraisal Tools Worksheets to help you appraise the reliability, importance and applicability of clinical evidence from the Centre for Evidence Based Medicine

PRISMA  provides a list of items to consider when reporting results. 

  • Study selection:   Give numbers of studies screened, assessed for eligibility, & included in the review, with reasons for exclusions at each stage, ideally with a flow diagram.
  • Study characteristics:   For each study, present characteristics for which data were extracted (e.g., study size, PICOs, follow-up period) & provide the citations.
  • Risk of bias within studies:   Present data on risk of bias of each study &, if available, any outcome level assessment.
  • Results of individual studies:   For all outcomes considered (benefits or harms), present, for each study: (a) simple summary data for each intervention group  (b) effect estimates & confidence intervals, ideally with a forest plot. 
  • Synthesis of results:   Present results of each meta-analysis done, including confidence intervals & measures of consistency.
  • Risk of bias across studies:   Present results of any assessment of risk of bias across studies.
  • Additional analysis:   Give results of additional analyses, if done (e.g., sensitivity or subgroup analyses, meta-regression).

Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement

PRISMA Diagram Generators

  • Flow Diagram Generator This is an updated version of the original PRISMA flow generator. Includes a downloadable PDF version.
  • Flow Diagram PRISMA Contains both PDF & Word versions. From PRISMA.

Other Reporting Templates

  • Equator Netowrk "Enhancing the QUAlity and Transparency Of health Research is an international initiative that seeks to improve the reliability and value of published health research literature by promoting transparent and accurate reporting and wider use of robust reporting guidelines"

Information in this section reproduced under a  Creative Commons Attribution 4.0 license  from the University of Michigan Libguide, " Systematic Reviews "

  • << Previous: What is a Systematic Review and where can you find them?
  • Next: Are there any guidelines or standards for publishing reviews? >>
  • Last Updated: Feb 15, 2024 3:52 PM
  • URL: https://pitt.libguides.com/SystematicReviews

Jump to navigation

Home

Cochrane Training

Chapter 1: starting a review.

Toby J Lasserson, James Thomas, Julian PT Higgins

Key Points:

  • Systematic reviews address a need for health decision makers to be able to access high quality, relevant, accessible and up-to-date information.
  • Systematic reviews aim to minimize bias through the use of pre-specified research questions and methods that are documented in protocols, and by basing their findings on reliable research.
  • Systematic reviews should be conducted by a team that includes domain expertise and methodological expertise, who are free of potential conflicts of interest.
  • People who might make – or be affected by – decisions around the use of interventions should be involved in important decisions about the review.
  • Good data management, project management and quality assurance mechanisms are essential for the completion of a successful systematic review.

Cite this chapter as: Lasserson TJ, Thomas J, Higgins JPT. Chapter 1: Starting a review. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.4 (updated August 2023). Cochrane, 2023. Available from www.training.cochrane.org/handbook .

1.1 Why do a systematic review?

Systematic reviews were developed out of a need to ensure that decisions affecting people’s lives can be informed by an up-to-date and complete understanding of the relevant research evidence. With the volume of research literature growing at an ever-increasing rate, it is impossible for individual decision makers to assess this vast quantity of primary research to enable them to make the most appropriate healthcare decisions that do more good than harm. By systematically assessing this primary research, systematic reviews aim to provide an up-to-date summary of the state of research knowledge on an intervention, diagnostic test, prognostic factor or other health or healthcare topic. Systematic reviews address the main problem with ad hoc searching and selection of research, namely that of bias. Just as primary research studies use methods to avoid bias, so should summaries and syntheses of that research.

A systematic review attempts to collate all the empirical evidence that fits pre-specified eligibility criteria in order to answer a specific research question. It uses explicit, systematic methods that are selected with a view to minimizing bias, thus providing more reliable findings from which conclusions can be drawn and decisions made (Antman et al 1992, Oxman and Guyatt 1993). Systematic review methodology, pioneered and developed by Cochrane, sets out a highly structured, transparent and reproducible methodology (Chandler and Hopewell 2013). This involves: the a priori specification of a research question; clarity on the scope of the review and which studies are eligible for inclusion; making every effort to find all relevant research and to ensure that issues of bias in included studies are accounted for; and analysing the included studies in order to draw conclusions based on all the identified research in an impartial and objective way.

This Handbook is about systematic reviews on the effects of interventions, and specifically about methods used by Cochrane to undertake them. Cochrane Reviews use primary research to generate new knowledge about the effects of an intervention (or interventions) used in clinical, public health or policy settings. They aim to provide users with a balanced summary of the potential benefits and harms of interventions and give an indication of how certain they can be of the findings. They can also compare the effectiveness of different interventions with one another and so help users to choose the most appropriate intervention in particular situations. The primary purpose of Cochrane Reviews is therefore to inform people making decisions about health or health care.

Systematic reviews are important for other reasons. New research should be designed or commissioned only if it does not unnecessarily duplicate existing research (Chalmers et al 2014). Therefore, a systematic review should typically be undertaken before embarking on new primary research. Such a review will identify current and ongoing studies, as well as indicate where specific gaps in knowledge exist, or evidence is lacking; for example, where existing studies have not used outcomes that are important to users of research (Macleod et al 2014). A systematic review may also reveal limitations in the conduct of previous studies that might be addressed in the new study or studies.

Systematic reviews are important, often rewarding and, at times, exciting research projects. They offer the opportunity for authors to make authoritative statements about the extent of human knowledge in important areas and to identify priorities for further research. They sometimes cover issues high on the political agenda and receive attention from the media. Conducting research with these impacts is not without its challenges, however, and completing a high-quality systematic review is often demanding and time-consuming. In this chapter we introduce some of the key considerations for potential review authors who are about to start a systematic review.

1.2 What is the review question?

Getting the research question right is critical for the success of a systematic review. Review authors should ensure that the review addresses an important question to those who are expected to use and act upon its conclusions.

We discuss the formulation of questions in detail in Chapter 2 . For a question about the effects of an intervention, the PICO approach is usually used, which is an acronym for Population, Intervention, Comparison(s) and Outcome. Reviews may have additional questions, for example about how interventions were implemented, economic issues, equity issues or patient experience.

To ensure that the review addresses a relevant question in a way that benefits users, it is important to ensure wide input. In most cases, question formulation should therefore be informed by people with various relevant – but potentially different – perspectives (see Chapter 2, Section 2.4 ).

1.3 Who should do a systematic review?

Systematic reviews should be undertaken by a team. Indeed, Cochrane will not publish a review that is proposed to be undertaken by a single person. Working as a team not only spreads the effort, but ensures that tasks such as the selection of studies for eligibility, data extraction and rating the certainty of the evidence will be performed by at least two people independently, minimizing the likelihood of errors. First-time review authors are encouraged to work with others who are experienced in the process of systematic reviews and to attend relevant training.

Review teams must include expertise in the topic area under review. Topic expertise should not be overly narrow, to ensure that all relevant perspectives are considered. Perspectives from different disciplines can help to avoid assumptions or terminology stemming from an over-reliance on a single discipline. Review teams should also include expertise in systematic review methodology, including statistical expertise.

Arguments have been made that methodological expertise is sufficient to perform a review, and that content expertise should be avoided because of the risk of preconceptions about the effects of interventions (Gøtzsche and Ioannidis 2012). However, it is important that both topic and methodological expertise is present to ensure a good mix of skills, knowledge and objectivity, because topic expertise provides important insight into the implementation of the intervention(s), the nature of the condition being treated or prevented, the relationships between outcomes measured, and other factors that may have an impact on decision making.

A Cochrane Review should represent an independent assessment of the evidence and avoiding financial and non-financial conflicts of interest often requires careful management. It will be important to consider if there are any relevant interests that may constitute a conflict of interest. There are situations where employment, holding of patents and other financial support should prevent people joining an author team. Funding of Cochrane Reviews by commercial organizations with an interest in the outcome of the review is not permitted. To ensure that any issues are identified early in the process, authors planning Cochrane Reviews should consult the Conflict of Interest Policy . Authors should make complete declarations of interest before registration of the review, and refresh these annually thereafter until publication and just prior to publication of the protocol and the review. For authors of review updates, this must be done at the time of the decision to update the review, annually thereafter until publication, and just prior to publication. Authors should also update declarations of interest at any point when their circumstances change.

1.3.1 Involving consumers and other stakeholders

Because the priorities of decision makers and consumers may be different from those of researchers, it is important that review authors consider carefully what questions are important to these different stakeholders. Systematic reviews are more likely to be relevant to a broad range of end users if they are informed by the involvement of people with a range of experiences, in terms of both the topic and the methodology (Thomas et al 2004, Rees and Oliver 2017). Engaging consumers and other stakeholders, such as policy makers, research funders and healthcare professionals, increases relevance, promotes mutual learning, improved uptake and decreases research waste.

Mapping out all potential stakeholders specific to the review question is a helpful first step to considering who might be invited to be involved in a review. Stakeholders typically include: patients and consumers; consumer advocates; policy makers and other public officials; guideline developers; professional organizations; researchers; funders of health services and research; healthcare practitioners, and, on occasion, journalists and other media professionals. Balancing seniority, credibility within the given field, and diversity should be considered. Review authors should also take account of the needs of resource-poor countries and regions in the review process (see Chapter 16 ) and invite appropriate input on the scope of the review and the questions it will address.

It is established good practice to ensure that consumers are involved and engaged in health research, including systematic reviews. Cochrane uses the term ‘consumers’ to refer to a wide range of people, including patients or people with personal experience of a healthcare condition, carers and family members, representatives of patients and carers, service users and members of the public. In 2017, a Statement of Principles for consumer involvement in Cochrane was agreed. This seeks to change the culture of research practice to one where both consumers and other stakeholders are joint partners in research from planning, conduct, and reporting to dissemination. Systematic reviews that have had consumer involvement should be more directly applicable to decision makers than those that have not (see online Chapter II ).

1.3.2 Working with consumers and other stakeholders

Methods for working with consumers and other stakeholders include surveys, workshops, focus groups and involvement in advisory groups. Decisions about what methods to use will typically be based on resource availability, but review teams should be aware of the merits and limitations of such methods. Authors will need to decide who to involve and how to provide adequate support for their involvement. This can include financial reimbursement, the provision of training, and stating clearly expectations of involvement, possibly in the form of terms of reference.

While a small number of consumers or other stakeholders may be part of the review team and become co-authors of the subsequent review, it is sometimes important to bring in a wider range of perspectives and to recognize that not everyone has the capacity or interest in becoming an author. Advisory groups offer a convenient approach to involving consumers and other relevant stakeholders, especially for topics in which opinions differ. Important points to ensure successful involvement include the following.

  • The review team should co-ordinate the input of the advisory group to inform key review decisions.
  • The advisory group’s input should continue throughout the systematic review process to ensure relevance of the review to end users is maintained.
  • Advisory group membership should reflect the breadth of the review question, and consideration should be given to involving vulnerable and marginalized people (Steel 2004) to ensure that conclusions on the value of the interventions are well-informed and applicable to all groups in society (see Chapter 16 ).

Templates such as terms of reference, job descriptions, or person specifications for an advisory group help to ensure clarity about the task(s) required and are available from INVOLVE . The website also gives further information on setting and organizing advisory groups. See also the Cochrane training website for further resources to support consumer involvement.

1.4 The importance of reliability

Systematic reviews aim to be an accurate representation of the current state of knowledge about a given issue. As understanding improves, the review can be updated. Nevertheless, it is important that the review itself is accurate at the time of publication. There are two main reasons for this imperative for accuracy. First, health decisions that affect people’s lives are increasingly taken based on systematic review findings. Current knowledge may be imperfect, but decisions will be better informed when taken in the light of the best of current knowledge. Second, systematic reviews form a critical component of legal and regulatory frameworks; for example, drug licensing or insurance coverage. Here, systematic reviews also need to hold up as auditable processes for legal examination. As systematic reviews need to be both correct, and be seen to be correct, detailed evidence-based methods have been developed to guide review authors as to the most appropriate procedures to follow, and what information to include in their reports to aid auditability.

1.4.1 Expectations for the conduct and reporting of Cochrane Reviews

Cochrane has developed methodological expectations for the conduct, reporting and updating of systematic reviews of interventions (MECIR) and their plain language summaries ( Plain Language Expectations for Authors of Cochrane Summaries ; PLEACS). Developed collaboratively by methodologists and Cochrane editors, they are intended to describe the desirable attributes of a Cochrane Review. The expectations are not all relevant at the same stage of review conduct, so care should be taken to identify those that are relevant at specific points during the review. Different methods should be used at different stages of the review in terms of the planning, conduct, reporting and updating of the review.

Each expectation has a title, a rationale and an elaboration. For the purposes of publication of a review with Cochrane, each has the status of either ‘mandatory’ or ‘highly desirable’. Items described as mandatory are expected to be applied, and if they are not then an appropriate justification should be provided; failure to implement such items may be used as a basis for deciding not to publish a review in the Cochrane Database of Systematic Reviews (CDSR). Items described as highly desirable should generally be implemented, but there are reasonable exceptions and justifications are not required.

All MECIR expectations for the conduct of a review are presented in the relevant chapters of this Handbook . Expectations for reporting of completed reviews (including PLEACS) are described in online Chapter III . The recommendations provided in the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) Statement have been incorporated into the Cochrane reporting expectations, ensuring compliance with the PRISMA recommendations and summarizing attributes of reporting that should allow a full assessment of the methods and findings of the review (Moher et al 2009).

1.5 Protocol development

Preparing a systematic review is complex and involves many judgements. To minimize the potential for bias in the review process, these judgements should be made as far as possible in ways that do not depend on the findings of the studies included in the review. Review authors’ prior knowledge of the evidence may, for example, influence the definition of a systematic review question, the choice of criteria for study eligibility, or the pre-specification of intervention comparisons and outcomes to analyse. It is important that the methods to be used should be established and documented in advance (see MECIR Box 1.5.a , MECIR Box 1.5.b and MECIR Box 1.5.c ).

Publication of a protocol for a review that is written without knowledge of the available studies reduces the impact of review authors’ biases, promotes transparency of methods and processes, reduces the potential for duplication, allows peer review of the planned methods before they have been completed, and offers an opportunity for the review team to plan resources and logistics for undertaking the review itself. All chapters in the Handbook should be consulted when drafting the protocol. Since systematic reviews are by their nature retrospective, an element of knowledge of the evidence is often inevitable. This is one reason why non-content experts such as methodologists should be part of the review team (see Section 1.3 ). Two exceptions to the retrospective nature of a systematic review are a meta-analysis of a prospectively planned series of trials and some living systematic reviews, as described in Chapter 22 .

The review question should determine the methods used in the review, and not vice versa. The question may concern a relatively straightforward comparison of one treatment with another; or it may necessitate plans to compare different treatments as part of a network meta-analysis, or assess differential effects of an intervention in different populations or delivered in different ways.

The protocol sets out the context in which the review is being conducted. It presents an opportunity to develop ideas that are foundational for the review. This concerns, most explicitly, definition of the eligibility criteria such as the study participants and the choice of comparators and outcomes. The eligibility criteria may also be defined following the development of a logic model (or an articulation of the aspects of an extent logic model that the review is addressing) to explain how the intervention might work (see Chapter 2, Section 2.5.1 ).

MECIR Box 1.5.a Relevant expectations for conduct of intervention reviews

A key purpose of the protocol is to make plans to minimize bias in the eventual findings of the review. Reliable synthesis of available evidence requires a planned, systematic approach. Threats to the validity of systematic reviews can come from the studies they include or the process by which reviews are conducted. Biases within the studies can arise from the method by which participants are allocated to the intervention groups, awareness of intervention group assignment, and the collection, analysis and reporting of data. Methods for examining these issues should be specified in the protocol. Review processes can generate bias through a failure to identify an unbiased (and preferably complete) set of studies, and poor quality assurance throughout the review. The availability of research may be influenced by the nature of the results (i.e. reporting bias). To reduce the impact of this form of bias, searching may need to include unpublished sources of evidence (Dwan et al 2013) ( MECIR Box 1.5.b ).

MECIR Box 1.5.b Relevant expectations for the conduct of intervention reviews

Developing a protocol for a systematic review has benefits beyond reducing bias. Investing effort in designing a systematic review will make the process more manageable and help to inform key priorities for the review. Defining the question, referring to it throughout, and using appropriate methods to address the question focuses the analysis and reporting, ensuring the review is most likely to inform treatment decisions for funders, policy makers, healthcare professionals and consumers. Details of the planned analyses, including investigations of variability across studies, should be specified in the protocol, along with methods for interpreting the results through the systematic consideration of factors that affect confidence in estimates of intervention effect ( MECIR Box 1.5.c ).

MECIR Box 1.5.c Relevant expectations for conduct of intervention reviews

While the intention should be that a review will adhere to the published protocol, changes in a review protocol are sometimes necessary. This is also the case for a protocol for a randomized trial, which must sometimes be changed to adapt to unanticipated circumstances such as problems with participant recruitment, data collection or event rates. While every effort should be made to adhere to a predetermined protocol, this is not always possible or appropriate. It is important, however, that changes in the protocol should not be made based on how they affect the outcome of the research study, whether it is a randomized trial or a systematic review. Post hoc decisions made when the impact on the results of the research is known, such as excluding selected studies from a systematic review, or changing the statistical analysis, are highly susceptible to bias and should therefore be avoided unless there are reasonable grounds for doing this.

Enabling access to a protocol through publication (all Cochrane Protocols are published in the CDSR ) and registration on the PROSPERO register of systematic reviews reduces duplication of effort, research waste, and promotes accountability. Changes to the methods outlined in the protocol should be transparently declared.

This Handbook provides details of the systematic review methods developed or selected by Cochrane. They are intended to address the need for rigour, comprehensiveness and transparency in preparing a Cochrane systematic review. All relevant chapters – including those describing procedures to be followed in the later stages of the review – should be consulted during the preparation of the protocol. A more specific description of the structure of Cochrane Protocols is provide in online Chapter II .

1.6 Data management and quality assurance

Systematic reviews should be replicable, and retaining a record of the inclusion decisions, data collection, transformations or adjustment of data will help to establish a secure and retrievable audit trail. They can be operationally complex projects, often involving large research teams operating in different sites across the world. Good data management processes are essential to ensure that data are not inadvertently lost, facilitating the identification and correction of errors and supporting future efforts to update and maintain the review. Transparent reporting of review decisions enables readers to assess the reliability of the review for themselves.

Review management software, such as Covidence and EPPI-Reviewer , can be used to assist data management and maintain consistent and standardized records of decisions made throughout the review. These tools offer a central repository for review data that can be accessed remotely throughout the world by members of the review team. They record independent assessment of studies for inclusion, risk of bias and extraction of data, enabling checks to be made later in the process if needed. Research has shown that even experienced reviewers make mistakes and disagree with one another on risk-of-bias assessments, so it is particularly important to maintain quality assurance here, despite its cost in terms of author time. As more sophisticated information technology tools begin to be deployed in reviews (see Chapter 4, Section 4.6.6.2 and Chapter 22, Section 22.2.4 ), it is increasingly apparent that all review data – including the initial decisions about study eligibility – have value beyond the scope of the individual review. For example, review updates can be made more efficient through (semi-) automation when data from the original review are available for machine learning.

1.7 Chapter information

Authors: Toby J Lasserson, James Thomas, Julian PT Higgins

Acknowledgements: This chapter builds on earlier versions of the Handbook . We would like to thank Ruth Foxlee, Richard Morley, Soumyadeep Bhaumik, Mona Nasser, Dan Fox and Sally Crowe for their contributions to Section 1.3 .

Funding: JT is supported by the National Institute for Health Research (NIHR) Collaboration for Leadership in Applied Health Research and Care North Thames at Barts Health NHS Trust. JPTH is a member of the NIHR Biomedical Research Centre at University Hospitals Bristol NHS Foundation Trust and the University of Bristol. JPTH received funding from National Institute for Health Research Senior Investigator award NF-SI-0617-10145. The views expressed are those of the author(s) and not necessarily those of the NHS, the NIHR or the Department of Health.

1.8 References

Antman E, Lau J, Kupelnick B, Mosteller F, Chalmers T. A comparison of results of meta-analyses of randomized control trials and recommendations of clinical experts: treatment for myocardial infarction. JAMA 1992; 268 : 240–248.

Chalmers I, Bracken MB, Djulbegovic B, Garattini S, Grant J, Gulmezoglu AM, Howells DW, Ioannidis JP, Oliver S. How to increase value and reduce waste when research priorities are set. Lancet 2014; 383 : 156–165.

Chandler J, Hopewell S. Cochrane methods – twenty years experience in developing systematic review methods. Systematic Reviews 2013; 2 : 76.

Dwan K, Gamble C, Williamson PR, Kirkham JJ, Reporting Bias Group. Systematic review of the empirical evidence of study publication bias and outcome reporting bias: an updated review. PloS One 2013; 8 : e66844.

Gøtzsche PC, Ioannidis JPA. Content area experts as authors: helpful or harmful for systematic reviews and meta-analyses? BMJ 2012; 345 .

Macleod MR, Michie S, Roberts I, Dirnagl U, Chalmers I, Ioannidis JP, Al-Shahi Salman R, Chan AW, Glasziou P. Biomedical research: increasing value, reducing waste. Lancet 2014; 383 : 101–104.

Moher D, Liberati A, Tetzlaff J, Altman D, PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Medicine 2009; 6 : e1000097.

Oxman A, Guyatt G. The science of reviewing research. Annals of the New York Academy of Sciences 1993; 703 : 125–133.

Rees R, Oliver S. Stakeholder perspectives and participation in reviews. In: Gough D, Oliver S, Thomas J, editors. An Introduction to Systematic Reviews . 2nd ed. London: Sage; 2017. p. 17–34.

Steel R. Involving marginalised and vulnerable people in research: a consultation document (2nd revision). INVOLVE; 2004.

Thomas J, Harden A, Oakley A, Oliver S, Sutcliffe K, Rees R, Brunton G, Kavanagh J. Integrating qualitative research with trials in systematic reviews. BMJ 2004; 328 : 1010–1012.

For permission to re-use material from the Handbook (either academic or commercial), please see here for full details.

Systematic Reviews: Steps in a Systematic Review

  • What Type of Review is Right for You?
  • What is in a Systematic Review
  • Finding and Appraising Systematic Reviews
  • Formulating Your Research Question
  • Inclusion and Exclusion Criteria
  • Creating a Protocol
  • Results and PRISMA Flow Diagram
  • Searching the Published Literature
  • Searching the Gray Literature
  • Methodology and Documentation
  • Managing the Process

Systematic review process diagram

Identify the issue and determine the question. Write a plan for the review (protocol). Search for studies. Sift and select studies. Extract data from the studies. Assess the quality of the studies. Combine the data (synthesis or meta-analysis). Discuss and conclude overall findings), systematic review and dissemination.

This diagram illustrates in a visual way and in plain language what review authors actually do in the process of undertaking a systematic review.

Designed by Jessica Kaufman, Cochrane Consumers & Communication Review Group, Centre for Health Communication & Participation, La Trobe University, 2011. CC-BY-SA License.

Steps in the Systematic Review Process

  • Check for existing reviews/protocols.  Is the review still needed? Can you change the question to answer a different altered question? Was the review done well? When was it done? Has there been a development in the research since then? Is it broad enough? 
  • I dentify your research question .  Formulate a clear, well-defined research question of appropriate scope. Define your terminology. Find existing reviews on your topic to inform the development of your research question, identify gaps, and confirm that you are not duplicating the efforts of previous reviews. Consider using a framework to define you question scope.
  • Define inclusion and exclusion criteria . This is also known as creating a review protocol . Clearly, state the criteria you will use to determine whether or not a study will be included in your search.  Consider study populations, study design, intervention types, comparison groups, measured outcomes.
  • Search for studies . Run your searches in the databases that you've identified as relevant to your topic. Work with a librarian to help you design comprehensive search strategies across a variety of databases. Approach the gray literature methodically and purposefully . Collect ALL of the retrieved records from each search into a reference manager, such as Endnote, and de-duplicate the library prior to screening.
  • Select studies for inclusion based on pre-defined criteria . Start with a title/abstract screening to remove studies that are clearly not related to your topic. Use your inclusion/exclusion criteria to screen the full-text of studies. It is highly recommended that two independent reviewers screen all studies, resolving areas of disagreement by consensus.
  • Extract data from included studies . Use a spreadsheet, or systematic review software, to extract all relevant data from each included study. It is recommended that you pilot your data extraction tool, to determine if other fields should be included or existing fields clarified.
  • Evaluate the risk of bias of included studies . Use a Risk of Bias tool (such as the Cochrane RoB Tool) to assess the potential biases of studies in regards to study design and other factors. You can adapt existing tools to best meet the needs of your review, depending on the types of studies included.
  • Present results and assess the quality of evidence . Clearly present your findings, including detailed methodology (such as search strategies used, selection criteria, etc.) such that your review can be easily updated in the future with new research findings. Perform a meta-analysis if the studies allow. Provide recommendations for practice and policy-making if sufficient, high-quality evidence exists, or future directions for research to fill existing gaps in knowledge or to strengthen the body of evidence
  • << Previous: What Type of Review is Right for You?
  • Next: What is in a Systematic Review >>
  • Last Updated: Dec 12, 2023 3:35 PM
  • URL: https://guides.lib.lsu.edu/Systematic_Reviews

Provide Website Feedback Accessibility Statement

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • View all journals
  • My Account Login
  • Explore content
  • About the journal
  • Publish with us
  • Sign up for alerts
  • Open access
  • Published: 21 May 2013

A step-by-step guide to the systematic review and meta-analysis of diagnostic and prognostic test accuracy evaluations

  • Z Liu 1 , 2 ,
  • H Chen 1 &

British Journal of Cancer volume  108 ,  pages 2299–2303 ( 2013 ) Cite this article

12k Accesses

24 Citations

1 Altmetric

Metrics details

  • Medical research
  • Tumour biomarkers

This article has been updated

In evidence-based medicine (EBM), systematic reviews and meta-analyses have been widely applied in biological and medical research. Moreover, the most popular application of meta-analyses in this field may be to examine diagnostic (sensitivity and specificity) and prognostic (hazard ratio (HR) and its variance, standard error (SE) or confidence interval (CI)) test accuracy. However, conducting such analyses requires not only a great deal of time but also an advanced professional knowledge of mathematics, statistics and computer science. Regarding the practical application of meta-analyses for diagnostic and prognostic markers, the majority of users are clinicians and biologists, most of whom are not skilled at mathematics and computer science in particular. Hence, it is necessary for these users to have a simplified version of a protocol to help them to quickly conduct meta-analyses of the accuracy of diagnostic and prognostic tests. The aim of this paper is to enable individuals who have never performed a meta-analysis to do so from scratch. The paper does not attempt to serve as a comprehensive theoretical guide but instead describes one rigorous way of conducting a meta-analysis for diagnostic and prognostic markers. Investigators who follow the outlined methods should be able to understand the basic ideas behind the steps taken, the meaning of the meta-analysis results obtained for diagnostic and prognostic markers and the scope of questions that can be answered with Systematic Reviews and Meta-Analyses (SRMA). The presented protocols have been successfully tested by clinicians without meta-analysis experience.

Systematic reviews of high-quality randomised controlled trials are crucial in evidence-based medicine (EBM), as they are particularly useful for overcoming the difficulties faced by clinicians when they wish to extract and analyse dates to guide their practice. Systematic reviews are of special value in aggregating and synthesising the findings of many separately conducted studies, sometimes with conflicting results ( Clarke, 2007 ). The purpose of the preferred reporting Items for Systematic Reviews and Meta-Analyses (SRMA) guidelines is to aid researchers in the reporting of SRMA ( Liberati et al, 2009 ; Moher et al, 2010 ). When a review makes an effort to comprehensively identify and trace all of the literature on a given topic (also referred to as a systematic literature review), meta-analysis is a particular statistical strategy for bringing together the results of several studies to produce a single estimate ( Sackett et al, 2007 ).

Numerous reports and books have been published that describe SRMA, but several predominant problems still exist, as stated below.

Dispersed and fragmentary documents that describe SRMA are much more numerous than systematic and comprehensive reports. Additionally, the materials available that introduce the theory of SRMA are much more numerous than those addressing methodology, and the theoretical and methodological papers relevant to SRMA are often independent of each other. Hence, obtaining comprehensive collections of the materials related to SRMA is time-consuming work.

The articles and books related to SRMA focus primarily on techniques, but not practices, necessitating the need for a strong background in fields such as mathematics, statistics and computer science. For non-technical readers and the majority of biologists and clinicians, these materials are daunting and will lead to a natural aversion to meta-analysis, thus hindering the wide application of SRMA.

Biomarkers (especially disease markers) have been widely applied in biological and clinical analyses. The characteristic of a single marker are usually reported in many articles, and an emergent task is the integration of the effect sizes of markers (i.e., sensitivity and specificity of a diagnostic/screening marker or the hazard ratio (HR) and its variance, SE or CI as a prognostic/monitoring marker) using SRMA. Nevertheless, according to our investigations, few articles and software tools are available that fully elaborate a procedure to combine the effect sizes of markers.

To address the above problems, we collected various materials and compiled a protocol using non-technical language as much as possible to guide common audiences in a step-by-step manner to realise SRMA, aiming at effect sizes of biomarkers with zero barriers. Our goal is to make SRMA accessible to most audiences, including biologists, clinicians and novices. We believe that investigators who read our paper will benefit from our protocol.

In this article, given that our protocol only focuses on SRMA of biomarker test accuracy, the following descriptions are not strictly in accordance with the above-mentioned general eight-step method. Based on the characteristics of the effect size of the accuracy of diagnostic and prognostic tests, we provide a five-step workflow.

Steps for SRMA of biomarkers

Search strategy.

The goal of the literature search is to be sufficiently exhaustive to develop a comprehensive list of potentially relevant studies. The first step in a meta-analysis is to find all of the pertinent articles on your topic. Important sources of information for meta-analyses include MEDLINE ( http://www.ncbi.nlm.nih.gov/pubmed ), EMBASE ( http://www.ncbi.nlm.nih.gov/pubmed ), OvidSP ( http://www.ovid.com/ ) and CancerLit ( http://www.twu.ca/library/cancerlit.htm ). The Cochrane Collaboration Controlled Trials Register, established in 1993, is also an important source of studies for a meta-analysis. It includes all of the controlled trials in the MEDLINE and EMBASE as well as the results of manual searches conducted by Cochrane Collaboration volunteers of thousands of journals not indexed by MEDLINE or EMBASE. Before applying the literature search strategy, the basic information and search syntax should be mastered; key words related to your topic should be listed; and the ‘associated words’ for each key word must also be prepared (see Tamara Durec BSc(Pharm), 2013 and Literature Searching and Systematic Reviews (2013) for more details and Appendix 1 as an example in Supplementary Materials ).

Inclusion and exclusion criteria

Biomarkers are commonly categorised into four types: screening, diagnostic, prognostic and monitoring (surveillance). The first two types of markers are assessed based on sensitivity and specificity in most cases, while prognostic and monitoring markers are estimated based on the HR and its variance or standard error (SE) because they are time-to-event markers. Once the author of a meta-analysis has assembled a large number of studies, it is important to select the right ones. Which studies are included or excluded depends on various factors, such as whether or not there is sufficient information in a study to conduct an analysis, in addition to the study design, dosage used in the study, sample size, patient age, and even the year of the study. The following general criteria are provided only for reference:

The study should be an original report (i.e., letters, editorials, case reports, tutorials and reviews are excluded), and both English and non-English studies should be included in case of a publication bias.

The study should assess the ability of one or more markers to detect the presence of a particular disease.

The study should provide sufficient data to allow estimation of a marker’s accuracy, for a diagnostic marker, the study must directly or indirectly provide at least four values, which are the following: the number of true positives (TP), false positives (FP), true negatives (TN) and false negatives (FN), to (re)construct a two-by-two table. (See the diagnostic marker sheet of Supplementary Table 1 for other relevant information requirements.) For a prognostic marker, the study must directly or indirectly provide at least two values, the HR, and its variance and/or SE and/or confidence interval (CI). (See the prognostic marker sheet of Supplementary Table 1 for other relevant information requirements.)

Combination marker and review articles are excluded.

If multiple papers are published based on the same or overlapping data sets, then only the paper with the largest number of specimens, the most detailed results and the longest follow-up time is included.

A minimum of two reviewers perform a first-stage screening of titles and abstracts based on the research question and the study design, population, intervention and outcome to be studied. Based on the initial screening, selected full-text articles are obtained for the second-stage screening. Including two reviewers minimises the introduction of bias by either reviewer. Any study identified by either reviewer should be included. Using the full text, a second-stage screening is performed by at least two reviewers. The studies selected are then submitted for data extraction.

Data extraction

Once an appropriate group of studies has been identified, the relevant data from candidate studies must be correctly extracted. To minimise errors, the following conventions should be considered: (1) all reviewers must be trained under a consensus standard and then practice using several articles for ‘calibration’; (2) a consensus form or database that constrains entries to the expected range should be determined in advance; (3) at least two independent reviewers should check and extract data from a given article, and if the extracted data are not same, conflicts are resolved by reaching a consensus; and (4) to prevent bias creeping into a meta-analysis, the reviewers should not be biased in favor of (or against) well-known researchers or prominent journals as far as possible.

For studies related to biomarkers, the reviewers should also pay attention to the following matters in addition to the preceding items. Among the four types of markers, screening and diagnostic markers mainly focus on sensitivity and specificity, while prognostic and monitoring markers are usually focused on the HR and its variance, SE or CI. Consequently, we will classify the markers into two different categories and describe how to abstract relevant information: (1) if more than one marker is used in a given study, then the relevant data for each eligible marker must be individually extracted; (2) if one marker has multiple functions (i.e., one marker for one disease is used for screening, diagnosis, prognosis and/or monitoring), then the data sets corresponding to multiple functions must be extracted separately; and (3) if there are multiple markers and diseases addressed in one study, then only the relevant data from the marker(s) corresponding to each disease of interest for the author(s) should be extracted.

For screening or diagnostic markers For diagnostic (or screening) markers, the data abstraction phase involves an assessment of study quality. Whiting et al (2003) proposed a set of criteria for Quality Assessment of Diagnostic Accuracy Studies (QUADAS) that applies well to diagnostic marker studies ( Whiting et al, 2003 ). Data extraction (see Appendix 2 in Supplementary Materials for details) with QUADAS assessment is often completed at the same time.

For prognostic or monitoring markers In contrast, data extraction and conversion for prognostic (or monitoring) markers are much more complex than for diagnostic markers because prognostic markers provide time-to-event data, which indicate the distinctions between the two groups of studies, and time to event needs to be scrutinised very carefully since the data may not only be right censored (patient was not followed until the event), but also left censored (patients were not all followed starting from a comparable point). Meta-analyses of this type of marker often require one of two kinds of data, that is, the log of the HR (namely, log e (HR)) and its variance or SE or the HR and its CI. For major prognostic marker studies, the two kinds of data cannot be extracted directly. Parmar et al (1998) presented a series of simple methods to extract relevant data from publications with the aim of performing a meta-analysis of survival-type data. The methods focus on approaches for extracting these data from publications and are illustrated throughout this publication with real examples. Riley et al (2003) summarised 11 methods (see Appendix 3 in Supplementary Materials for details) that are available for directly or indirectly estimating these data and the approximate normal log e (HR) distribution for large samples. In addition, Tierney et al (2007) provided step-by-step guidance for how to calculate an HR and the associated statistics for individual trials, according to the information presented in the trial report.

Statistical methods

When studies used a similar design, we often combine the information they provide to increase precision and to investigate consistencies and discrepancies between the results. There has been great growth in this kind of analysis in several fields in recent years, particularly in medicine. In medicine, such studies usually involve controlled therapeutic trials. We apply the same principles in any scientific area, such as epidemiology, psychology or educational research. The essence of meta-analysis is obtaining a single estimate of the effect size from each similar study. There are many issues and controversies regarding meta-analysis data. First, we have to define two important terms, homogeneity and heterogeneity, to describe the degree of between-study variability in a group of studies. Fixed-effect models consider only within-study variability. The assumption is that studies use identical methods, patients and measurements; that they should produce identical results; and that any differences are only due to within-study variation. Random-effect models consider both the between-study and within-study variability. It is assumed that studies provide a random sample from the universe of all possible studies. If the studies are heterogeneous, then a random-effect model is applied for meta-analysis of the effect size in a group of studies; otherwise, a fixed-effect model is selected (see Appendix 4 in Supplementary Materials for detailed interpretations). A meta-analysis will customarily include a forest plot, in which the results from each study are displayed as a square and a horizontal line, representing the intervention effect estimated together with its CI. The area of the square reflects the weight that the study contributes to the meta-analysis. The combined-effects estimate and its CI are represented by a diamond. Biomarkers generally include screening, diagnostic, prognostic and monitoring markers. The first two types of markers correspond to diagnostic tests, and the last two provide time-to-event data. In meta-analyses of the two kinds of tests, there are significant differences in terms of both the combined objects and methods. Hence, descriptions of the two kinds of meta-analyses are provided below.

Analysis of diagnostic (or screening) test accuracy. The meta-analysis of diagnostic test accuracy represents an area of growing interest. These analyses often consist of three steps: assessment of study quality, creation of forest plots for sensitivity and specificity for each study, and summarisation of estimates of sensitivity and specificity using two types of models.

Quality assessment for diagnostic articles: Quality assessment is as important in systematic reviews of diagnostic accuracy studies as it is in any other type of review, and the methodological quality of each study was assessed as recommended by the Cochrane Diagnostic Test Accuracy Working Group. These recommendations were adapted from the QUADAS guidelines ( Whiting et al, 2003 ; Macaskill et al, 2010 ). All of the criteria were classified as Yes, No or Unclear based on information available in this publication (see Appendix 5 in Supplementary Materials ). The studies were judged according to the data used for the meta-analysis, which may not include all of the data available in the publication.

Meta-analysis of the accuracy of diagnostic tests: Meta-analyses of diagnostic test accuracy present many challengers: (1) even in the simplest case, a minimum of two summary statistics (sensitivity and specificity) must be addressed simultaneously; (2) meta-analysis methods allow studies to be combined that have applied tests at different thresholds; and (3) random-effect methods are recommended when data are heterogeneous (this is the rule for diagnostic studies). Therefore, in a meta-analysis of diagnostic accuracy, two analysis steps must be completed: (1) forest plots for pooling the sensitivity and specificity of all of the selected studies are first created; and (2) two statistical methods to calculate summary estimates of sensitivity and specificity are proposed to account for the correlation between sensitivity and specificity across studies caused by the relationship between sensitivity and specificity within each study ( Moses et al, 1993 ). The two statistical methods, that is, the hierarchical summary receiver operating characteristic (HSROC) model ( Rutter and Gatsonis, 2001 ) and bivariate model ( Reitsma et al, 2005 ), are statistically rigorous ( Appendix 6 in Supplementary Materials introduces two ways to perform a meta-analysis of the accuracy of diagnostic tests).

Meta-analysis of the accuracy of prognostic (or monitoring) tests. For prognostic (monitoring) markers, as described above, there are two types of extracted data: the HR and its CI (lower limit and upper limit); and log e (HR) and its variance (var[log e (HR)]) or standard error (SE[log e (HR)]=the reciprocal of the square root of var[log e (HR)]). Among the existing meta-analysis software tools, RevMan 5.1 ( Review Manager, 2011 and MetaDisc 1.4 ( Zamora et al, 2006 ) cannot be used to implement a meta-analysis of the accuracy of prognostic tests. The mada package in the R language can be used to perform meta-analysis of prognostic test accuracy, but the R language requires complete user-entry of codes. In contrast, the mode of operation of STATA involves an interface of window plus commands that can be used by common audiences, making the performance of a meta-analysis using STATA (see Appendix 7 in Supplementary Materials for STATA installation and 14 STATA meta-analysis commands) much easier than using the R language. Next, we will use non-technical language to interpret how to perform a meta-analysis of the accuracy of prognostic tests.

Meta-analysis of prognostic test accuracy: Meta-analysis is a two-stage process involving the estimation of an appropriate summary statistic for each of a set of studies, followed by the calculation of a weighted average of these statistics across the studies ( Deeks et al, 2008 ).

The summary statistics from each study can be combined using a variety of meta-analytical methods, which are classified as fixed-effect models in which studies are weighted according to the amount of information they contain, or random-effects models, which incorporate an estimate of between-study variation (heterogeneity) in the weighting. A meta-analysis will customarily include a forest plot, where the results from each study are displayed as a square and a horizontal line, representing the intervention effect estimate together with its CI. The area of the square reflects the weight that the study contributes to the meta-analysis. The combined-effect estimate and its CI are represented by a diamond. Here, we present updates to the metan command in STATA to perform a meta-analysis of prognostic test accuracy. metan provides methods for the meta-analysis of studies with two groups, and either fixed-effect or random-effect models can be fitted ( Fleiss, 1993 ). The following is the syntax for metan: metan [varlist] [option] . In Supplementary Materials document, we used two different data types as examples (see examples 1 and 2 in Supplementary Materials for the detailed operation flows) to present the metan command for the meta-analysis of prognostic test accuracy.

Publication bias regarding prognostic test accuracy: Publication bias is the phenomenon of studies with uninteresting or unfavorable results being less likely to be published than those with more favorable results ( Rothstein et al, 2005 ). If a publication bias exists, then the published literature is a biased sample of all studies on a topic, and any meta-analysis based on it will be similarly biased. Funnel plots are commonly used to investigate publication and related biases in meta-analyses ( Sterne et al, 2005 ). metabias performs the Begg and Mazumdar (1994) ) adjusted rank correlation test for publication bias as well as the Egger et al (1997) regression asymmetry test for publication bias. As options, it provides a funnel graph of the data or the regression asymmetry plot. The Begg adjusted rank correlation test is more popular in common applications for publication bias analysis (see examples 3 and 4 in Supplementary Materials ).

Non-parametric trim and fill analysis of publication bias. Meta-analysis is a popular technique for numerically synthesising information from published studies. One of the many concerns that must be addressed when performing a meta-analysis is whether selective publication of studies could lead to a bias in estimating the overall meta-analytical effect and in the inferences derived from the analysis. If a publication bias appears to exist, then it is desirable to consider what the unbiased data set might look like and then to re-estimate the overall meta-analytical effect after any apparently ‘missing’ studies are included. Duval and Tweedie’s ‘non-parametric ‘trim and fill’ method’ is designed to meet these objectives ( Duval and Tweedie, 2000 ). The command metatrim is used to implement the Duval and Tweedie non-parametric ‘trim and fill’ method (see examples 5 and 6 in Supplementary Materials ).

Cumulative meta-analysis of prognostic test accuracy: In a cumulative meta-analysis ( Rothstein et al, 2005 ), the pooled estimate of the treatment effect is updated each time the results of a new study are published. This makes it possible to track the accumulation of evidence related to the effect of a particular treatment. The command metacum performs a cumulative meta-analysis (using fixed- or random-effect models), and optionally, the results can be graphed. A user supplies the preceding two types of data on prognostic test accuracy. The full metacum command is very similar to the metan command. The detailed commands are as follows:

metacum lnhr lnll lnul, eform label (namevar=studyid) title (“random-effect model”) boxsca (0.9) random effect (Hazard Ratio)

(The output and forest plot are omitted)

metacum logehr selogehr, eform effect (Hazard Ratio) title (“Fixed-effect meta-analysis”) boxsca(0.9) label (namevar=paperno)

(The output and forest plot are omitted).

Subgroup for prognostic test accuracy: Dividing results between different types of patients and outcomes requires cautious interpretation. If these analyses are to be conducted, then more subgroup analyses must be performed. It is often more reliable to assume that the overall result is as good an estimate (if not a better one) for a particular group of patients than that obtained by examining those patients within the meta-analysis. In prognostic test data, it is very common for the data to be classified into disease-free survival (DFS) and overall survival (OS) subgroups, as in the data in Example data 1. In fact, performing both subgroup analyses in meta-analysis in STATA is very simple, and a major addition to metan is the ability to perform stratified or subgroup analyses. Subgroup analyses in meta-analysis may be used to investigate the possibility that treatment effects vary between subgroups. Subgrouping in meta-analysis can be completed by adding one option: by (grouping variable name) , to all meta-analysis commands for diagnostic or prognostic test accuracy (all examples are omitted).

In EBM, SRMA have been widely applied in biological and medical research. Moreover, the most popular application of meta-analysis in this field may be to assess diagnostic (sensitivity and specificity) and prognostic (the HR and its precision) test accuracy. With the growth of clinical renal studies, an increasing number of these types of summary publications will certainly become available to nephrologists, researchers, administrators and policy makers who seek to keep abreast of recent developments. To maximise the advantages of these studies, it is necessary for these individuals to have a simplified version of a protocol to aid them in rapidly conducting meta-analyses of the accuracy of diagnostic and prognostic tests. In this article, we first presented a simplified and practical protocol to guide non-professional academicians and clinicians to perform systematic reviews of diagnostic and prognostic accuracies in a step-by-step manner, and we confirmed that once an individual studies our article, even a novice, they are soon able to accomplish complex systematic reviews and meta-analyses. The protocols have been successfully tested by clinicians without meta-analysis experience.

Change history

11 june 2013.

This paper was modified 12 months after initial publication to switch to Creative Commons licence terms, as noted at publication

Begg CB, Mazumdar M (1994) Operating characteristics of a rank correlation test for publication bias. Biometrics 50 : 1088–1101

Article   CAS   Google Scholar  

Clarke M (2007) The Cochrane Collaboration and systematic reviews. Br J Surg 94 (4): 391–392

Deeks JJ, Altman DG, Bradburn MJ (2008) Statistical methods for examining heterogeneity and combining results from several studies in meta-analysis. In Systematic Reviews in Health Care: Meta-analysis in Context 2nd edn Egger M, Smith GD, Altman DG (eds) BMJ Publishing Group: London, UK doi: 10.1002/9780470693926.ch15)

Duval S, Tweedie R (2000) A nonparametric ‘trim and fill’ method of accounting for publication bias in meta-analysis. J Am Stat Assoc 95 : 89–98

Google Scholar  

Egger M, Smith GD, Schneider M, Minder C (1997) Bias in meta-analysis detected by a simple, graphical test. Br Med J 315 : 629–634

Fleiss JL (1993) The statistical basis of meta-analysis. Stat Methods Med Res 2 : 121–145

Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gøtzsche PC, Ioannidis JP, Clarke M, Devereaux PJ, Kleijnen J, Moher D (2009) The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration. BMJ 21 : 339 b2700

Literature Searching and Systematic Reviews (2013) Available from www.rds-eoe.nihr.ac.uk/activity/docs/infosheet-5.doc (accessed 12 March 2013)

Macaskill P, Gatsonis C, Deeks JJ, Harbord RM, Takwoingi Y (2010) Chapter 10: analysing and presenting results. In: Deeks JJ, Bossuyt PM, Gatsonis C (eds), Cochrane Handbook for Systematic Reviews of Diagnostic Test Accuracy Version 1.0 . The Cochrane Collaboration: Oxford, UK. 1–47., Available from http://srdta.cochrane.org/

Moher D, Liberati A, Tetzlaff J, Altman DG PRISMA Group (2010) Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Int J Surg 8 (5): 336–341

Article   Google Scholar  

Moses LE, Shapiro D, Littenberg B (1993) Combining independent studies of a diagnostic test into a summary ROC curve: data-analytic approaches and some additional considerations. Stat Med 12 (14): 1293–1316

Parmar MK, Torri V, Stewart L (1998) Extracting summary statistics to perform meta-analyses of the published literature for survival endpoints. Stat Med 17 (24): 2815–2834

Reitsma JB, Glas AS, Rutjes AW, Scholten RJ, Bossuyt PM, Zwinderman AH (2005) Bivariate analysis of sensitivity and specificity produces informative summary measures in diagnostic reviews. J Clin Epidemiol 58 (10): 982–990

Review Manager (RevMan) [computer software] (2011) Version 5.1 . Cochrane Collaboration: Copenhagen, Denmark

Riley RD, Burchill SA, Abrams KR, Heney D, Lambert PC, Jones DR, Sutton AJ, Young B, Wailoo AJ, Lewis IJ (2003) A systematic review and evaluation of the use of tumour markers in paediatric oncology: Ewing's sarcoma and neuroblastoma. Health Technol Assess 7 (5): 1–162

Rothstein HR, Sutton AJ, Borenstein M (2005) Publication bias in meta-analysis: prevention. Assessment and Adjustments . Chichester, UK, : Wiley

Rutter CM, Gatsonis CA (2001) A hierarchical regression approach to meta-analysis of diagnostic test accuracy evaluations. Stat Med 20 (19): 2865–2884

Sackett DL, Rosenberg WM, Gray JA, Haynes RB, Richardson WS (2007) Evidence based medicine: what it is and what it isn’t.2006. Clin Orthop Relat Res 455 : 3–5

PubMed   Google Scholar  

Sterne JA, Becker BJ, Egger M (2005) The funnel plot. In Publication Bias in Meta-Analysis: Prevention, Assessment and Adjustments , Rothstein HR, Sutton AJ, Borenstein M (eds) pp 75–98. Wiley: Chichester, UK

Tamara Durec BSc(Pharm) (2013) Sarah Curtis—Online EBM Tutorial. Literature Searching For Systematic Reviews. Available from www.columbia.edu/~mvp19/RMC/M2/Files/LitSearch.doc (accessed 12 March 2013)

Tierney JF, Stewart LA, Ghersi D, Burdett S, Sydes MR (2007) Practical methods for incorporating summary time-to-event data into meta-analysis. Trials 8 : 16

Whiting P, Rutjes AW, Reitsma JB, Bossuyt PM, Kleijnen J (2003) The development of QUADAS: a tool for the quality assessment of studies of diagnostic accuracy included in systematic reviews. BMC Med Res Methodol 3 : 25

Zamora J, Abraira V, Muriel A, Khan K, Coomarasamy A (2006) Meta-DiSc: a software for meta-analysis of test accuracy data. BMC Med Res Methodol 6 : 31

Download references

Acknowledgements

We thank the anonymous reviewers for constructive comments on the manuscript. Funding for this work was provided by China Postdoctoral Science Foundation (201150M1569 and 2012T50893 to ZL).

Author information

Authors and affiliations.

Anal-Colorectal Surgery Institute, 150th Central Hospital of PLA, Luoyang, 471031, Henan, China

Z Liu, Z Yao, X Liu, H Chen & C Gao

Jinan Military General Hospital, Jinan, 250031, China

Division of head and neck surgery, Cancer Hospital of Sichuan, Sichuan, 610000, China

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to C Gao .

Ethics declarations

Competing interests.

The authors declare no conflict of interest.

Additional information

Author contributions

ZL contributed to the design, preparation and editing of the document. CG was responsible for the final review and approval for submission.

Supplementary Information accompanies this paper on British Journal of Cancer website

Supplementary information

Supplementary table 1, supplementary information, rights and permissions.

From twelve months after its original publication, this work is licensed under the Creative Commons Attribution-NonCommercial-Share Alike 3.0 Unported License. To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-sa/3.0/

Reprints and permissions

About this article

Cite this article.

Liu, Z., Yao, Z., Li, C. et al. A step-by-step guide to the systematic review and meta-analysis of diagnostic and prognostic test accuracy evaluations. Br J Cancer 108 , 2299–2303 (2013). https://doi.org/10.1038/bjc.2013.185

Download citation

Received : 02 January 2013

Revised : 22 March 2013

Accepted : 24 March 2013

Published : 21 May 2013

Issue Date : 11 June 2013

DOI : https://doi.org/10.1038/bjc.2013.185

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

  • systematic review
  • meta-analysis
  • prognostic marker
  • diagnostic marker
  • sensitivity and specificity
  • hazard ratio

This article is cited by

The diagnostic potential of oxidative stress biomarkers for preeclampsia: systematic review and meta-analysis.

  • Dinara Afrose
  • Lana McClements

Biology of Sex Differences (2022)

Surveillance for second breast cancer events in women with a personal history of breast cancer using breast MRI: a systematic review and meta-analysis

  • Cameron B. Haas
  • Larissa Nekhlyudov
  • Karen J. Wernli

Breast Cancer Research and Treatment (2020)

Postmortem computed tomography angiography (PMCTA) and traditional autopsy in cases of sudden cardiac death due to coronary artery disease: a systematic review and meta-analysis

  • Raffaele La Russa
  • Carlo Catalano
  • Vittorio Fineschi

La radiologia medica (2019)

The value of prostate-specific antigen monitoring during salvage radiotherapy: a retrospective study and systematic review with meta-analysis

  • Riccardo Vigna-Taglianti
  • Alberto Boriano
  • Elvio G. Russi

Journal of Radiation Oncology (2019)

Optic nerve sheath diameter measured sonographically as non-invasive estimator of intracranial pressure: a systematic review and meta-analysis

  • Chiara Robba
  • Gregorio Santori
  • Giuseppe Citerio

Intensive Care Medicine (2018)

Quick links

  • Explore articles by subject
  • Guide to authors
  • Editorial policies

systematic review step by step guide

U.S. flag

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

  • Publications
  • Account settings
  • Advanced Search
  • Journal List
  • Trop Med Health

Logo of tropmedhealth

A step by step guide for conducting a systematic review and meta-analysis with simulation data

Gehad mohamed tawfik.

1 Faculty of Medicine, Ain Shams University, Cairo, Egypt

2 Online research Club, http://www.onlineresearchclub.org/

Kadek Agus Surya Dila

3 Pratama Giri Emas Hospital, Singaraja-Amlapura street, Giri Emas village, Sawan subdistrict, Singaraja City, Buleleng, Bali 81171 Indonesia

Muawia Yousif Fadlelmola Mohamed

4 Faculty of Medicine, University of Khartoum, Khartoum, Sudan

Dao Ngoc Hien Tam

5 Nanogen Pharmaceutical Biotechnology Joint Stock Company, Ho Chi Minh City, Vietnam

Nguyen Dang Kien

6 Department of Obstetrics and Gynecology, Thai Binh University of Medicine and Pharmacy, Thai Binh, Vietnam

Ali Mahmoud Ahmed

7 Faculty of Medicine, Al-Azhar University, Cairo, Egypt

Nguyen Tien Huy

8 Evidence Based Medicine Research Group & Faculty of Applied Sciences, Ton Duc Thang University, Ho Chi Minh City, 70000 Vietnam

9 Faculty of Applied Sciences, Ton Duc Thang University, Ho Chi Minh City, 70000 Vietnam

10 Department of Clinical Product Development, Institute of Tropical Medicine (NEKKEN), Leading Graduate School Program, and Graduate School of Biomedical Sciences, Nagasaki University, 1-12-4 Sakamoto, Nagasaki, 852-8523 Japan

Associated Data

Not applicable.

The massive abundance of studies relating to tropical medicine and health has increased strikingly over the last few decades. In the field of tropical medicine and health, a well-conducted systematic review and meta-analysis (SR/MA) is considered a feasible solution for keeping clinicians abreast of current evidence-based medicine. Understanding of SR/MA steps is of paramount importance for its conduction. It is not easy to be done as there are obstacles that could face the researcher. To solve those hindrances, this methodology study aimed to provide a step-by-step approach mainly for beginners and junior researchers, in the field of tropical medicine and other health care fields, on how to properly conduct a SR/MA, in which all the steps here depicts our experience and expertise combined with the already well-known and accepted international guidance.

We suggest that all steps of SR/MA should be done independently by 2–3 reviewers’ discussion, to ensure data quality and accuracy.

SR/MA steps include the development of research question, forming criteria, search strategy, searching databases, protocol registration, title, abstract, full-text screening, manual searching, extracting data, quality assessment, data checking, statistical analysis, double data checking, and manuscript writing.

Electronic supplementary material

The online version of this article (10.1186/s41182-019-0165-6) contains supplementary material, which is available to authorized users.

Introduction

The amount of studies published in the biomedical literature, especially tropical medicine and health, has increased strikingly over the last few decades. This massive abundance of literature makes clinical medicine increasingly complex, and knowledge from various researches is often needed to inform a particular clinical decision. However, available studies are often heterogeneous with regard to their design, operational quality, and subjects under study and may handle the research question in a different way, which adds to the complexity of evidence and conclusion synthesis [ 1 ].

Systematic review and meta-analyses (SR/MAs) have a high level of evidence as represented by the evidence-based pyramid. Therefore, a well-conducted SR/MA is considered a feasible solution in keeping health clinicians ahead regarding contemporary evidence-based medicine.

Differing from a systematic review, unsystematic narrative review tends to be descriptive, in which the authors select frequently articles based on their point of view which leads to its poor quality. A systematic review, on the other hand, is defined as a review using a systematic method to summarize evidence on questions with a detailed and comprehensive plan of study. Furthermore, despite the increasing guidelines for effectively conducting a systematic review, we found that basic steps often start from framing question, then identifying relevant work which consists of criteria development and search for articles, appraise the quality of included studies, summarize the evidence, and interpret the results [ 2 , 3 ]. However, those simple steps are not easy to be reached in reality. There are many troubles that a researcher could be struggled with which has no detailed indication.

Conducting a SR/MA in tropical medicine and health may be difficult especially for young researchers; therefore, understanding of its essential steps is crucial. It is not easy to be done as there are obstacles that could face the researcher. To solve those hindrances, we recommend a flow diagram (Fig. ​ (Fig.1) 1 ) which illustrates a detailed and step-by-step the stages for SR/MA studies. This methodology study aimed to provide a step-by-step approach mainly for beginners and junior researchers, in the field of tropical medicine and other health care fields, on how to properly and succinctly conduct a SR/MA; all the steps here depicts our experience and expertise combined with the already well known and accepted international guidance.

An external file that holds a picture, illustration, etc.
Object name is 41182_2019_165_Fig1_HTML.jpg

Detailed flow diagram guideline for systematic review and meta-analysis steps. Note : Star icon refers to “2–3 reviewers screen independently”

Methods and results

Detailed steps for conducting any systematic review and meta-analysis.

We searched the methods reported in published SR/MA in tropical medicine and other healthcare fields besides the published guidelines like Cochrane guidelines {Higgins, 2011 #7} [ 4 ] to collect the best low-bias method for each step of SR/MA conduction steps. Furthermore, we used guidelines that we apply in studies for all SR/MA steps. We combined these methods in order to conclude and conduct a detailed flow diagram that shows the SR/MA steps how being conducted.

Any SR/MA must follow the widely accepted Preferred Reporting Items for Systematic Review and Meta-analysis statement (PRISMA checklist 2009) (Additional file 5 : Table S1) [ 5 ].

We proposed our methods according to a valid explanatory simulation example choosing the topic of “evaluating safety of Ebola vaccine,” as it is known that Ebola is a very rare tropical disease but fatal. All the explained methods feature the standards followed internationally, with our compiled experience in the conduct of SR beside it, which we think proved some validity. This is a SR under conduct by a couple of researchers teaming in a research group, moreover, as the outbreak of Ebola which took place (2013–2016) in Africa resulted in a significant mortality and morbidity. Furthermore, since there are many published and ongoing trials assessing the safety of Ebola vaccines, we thought this would provide a great opportunity to tackle this hotly debated issue. Moreover, Ebola started to fire again and new fatal outbreak appeared in the Democratic Republic of Congo since August 2018, which caused infection to more than 1000 people according to the World Health Organization, and 629 people have been killed till now. Hence, it is considered the second worst Ebola outbreak, after the first one in West Africa in 2014 , which infected more than 26,000 and killed about 11,300 people along outbreak course.

Research question and objectives

Like other study designs, the research question of SR/MA should be feasible, interesting, novel, ethical, and relevant. Therefore, a clear, logical, and well-defined research question should be formulated. Usually, two common tools are used: PICO or SPIDER. PICO (Population, Intervention, Comparison, Outcome) is used mostly in quantitative evidence synthesis. Authors demonstrated that PICO holds more sensitivity than the more specific SPIDER approach [ 6 ]. SPIDER (Sample, Phenomenon of Interest, Design, Evaluation, Research type) was proposed as a method for qualitative and mixed methods search.

We here recommend a combined approach of using either one or both the SPIDER and PICO tools to retrieve a comprehensive search depending on time and resources limitations. When we apply this to our assumed research topic, being of qualitative nature, the use of SPIDER approach is more valid.

PICO is usually used for systematic review and meta-analysis of clinical trial study. For the observational study (without intervention or comparator), in many tropical and epidemiological questions, it is usually enough to use P (Patient) and O (outcome) only to formulate a research question. We must indicate clearly the population (P), then intervention (I) or exposure. Next, it is necessary to compare (C) the indicated intervention with other interventions, i.e., placebo. Finally, we need to clarify which are our relevant outcomes.

To facilitate comprehension, we choose the Ebola virus disease (EVD) as an example. Currently, the vaccine for EVD is being developed and under phase I, II, and III clinical trials; we want to know whether this vaccine is safe and can induce sufficient immunogenicity to the subjects.

An example of a research question for SR/MA based on PICO for this issue is as follows: How is the safety and immunogenicity of Ebola vaccine in human? (P: healthy subjects (human), I: vaccination, C: placebo, O: safety or adverse effects)

Preliminary research and idea validation

We recommend a preliminary search to identify relevant articles, ensure the validity of the proposed idea, avoid duplication of previously addressed questions, and assure that we have enough articles for conducting its analysis. Moreover, themes should focus on relevant and important health-care issues, consider global needs and values, reflect the current science, and be consistent with the adopted review methods. Gaining familiarity with a deep understanding of the study field through relevant videos and discussions is of paramount importance for better retrieval of results. If we ignore this step, our study could be canceled whenever we find out a similar study published before. This means we are wasting our time to deal with a problem that has been tackled for a long time.

To do this, we can start by doing a simple search in PubMed or Google Scholar with search terms Ebola AND vaccine. While doing this step, we identify a systematic review and meta-analysis of determinant factors influencing antibody response from vaccination of Ebola vaccine in non-human primate and human [ 7 ], which is a relevant paper to read to get a deeper insight and identify gaps for better formulation of our research question or purpose. We can still conduct systematic review and meta-analysis of Ebola vaccine because we evaluate safety as a different outcome and different population (only human).

Inclusion and exclusion criteria

Eligibility criteria are based on the PICO approach, study design, and date. Exclusion criteria mostly are unrelated, duplicated, unavailable full texts, or abstract-only papers. These exclusions should be stated in advance to refrain the researcher from bias. The inclusion criteria would be articles with the target patients, investigated interventions, or the comparison between two studied interventions. Briefly, it would be articles which contain information answering our research question. But the most important is that it should be clear and sufficient information, including positive or negative, to answer the question.

For the topic we have chosen, we can make inclusion criteria: (1) any clinical trial evaluating the safety of Ebola vaccine and (2) no restriction regarding country, patient age, race, gender, publication language, and date. Exclusion criteria are as follows: (1) study of Ebola vaccine in non-human subjects or in vitro studies; (2) study with data not reliably extracted, duplicate, or overlapping data; (3) abstract-only papers as preceding papers, conference, editorial, and author response theses and books; (4) articles without available full text available; and (5) case reports, case series, and systematic review studies. The PRISMA flow diagram template that is used in SR/MA studies can be found in Fig. ​ Fig.2 2 .

An external file that holds a picture, illustration, etc.
Object name is 41182_2019_165_Fig2_HTML.jpg

PRISMA flow diagram of studies’ screening and selection

Search strategy

A standard search strategy is used in PubMed, then later it is modified according to each specific database to get the best relevant results. The basic search strategy is built based on the research question formulation (i.e., PICO or PICOS). Search strategies are constructed to include free-text terms (e.g., in the title and abstract) and any appropriate subject indexing (e.g., MeSH) expected to retrieve eligible studies, with the help of an expert in the review topic field or an information specialist. Additionally, we advise not to use terms for the Outcomes as their inclusion might hinder the database being searched to retrieve eligible studies because the used outcome is not mentioned obviously in the articles.

The improvement of the search term is made while doing a trial search and looking for another relevant term within each concept from retrieved papers. To search for a clinical trial, we can use these descriptors in PubMed: “clinical trial”[Publication Type] OR “clinical trials as topic”[MeSH terms] OR “clinical trial”[All Fields]. After some rounds of trial and refinement of search term, we formulate the final search term for PubMed as follows: (ebola OR ebola virus OR ebola virus disease OR EVD) AND (vaccine OR vaccination OR vaccinated OR immunization) AND (“clinical trial”[Publication Type] OR “clinical trials as topic”[MeSH Terms] OR “clinical trial”[All Fields]). Because the study for this topic is limited, we do not include outcome term (safety and immunogenicity) in the search term to capture more studies.

Search databases, import all results to a library, and exporting to an excel sheet

According to the AMSTAR guidelines, at least two databases have to be searched in the SR/MA [ 8 ], but as you increase the number of searched databases, you get much yield and more accurate and comprehensive results. The ordering of the databases depends mostly on the review questions; being in a study of clinical trials, you will rely mostly on Cochrane, mRCTs, or International Clinical Trials Registry Platform (ICTRP). Here, we propose 12 databases (PubMed, Scopus, Web of Science, EMBASE, GHL, VHL, Cochrane, Google Scholar, Clinical trials.gov , mRCTs, POPLINE, and SIGLE), which help to cover almost all published articles in tropical medicine and other health-related fields. Among those databases, POPLINE focuses on reproductive health. Researchers should consider to choose relevant database according to the research topic. Some databases do not support the use of Boolean or quotation; otherwise, there are some databases that have special searching way. Therefore, we need to modify the initial search terms for each database to get appreciated results; therefore, manipulation guides for each online database searches are presented in Additional file 5 : Table S2. The detailed search strategy for each database is found in Additional file 5 : Table S3. The search term that we created in PubMed needs customization based on a specific characteristic of the database. An example for Google Scholar advanced search for our topic is as follows:

With at least one of the words: vaccine vaccination vaccinated immunization

Where my words occur: in the title of the article

Finally, all records are collected into one Endnote library in order to delete duplicates and then to it export into an excel sheet. Using remove duplicating function with two options is mandatory. All references which have (1) the same title and author, and published in the same year, and (2) the same title and author, and published in the same journal, would be deleted. References remaining after this step should be exported to an excel file with essential information for screening. These could be the authors’ names, publication year, journal, DOI, URL link, and abstract.

Protocol writing and registration

Protocol registration at an early stage guarantees transparency in the research process and protects from duplication problems. Besides, it is considered a documented proof of team plan of action, research question, eligibility criteria, intervention/exposure, quality assessment, and pre-analysis plan. It is recommended that researchers send it to the principal investigator (PI) to revise it, then upload it to registry sites. There are many registry sites available for SR/MA like those proposed by Cochrane and Campbell collaborations; however, we recommend registering the protocol into PROSPERO as it is easier. The layout of a protocol template, according to PROSPERO, can be found in Additional file 5 : File S1.

Title and abstract screening

Decisions to select retrieved articles for further assessment are based on eligibility criteria, to minimize the chance of including non-relevant articles. According to the Cochrane guidance, two reviewers are a must to do this step, but as for beginners and junior researchers, this might be tiresome; thus, we propose based on our experience that at least three reviewers should work independently to reduce the chance of error, particularly in teams with a large number of authors to add more scrutiny and ensure proper conduct. Mostly, the quality with three reviewers would be better than two, as two only would have different opinions from each other, so they cannot decide, while the third opinion is crucial. And here are some examples of systematic reviews which we conducted following the same strategy (by a different group of researchers in our research group) and published successfully, and they feature relevant ideas to tropical medicine and disease [ 9 – 11 ].

In this step, duplications will be removed manually whenever the reviewers find them out. When there is a doubt about an article decision, the team should be inclusive rather than exclusive, until the main leader or PI makes a decision after discussion and consensus. All excluded records should be given exclusion reasons.

Full text downloading and screening

Many search engines provide links for free to access full-text articles. In case not found, we can search in some research websites as ResearchGate, which offer an option of direct full-text request from authors. Additionally, exploring archives of wanted journals, or contacting PI to purchase it if available. Similarly, 2–3 reviewers work independently to decide about included full texts according to eligibility criteria, with reporting exclusion reasons of articles. In case any disagreement has occurred, the final decision has to be made by discussion.

Manual search

One has to exhaust all possibilities to reduce bias by performing an explicit hand-searching for retrieval of reports that may have been dropped from first search [ 12 ]. We apply five methods to make manual searching: searching references from included studies/reviews, contacting authors and experts, and looking at related articles/cited articles in PubMed and Google Scholar.

We describe here three consecutive methods to increase and refine the yield of manual searching: firstly, searching reference lists of included articles; secondly, performing what is known as citation tracking in which the reviewers track all the articles that cite each one of the included articles, and this might involve electronic searching of databases; and thirdly, similar to the citation tracking, we follow all “related to” or “similar” articles. Each of the abovementioned methods can be performed by 2–3 independent reviewers, and all the possible relevant article must undergo further scrutiny against the inclusion criteria, after following the same records yielded from electronic databases, i.e., title/abstract and full-text screening.

We propose an independent reviewing by assigning each member of the teams a “tag” and a distinct method, to compile all the results at the end for comparison of differences and discussion and to maximize the retrieval and minimize the bias. Similarly, the number of included articles has to be stated before addition to the overall included records.

Data extraction and quality assessment

This step entitles data collection from included full-texts in a structured extraction excel sheet, which is previously pilot-tested for extraction using some random studies. We recommend extracting both adjusted and non-adjusted data because it gives the most allowed confounding factor to be used in the analysis by pooling them later [ 13 ]. The process of extraction should be executed by 2–3 independent reviewers. Mostly, the sheet is classified into the study and patient characteristics, outcomes, and quality assessment (QA) tool.

Data presented in graphs should be extracted by software tools such as Web plot digitizer [ 14 ]. Most of the equations that can be used in extraction prior to analysis and estimation of standard deviation (SD) from other variables is found inside Additional file 5 : File S2 with their references as Hozo et al. [ 15 ], Xiang et al. [ 16 ], and Rijkom et al. [ 17 ]. A variety of tools are available for the QA, depending on the design: ROB-2 Cochrane tool for randomized controlled trials [ 18 ] which is presented as Additional file 1 : Figure S1 and Additional file 2 : Figure S2—from a previous published article data—[ 19 ], NIH tool for observational and cross-sectional studies [ 20 ], ROBINS-I tool for non-randomize trials [ 21 ], QUADAS-2 tool for diagnostic studies, QUIPS tool for prognostic studies, CARE tool for case reports, and ToxRtool for in vivo and in vitro studies. We recommend that 2–3 reviewers independently assess the quality of the studies and add to the data extraction form before the inclusion into the analysis to reduce the risk of bias. In the NIH tool for observational studies—cohort and cross-sectional—as in this EBOLA case, to evaluate the risk of bias, reviewers should rate each of the 14 items into dichotomous variables: yes, no, or not applicable. An overall score is calculated by adding all the items scores as yes equals one, while no and NA equals zero. A score will be given for every paper to classify them as poor, fair, or good conducted studies, where a score from 0–5 was considered poor, 6–9 as fair, and 10–14 as good.

In the EBOLA case example above, authors can extract the following information: name of authors, country of patients, year of publication, study design (case report, cohort study, or clinical trial or RCT), sample size, the infected point of time after EBOLA infection, follow-up interval after vaccination time, efficacy, safety, adverse effects after vaccinations, and QA sheet (Additional file 6 : Data S1).

Data checking

Due to the expected human error and bias, we recommend a data checking step, in which every included article is compared with its counterpart in an extraction sheet by evidence photos, to detect mistakes in data. We advise assigning articles to 2–3 independent reviewers, ideally not the ones who performed the extraction of those articles. When resources are limited, each reviewer is assigned a different article than the one he extracted in the previous stage.

Statistical analysis

Investigators use different methods for combining and summarizing findings of included studies. Before analysis, there is an important step called cleaning of data in the extraction sheet, where the analyst organizes extraction sheet data in a form that can be read by analytical software. The analysis consists of 2 types namely qualitative and quantitative analysis. Qualitative analysis mostly describes data in SR studies, while quantitative analysis consists of two main types: MA and network meta-analysis (NMA). Subgroup, sensitivity, cumulative analyses, and meta-regression are appropriate for testing whether the results are consistent or not and investigating the effect of certain confounders on the outcome and finding the best predictors. Publication bias should be assessed to investigate the presence of missing studies which can affect the summary.

To illustrate basic meta-analysis, we provide an imaginary data for the research question about Ebola vaccine safety (in terms of adverse events, 14 days after injection) and immunogenicity (Ebola virus antibodies rise in geometric mean titer, 6 months after injection). Assuming that from searching and data extraction, we decided to do an analysis to evaluate Ebola vaccine “A” safety and immunogenicity. Other Ebola vaccines were not meta-analyzed because of the limited number of studies (instead, it will be included for narrative review). The imaginary data for vaccine safety meta-analysis can be accessed in Additional file 7 : Data S2. To do the meta-analysis, we can use free software, such as RevMan [ 22 ] or R package meta [ 23 ]. In this example, we will use the R package meta. The tutorial of meta package can be accessed through “General Package for Meta-Analysis” tutorial pdf [ 23 ]. The R codes and its guidance for meta-analysis done can be found in Additional file 5 : File S3.

For the analysis, we assume that the study is heterogenous in nature; therefore, we choose a random effect model. We did an analysis on the safety of Ebola vaccine A. From the data table, we can see some adverse events occurring after intramuscular injection of vaccine A to the subject of the study. Suppose that we include six studies that fulfill our inclusion criteria. We can do a meta-analysis for each of the adverse events extracted from the studies, for example, arthralgia, from the results of random effect meta-analysis using the R meta package.

From the results shown in Additional file 3 : Figure S3, we can see that the odds ratio (OR) of arthralgia is 1.06 (0.79; 1.42), p value = 0.71, which means that there is no association between the intramuscular injection of Ebola vaccine A and arthralgia, as the OR is almost one, and besides, the P value is insignificant as it is > 0.05.

In the meta-analysis, we can also visualize the results in a forest plot. It is shown in Fig. ​ Fig.3 3 an example of a forest plot from the simulated analysis.

An external file that holds a picture, illustration, etc.
Object name is 41182_2019_165_Fig3_HTML.jpg

Random effect model forest plot for comparison of vaccine A versus placebo

From the forest plot, we can see six studies (A to F) and their respective OR (95% CI). The green box represents the effect size (in this case, OR) of each study. The bigger the box means the study weighted more (i.e., bigger sample size). The blue diamond shape represents the pooled OR of the six studies. We can see the blue diamond cross the vertical line OR = 1, which indicates no significance for the association as the diamond almost equalized in both sides. We can confirm this also from the 95% confidence interval that includes one and the p value > 0.05.

For heterogeneity, we see that I 2 = 0%, which means no heterogeneity is detected; the study is relatively homogenous (it is rare in the real study). To evaluate publication bias related to the meta-analysis of adverse events of arthralgia, we can use the metabias function from the R meta package (Additional file 4 : Figure S4) and visualization using a funnel plot. The results of publication bias are demonstrated in Fig. ​ Fig.4. 4 . We see that the p value associated with this test is 0.74, indicating symmetry of the funnel plot. We can confirm it by looking at the funnel plot.

An external file that holds a picture, illustration, etc.
Object name is 41182_2019_165_Fig4_HTML.jpg

Publication bias funnel plot for comparison of vaccine A versus placebo

Looking at the funnel plot, the number of studies at the left and right side of the funnel plot is the same; therefore, the plot is symmetry, indicating no publication bias detected.

Sensitivity analysis is a procedure used to discover how different values of an independent variable will influence the significance of a particular dependent variable by removing one study from MA. If all included study p values are < 0.05, hence, removing any study will not change the significant association. It is only performed when there is a significant association, so if the p value of MA done is 0.7—more than one—the sensitivity analysis is not needed for this case study example. If there are 2 studies with p value > 0.05, removing any of the two studies will result in a loss of the significance.

Double data checking

For more assurance on the quality of results, the analyzed data should be rechecked from full-text data by evidence photos, to allow an obvious check for the PI of the study.

Manuscript writing, revision, and submission to a journal

Writing based on four scientific sections: introduction, methods, results, and discussion, mostly with a conclusion. Performing a characteristic table for study and patient characteristics is a mandatory step which can be found as a template in Additional file 5 : Table S3.

After finishing the manuscript writing, characteristics table, and PRISMA flow diagram, the team should send it to the PI to revise it well and reply to his comments and, finally, choose a suitable journal for the manuscript which fits with considerable impact factor and fitting field. We need to pay attention by reading the author guidelines of journals before submitting the manuscript.

The role of evidence-based medicine in biomedical research is rapidly growing. SR/MAs are also increasing in the medical literature. This paper has sought to provide a comprehensive approach to enable reviewers to produce high-quality SR/MAs. We hope that readers could gain general knowledge about how to conduct a SR/MA and have the confidence to perform one, although this kind of study requires complex steps compared to narrative reviews.

Having the basic steps for conduction of MA, there are many advanced steps that are applied for certain specific purposes. One of these steps is meta-regression which is performed to investigate the association of any confounder and the results of the MA. Furthermore, there are other types rather than the standard MA like NMA and MA. In NMA, we investigate the difference between several comparisons when there were not enough data to enable standard meta-analysis. It uses both direct and indirect comparisons to conclude what is the best between the competitors. On the other hand, mega MA or MA of patients tend to summarize the results of independent studies by using its individual subject data. As a more detailed analysis can be done, it is useful in conducting repeated measure analysis and time-to-event analysis. Moreover, it can perform analysis of variance and multiple regression analysis; however, it requires homogenous dataset and it is time-consuming in conduct [ 24 ].

Conclusions

Systematic review/meta-analysis steps include development of research question and its validation, forming criteria, search strategy, searching databases, importing all results to a library and exporting to an excel sheet, protocol writing and registration, title and abstract screening, full-text screening, manual searching, extracting data and assessing its quality, data checking, conducting statistical analysis, double data checking, manuscript writing, revising, and submitting to a journal.

Additional files

Figure S1. Risk of bias assessment graph of included randomized controlled trials. (TIF 20 kb)

Figure S2. Risk of bias assessment summary. (TIF 69 kb)

Figure S3. Arthralgia results of random effect meta-analysis using R meta package. (TIF 20 kb)

Figure S4. Arthralgia linear regression test of funnel plot asymmetry using R meta package. (TIF 13 kb)

Table S1. PRISMA 2009 Checklist. Table S2. Manipulation guides for online database searches. Table S3. Detailed search strategy for twelve database searches. Table S4. Baseline characteristics of the patients in the included studies. File S1. PROSPERO protocol template file. File S2. Extraction equations that can be used prior to analysis to get missed variables. File S3. R codes and its guidance for meta-analysis done for comparison between EBOLA vaccine A and placebo. (DOCX 49 kb)

Data S1. Extraction and quality assessment data sheets for EBOLA case example. (XLSX 1368 kb)

Data S2. Imaginary data for EBOLA case example. (XLSX 10 kb)

Acknowledgements

Abbreviations, authors’ contributions.

NTH and GMT were responsible for the idea and its design. The figure was done by GMT. All authors contributed to the manuscript writing and approval of the final version.

This study was conducted (in part) at the Joint Usage/Research Center on Tropical Disease, Institute of Tropical Medicine, Nagasaki University, Japan.

Availability of data and materials

Ethics approval and consent to participate, consent for publication, competing interests.

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Contributor Information

Gehad Mohamed Tawfik, Email: [email protected] .

Kadek Agus Surya Dila, Email: moc.liamg@alidayrussugakedak .

Muawia Yousif Fadlelmola Mohamed, Email: moc.oohay@fusuyaiwaom .

Dao Ngoc Hien Tam, Email: moc.liamg@matneihcognoad .

Nguyen Dang Kien, Email: moc.liamg@1915neikgnadneyugn .

Ali Mahmoud Ahmed, Email: [email protected] .

Nguyen Tien Huy, Phone: 81-95-819-7558, Email: moc.oohay@yuhneitgn , Email: nv.ude.utdt@yuhneitneyugn .

A step by step guide for conducting a systematic review and meta-analysis with simulation data

Affiliations.

  • 1 1Faculty of Medicine, Ain Shams University, Cairo, Egypt.
  • 2 Online research Club, http://www.onlineresearchclub.org/.
  • 3 Pratama Giri Emas Hospital, Singaraja-Amlapura street, Giri Emas village, Sawan subdistrict, Singaraja City, Buleleng, Bali 81171 Indonesia.
  • 4 4Faculty of Medicine, University of Khartoum, Khartoum, Sudan.
  • 5 Nanogen Pharmaceutical Biotechnology Joint Stock Company, Ho Chi Minh City, Vietnam.
  • 6 6Department of Obstetrics and Gynecology, Thai Binh University of Medicine and Pharmacy, Thai Binh, Vietnam.
  • 7 7Faculty of Medicine, Al-Azhar University, Cairo, Egypt.
  • 8 8Evidence Based Medicine Research Group & Faculty of Applied Sciences, Ton Duc Thang University, Ho Chi Minh City, 70000 Vietnam.
  • 9 9Faculty of Applied Sciences, Ton Duc Thang University, Ho Chi Minh City, 70000 Vietnam.
  • 10 10Department of Clinical Product Development, Institute of Tropical Medicine (NEKKEN), Leading Graduate School Program, and Graduate School of Biomedical Sciences, Nagasaki University, 1-12-4 Sakamoto, Nagasaki, 852-8523 Japan.
  • PMID: 31388330
  • PMCID: PMC6670166
  • DOI: 10.1186/s41182-019-0165-6

Background: The massive abundance of studies relating to tropical medicine and health has increased strikingly over the last few decades. In the field of tropical medicine and health, a well-conducted systematic review and meta-analysis (SR/MA) is considered a feasible solution for keeping clinicians abreast of current evidence-based medicine. Understanding of SR/MA steps is of paramount importance for its conduction. It is not easy to be done as there are obstacles that could face the researcher. To solve those hindrances, this methodology study aimed to provide a step-by-step approach mainly for beginners and junior researchers, in the field of tropical medicine and other health care fields, on how to properly conduct a SR/MA, in which all the steps here depicts our experience and expertise combined with the already well-known and accepted international guidance.We suggest that all steps of SR/MA should be done independently by 2-3 reviewers' discussion, to ensure data quality and accuracy.

Conclusion: SR/MA steps include the development of research question, forming criteria, search strategy, searching databases, protocol registration, title, abstract, full-text screening, manual searching, extracting data, quality assessment, data checking, statistical analysis, double data checking, and manuscript writing.

Keywords: Analysis; Data; Extraction; Results; Search; Study.

Publication types

COMMENTS

  1. Steps of a Systematic Review

    Steps of SR (PIECES) Image by TraceyChandler Steps to conducting a systematic review Quick overview of the process: Steps and resources from the UMB HSHSL Guide. YouTube video (26 min) Another detailed guide on how to conduct and write a systematic review from RMIT University A roadmap for searching literature in PubMed from the VU Amsterdam

  2. PDF How to Write a Systematic Review: A Step-by-Step Guide

    1. Population: Define your subject group. Think about the age, sex, race and other patient characteristics, as well as relevant co-morbidities, pathology, and outcomes. 2. Intervention: Consider the prognostic factor or exposure (includes intervention) of interest. 3.

  3. Systematic Review

    Knowledge Base Methodology Systematic Review | Definition, Example, & Guide Systematic Review | Definition, Example & Guide Published on June 15, 2022 by Shaun Turney . Revised on November 20, 2023. A systematic review is a type of review that uses repeatable methods to find, select, and synthesize all available evidence.

  4. Five steps to conducting a systematic review

    STEP 1: FRAMING THE QUESTION The research question may initially be stated as a query in free form but reviewers prefer to pose it in a structured and explicit way. The relations between various components of the question and the structure of the research design are shown in Figure 1 .

  5. How to Do a Systematic Review: A Best Practice Guide for Conducting and

    How to Do a Systematic Review: A Best Practice Guide for Conducting and Reporting Narrative Reviews, Meta-Analyses, and Meta-Syntheses Annu Rev Psychol2019 Jan 4:70:747-770. doi: 10.1146/annurev-psych-010418-102803. Epub 2018 Aug 8. Andy P Siddaway 1 , Alex M Wood 2 , Larry V Hedges 3 30089228 10.1146/annurev-psych-010418-102803

  6. Home

    Publication Date: 2019-12-02 In this open access edited volume, international researchers of the field describe and discuss the systematic review method in its application to research in education.

  7. LibGuides: Systematic Reviews: Review Process: Step by Step

    The below handbooks provide step by step guidance for conducting a systematic review. You may be required to follow a specific review standard or protocol, or you may use handbooks for more general guidance. This guide frequently references the Cochrane Handbook. Cochrane Handbook for Systematic Reviews of Interventions

  8. Systematic Reviews & Evidence Synthesis Methods

    Steps in a Systematic Review 0. Develop a Protocol 1. Draft your Research Question 2. Select Databases 3. Select Grey Literature Sources 4. Write a Search Strategy 5. Register a Protocol 6. Translate Search Strategies 7. Citation Management 8.

  9. Systematic Reviews: Step 8: Write the Review

    You will: Review the reporting standards you will use, such as PRISMA. Gather your completed data tables and PRISMA chart. Write the Introduction to the topic and your study, Methods of your research, Results of your research, and Discussion of your results. Write an Abstract describing your study and a Conclusion summarizing your paper.

  10. Conducting a Systematic Review: A Practical Guide

    Abstract. It can be challenging to conduct a systematic review with limited experience and skills in undertaking such a task. This chapter provides a practical guide to undertaking a systematic review, providing step-by-step instructions to guide the individual through the process from start to finish. The chapter begins with defining what a ...

  11. A step by step guide for conducting a systematic review and meta

    SR/MA steps include the development of research question, forming criteria, search strategy, searching databases, protocol registration, title, abstract, full-text screening, manual searching, extracting data, quality assessment, data checking, statistical analysis, double data checking, and manuscript writing. Introduction

  12. Easy guide to conducting a systematic review

    This paper provides a brief step by step summary of how to conduct a systematic review, which may be of interest for clinicians and researchers. Keywords: research; research design; systematic review. © 2020 Paediatrics and Child Health Division (The Royal Australasian College of Physicians).

  13. Conduct a systematic review: step-by-step guide: Home

    This page provides a step-by-step guide focusing on how to conduct a systematic review (Synthesis without a Meta-Analysis (SWiM) or Systematic Review with a Meta-Analysis). If you are looking for information focused on other types of reviews, such as rapid reviews, scoping reviews, umbrella reviews and living systematic reviews, please also ...

  14. What are the steps in a Systematic Review?

    STEP 1. Identify your research question A well defined research question should address a gap in the current literature and is the essential starting point of your synthesis. You can use the following frameworks to help construct your research question. PICO for Quantitative Studies P Population/Problem I Intervention/Exposure C Comparison

  15. Chapter 1: Starting a review

    Cochrane Training Trusted evidence. Informed decisions. Better health. Chapter 1: Starting a review Toby J Lasserson, James Thomas, Julian PT Higgins Key Points: Systematic reviews address a need for health decision makers to be able to access high quality, relevant, accessible and up-to-date information.

  16. Systematic Reviews: Steps in a Systematic Review

    Use a spreadsheet, or systematic review software, to extract all relevant data from each included study. It is recommended that you pilot your data extraction tool, to determine if other fields should be included or existing fields clarified. Evaluate the risk of bias of included studies . Use a Risk of Bias tool (such as the Cochrane RoB Tool ...

  17. Systematic Literature Review Made EASY: A Step-by-Step Guide

    You will learn everything you need about systematic literature review. Following my instructions, many published their systematic literature reviews in top t...

  18. Systematic reviews and meta-analyses: an illustrated, step-by-step guide

    PMID: 15141602 Abstract Systematic reviews and meta-analyses synthesize data from existing primary research, and well-conducted reviews offer clinicians a practical solution to the problem of staying current in their fields of interest.

  19. A step-by-step guide to the systematic review and meta-analysis of

    A step-by-step guide to the systematic review and meta-analysis of diagnostic and prognostic test accuracy evaluations Download PDF. Download PDF. Review; Open access; Published: 21 ...

  20. A step by step guide for conducting a systematic review and meta

    While doing this step, we identify a systematic review and meta-analysis of determinant factors influencing antibody response from vaccination of Ebola vaccine in non-human primate and human , which is a relevant paper to read to get a deeper insight and identify gaps for better formulation of our research question or purpose. We can still ...

  21. (PDF) BEGINNER'S GUIDE FOR SYSTEMATIC REVIEWS A step by step guide to

    BEGINNER'S GUIDE FOR SYSTEMATIC REVIEWS A step by step guide to conduct systematic reviews and Meta-Analysis (An ICMR Publication) Authors: Anju Sinha Indian Council of Medical...

  22. A step by step guide for conducting a systematic review and meta

    SR/MA steps include the development of research question, forming criteria, search strategy, searching databases, protocol registration, title, abstract, full-text screening, manual searching, extracting data, quality assessment, data checking, statistical analysis, double data checking, and manuscr …

  23. A systematic review of creativity evaluation and creativity selection

    Evaluating and selecting creative ideas are vital steps in the transition from creativity to innovation. However, research has only recently turned its focus on performance in idea evaluation (i.e., assessment of available options against certain standards such as originality and usefulness) and idea selection (i.e., a dichotomous decision whether to select ideas for further development or ...

  24. AA Nowbuth on Instagram: "Reading research papers can be a daunting

    dr_anyanowbuth on April 26, 2023: "Reading research papers can be a daunting task, especially for those who are new to the field or ..."