Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

  • Knowledge Base

Methodology

  • Systematic Review | Definition, Example, & Guide

Systematic Review | Definition, Example & Guide

Published on June 15, 2022 by Shaun Turney . Revised on November 20, 2023.

A systematic review is a type of review that uses repeatable methods to find, select, and synthesize all available evidence. It answers a clearly formulated research question and explicitly states the methods used to arrive at the answer.

They answered the question “What is the effectiveness of probiotics in reducing eczema symptoms and improving quality of life in patients with eczema?”

In this context, a probiotic is a health product that contains live microorganisms and is taken by mouth. Eczema is a common skin condition that causes red, itchy skin.

Table of contents

What is a systematic review, systematic review vs. meta-analysis, systematic review vs. literature review, systematic review vs. scoping review, when to conduct a systematic review, pros and cons of systematic reviews, step-by-step example of a systematic review, other interesting articles, frequently asked questions about systematic reviews.

A review is an overview of the research that’s already been completed on a topic.

What makes a systematic review different from other types of reviews is that the research methods are designed to reduce bias . The methods are repeatable, and the approach is formal and systematic:

  • Formulate a research question
  • Develop a protocol
  • Search for all relevant studies
  • Apply the selection criteria
  • Extract the data
  • Synthesize the data
  • Write and publish a report

Although multiple sets of guidelines exist, the Cochrane Handbook for Systematic Reviews is among the most widely used. It provides detailed guidelines on how to complete each step of the systematic review process.

Systematic reviews are most commonly used in medical and public health research, but they can also be found in other disciplines.

Systematic reviews typically answer their research question by synthesizing all available evidence and evaluating the quality of the evidence. Synthesizing means bringing together different information to tell a single, cohesive story. The synthesis can be narrative ( qualitative ), quantitative , or both.

Here's why students love Scribbr's proofreading services

Discover proofreading & editing

Systematic reviews often quantitatively synthesize the evidence using a meta-analysis . A meta-analysis is a statistical analysis, not a type of review.

A meta-analysis is a technique to synthesize results from multiple studies. It’s a statistical analysis that combines the results of two or more studies, usually to estimate an effect size .

A literature review is a type of review that uses a less systematic and formal approach than a systematic review. Typically, an expert in a topic will qualitatively summarize and evaluate previous work, without using a formal, explicit method.

Although literature reviews are often less time-consuming and can be insightful or helpful, they have a higher risk of bias and are less transparent than systematic reviews.

Similar to a systematic review, a scoping review is a type of review that tries to minimize bias by using transparent and repeatable methods.

However, a scoping review isn’t a type of systematic review. The most important difference is the goal: rather than answering a specific question, a scoping review explores a topic. The researcher tries to identify the main concepts, theories, and evidence, as well as gaps in the current research.

Sometimes scoping reviews are an exploratory preparation step for a systematic review, and sometimes they are a standalone project.

Receive feedback on language, structure, and formatting

Professional editors proofread and edit your paper by focusing on:

  • Academic style
  • Vague sentences
  • Style consistency

See an example

what is a qualitative systematic literature review

A systematic review is a good choice of review if you want to answer a question about the effectiveness of an intervention , such as a medical treatment.

To conduct a systematic review, you’ll need the following:

  • A precise question , usually about the effectiveness of an intervention. The question needs to be about a topic that’s previously been studied by multiple researchers. If there’s no previous research, there’s nothing to review.
  • If you’re doing a systematic review on your own (e.g., for a research paper or thesis ), you should take appropriate measures to ensure the validity and reliability of your research.
  • Access to databases and journal archives. Often, your educational institution provides you with access.
  • Time. A professional systematic review is a time-consuming process: it will take the lead author about six months of full-time work. If you’re a student, you should narrow the scope of your systematic review and stick to a tight schedule.
  • Bibliographic, word-processing, spreadsheet, and statistical software . For example, you could use EndNote, Microsoft Word, Excel, and SPSS.

A systematic review has many pros .

  • They minimize research bias by considering all available evidence and evaluating each study for bias.
  • Their methods are transparent , so they can be scrutinized by others.
  • They’re thorough : they summarize all available evidence.
  • They can be replicated and updated by others.

Systematic reviews also have a few cons .

  • They’re time-consuming .
  • They’re narrow in scope : they only answer the precise research question.

The 7 steps for conducting a systematic review are explained with an example.

Step 1: Formulate a research question

Formulating the research question is probably the most important step of a systematic review. A clear research question will:

  • Allow you to more effectively communicate your research to other researchers and practitioners
  • Guide your decisions as you plan and conduct your systematic review

A good research question for a systematic review has four components, which you can remember with the acronym PICO :

  • Population(s) or problem(s)
  • Intervention(s)
  • Comparison(s)

You can rearrange these four components to write your research question:

  • What is the effectiveness of I versus C for O in P ?

Sometimes, you may want to include a fifth component, the type of study design . In this case, the acronym is PICOT .

  • Type of study design(s)
  • The population of patients with eczema
  • The intervention of probiotics
  • In comparison to no treatment, placebo , or non-probiotic treatment
  • The outcome of changes in participant-, parent-, and doctor-rated symptoms of eczema and quality of life
  • Randomized control trials, a type of study design

Their research question was:

  • What is the effectiveness of probiotics versus no treatment, a placebo, or a non-probiotic treatment for reducing eczema symptoms and improving quality of life in patients with eczema?

Step 2: Develop a protocol

A protocol is a document that contains your research plan for the systematic review. This is an important step because having a plan allows you to work more efficiently and reduces bias.

Your protocol should include the following components:

  • Background information : Provide the context of the research question, including why it’s important.
  • Research objective (s) : Rephrase your research question as an objective.
  • Selection criteria: State how you’ll decide which studies to include or exclude from your review.
  • Search strategy: Discuss your plan for finding studies.
  • Analysis: Explain what information you’ll collect from the studies and how you’ll synthesize the data.

If you’re a professional seeking to publish your review, it’s a good idea to bring together an advisory committee . This is a group of about six people who have experience in the topic you’re researching. They can help you make decisions about your protocol.

It’s highly recommended to register your protocol. Registering your protocol means submitting it to a database such as PROSPERO or ClinicalTrials.gov .

Step 3: Search for all relevant studies

Searching for relevant studies is the most time-consuming step of a systematic review.

To reduce bias, it’s important to search for relevant studies very thoroughly. Your strategy will depend on your field and your research question, but sources generally fall into these four categories:

  • Databases: Search multiple databases of peer-reviewed literature, such as PubMed or Scopus . Think carefully about how to phrase your search terms and include multiple synonyms of each word. Use Boolean operators if relevant.
  • Handsearching: In addition to searching the primary sources using databases, you’ll also need to search manually. One strategy is to scan relevant journals or conference proceedings. Another strategy is to scan the reference lists of relevant studies.
  • Gray literature: Gray literature includes documents produced by governments, universities, and other institutions that aren’t published by traditional publishers. Graduate student theses are an important type of gray literature, which you can search using the Networked Digital Library of Theses and Dissertations (NDLTD) . In medicine, clinical trial registries are another important type of gray literature.
  • Experts: Contact experts in the field to ask if they have unpublished studies that should be included in your review.

At this stage of your review, you won’t read the articles yet. Simply save any potentially relevant citations using bibliographic software, such as Scribbr’s APA or MLA Generator .

  • Databases: EMBASE, PsycINFO, AMED, LILACS, and ISI Web of Science
  • Handsearch: Conference proceedings and reference lists of articles
  • Gray literature: The Cochrane Library, the metaRegister of Controlled Trials, and the Ongoing Skin Trials Register
  • Experts: Authors of unpublished registered trials, pharmaceutical companies, and manufacturers of probiotics

Step 4: Apply the selection criteria

Applying the selection criteria is a three-person job. Two of you will independently read the studies and decide which to include in your review based on the selection criteria you established in your protocol . The third person’s job is to break any ties.

To increase inter-rater reliability , ensure that everyone thoroughly understands the selection criteria before you begin.

If you’re writing a systematic review as a student for an assignment, you might not have a team. In this case, you’ll have to apply the selection criteria on your own; you can mention this as a limitation in your paper’s discussion.

You should apply the selection criteria in two phases:

  • Based on the titles and abstracts : Decide whether each article potentially meets the selection criteria based on the information provided in the abstracts.
  • Based on the full texts: Download the articles that weren’t excluded during the first phase. If an article isn’t available online or through your library, you may need to contact the authors to ask for a copy. Read the articles and decide which articles meet the selection criteria.

It’s very important to keep a meticulous record of why you included or excluded each article. When the selection process is complete, you can summarize what you did using a PRISMA flow diagram .

Next, Boyle and colleagues found the full texts for each of the remaining studies. Boyle and Tang read through the articles to decide if any more studies needed to be excluded based on the selection criteria.

When Boyle and Tang disagreed about whether a study should be excluded, they discussed it with Varigos until the three researchers came to an agreement.

Step 5: Extract the data

Extracting the data means collecting information from the selected studies in a systematic way. There are two types of information you need to collect from each study:

  • Information about the study’s methods and results . The exact information will depend on your research question, but it might include the year, study design , sample size, context, research findings , and conclusions. If any data are missing, you’ll need to contact the study’s authors.
  • Your judgment of the quality of the evidence, including risk of bias .

You should collect this information using forms. You can find sample forms in The Registry of Methods and Tools for Evidence-Informed Decision Making and the Grading of Recommendations, Assessment, Development and Evaluations Working Group .

Extracting the data is also a three-person job. Two people should do this step independently, and the third person will resolve any disagreements.

They also collected data about possible sources of bias, such as how the study participants were randomized into the control and treatment groups.

Step 6: Synthesize the data

Synthesizing the data means bringing together the information you collected into a single, cohesive story. There are two main approaches to synthesizing the data:

  • Narrative ( qualitative ): Summarize the information in words. You’ll need to discuss the studies and assess their overall quality.
  • Quantitative : Use statistical methods to summarize and compare data from different studies. The most common quantitative approach is a meta-analysis , which allows you to combine results from multiple studies into a summary result.

Generally, you should use both approaches together whenever possible. If you don’t have enough data, or the data from different studies aren’t comparable, then you can take just a narrative approach. However, you should justify why a quantitative approach wasn’t possible.

Boyle and colleagues also divided the studies into subgroups, such as studies about babies, children, and adults, and analyzed the effect sizes within each group.

Step 7: Write and publish a report

The purpose of writing a systematic review article is to share the answer to your research question and explain how you arrived at this answer.

Your article should include the following sections:

  • Abstract : A summary of the review
  • Introduction : Including the rationale and objectives
  • Methods : Including the selection criteria, search method, data extraction method, and synthesis method
  • Results : Including results of the search and selection process, study characteristics, risk of bias in the studies, and synthesis results
  • Discussion : Including interpretation of the results and limitations of the review
  • Conclusion : The answer to your research question and implications for practice, policy, or research

To verify that your report includes everything it needs, you can use the PRISMA checklist .

Once your report is written, you can publish it in a systematic review database, such as the Cochrane Database of Systematic Reviews , and/or in a peer-reviewed journal.

In their report, Boyle and colleagues concluded that probiotics cannot be recommended for reducing eczema symptoms or improving quality of life in patients with eczema. Note Generative AI tools like ChatGPT can be useful at various stages of the writing and research process and can help you to write your systematic review. However, we strongly advise against trying to pass AI-generated text off as your own work.

If you want to know more about statistics , methodology , or research bias , make sure to check out some of our other articles with explanations and examples.

  • Student’s  t -distribution
  • Normal distribution
  • Null and Alternative Hypotheses
  • Chi square tests
  • Confidence interval
  • Quartiles & Quantiles
  • Cluster sampling
  • Stratified sampling
  • Data cleansing
  • Reproducibility vs Replicability
  • Peer review
  • Prospective cohort study

Research bias

  • Implicit bias
  • Cognitive bias
  • Placebo effect
  • Hawthorne effect
  • Hindsight bias
  • Affect heuristic
  • Social desirability bias

A literature review is a survey of scholarly sources (such as books, journal articles, and theses) related to a specific topic or research question .

It is often written as part of a thesis, dissertation , or research paper , in order to situate your work in relation to existing knowledge.

A literature review is a survey of credible sources on a topic, often used in dissertations , theses, and research papers . Literature reviews give an overview of knowledge on a subject, helping you identify relevant theories and methods, as well as gaps in existing research. Literature reviews are set up similarly to other  academic texts , with an introduction , a main body, and a conclusion .

An  annotated bibliography is a list of  source references that has a short description (called an annotation ) for each of the sources. It is often assigned as part of the research process for a  paper .  

A systematic review is secondary research because it uses existing research. You don’t collect new data yourself.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

Turney, S. (2023, November 20). Systematic Review | Definition, Example & Guide. Scribbr. Retrieved February 19, 2024, from https://www.scribbr.com/methodology/systematic-review/

Is this article helpful?

Shaun Turney

Shaun Turney

Other students also liked, how to write a literature review | guide, examples, & templates, how to write a research proposal | examples & templates, what is critical thinking | definition & examples, what is your plagiarism score.

  • Locations and Hours
  • UCLA Library
  • Research Guides
  • Biomedical Library Guides

Systematic Reviews

  • Types of Literature Reviews

What Makes a Systematic Review Different from Other Types of Reviews?

  • Planning Your Systematic Review
  • Database Searching
  • Creating the Search
  • Search Filters & Hedges
  • Grey Literature
  • Managing & Appraising Results
  • Further Resources

Reproduced from Grant, M. J. and Booth, A. (2009), A typology of reviews: an analysis of 14 review types and associated methodologies. Health Information & Libraries Journal, 26: 91–108. doi:10.1111/j.1471-1842.2009.00848.x

  • << Previous: Home
  • Next: Planning Your Systematic Review >>
  • Last Updated: Feb 20, 2024 9:33 AM
  • URL: https://guides.library.ucla.edu/systematicreviews

Duke University Libraries

Literature Reviews

  • Getting started

What is a literature review?

Why conduct a literature review, stages of a literature review, lit reviews: an overview (video), check out these books.

  • Types of reviews
  • 1. Define your research question
  • 2. Plan your search
  • 3. Search the literature
  • 4. Organize your results
  • 5. Synthesize your findings
  • 6. Write the review
  • Thompson Writing Studio This link opens in a new window
  • Need to write a systematic review? This link opens in a new window

what is a qualitative systematic literature review

Contact a Librarian

Ask a Librarian

what is a qualitative systematic literature review

Definition: A literature review is a systematic examination and synthesis of existing scholarly research on a specific topic or subject.

Purpose: It serves to provide a comprehensive overview of the current state of knowledge within a particular field.

Analysis: Involves critically evaluating and summarizing key findings, methodologies, and debates found in academic literature.

Identifying Gaps: Aims to pinpoint areas where there is a lack of research or unresolved questions, highlighting opportunities for further investigation.

Contextualization: Enables researchers to understand how their work fits into the broader academic conversation and contributes to the existing body of knowledge.

what is a qualitative systematic literature review

tl;dr  A literature review critically examines and synthesizes existing scholarly research and publications on a specific topic to provide a comprehensive understanding of the current state of knowledge in the field.

What is a literature review NOT?

❌ An annotated bibliography

❌ Original research

❌ A summary

❌ Something to be conducted at the end of your research

❌ An opinion piece

❌ A chronological compilation of studies

The reason for conducting a literature review is to:

what is a qualitative systematic literature review

Literature Reviews: An Overview for Graduate Students

While this 9-minute video from NCSU is geared toward graduate students, it is useful for anyone conducting a literature review.

what is a qualitative systematic literature review

Writing the literature review: A practical guide

Available 3rd floor of Perkins

what is a qualitative systematic literature review

Writing literature reviews: A guide for students of the social and behavioral sciences

Available online!

what is a qualitative systematic literature review

So, you have to write a literature review: A guided workbook for engineers

what is a qualitative systematic literature review

Telling a research story: Writing a literature review

what is a qualitative systematic literature review

The literature review: Six steps to success

what is a qualitative systematic literature review

Systematic approaches to a successful literature review

Request from Duke Medical Center Library

what is a qualitative systematic literature review

Doing a systematic review: A student's guide

  • Next: Types of reviews >>
  • Last Updated: Feb 15, 2024 1:45 PM
  • URL: https://guides.library.duke.edu/lit-reviews

Duke University Libraries

Services for...

  • Faculty & Instructors
  • Graduate Students
  • Undergraduate Students
  • International Students
  • Patrons with Disabilities

Twitter

  • Harmful Language Statement
  • Re-use & Attribution / Privacy
  • Support the Libraries

Creative Commons License

Banner

Systematic Reviews

Who is this guide for and what can be found in it, what are systematic reviews, how do systematic reviews differ from narrative literature reviews.

  • Types of Systematic Reviews
  • Reading Systematic Reviews
  • Resources for Conducting Systematic Reviews
  • Getting Help with Systematic Reviews from the Library
  • History of Systematic Reviews
  • Acknowledgements

Contact Us!

Ask a question

This guide aims to support all OHSU members' systematic review education and activities, orienting OHSU members who are new to systematic reviews and facilitating the quality, rigor, and reproducibility of systematic reviews produced by OHSU members.

In it you will find:

  • A definition of what systematic reviews are, how they compare to other evidence, and how they differ from narrative literature reviews
  • Descriptions of the different types of systematic reviews , with links to resources on methods, protocols, reporting, additional information, and selecting the right type of systematic review for your research question
  • Guidance on how to read and evaluate systematic reviews for strength, quality, and potential for bias
  • A high-level overview of how systematic reviews are conducted , including team size and roles, standards, and processes
  • Links to resources and tools for conducting systematic reviews
  • Information about how to get assistance with conducting a systematic review from the OHSU Library
  • A history of systematic reviews to provide contextual understanding of how they have developed over time
"A systematic review is a summary of the medical literature that uses explicit and reproducible methods to systematically search, critically appraise, and synthesize on a specific issue. It synthesizes the results of multiple primary studies related to each other by using strategies that reduce biases and random errors."

Gopalakrishnan S, Ganeshkumar P. Systematic Reviews and Meta-analysis: Understanding the Best Evidence in Primary Healthcare . J Family Med Prim Care . 2013;2(1):9-14. doi:10.4103/2249-4863.109934

Systematic Reviews are a vital resource used in the pursuit of Evidence-Based Practice (EBP):

  • These studies can be found near the top of the Evidence Pyramid , which ranks sources of information and study designs by the level of evidence contained within them
  • This ranking is based on the level of scientific rigor employed in their methods and the quality and reliability of the evidence contained within these sources
  • A higher ranking means that we can be more confident that their conclusions are accurate and have taken measures to limit bias

Research design and evidence , by CFCF , CC BY-SA 4.0 , via Wikimedia Commons

Things to know about systematic reviews:

  • Systematic reviews are a type of research study
  • Systematic reviews aim to provide a comprehensive and unbiased summary of the existing evidence on a particular research question
  • There are many types of systematic reviews , each designed to address a specific type of research purpose and with their own strengths and weaknesses
  • The choice of what type of review to produce typically will depend on the nature of the research question and the resources that are available on the topic

The practice of producing systematic reviews is sometimes referred to by other names such as:

  • Evidence Synthesis
  • Knowledge Synthesis
  • Research Synthesis

This guide tries to stick with the term "Systematic Reviews" unless a specific type of systematic review is being discussed.

While all reviews combat information overload in the health sciences by summarizing the literature on a topic, different types of reviews have different approaches. The term systematic review is often conflated with narrative literature reviews , which can lead to confusion and misunderstandings when seeking help with conducting them. This table helps clarify the differences.

  • Next: Types of Systematic Reviews >>
  • Last Updated: Feb 12, 2024 5:59 PM
  • URL: https://libguides.ohsu.edu/systematic-reviews
  • Open access
  • Published: 15 December 2015

Qualitative and mixed methods in systematic reviews

  • David Gough 1  

Systematic Reviews volume  4 , Article number:  181 ( 2015 ) Cite this article

21k Accesses

47 Citations

23 Altmetric

Metrics details

Expanding the range of methods of systematic review

The logic of systematic reviews is very simple. We use transparent rigorous approaches to undertake primary research, and so we should do the same in bringing together studies to describe what has been studied (a research map) or to integrate the findings of the different studies to answer a research question (a research synthesis). We should not really need to use the term ‘systematic’ as it should be assumed that researchers are using and reporting systematic methods in all of their research, whether primary or secondary. Despite the universality of this logic, systematic reviews (maps and syntheses) are much better known in health research and for answering questions of the effectiveness of interventions (what works). Systematic reviews addressing other sorts of questions have been around for many years, as in, for example, meta ethnography [ 1 ] and other forms of conceptual synthesis [ 2 ], but only recently has there been a major increase in the use of systematic review approaches to answer other sorts of research questions.

There are probably several reasons for this broadening of approach. One may be that the increased awareness of systematic reviews has made people consider the possibilities for all areas of research. A second related factor may be that more training and funding resources have become available and increased the capacity to undertake such varied review work.

A third reason could be that some of the initial anxieties about systematic reviews have subsided. Initially, there were concerns that their use was being promoted by a new managerialism where reviews, particularly effectiveness reviews, were being used to promote particular ideological and theoretical assumptions and to indirectly control research agendas. However, others like me believe that explicit methods should be used to enable transparency of perspectives driving research and to open up access to and participation in research agendas and priority setting [ 3 ] as illustrated, for example, by the James Lind Alliance (see http://www.jla.nihr.ac.uk/ ).

A fourth possible reason for the development of new approaches is that effectiveness reviews have themselves broadened. Some ‘what works’ reviews can be open to criticism for only testing a ‘black box’ hypothesis of what works with little theorizing or any logic model about why any such hypothesis should be true and the mechanisms involved in such processes. There is now more concern to develop theory and to test how variables combine and interact. In primary research, qualitative strategies are advised prior to undertaking experimental trials [ 4 , 5 ] and similar approaches are being advocated to address complexity in reviews [ 6 ], in order to ask questions and use methods that address theories and processes that enable an understanding of both impact and context.

This Special Issue of Systematic Reviews Journal is providing a focus for these new methods of review whether these use qualitative review methods on their own or mixed together with more quantitative approaches. We are linking together with the sister journal Trials for this Special Issue as there is a similar interest in what qualitative approaches can and should contribute to primary research using experimentally controlled trials (see Trials Special Issue editorial by Claire Snowdon).

Dimensions of difference in reviews

Developing the range of methods to address different questions for review creates a challenge in describing and understanding such methods. There are many names and brands for the new methods which may or may not withstand the changes of historical time, but another way to comprehend the changes and new developments is to consider the dimensions on which the approaches to review differ [ 7 , 8 ].

One important distinction is the research question being asked and the associated paradigm underlying the method used to address this question. Research assumes a particular theoretical position and then gathers data within this conceptual lens. In some cases, this is a very specific hypothesis that is then tested empirically, and sometimes, the research is more exploratory and iterative with concepts being emergent and constructed during the research process. This distinction is often labelled as quantitative or positivist versus qualitative or constructionist. However, this can be confusing as much research taking a ‘quantitative’ perspective does not have the necessary numeric data to analyse. Even if it does have such data, this might be explored for emergent properties. Similarly, research taking a ‘qualitative’ perspective may include implicit quantitative themes in terms of the extent of different qualitative findings reported by a study.

Sandelowski and colleagues’ solution is to consider the analytic activity and whether this aggregates (adds up) or configures (arranges) the data [ 9 ]. In a randomized controlled trial and an effectiveness review of such studies, the main analysis is the aggregation of data using a priori non-emergent strategies with little iteration. However, there may also be post hoc analysis that is more exploratory in arranging (configuring) data to identify patterns as in, for example, meta regression or qualitative comparative analysis aiming to identify the active ingredients of effective interventions [ 10 ]. Similarly, qualitative primary research or reviews of such research are predominantly exploring emergent patterns and developing concepts iteratively, yet there may be some aggregation of data to make statements of generalizations of extent.

Even where the analysis is predominantly configuration, there can be a wide variation in the dimensions of difference of iteration of theories and concepts. In thematic synthesis [ 11 ], there may be few presumptions about the concepts that will be configured. In meta ethnography which can be richer in theory, there may be theoretical assumptions underlying the review question framing the analysis. In framework synthesis, there is an explicit conceptual framework that is iteratively developed and changed through the review process [ 12 , 13 ].

In addition to the variation in question, degree of configuration, complexity of theory, and iteration are many other dimensions of difference between reviews. Some of these differences follow on from the research questions being asked and the research paradigm being used such as in the approach to searching (exhaustive or based on exploration or saturation) and the appraisal of the quality and relevance of included studies (based more on risk of bias or more on meaning). Others include the extent that reviews have a broad question, depth of analysis, and the extent of resultant ‘work done’ in terms of progressing a field of inquiry [ 7 , 8 ].

Mixed methods reviews

As one reason for the growth in qualitative synthesis is what they can add to quantitative reviews, it is not surprising that there is also growing interest in mixed methods reviews. This reflects similar developments in primary research in mixing methods to examine the relationship between theory and empirical data which is of course the cornerstone of much research. But, both primary and secondary mixed methods research also face similar challenges in examining complex questions at different levels of analysis and of combining research findings investigated in different ways and may be based on very different epistemological assumptions [ 14 , 15 ].

Some mixed methods approaches are convergent in that they integrate different data and methods of analysis together at the same time [ 16 , 17 ]. Convergent systematic reviews could be described as having broad inclusion criteria (or two or more different sets of criteria) for methods of primary studies and have special methods for the synthesis of the resultant variation in data. Other reviews (and also primary mixed methods studies) are sequences of sub-reviews in that one sub-study using one research paradigm is followed by another sub-study with a different research paradigm. In other words, a qualitative synthesis might be used to explore the findings of a prior quantitative synthesis or vice versa [ 16 , 17 ].

An example of a predominantly aggregative sub-review followed by a configuring sub-review is the EPPI-Centre’s mixed methods review of barriers to healthy eating [ 18 ]. A sub-review on the effectiveness of public health interventions showed a modest effect size. A configuring review of studies of children and young people’s understanding and views about eating provided evidence that the public health interventions did not take good account of such user views research, and that the interventions most closely aligned to the user views were the most effective. The already mentioned qualitative comparative analysis to identify the active ingredients within interventions leading to impact could also be considered a qualitative configuring investigation of an existing quantitative aggregative review [ 10 ].

An example of a predominantly configurative review followed by an aggregative review is realist synthesis. Realist reviews examine the evidence in support of mid-range theories [ 19 ] with a first stage of a configuring review of what is proposed by the theory or proposal (what would need to be in place and what casual pathways would have to be effective for the outcomes proposed by the theory to be supported?) and a second stage searching for empirical evidence to test for those necessary conditions and effectiveness of the pathways. The empirical testing does not however use a standard ‘what works’ a priori methods approach but rather a more iterative seeking out of evidence that confirms or undermines the theory being evaluated [ 20 ].

Although sequential mixed methods approaches are considered to be sub-parts of one larger study, they could be separate studies as part of a long-term strategic approach to studying an issue. We tend to see both primary studies and reviews as one-off events, yet reviews are a way of examining what we know and what more we want to know as a strategic approach to studying an issue over time. If we are in favour of mixing paradigms of research to enable multiple levels and perspectives and mixing of theory development and empirical evaluation, then we are really seeking mixed methods research strategies rather than simply mixed methods studies and reviews.

Noblit G. Hare RD: meta-ethnography: synthesizing qualitative studies. Newbury Park NY: Sage Publications; 1988.

Google Scholar  

Barnett-Page E, Thomas J. Methods for the synthesis of qualitative research: a critical review. BMC Med Res Methodol. 2009;9:59.

Article   PubMed   PubMed Central   Google Scholar  

Gough D, Elbourne D. Systematic research synthesis to inform policy, practice and democratic debate. Soc Pol Soc. 2002;2002:1.

Moore GF, Audrey S, Barker M, Bond L, Bonell C, Hardeman W, et al. Process evaluation of complex interventions: Medical Research Council guidance 2015. BMJ. 2015;350:h1258

Candy B, Jone L, King M, Oliver S. Using qualitative evidence to help understand complex palliative care interventions: a novel evidence synthesis approach. BMJ Support Palliat Care. 2014;4:Supp A41–A42.

Article   Google Scholar  

Noyes J, Gough D, Lewin S, Mayhew A, Michie S, Pantoja T, et al. A research and development agenda for systematic reviews that ask complex questions about complex interventions. J Clin Epidemiol. 2013;66:11.

Gough D, Oliver S, Thomas J. Introduction to systematic reviews. London: Sage; 2012.

Gough D, Thomas J, Oliver S. Clarifying differences between review designs and methods. Syst Rev. 2012;1:28.

Sandelowski M, Voils CJ, Leeman J, Crandlee JL. Mapping the mixed methods-mixed research synthesis terrain. J Mix Methods Res. 2012;6:4.

Thomas J, O’Mara-Eves A, Brunton G. Using qualitative comparative analysis (QCA) in systematic reviews of complex interventions: a worked example. Syst Rev. 2014;3:67.

Thomas J, Harden A. Methods for the thematic synthesis of qualitative research in systematic reviews. BMC Med Res Methodol. 2008;8:45.

Oliver S, Rees R, Clarke-Jones L, Milne R, Oakley AR, Gabbay J, et al. A multidimensional conceptual framework for analysing public involvement in health services research. Health Exp. 2008;11:72–84.

Booth A, Carroll C. How to build up the actionable knowledge base: the role of ‘best fit’ framework synthesis for studies of improvement in healthcare. BMJ Qual Saf. 2015. 2014-003642.

Brannen J. Mixed methods research: a discussion paper. NCRM Methods Review Papers, 2006. NCRM/005.

Creswell J. Mapping the developing landscape of mixed methods research. In: Teddlie C, Tashakkori A, editors. SAGE handbook of mixed methods in social & behavioral research. New York: Sage; 2011.

Morse JM. Principles of mixed method and multi-method research design. In: Teddlie C, Tashakkori A, editors. Handbook of mixed methods in social and behavioural research. London: Sage; 2003.

Pluye P, Hong QN. Combining the power of stories and the power of numbers: mixed methods research and mixed studies reviews. Annu Rev Public Health. 2014;35:29–45.

Harden A, Thomas J. Mixed methods and systematic reviews: examples and emerging issues. In: Tashakkori A, Teddlie C, editors. Handbook of mixed methods in the social and behavioral sciences. 2nd ed. London: Sage; 2010. p. 749–74.

Chapter   Google Scholar  

Pawson R. Evidenced-based policy: a realist perspective. London: Sage; 2006.

Book   Google Scholar  

Gough D. Meta-narrative and realist reviews: guidance, rules, publication standards and quality appraisal. BMC Med. 2013;11:22.

Download references

Author information

Authors and affiliations.

EPPI-Centre, Social Science Research Unit, University College London, London, WC1H 0NR, UK

David Gough

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to David Gough .

Additional information

Competing interests.

The author is a writer and researcher in this area. The author declares that he has no other competing interests.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Cite this article.

Gough, D. Qualitative and mixed methods in systematic reviews. Syst Rev 4 , 181 (2015). https://doi.org/10.1186/s13643-015-0151-y

Download citation

Received : 13 October 2015

Accepted : 29 October 2015

Published : 15 December 2015

DOI : https://doi.org/10.1186/s13643-015-0151-y

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Systematic Reviews

ISSN: 2046-4053

  • Submission enquiries: Access here and click Contact Us
  • General enquiries: [email protected]

what is a qualitative systematic literature review

Qualitative systematic reviews: their importance for our understanding of research relevant to pain

Affiliation.

  • 1 RCN Research Institute, Warwick Medical School, University of Warwick, Coventry, UK.
  • PMID: 26516555
  • PMCID: PMC4616987
  • DOI: 10.1177/2049463714549777

This article outlines what a qualitative systematic review is and explores what it can contribute to our understanding of pain. Many of us use evidence of effectiveness for various interventions when working with people in pain. A good systematic review can be invaluable in bringing together research evidence to help inform our practice and help us understand what works. In addition to evidence of effectiveness, understanding how people with pain experience both their pain and their care can help us when we are working with them to provide care that meets their needs. A rigorous qualitative systematic review can also uncover new understandings, often helping illuminate 'why' and can help build theory. Such a review can answer the question 'What is it like to have chronic pain?' This article presents the different stages of meta-ethnography, which is the most common methodology used for qualitative systematic reviews. It presents evidence from four meta-ethnographies relevant to pain to illustrate the types of findings that can emerge from this approach. It shows how new understandings may emerge and gives an example of chronic musculoskeletal pain being experienced as 'an adversarial struggle' across many aspects of the person's life. This article concludes that evidence from qualitative systematic reviews has its place alongside or integrated with evidence from more quantitative approaches.

Keywords: Qualitative systematic review; meta-ethnography; qualitative synthesis.

  • Systematic review
  • Open access
  • Published: 19 February 2024

‘It depends’: what 86 systematic reviews tell us about what strategies to use to support the use of research in clinical practice

  • Annette Boaz   ORCID: orcid.org/0000-0003-0557-1294 1 ,
  • Juan Baeza 2 ,
  • Alec Fraser   ORCID: orcid.org/0000-0003-1121-1551 2 &
  • Erik Persson 3  

Implementation Science volume  19 , Article number:  15 ( 2024 ) Cite this article

19 Altmetric

Metrics details

The gap between research findings and clinical practice is well documented and a range of strategies have been developed to support the implementation of research into clinical practice. The objective of this study was to update and extend two previous reviews of systematic reviews of strategies designed to implement research evidence into clinical practice.

We developed a comprehensive systematic literature search strategy based on the terms used in the previous reviews to identify studies that looked explicitly at interventions designed to turn research evidence into practice. The search was performed in June 2022 in four electronic databases: Medline, Embase, Cochrane and Epistemonikos. We searched from January 2010 up to June 2022 and applied no language restrictions. Two independent reviewers appraised the quality of included studies using a quality assessment checklist. To reduce the risk of bias, papers were excluded following discussion between all members of the team. Data were synthesised using descriptive and narrative techniques to identify themes and patterns linked to intervention strategies, targeted behaviours, study settings and study outcomes.

We identified 32 reviews conducted between 2010 and 2022. The reviews are mainly of multi-faceted interventions ( n  = 20) although there are reviews focusing on single strategies (ICT, educational, reminders, local opinion leaders, audit and feedback, social media and toolkits). The majority of reviews report strategies achieving small impacts (normally on processes of care). There is much less evidence that these strategies have shifted patient outcomes. Furthermore, a lot of nuance lies behind these headline findings, and this is increasingly commented upon in the reviews themselves.

Combined with the two previous reviews, 86 systematic reviews of strategies to increase the implementation of research into clinical practice have been identified. We need to shift the emphasis away from isolating individual and multi-faceted interventions to better understanding and building more situated, relational and organisational capability to support the use of research in clinical practice. This will involve drawing on a wider range of research perspectives (including social science) in primary studies and diversifying the types of synthesis undertaken to include approaches such as realist synthesis which facilitate exploration of the context in which strategies are employed.

Peer Review reports

Contribution to the literature

Considerable time and money is invested in implementing and evaluating strategies to increase the implementation of research into clinical practice.

The growing body of evidence is not providing the anticipated clear lessons to support improved implementation.

Instead what is needed is better understanding and building more situated, relational and organisational capability to support the use of research in clinical practice.

This would involve a more central role in implementation science for a wider range of perspectives, especially from the social, economic, political and behavioural sciences and for greater use of different types of synthesis, such as realist synthesis.

Introduction

The gap between research findings and clinical practice is well documented and a range of interventions has been developed to increase the implementation of research into clinical practice [ 1 , 2 ]. In recent years researchers have worked to improve the consistency in the ways in which these interventions (often called strategies) are described to support their evaluation. One notable development has been the emergence of Implementation Science as a field focusing explicitly on “the scientific study of methods to promote the systematic uptake of research findings and other evidence-based practices into routine practice” ([ 3 ] p. 1). The work of implementation science focuses on closing, or at least narrowing, the gap between research and practice. One contribution has been to map existing interventions, identifying 73 discreet strategies to support research implementation [ 4 ] which have been grouped into 9 clusters [ 5 ]. The authors note that they have not considered the evidence of effectiveness of the individual strategies and that a next step is to understand better which strategies perform best in which combinations and for what purposes [ 4 ]. Other authors have noted that there is also scope to learn more from other related fields of study such as policy implementation [ 6 ] and to draw on methods designed to support the evaluation of complex interventions [ 7 ].

The increase in activity designed to support the implementation of research into practice and improvements in reporting provided the impetus for an update of a review of systematic reviews of the effectiveness of interventions designed to support the use of research in clinical practice [ 8 ] which was itself an update of the review conducted by Grimshaw and colleagues in 2001. The 2001 review [ 9 ] identified 41 reviews considering a range of strategies including educational interventions, audit and feedback, computerised decision support to financial incentives and combined interventions. The authors concluded that all the interventions had the potential to promote the uptake of evidence in practice, although no one intervention seemed to be more effective than the others in all settings. They concluded that combined interventions were more likely to be effective than single interventions. The 2011 review identified a further 13 systematic reviews containing 313 discrete primary studies. Consistent with the previous review, four main strategy types were identified: audit and feedback; computerised decision support; opinion leaders; and multi-faceted interventions (MFIs). Nine of the reviews reported on MFIs. The review highlighted the small effects of single interventions such as audit and feedback, computerised decision support and opinion leaders. MFIs claimed an improvement in effectiveness over single interventions, although effect sizes remained small to moderate and this improvement in effectiveness relating to MFIs has been questioned in a subsequent review [ 10 ]. In updating the review, we anticipated a larger pool of reviews and an opportunity to consolidate learning from more recent systematic reviews of interventions.

This review updates and extends our previous review of systematic reviews of interventions designed to implement research evidence into clinical practice. To identify potentially relevant peer-reviewed research papers, we developed a comprehensive systematic literature search strategy based on the terms used in the Grimshaw et al. [ 9 ] and Boaz, Baeza and Fraser [ 8 ] overview articles. To ensure optimal retrieval, our search strategy was refined with support from an expert university librarian, considering the ongoing improvements in the development of search filters for systematic reviews since our first review [ 11 ]. We also wanted to include technology-related terms (e.g. apps, algorithms, machine learning, artificial intelligence) to find studies that explored interventions based on the use of technological innovations as mechanistic tools for increasing the use of evidence into practice (see Additional file 1 : Appendix A for full search strategy).

The search was performed in June 2022 in the following electronic databases: Medline, Embase, Cochrane and Epistemonikos. We searched for articles published since the 2011 review. We searched from January 2010 up to June 2022 and applied no language restrictions. Reference lists of relevant papers were also examined.

We uploaded the results using EPPI-Reviewer, a web-based tool that facilitated semi-automation of the screening process and removal of duplicate studies. We made particular use of a priority screening function to reduce screening workload and avoid ‘data deluge’ [ 12 ]. Through machine learning, one reviewer screened a smaller number of records ( n  = 1200) to train the software to predict whether a given record was more likely to be relevant or irrelevant, thus pulling the relevant studies towards the beginning of the screening process. This automation did not replace manual work but helped the reviewer to identify eligible studies more quickly. During the selection process, we included studies that looked explicitly at interventions designed to turn research evidence into practice. Studies were included if they met the following pre-determined inclusion criteria:

The study was a systematic review

Search terms were included

Focused on the implementation of research evidence into practice

The methodological quality of the included studies was assessed as part of the review

Study populations included healthcare providers and patients. The EPOC taxonomy [ 13 ] was used to categorise the strategies. The EPOC taxonomy has four domains: delivery arrangements, financial arrangements, governance arrangements and implementation strategies. The implementation strategies domain includes 20 strategies targeted at healthcare workers. Numerous EPOC strategies were assessed in the review including educational strategies, local opinion leaders, reminders, ICT-focused approaches and audit and feedback. Some strategies that did not fit easily within the EPOC categories were also included. These were social media strategies and toolkits, and multi-faceted interventions (MFIs) (see Table  2 ). Some systematic reviews included comparisons of different interventions while other reviews compared one type of intervention against a control group. Outcomes related to improvements in health care processes or patient well-being. Numerous individual study types (RCT, CCT, BA, ITS) were included within the systematic reviews.

We excluded papers that:

Focused on changing patient rather than provider behaviour

Had no demonstrable outcomes

Made unclear or no reference to research evidence

The last of these criteria was sometimes difficult to judge, and there was considerable discussion amongst the research team as to whether the link between research evidence and practice was sufficiently explicit in the interventions analysed. As we discussed in the previous review [ 8 ] in the field of healthcare, the principle of evidence-based practice is widely acknowledged and tools to change behaviour such as guidelines are often seen to be an implicit codification of evidence, despite the fact that this is not always the case.

Reviewers employed a two-stage process to select papers for inclusion. First, all titles and abstracts were screened by one reviewer to determine whether the study met the inclusion criteria. Two papers [ 14 , 15 ] were identified that fell just before the 2010 cut-off. As they were not identified in the searches for the first review [ 8 ] they were included and progressed to assessment. Each paper was rated as include, exclude or maybe. The full texts of 111 relevant papers were assessed independently by at least two authors. To reduce the risk of bias, papers were excluded following discussion between all members of the team. 32 papers met the inclusion criteria and proceeded to data extraction. The study selection procedure is documented in a PRISMA literature flow diagram (see Fig.  1 ). We were able to include French, Spanish and Portuguese papers in the selection reflecting the language skills in the study team, but none of the papers identified met the inclusion criteria. Other non- English language papers were excluded.

figure 1

PRISMA flow diagram. Source: authors

One reviewer extracted data on strategy type, number of included studies, local, target population, effectiveness and scope of impact from the included studies. Two reviewers then independently read each paper and noted key findings and broad themes of interest which were then discussed amongst the wider authorial team. Two independent reviewers appraised the quality of included studies using a Quality Assessment Checklist based on Oxman and Guyatt [ 16 ] and Francke et al. [ 17 ]. Each study was rated a quality score ranging from 1 (extensive flaws) to 7 (minimal flaws) (see Additional file 2 : Appendix B). All disagreements were resolved through discussion. Studies were not excluded in this updated overview based on methodological quality as we aimed to reflect the full extent of current research into this topic.

The extracted data were synthesised using descriptive and narrative techniques to identify themes and patterns in the data linked to intervention strategies, targeted behaviours, study settings and study outcomes.

Thirty-two studies were included in the systematic review. Table 1. provides a detailed overview of the included systematic reviews comprising reference, strategy type, quality score, number of included studies, local, target population, effectiveness and scope of impact (see Table  1. at the end of the manuscript). Overall, the quality of the studies was high. Twenty-three studies scored 7, six studies scored 6, one study scored 5, one study scored 4 and one study scored 3. The primary focus of the review was on reviews of effectiveness studies, but a small number of reviews did include data from a wider range of methods including qualitative studies which added to the analysis in the papers [ 18 , 19 , 20 , 21 ]. The majority of reviews report strategies achieving small impacts (normally on processes of care). There is much less evidence that these strategies have shifted patient outcomes. In this section, we discuss the different EPOC-defined implementation strategies in turn. Interestingly, we found only two ‘new’ approaches in this review that did not fit into the existing EPOC approaches. These are a review focused on the use of social media and a review considering toolkits. In addition to single interventions, we also discuss multi-faceted interventions. These were the most common intervention approach overall. A summary is provided in Table  2 .

Educational strategies

The overview identified three systematic reviews focusing on educational strategies. Grudniewicz et al. [ 22 ] explored the effectiveness of printed educational materials on primary care physician knowledge, behaviour and patient outcomes and concluded they were not effective in any of these aspects. Koota, Kääriäinen and Melender [ 23 ] focused on educational interventions promoting evidence-based practice among emergency room/accident and emergency nurses and found that interventions involving face-to-face contact led to significant or highly significant effects on patient benefits and emergency nurses’ knowledge, skills and behaviour. Interventions using written self-directed learning materials also led to significant improvements in nurses’ knowledge of evidence-based practice. Although the quality of the studies was high, the review primarily included small studies with low response rates, and many of them relied on self-assessed outcomes; consequently, the strength of the evidence for these outcomes is modest. Wu et al. [ 20 ] questioned if educational interventions aimed at nurses to support the implementation of evidence-based practice improve patient outcomes. Although based on evaluation projects and qualitative data, their results also suggest that positive changes on patient outcomes can be made following the implementation of specific evidence-based approaches (or projects). The differing positive outcomes for educational strategies aimed at nurses might indicate that the target audience is important.

Local opinion leaders

Flodgren et al. [ 24 ] was the only systemic review focusing solely on opinion leaders. The review found that local opinion leaders alone, or in combination with other interventions, can be effective in promoting evidence‐based practice, but this varies both within and between studies and the effect on patient outcomes is uncertain. The review found that, overall, any intervention involving opinion leaders probably improves healthcare professionals’ compliance with evidence-based practice but varies within and across studies. However, how opinion leaders had an impact could not be determined because of insufficient details were provided, illustrating that reporting specific details in published studies is important if diffusion of effective methods of increasing evidence-based practice is to be spread across a system. The usefulness of this review is questionable because it cannot provide evidence of what is an effective opinion leader, whether teams of opinion leaders or a single opinion leader are most effective, or the most effective methods used by opinion leaders.

Pantoja et al. [ 26 ] was the only systemic review focusing solely on manually generated reminders delivered on paper included in the overview. The review explored how these affected professional practice and patient outcomes. The review concluded that manually generated reminders delivered on paper as a single intervention probably led to small to moderate increases in adherence to clinical recommendations, and they could be used as a single quality improvement intervention. However, the authors indicated that this intervention would make little or no difference to patient outcomes. The authors state that such a low-tech intervention may be useful in low- and middle-income countries where paper records are more likely to be the norm.

ICT-focused approaches

The three ICT-focused reviews [ 14 , 27 , 28 ] showed mixed results. Jamal, McKenzie and Clark [ 14 ] explored the impact of health information technology on the quality of medical and health care. They examined the impact of electronic health record, computerised provider order-entry, or decision support system. This showed a positive improvement in adherence to evidence-based guidelines but not to patient outcomes. The number of studies included in the review was low and so a conclusive recommendation could not be reached based on this review. Similarly, Brown et al. [ 28 ] found that technology-enabled knowledge translation interventions may improve knowledge of health professionals, but all eight studies raised concerns of bias. The De Angelis et al. [ 27 ] review was more promising, reporting that ICT can be a good way of disseminating clinical practice guidelines but conclude that it is unclear which type of ICT method is the most effective.

Audit and feedback

Sykes, McAnuff and Kolehmainen [ 29 ] examined whether audit and feedback were effective in dementia care and concluded that it remains unclear which ingredients of audit and feedback are successful as the reviewed papers illustrated large variations in the effectiveness of interventions using audit and feedback.

Non-EPOC listed strategies: social media, toolkits

There were two new (non-EPOC listed) intervention types identified in this review compared to the 2011 review — fewer than anticipated. We categorised a third — ‘care bundles’ [ 36 ] as a multi-faceted intervention due to its description in practice and a fourth — ‘Technology Enhanced Knowledge Transfer’ [ 28 ] was classified as an ICT-focused approach. The first new strategy was identified in Bhatt et al.’s [ 30 ] systematic review of the use of social media for the dissemination of clinical practice guidelines. They reported that the use of social media resulted in a significant improvement in knowledge and compliance with evidence-based guidelines compared with more traditional methods. They noted that a wide selection of different healthcare professionals and patients engaged with this type of social media and its global reach may be significant for low- and middle-income countries. This review was also noteworthy for developing a simple stepwise method for using social media for the dissemination of clinical practice guidelines. However, it is debatable whether social media can be classified as an intervention or just a different way of delivering an intervention. For example, the review discussed involving opinion leaders and patient advocates through social media. However, this was a small review that included only five studies, so further research in this new area is needed. Yamada et al. [ 31 ] draw on 39 studies to explore the application of toolkits, 18 of which had toolkits embedded within larger KT interventions, and 21 of which evaluated toolkits as standalone interventions. The individual component strategies of the toolkits were highly variable though the authors suggest that they align most closely with educational strategies. The authors conclude that toolkits as either standalone strategies or as part of MFIs hold some promise for facilitating evidence use in practice but caution that the quality of many of the primary studies included is considered weak limiting these findings.

Multi-faceted interventions

The majority of the systematic reviews ( n  = 20) reported on more than one intervention type. Some of these systematic reviews focus exclusively on multi-faceted interventions, whilst others compare different single or combined interventions aimed at achieving similar outcomes in particular settings. While these two approaches are often described in a similar way, they are actually quite distinct from each other as the former report how multiple strategies may be strategically combined in pursuance of an agreed goal, whilst the latter report how different strategies may be incidentally used in sometimes contrasting settings in the pursuance of similar goals. Ariyo et al. [ 35 ] helpfully summarise five key elements often found in effective MFI strategies in LMICs — but which may also be transferrable to HICs. First, effective MFIs encourage a multi-disciplinary approach acknowledging the roles played by different professional groups to collectively incorporate evidence-informed practice. Second, they utilise leadership drawing on a wide set of clinical and non-clinical actors including managers and even government officials. Third, multiple types of educational practices are utilised — including input from patients as stakeholders in some cases. Fourth, protocols, checklists and bundles are used — most effectively when local ownership is encouraged. Finally, most MFIs included an emphasis on monitoring and evaluation [ 35 ]. In contrast, other studies offer little information about the nature of the different MFI components of included studies which makes it difficult to extrapolate much learning from them in relation to why or how MFIs might affect practice (e.g. [ 28 , 38 ]). Ultimately, context matters, which some review authors argue makes it difficult to say with real certainty whether single or MFI strategies are superior (e.g. [ 21 , 27 ]). Taking all the systematic reviews together we may conclude that MFIs appear to be more likely to generate positive results than single interventions (e.g. [ 34 , 45 ]) though other reviews should make us cautious (e.g. [ 32 , 43 ]).

While multi-faceted interventions still seem to be more effective than single-strategy interventions, there were important distinctions between how the results of reviews of MFIs are interpreted in this review as compared to the previous reviews [ 8 , 9 ], reflecting greater nuance and debate in the literature. This was particularly noticeable where the effectiveness of MFIs was compared to single strategies, reflecting developments widely discussed in previous studies [ 10 ]. We found that most systematic reviews are bounded by their clinical, professional, spatial, system, or setting criteria and often seek to draw out implications for the implementation of evidence in their areas of specific interest (such as nursing or acute care). Frequently this means combining all relevant studies to explore the respective foci of each systematic review. Therefore, most reviews we categorised as MFIs actually include highly variable numbers and combinations of intervention strategies and highly heterogeneous original study designs. This makes statistical analyses of the type used by Squires et al. [ 10 ] on the three reviews in their paper not possible. Further, it also makes extrapolating findings and commenting on broad themes complex and difficult. This may suggest that future research should shift its focus from merely examining ‘what works’ to ‘what works where and what works for whom’ — perhaps pointing to the value of realist approaches to these complex review topics [ 48 , 49 ] and other more theory-informed approaches [ 50 ].

Some reviews have a relatively small number of studies (i.e. fewer than 10) and the authors are often understandably reluctant to engage with wider debates about the implications of their findings. Other larger studies do engage in deeper discussions about internal comparisons of findings across included studies and also contextualise these in wider debates. Some of the most informative studies (e.g. [ 35 , 40 ]) move beyond EPOC categories and contextualise MFIs within wider systems thinking and implementation theory. This distinction between MFIs and single interventions can actually be very useful as it offers lessons about the contexts in which individual interventions might have bounded effectiveness (i.e. educational interventions for individual change). Taken as a whole, this may also then help in terms of how and when to conjoin single interventions into effective MFIs.

In the two previous reviews, a consistent finding was that MFIs were more effective than single interventions [ 8 , 9 ]. However, like Squires et al. [ 10 ] this overview is more equivocal on this important issue. There are four points which may help account for the differences in findings in this regard. Firstly, the diversity of the systematic reviews in terms of clinical topic or setting is an important factor. Secondly, there is heterogeneity of the studies within the included systematic reviews themselves. Thirdly, there is a lack of consistency with regards to the definition and strategies included within of MFIs. Finally, there are epistemological differences across the papers and the reviews. This means that the results that are presented depend on the methods used to measure, report, and synthesise them. For instance, some reviews highlight that education strategies can be useful to improve provider understanding — but without wider organisational or system-level change, they may struggle to deliver sustained transformation [ 19 , 44 ].

It is also worth highlighting the importance of the theory of change underlying the different interventions. Where authors of the systematic reviews draw on theory, there is space to discuss/explain findings. We note a distinction between theoretical and atheoretical systematic review discussion sections. Atheoretical reviews tend to present acontextual findings (for instance, one study found very positive results for one intervention, and this gets highlighted in the abstract) whilst theoretically informed reviews attempt to contextualise and explain patterns within the included studies. Theory-informed systematic reviews seem more likely to offer more profound and useful insights (see [ 19 , 35 , 40 , 43 , 45 ]). We find that the most insightful systematic reviews of MFIs engage in theoretical generalisation — they attempt to go beyond the data of individual studies and discuss the wider implications of the findings of the studies within their reviews drawing on implementation theory. At the same time, they highlight the active role of context and the wider relational and system-wide issues linked to implementation. It is these types of investigations that can help providers further develop evidence-based practice.

This overview has identified a small, but insightful set of papers that interrogate and help theorise why, how, for whom, and in which circumstances it might be the case that MFIs are superior (see [ 19 , 35 , 40 ] once more). At the level of this overview — and in most of the systematic reviews included — it appears to be the case that MFIs struggle with the question of attribution. In addition, there are other important elements that are often unmeasured, or unreported (e.g. costs of the intervention — see [ 40 ]). Finally, the stronger systematic reviews [ 19 , 35 , 40 , 43 , 45 ] engage with systems issues, human agency and context [ 18 ] in a way that was not evident in the systematic reviews identified in the previous reviews [ 8 , 9 ]. The earlier reviews lacked any theory of change that might explain why MFIs might be more effective than single ones — whereas now some systematic reviews do this, which enables them to conclude that sometimes single interventions can still be more effective.

As Nilsen et al. ([ 6 ] p. 7) note ‘Study findings concerning the effectiveness of various approaches are continuously synthesized and assembled in systematic reviews’. We may have gone as far as we can in understanding the implementation of evidence through systematic reviews of single and multi-faceted interventions and the next step would be to conduct more research exploring the complex and situated nature of evidence used in clinical practice and by particular professional groups. This would further build on the nuanced discussion and conclusion sections in a subset of the papers we reviewed. This might also support the field to move away from isolating individual implementation strategies [ 6 ] to explore the complex processes involving a range of actors with differing capacities [ 51 ] working in diverse organisational cultures. Taxonomies of implementation strategies do not fully account for the complex process of implementation, which involves a range of different actors with different capacities and skills across multiple system levels. There is plenty of work to build on, particularly in the social sciences, which currently sits at the margins of debates about evidence implementation (see for example, Normalisation Process Theory [ 52 ]).

There are several changes that we have identified in this overview of systematic reviews in comparison to the review we published in 2011 [ 8 ]. A consistent and welcome finding is that the overall quality of the systematic reviews themselves appears to have improved between the two reviews, although this is not reflected upon in the papers. This is exhibited through better, clearer reporting mechanisms in relation to the mechanics of the reviews, alongside a greater attention to, and deeper description of, how potential biases in included papers are discussed. Additionally, there is an increased, but still limited, inclusion of original studies conducted in low- and middle-income countries as opposed to just high-income countries. Importantly, we found that many of these systematic reviews are attuned to, and comment upon the contextual distinctions of pursuing evidence-informed interventions in health care settings in different economic settings. Furthermore, systematic reviews included in this updated article cover a wider set of clinical specialities (both within and beyond hospital settings) and have a focus on a wider set of healthcare professions — discussing both similarities, differences and inter-professional challenges faced therein, compared to the earlier reviews. These wider ranges of studies highlight that a particular intervention or group of interventions may work well for one professional group but be ineffective for another. This diversity of study settings allows us to consider the important role context (in its many forms) plays on implementing evidence into practice. Examining the complex and varied context of health care will help us address what Nilsen et al. ([ 6 ] p. 1) described as, ‘society’s health problems [that] require research-based knowledge acted on by healthcare practitioners together with implementation of political measures from governmental agencies’. This will help us shift implementation science to move, ‘beyond a success or failure perspective towards improved analysis of variables that could explain the impact of the implementation process’ ([ 6 ] p. 2).

This review brings together 32 papers considering individual and multi-faceted interventions designed to support the use of evidence in clinical practice. The majority of reviews report strategies achieving small impacts (normally on processes of care). There is much less evidence that these strategies have shifted patient outcomes. Combined with the two previous reviews, 86 systematic reviews of strategies to increase the implementation of research into clinical practice have been conducted. As a whole, this substantial body of knowledge struggles to tell us more about the use of individual and MFIs than: ‘it depends’. To really move forwards in addressing the gap between research evidence and practice, we may need to shift the emphasis away from isolating individual and multi-faceted interventions to better understanding and building more situated, relational and organisational capability to support the use of research in clinical practice. This will involve drawing on a wider range of perspectives, especially from the social, economic, political and behavioural sciences in primary studies and diversifying the types of synthesis undertaken to include approaches such as realist synthesis which facilitate exploration of the context in which strategies are employed. Harvey et al. [ 53 ] suggest that when context is likely to be critical to implementation success there are a range of primary research approaches (participatory research, realist evaluation, developmental evaluation, ethnography, quality/ rapid cycle improvement) that are likely to be appropriate and insightful. While these approaches often form part of implementation studies in the form of process evaluations, they are usually relatively small scale in relation to implementation research as a whole. As a result, the findings often do not make it into the subsequent systematic reviews. This review provides further evidence that we need to bring qualitative approaches in from the periphery to play a central role in many implementation studies and subsequent evidence syntheses. It would be helpful for systematic reviews, at the very least, to include more detail about the interventions and their implementation in terms of how and why they worked.

Availability of data and materials

The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.

Abbreviations

Before and after study

Controlled clinical trial

Effective Practice and Organisation of Care

High-income countries

Information and Communications Technology

Interrupted time series

Knowledge translation

Low- and middle-income countries

Randomised controlled trial

Grol R, Grimshaw J. From best evidence to best practice: effective implementation of change in patients’ care. Lancet. 2003;362:1225–30. https://doi.org/10.1016/S0140-6736(03)14546-1 .

Article   PubMed   Google Scholar  

Green LA, Seifert CM. Translation of research into practice: why we can’t “just do it.” J Am Board Fam Pract. 2005;18:541–5. https://doi.org/10.3122/jabfm.18.6.541 .

Eccles MP, Mittman BS. Welcome to Implementation Science. Implement Sci. 2006;1:1–3. https://doi.org/10.1186/1748-5908-1-1 .

Article   PubMed Central   Google Scholar  

Powell BJ, Waltz TJ, Chinman MJ, Damschroder LJ, Smith JL, Matthieu MM, et al. A refined compilation of implementation strategies: results from the Expert Recommendations for Implementing Change (ERIC) project. Implement Sci. 2015;10:2–14. https://doi.org/10.1186/s13012-015-0209-1 .

Article   Google Scholar  

Waltz TJ, Powell BJ, Matthieu MM, Damschroder LJ, et al. Use of concept mapping to characterize relationships among implementation strategies and assess their feasibility and importance: results from the Expert Recommendations for Implementing Change (ERIC) study. Implement Sci. 2015;10:1–8. https://doi.org/10.1186/s13012-015-0295-0 .

Nilsen P, Ståhl C, Roback K, et al. Never the twain shall meet? - a comparison of implementation science and policy implementation research. Implementation Sci. 2013;8:2–12. https://doi.org/10.1186/1748-5908-8-63 .

Rycroft-Malone J, Seers K, Eldh AC, et al. A realist process evaluation within the Facilitating Implementation of Research Evidence (FIRE) cluster randomised controlled international trial: an exemplar. Implementation Sci. 2018;13:1–15. https://doi.org/10.1186/s13012-018-0811-0 .

Boaz A, Baeza J, Fraser A, European Implementation Score Collaborative Group (EIS). Effective implementation of research into practice: an overview of systematic reviews of the health literature. BMC Res Notes. 2011;4:212. https://doi.org/10.1186/1756-0500-4-212 .

Article   PubMed   PubMed Central   Google Scholar  

Grimshaw JM, Shirran L, Thomas R, Mowatt G, Fraser C, Bero L, et al. Changing provider behavior – an overview of systematic reviews of interventions. Med Care. 2001;39 8Suppl 2:II2–45.

Google Scholar  

Squires JE, Sullivan K, Eccles MP, et al. Are multifaceted interventions more effective than single-component interventions in changing health-care professionals’ behaviours? An overview of systematic reviews. Implement Sci. 2014;9:1–22. https://doi.org/10.1186/s13012-014-0152-6 .

Salvador-Oliván JA, Marco-Cuenca G, Arquero-Avilés R. Development of an efficient search filter to retrieve systematic reviews from PubMed. J Med Libr Assoc. 2021;109:561–74. https://doi.org/10.5195/jmla.2021.1223 .

Thomas JM. Diffusion of innovation in systematic review methodology: why is study selection not yet assisted by automation? OA Evid Based Med. 2013;1:1–6.

Effective Practice and Organisation of Care (EPOC). The EPOC taxonomy of health systems interventions. EPOC Resources for review authors. Oslo: Norwegian Knowledge Centre for the Health Services; 2016. epoc.cochrane.org/epoc-taxonomy . Accessed 9 Oct 2023.

Jamal A, McKenzie K, Clark M. The impact of health information technology on the quality of medical and health care: a systematic review. Health Inf Manag. 2009;38:26–37. https://doi.org/10.1177/183335830903800305 .

Menon A, Korner-Bitensky N, Kastner M, et al. Strategies for rehabilitation professionals to move evidence-based knowledge into practice: a systematic review. J Rehabil Med. 2009;41:1024–32. https://doi.org/10.2340/16501977-0451 .

Oxman AD, Guyatt GH. Validation of an index of the quality of review articles. J Clin Epidemiol. 1991;44:1271–8. https://doi.org/10.1016/0895-4356(91)90160-b .

Article   CAS   PubMed   Google Scholar  

Francke AL, Smit MC, de Veer AJ, et al. Factors influencing the implementation of clinical guidelines for health care professionals: a systematic meta-review. BMC Med Inform Decis Mak. 2008;8:1–11. https://doi.org/10.1186/1472-6947-8-38 .

Jones CA, Roop SC, Pohar SL, et al. Translating knowledge in rehabilitation: systematic review. Phys Ther. 2015;95:663–77. https://doi.org/10.2522/ptj.20130512 .

Scott D, Albrecht L, O’Leary K, Ball GDC, et al. Systematic review of knowledge translation strategies in the allied health professions. Implement Sci. 2012;7:1–17. https://doi.org/10.1186/1748-5908-7-70 .

Wu Y, Brettle A, Zhou C, Ou J, et al. Do educational interventions aimed at nurses to support the implementation of evidence-based practice improve patient outcomes? A systematic review. Nurse Educ Today. 2018;70:109–14. https://doi.org/10.1016/j.nedt.2018.08.026 .

Yost J, Ganann R, Thompson D, Aloweni F, et al. The effectiveness of knowledge translation interventions for promoting evidence-informed decision-making among nurses in tertiary care: a systematic review and meta-analysis. Implement Sci. 2015;10:1–15. https://doi.org/10.1186/s13012-015-0286-1 .

Grudniewicz A, Kealy R, Rodseth RN, Hamid J, et al. What is the effectiveness of printed educational materials on primary care physician knowledge, behaviour, and patient outcomes: a systematic review and meta-analyses. Implement Sci. 2015;10:2–12. https://doi.org/10.1186/s13012-015-0347-5 .

Koota E, Kääriäinen M, Melender HL. Educational interventions promoting evidence-based practice among emergency nurses: a systematic review. Int Emerg Nurs. 2018;41:51–8. https://doi.org/10.1016/j.ienj.2018.06.004 .

Flodgren G, O’Brien MA, Parmelli E, et al. Local opinion leaders: effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev. 2019. https://doi.org/10.1002/14651858.CD000125.pub5 .

Arditi C, Rège-Walther M, Durieux P, et al. Computer-generated reminders delivered on paper to healthcare professionals: effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev. 2017. https://doi.org/10.1002/14651858.CD001175.pub4 .

Pantoja T, Grimshaw JM, Colomer N, et al. Manually-generated reminders delivered on paper: effects on professional practice and patient outcomes. Cochrane Database Syst Rev. 2019. https://doi.org/10.1002/14651858.CD001174.pub4 .

De Angelis G, Davies B, King J, McEwan J, et al. Information and communication technologies for the dissemination of clinical practice guidelines to health professionals: a systematic review. JMIR Med Educ. 2016;2:e16. https://doi.org/10.2196/mededu.6288 .

Brown A, Barnes C, Byaruhanga J, McLaughlin M, et al. Effectiveness of technology-enabled knowledge translation strategies in improving the use of research in public health: systematic review. J Med Internet Res. 2020;22:e17274. https://doi.org/10.2196/17274 .

Sykes MJ, McAnuff J, Kolehmainen N. When is audit and feedback effective in dementia care? A systematic review. Int J Nurs Stud. 2018;79:27–35. https://doi.org/10.1016/j.ijnurstu.2017.10.013 .

Bhatt NR, Czarniecki SW, Borgmann H, et al. A systematic review of the use of social media for dissemination of clinical practice guidelines. Eur Urol Focus. 2021;7:1195–204. https://doi.org/10.1016/j.euf.2020.10.008 .

Yamada J, Shorkey A, Barwick M, Widger K, et al. The effectiveness of toolkits as knowledge translation strategies for integrating evidence into clinical care: a systematic review. BMJ Open. 2015;5:e006808. https://doi.org/10.1136/bmjopen-2014-006808 .

Afari-Asiedu S, Abdulai MA, Tostmann A, et al. Interventions to improve dispensing of antibiotics at the community level in low and middle income countries: a systematic review. J Glob Antimicrob Resist. 2022;29:259–74. https://doi.org/10.1016/j.jgar.2022.03.009 .

Boonacker CW, Hoes AW, Dikhoff MJ, Schilder AG, et al. Interventions in health care professionals to improve treatment in children with upper respiratory tract infections. Int J Pediatr Otorhinolaryngol. 2010;74:1113–21. https://doi.org/10.1016/j.ijporl.2010.07.008 .

Al Zoubi FM, Menon A, Mayo NE, et al. The effectiveness of interventions designed to increase the uptake of clinical practice guidelines and best practices among musculoskeletal professionals: a systematic review. BMC Health Serv Res. 2018;18:2–11. https://doi.org/10.1186/s12913-018-3253-0 .

Ariyo P, Zayed B, Riese V, Anton B, et al. Implementation strategies to reduce surgical site infections: a systematic review. Infect Control Hosp Epidemiol. 2019;3:287–300. https://doi.org/10.1017/ice.2018.355 .

Borgert MJ, Goossens A, Dongelmans DA. What are effective strategies for the implementation of care bundles on ICUs: a systematic review. Implement Sci. 2015;10:1–11. https://doi.org/10.1186/s13012-015-0306-1 .

Cahill LS, Carey LM, Lannin NA, et al. Implementation interventions to promote the uptake of evidence-based practices in stroke rehabilitation. Cochrane Database Syst Rev. 2020. https://doi.org/10.1002/14651858.CD012575.pub2 .

Pedersen ER, Rubenstein L, Kandrack R, Danz M, et al. Elusive search for effective provider interventions: a systematic review of provider interventions to increase adherence to evidence-based treatment for depression. Implement Sci. 2018;13:1–30. https://doi.org/10.1186/s13012-018-0788-8 .

Jenkins HJ, Hancock MJ, French SD, Maher CG, et al. Effectiveness of interventions designed to reduce the use of imaging for low-back pain: a systematic review. CMAJ. 2015;187:401–8. https://doi.org/10.1503/cmaj.141183 .

Bennett S, Laver K, MacAndrew M, Beattie E, et al. Implementation of evidence-based, non-pharmacological interventions addressing behavior and psychological symptoms of dementia: a systematic review focused on implementation strategies. Int Psychogeriatr. 2021;33:947–75. https://doi.org/10.1017/S1041610220001702 .

Noonan VK, Wolfe DL, Thorogood NP, et al. Knowledge translation and implementation in spinal cord injury: a systematic review. Spinal Cord. 2014;52:578–87. https://doi.org/10.1038/sc.2014.62 .

Albrecht L, Archibald M, Snelgrove-Clarke E, et al. Systematic review of knowledge translation strategies to promote research uptake in child health settings. J Pediatr Nurs. 2016;31:235–54. https://doi.org/10.1016/j.pedn.2015.12.002 .

Campbell A, Louie-Poon S, Slater L, et al. Knowledge translation strategies used by healthcare professionals in child health settings: an updated systematic review. J Pediatr Nurs. 2019;47:114–20. https://doi.org/10.1016/j.pedn.2019.04.026 .

Bird ML, Miller T, Connell LA, et al. Moving stroke rehabilitation evidence into practice: a systematic review of randomized controlled trials. Clin Rehabil. 2019;33:1586–95. https://doi.org/10.1177/0269215519847253 .

Goorts K, Dizon J, Milanese S. The effectiveness of implementation strategies for promoting evidence informed interventions in allied healthcare: a systematic review. BMC Health Serv Res. 2021;21:1–11. https://doi.org/10.1186/s12913-021-06190-0 .

Zadro JR, O’Keeffe M, Allison JL, Lembke KA, et al. Effectiveness of implementation strategies to improve adherence of physical therapist treatment choices to clinical practice guidelines for musculoskeletal conditions: systematic review. Phys Ther. 2020;100:1516–41. https://doi.org/10.1093/ptj/pzaa101 .

Van der Veer SN, Jager KJ, Nache AM, et al. Translating knowledge on best practice into improving quality of RRT care: a systematic review of implementation strategies. Kidney Int. 2011;80:1021–34. https://doi.org/10.1038/ki.2011.222 .

Pawson R, Greenhalgh T, Harvey G, et al. Realist review–a new method of systematic review designed for complex policy interventions. J Health Serv Res Policy. 2005;10Suppl 1:21–34. https://doi.org/10.1258/1355819054308530 .

Rycroft-Malone J, McCormack B, Hutchinson AM, et al. Realist synthesis: illustrating the method for implementation research. Implementation Sci. 2012;7:1–10. https://doi.org/10.1186/1748-5908-7-33 .

Johnson MJ, May CR. Promoting professional behaviour change in healthcare: what interventions work, and why? A theory-led overview of systematic reviews. BMJ Open. 2015;5:e008592. https://doi.org/10.1136/bmjopen-2015-008592 .

Metz A, Jensen T, Farley A, Boaz A, et al. Is implementation research out of step with implementation practice? Pathways to effective implementation support over the last decade. Implement Res Pract. 2022;3:1–11. https://doi.org/10.1177/26334895221105585 .

May CR, Finch TL, Cornford J, Exley C, et al. Integrating telecare for chronic disease management in the community: What needs to be done? BMC Health Serv Res. 2011;11:1–11. https://doi.org/10.1186/1472-6963-11-131 .

Harvey G, Rycroft-Malone J, Seers K, Wilson P, et al. Connecting the science and practice of implementation – applying the lens of context to inform study design in implementation research. Front Health Serv. 2023;3:1–15. https://doi.org/10.3389/frhs.2023.1162762 .

Download references

Acknowledgements

The authors would like to thank Professor Kathryn Oliver for her support in the planning the review, Professor Steve Hanney for reading and commenting on the final manuscript and the staff at LSHTM library for their support in planning and conducting the literature search.

This study was supported by LSHTM’s Research England QR strategic priorities funding allocation and the National Institute for Health and Care Research (NIHR) Applied Research Collaboration South London (NIHR ARC South London) at King’s College Hospital NHS Foundation Trust. Grant number NIHR200152. The views expressed are those of the author(s) and not necessarily those of the NIHR, the Department of Health and Social Care or Research England.

Author information

Authors and affiliations.

Health and Social Care Workforce Research Unit, The Policy Institute, King’s College London, Virginia Woolf Building, 22 Kingsway, London, WC2B 6LE, UK

Annette Boaz

King’s Business School, King’s College London, 30 Aldwych, London, WC2B 4BG, UK

Juan Baeza & Alec Fraser

Federal University of Santa Catarina (UFSC), Campus Universitário Reitor João Davi Ferreira Lima, Florianópolis, SC, 88.040-900, Brazil

Erik Persson

You can also search for this author in PubMed   Google Scholar

Contributions

AB led the conceptual development and structure of the manuscript. EP conducted the searches and data extraction. All authors contributed to screening and quality appraisal. EP and AF wrote the first draft of the methods section. AB, JB and AF performed result synthesis and contributed to the analyses. AB wrote the first draft of the manuscript and incorporated feedback and revisions from all other authors. All authors revised and approved the final manuscript.

Corresponding author

Correspondence to Annette Boaz .

Ethics declarations

Ethics approval and consent to participate.

Not applicable.

Consent for publication

Competing interests.

The authors declare that they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1: appendix a., additional file 2: appendix b., rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Boaz, A., Baeza, J., Fraser, A. et al. ‘It depends’: what 86 systematic reviews tell us about what strategies to use to support the use of research in clinical practice. Implementation Sci 19 , 15 (2024). https://doi.org/10.1186/s13012-024-01337-z

Download citation

Received : 01 November 2023

Accepted : 05 January 2024

Published : 19 February 2024

DOI : https://doi.org/10.1186/s13012-024-01337-z

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

  • Implementation
  • Interventions
  • Clinical practice
  • Research evidence
  • Multi-faceted

Implementation Science

ISSN: 1748-5908

  • Submission enquiries: Access here and click Contact Us
  • General enquiries: [email protected]

what is a qualitative systematic literature review

DistillerSR Logo

About Systematic Reviews

Are Systematic Reviews Qualitative or Quantitative?

what is a qualitative systematic literature review

Automate every stage of your literature review to produce evidence-based research faster and more accurately.

A systematic review is designed to be transparent and replicable. Therefore, systematic reviews are considered reliable tools in scientific research and clinical practice. They synthesize the results using multiple primary studies by using strategies that minimize bias and random errors. Depending on the research question and the objectives of the research, the reviews can either be qualitative or quantitative. Qualitative reviews deal with understanding concepts, thoughts, or experiences. Quantitative reviews are employed when researchers want to test or confirm a hypothesis or theory. Let’s look at some of the differences between these two types of reviews.

To learn more about how long it takes to do a systematic review , you can check out the link to our full article on the topic.

Differences between Qualitative and Quantitative Reviews

The differences lie in the scope of the research, the methodology followed, and the type of questions they attempt to answer. Some of these differences include:

Research Questions

As mentioned earlier qualitative reviews attempt to answer open-ended research questions to understand or formulate hypotheses. This type of research is used to gather in-depth insights into new topics. Quantitative reviews, on the other hand, test or confirm existing hypotheses. This type of research is used to establish generalizable facts about a topic.

Type of Sample Data

The data collected for both types of research differ significantly. For qualitative research, data is collected as words using observations, interviews, and interactions with study subjects or from literature reviews. Quantitative studies collect data as numbers, usually from a larger sample size.

Data Collection Methods

To collect data as words for a qualitative study, researchers can employ tools such as interviews, recorded observations, focused groups, videos, or by collecting literature reviews on the same subject. For quantitative studies, data from primary sources is collected as numbers using rating scales and counting frequencies. The data for these studies can also be collected as measurements of variables from a well-designed experiment carried out under pre-defined, monitored conditions.

Data Analysis Methods

Data by itself cannot prove or demonstrate anything unless it is analyzed. Qualitative data is more challenging to analyze than quantitative data. A few different approaches to analyzing qualitative data include content analysis, thematic analysis, and discourse analysis. The goal of all of these approaches is to carefully analyze textual data to identify patterns, themes, and the meaning of words or phrases.

Quantitative data, since it is in the form of numbers, is analyzed using simple math or statistical methods. There are several software programs that can be used for mathematical and statistical analysis of numerical data.

Presentation of Results

Learn more about distillersr.

(Article continues below)

what is a qualitative systematic literature review

Final Takeaway – Qualitative or Quantitative?

3 reasons to connect.

what is a qualitative systematic literature review

Log in using your username and password

  • Search More Search for this keyword Advanced search
  • Latest content
  • Current issue
  • BMJ Journals More You are viewing from: Google Indexer

You are here

  • Online First
  • Rapid reviews methods series: guidance on rapid qualitative evidence synthesis
  • Article Text
  • Article info
  • Citation Tools
  • Rapid Responses
  • Article metrics

Download PDF

  • http://orcid.org/0000-0003-4808-3880 Andrew Booth 1 , 2 ,
  • Isolde Sommer 3 , 4 ,
  • Jane Noyes 2 , 5 ,
  • Catherine Houghton 2 , 6 ,
  • Fiona Campbell 1 , 7
  • The Cochrane Rapid Reviews Methods Group and Cochrane Qualitative and Implementation Methods Group (CQIMG)
  • 1 EnSyGN Sheffield Evidence Synthesis Group , University of Sheffield , Sheffield , UK
  • 2 Cochrane Qualitative and Implementation Methods Group (CQIMG) , London , UK
  • 3 Department for Evidence-based Medicine and Evaluation , University for Continuing Education Krems , Krems , Austria
  • 4 Cochrane Rapid Reviews Group & Cochrane Austria , Krems , Austria
  • 5 Bangor University , Bangor , UK
  • 6 University of Galway , Galway , Ireland
  • 7 University of Newcastle upon Tyne , Newcastle upon Tyne , UK
  • Correspondence to Professor Andrew Booth, Univ Sheffield, Sheffield, UK; a.booth{at}sheffield.ac.uk

This paper forms part of a series of methodological guidance from the Cochrane Rapid Reviews Methods Group and addresses rapid qualitative evidence syntheses (QESs), which use modified systematic, transparent and reproducible methodsu to accelerate the synthesis of qualitative evidence when faced with resource constraints. This guidance covers the review process as it relates to synthesis of qualitative research. ‘Rapid’ or ‘resource-constrained’ QES require use of templates and targeted knowledge user involvement. Clear definition of perspectives and decisions on indirect evidence, sampling and use of existing QES help in targeting eligibility criteria. Involvement of an information specialist, especially in prioritising databases, targeting grey literature and planning supplemental searches, can prove invaluable. Use of templates and frameworks in study selection and data extraction can be accompanied by quality assurance procedures targeting areas of likely weakness. Current Cochrane guidance informs selection of tools for quality assessment and of synthesis method. Thematic and framework synthesis facilitate efficient synthesis of large numbers of studies or plentiful data. Finally, judicious use of Grading of Recommendations Assessment, Development and Evaluation approach for assessing the Confidence of Evidence from Reviews of Qualitative research assessments and of software as appropriate help to achieve a timely and useful review product.

  • Systematic Reviews as Topic
  • Patient Care

Data availability statement

No data are available. Not applicable. All data is from published articles.

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See:  http://creativecommons.org/licenses/by-nc/4.0/ .

https://doi.org/10.1136/bmjebm-2023-112620

Statistics from Altmetric.com

Request permissions.

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

WHAT IS ALREADY KNOWN ON THIS TOPIC

Rapid Qualitative Evidence Synthesis (QES) is a relatively recent innovation in evidence synthesis and few published examples currently exists.

Guidance for authoring a rapid QES is scattered and requires compilation and summary.

WHAT THIS STUDY ADDS

This paper represents the first attempt to compile current guidance, illustrated by the experience of several international review teams.

We identify features of rapid QES methods that could be accelerated or abbreviated and where methods resemble those for conventional QESs.

HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY

This paper offers guidance for researchers when conducting a rapid QES and informs commissioners of research and policy-makers what to expect when commissioning such a review.

Introduction

This paper forms part of a series from the Cochrane Rapid Reviews Methods Group providing methodological guidance for rapid reviews. While other papers in the series 1–4 focus on generic considerations, we aim to provide in-depth recommendations specific to a resource-constrained (or rapid) qualitative evidence synthesis (rQES). 5 This paper is accompanied by recommended resources ( online supplemental appendix A ) and an elaboration with practical considerations ( online supplemental appendix B ).

Supplemental material

The role of qualitative evidence in decision-making is increasingly recognised. 6 This, in turn, has led to appreciation of the value of qualitative evidence syntheses (QESs) that summarise findings across multiple contexts. 7 Recognition of the need for such syntheses to be available at the time most useful to decision-making has, in turn, driven demand for rapid qualitative evidence syntheses. 8 The breadth of potential rQES mirrors the versatility of QES in general (from focused questions to broad overviews) and outputs range from descriptive thematic maps through to theory-informed syntheses (see table 1 ).

  • View inline

Glossary of important terms (alphabetically)

As with other resource-constrained reviews, no one size fits all. A team should start by specifying the phenomenon of interest, the review question, 9 the perspectives to be included 9 and the sample to be determined and selected. 10 Subsequently, the team must finalise the appropriate choice of synthesis. 11 Above all, the review team should consider the intended knowledge users, 3 including requirements of the funder.

An rQES team, in particular, cannot afford any extra time or resource requirements that might arise from either a misunderstanding of the review question, an unclear picture of user requirements or an inappropriate choice of methods. The team seeks to align the review question and the requirements of the knowledge user with available time and resources. They also need to ensure that the choice of data and choice of synthesis are appropriate to the intended ‘knowledge claims’ (epistemology) made by the rQES. 11 This involves the team asking ‘what types of data are meaningful for this review question?’, ‘what types of data are trustworthy?’ and ‘is the favoured synthesis method appropriate for this type of data?’. 12 This paper aims to help rQES teams to choose methods that best fit their project while understanding the limitations of those choices. Our recommendations derive from current QES guidance, 5 evidence on modified QES methods, 8 13 and practical experience. 14 15

This paper presents an overview of considerations and recommendations as described in table 2 . Supplemental materials including additional resources details of our recommendations and practical examples are provided in online supplemental appendices A and B .

Recommendations for resource-constrained qualitative evidence synthesis (rQES)

Setting the review question and topic refinement

Rapid reviews summarise information from multiple research studies to produce evidence for ‘the public, researchers, policymakers and funders in a systematic, resource-efficient manner’. 16 Involvement of knowledge users is critical. 3 Given time constraints, individual knowledge users could be asked only to feedback on very specific decisions and tasks or on selective sections of the protocol. Specifically, whenever a QES is abbreviated or accelerated, a team should ensure that the review question is agreed by a minimum number of knowledge users with expertise or experience that reflects all the important review perspectives and with authority to approve the final version 2 5 11 ( table 2 , item R1).

Involvement of topic experts can ensure that the rQES is responsive to need. 14 17 One Cochrane rQES saved considerable time by agreeing the review topic within a single meeting and one-phase iteration. 9 Decisions on topics to be omitted are also informed by a knowledge of existing QESs. 17

An information specialist can help to manage the quantity and quality of available evidence by setting conceptual boundaries and logistic limits. A structured question format, such as Setting-Perspective-Interest, phenomenon of-Comparison-Evaluation or Population-Interest, phenomenon of-Context helps in communicating the scope and, subsequently, in operationalising study selection. 9 18

Scoping (of review parameters) and mapping (of key types of evidence and likely richness of data) helps when planning the review. 5 19 The option to choose purposive sampling over comprehensive sampling approaches, as offered by standard QES, may be particularly helpful in the context of a rapid QES. 8 Once a team knows the approximate number and distribution of studies, perhaps mapping them against country, age, ethnicity, etc), they can decide whether or not to use purposive sampling. 12 An rQES for the WHO combined purposive with variation sampling. Sampling in two stages started by reducing the initial number of studies to a more manageable sampling frame and then sampling approximately a third of the remaining studies from within the sampling frame. 20

Sampling may target richer studies and/or privilege diversity. 8 21 A rich qualitative study typically illustrates findings with verbatim extracts from transcripts from interviews or textual responses from questionnaires. Rich studies are often found in specialist qualitative research or social science journals. In contrast, less rich studies may itemise themes with an occasional indicative text extract and tend to summarise findings. In clinical or biomedical journals less rich findings may be placed within a single table or box.

No rule exists on an optimal number of studies; too many studies makes it challenging to ‘maintain insight’, 22 too few does not sustain rigorous analysis. 23 Guidance on sampling is available from the forthcoming Cochrane-Campbell QES Handbook.

A review team can use templates to fast-track writing of a protocol. The protocol should always be publicly available ( table 2 , item R2). 24 25 Formal registration may require that the team has not commenced data extraction but should be considered if it does not compromise the rQES timeframe. Time pressures may require that methods are left suitably flexible to allow well-justified changes to be made as a detailed picture of the studies and data emerge. 26 The first Cochrane rQES drew heavily on text from a joint protocol/review template previously produced within Cochrane. 24

Setting eligibility criteria

An rQES team may need to limit the number of perspectives, focusing on those most important for decision-making 5 9 27 ( table 2 , item R3). Beyond the patients/clients each additional perspective (eg, family members, health professionals, other professionals, etc) multiplies the additional effort involved.

A rapid QES may require strict date and setting restrictions 17 and language restrictions that accommodate the specific requirements of the review. Specifically, the team should consider whether changes in context over time or substantive differences between geographical regions could be used to justify a narrower date range or a limited coverage of countries and/or languages. The team should also decide if ‘indirect evidence’ is to substitute for the absence of direct evidence. An rQES typically focuses on direct evidence, except when only indirect evidence is available 28 ( table 2 , item R4). Decisions on relevance are challenging—precautions for swine influenza may inform precautions for bird influenza. 28 A smoking ban may operate similarly to seat belt legislation, etc. A review team should identify where such shared mechanisms might operate. 28 An rQES team must also decide whether to use frameworks or models to focus the review. Theories may be unearthed within the topic search or be already known to team members, fro example, Theory of Planned Behaviour. 29

Options for managing the quantity and quality of studies and data emerge during the scoping (see above). In summary, the review team should consider privileging rich qualitative studies 2 ; consider a stepwise approach to inclusion of qualitative data and explore the possibility of sampling ( table 2 , item R5). For example, where data is plentiful an rQES may be limited to qualitative research and/or to mixed methods studies. Where data is less plentiful then surveys or other qualitative data sources may need to be included. Where plentiful reviews already exist, a team may decide to conduct a review of reviews 5 by including multiple QES within a mega-synthesis 28 29 ( table 2 , item R6).

Searching for QES merits its own guidance, 21–23 30 this section reinforces important considerations from guidance specific to qualitative research. Generic guidance for rapid reviews in this series broadly applies to rapid QESs. 1

In addition to journal articles, by far the most plentiful source, qualitative research is found in book chapters, theses and in published and unpublished reports. 21 Searches to support an rQES can (a) limit the number of databases searched, deliberately selecting databases from diverse disciplines, (b) use abbreviated study filters to retrieve qualitative designs and (c) employ high yield complementary methods (eg, reference checking, citation searching and Related Articles features). An information specialist (eg, librarian) should be involved in prioritising sources and search methods ( table 2 , item R7). 11 14

According to empirical evidence optimal database combinations include Scopus plus CINAHL or Scopus plus ProQuest Dissertations and Theses Global (two-database combinations) and Scopus plus CINAHL plus ProQuest Dissertations and Theses Global (three-database combination) with both choices retrieving between 89% and 92% of relevant studies. 30

If resources allow, searches should include one or two specialised databases ( table 2 , item R8) from different disciplines or contexts 21 (eg, social science databases, specialist discipline databases or regional or institutional repositories). Even when resources are limited, the information specialist should factor in time for peer review of at least one search strategy ( table 2 , item R9). 31 Searches for ‘grey literature’ should selectively target appropriate types of grey literature (such as theses or process evaluations) and supplemental searches, including citation chaining or Related Articles features ( table 2 , item R10). 32 The first Cochrane rQES reported that searching reference lists of key papers yielded an extra 30 candidate papers for review. However, the team documented exclusion of grey literature as a limitation of their review. 15

Study selection

Consistency in study selection is achieved by using templates, by gaining a shared team understanding of the audience and purpose, and by ongoing communication within, and beyond, the team. 2 33 Individuals may work in parallel on the same task, as in the first Cochrane rQES, or follow a ‘segmented’ approach where each reviewer is allocated a different task. 14 The use of machine learning in the specific context of rQES remains experimental. However, the possibility of developing qualitative study classifiers comparable to those for randomised controlled trials offers an achievable aspiration. 34

Title and abstract screening

The entire screening team should use pre-prepared, pretested title and abstract templates to limit the scale of piloting, calibration and testing ( table 2 , item R11). 1 14 The first Cochrane rQES team double-screened titles and abstracts within Covidence review software. 14 Disagreements were resolved with reference to a third reviewer achieving a shared understanding of the eligibility criteria and enhancing familiarity with target studies and insight from data. 14 The team should target and prioritise identified risks of either over-zealous inclusion or over-exclusion specific to each rQES ( table 2 , item R12). 14 The team should maximise opportunities to capture divergent views and perspectives within study findings. 35

Full-text screening

Full-text screening similarly benefits from using a pre-prepared pretested standardised template where possible 1 14 ( table 2 , item R11). If a single reviewer undertakes full-text screening, 8 the team should identify likely risks to trustworthiness of findings and focus quality control procedures (eg, use of additional reviewers and percentages for double screening) on specific threats 14 ( table 2 , item R13). The Cochrane rQES team opted for double screening to assist their immersion within the topic. 14

Data extraction

Data extraction of descriptive/contextual data may be facilitated by review management software (eg, EPPI-Reviewer) or home-made approaches using Google Forms, or other survey software. 36 Where extraction of qualitative findings requires line-by-line coding with multiple iterations of the data then a qualitative data management analysis package, such as QSR NVivo, reaps dividends. 36 The team must decide if, collectively, they favour extracting data to a template or coding direct within an electronic version of an article.

Quality control must be fit for purpose but not excessive. Published examples typically use a single reviewer for data extraction 8 with use of two independent reviewers being the exception. The team could limit data extraction to minimal essential items. They may also consider re-using descriptive details and findings previously extracted within previous well-conducted QES ( table 2 , item R14). A pre-existing framework, where readily identified, may help to structure the data extraction template. 15 37 The same framework may be used to present the findings. Some organisations may specify a preferred framework, such as an evidence-to-decision-making framework. 38

Assessment of methodological limitations

The QES community assess ‘methodological limitations’ rather than use ‘risk of bias’ terminology. An rQES team should pick an approach appropriate to their specific review. For example, a thematic map may not require assessment of individual studies—a brief statement of the generic limitations of the set of studies may be sufficient. However, for any synthesis that underpins practice recommendations 39 assessment of included studies is integral to the credibility of findings. In any decision-making context that involves recommendations or guidelines, an assessment of methodological limitations is mandatory. 40 41

Each review team should work with knowledge users to determine a review-specific approach to quality assessment. 27 While ‘traffic lights’, similar to the outputs from the Cochrane Risk of Bias tool, may facilitate rapid interpretation, accompanying textual notes are invaluable in highlighting specific areas for concern. In particular, the rQES team should demonstrate that they are aware (a) that research designs for qualitative research seek to elicit divergent views, rather than control for variation; (b) that, for qualitative research, the selection of the sample is far more informative than the size of the sample; and (c) that researchers from primary research, and equally reviewers for the qualitative synthesis, need to be thoughtful and reflexive about their possible influences on interpretation of either the primary data or the synthesised findings.

Selection of checklist

Numerous scales and checklists exist for assessing the quality of qualitative studies. In the absence of validated risk of bias tools for qualitative studies, the team should choose a tool according to Cochrane Qualitative and Implementation Methods Group (CQIMG) guidance together with expediency (according to ease of use, prior familiarity, etc) ( table 2 , item R15). 41 In comparison to the Critical Appraisal Skills Programme checklist which was never designed for use in synthesis, 42 the Cochrane qualitative tool is similarly easy to use and was designed for QES use. Work is underway to identify an assessment process that is compatible with QESs that support decision-making. 41 For now the choice of a checklist remains determined by interim Cochrane guidance and, beyond this, by personal preference and experience. For an rQES a team could use a single reviewer to assess methodological limitations, with verification of judgements (and support statements) by a second reviewer ( table 2 , item R16).

The CQIMG endorses three types of synthesis; thematic synthesis, framework synthesis and meta-ethnography ( box 1 ). 43 44 Rapid QES favour descriptive thematic synthesis 45 or framework synthesis, 46 47 except when theory generation (meta-ethnography 48 49 or analytical thematic synthesis) is a priority ( table 2 , item R17).

Choosing a method for rapid qualitative synthesis

Thematic synthesis: first choice method for rQES. 45 For example, in their rapid QES Crooks and colleagues 44 used a thematic synthesis to understand the experiences of both academic and lived experience coresearchers within palliative and end of life research. 45

Framework synthesis: alternative where a suitable framework can be speedily identified. 46 For example, Bright and colleagues 46 considered ‘best-fit framework synthesis’ as appropriate for mapping study findings to an ‘a priori framework of dimensions measured by prenatal maternal anxiety tools’ within their ‘streamlined and time-limited evidence review’. 47

Less commonly, an adapted meta-ethnographical approach was used for an implementation model of social distancing where supportive data (29 studies) was plentiful. 48 However, this QES demonstrates several features that subsequently challenge its original identification as ‘rapid’. 49

Abbrevations: QES, qualitative evidence synthesis; rQES, resource-constrained qualitative evidence synthesis.

The team should consider whether a conceptual model, theory or framework offers a rapid way for organising, coding, interpreting and presenting findings ( table 2 , item R18). If the extracted data appears rich enough to sustain further interpretation, data from a thematic or framework synthesis can subsequently be explored within a subsequent meta-ethnography. 43 However, this requires a team with substantial interpretative expertise. 11

Assessments of confidence in the evidence 4 are central to any rQES that seeks to support decision-making and the QES-specific Grading of Recommendations Assessment, Development and Evaluation approach for assessing the Confidence of Evidence from Reviews of Qualitative research (GRADE-CERQual) approach is designed to assess confidence in qualitative evidence. 50 This can be performed by a single reviewer, confirmed by a second reviewer. 26 Additional reviewers could verify all, or a sample of, assessments. For a rapid assessment a team must prioritise findings, using objective criteria; a WHO rQES focused only on the three ‘highly synthesised findings’. 20 The team could consider reusing GRADE-CERQual assessments from published QESs if findings are relevant and of demonstrable high quality ( table 2 , item R19). 50 No rapid approach to full application of GRADE-CERQual currently exists.

Reporting and record management

Little is written on optimal use of technology. 8 A rapid review is not a good time to learn review management software or qualitative analysis management software. Using such software for all general QES processes ( table 2 , item R20), and then harnessing these skills and tools when specifically under resource pressures, is a sounder strategy. Good file labelling and folder management and a ‘develop once, re-use multi-times’ approach facilitates resource savings.

Reporting requirements include the meta-ethnography reporting guidance (eMERGe) 51 and the Enhancing transparency in reporting the synthesis of qualitative research (ENTREQ) statement. 52 An rQES should describe limitations and their implications for confidence in the evidence even more thoroughly than a regular QES; detailing the consequences of fast-tracking, streamlining or of omitting processes all together. 8 Time spent documenting reflexivity is similarly important. 27 If QES methodology is to remain credible rapid approaches must be applied with insight and documented with circumspection. 53 54 (56)

Ethics statements

Patient consent for publication.

Not applicable.

Ethics approval

  • Klerings I ,
  • Robalino S ,
  • Booth A , et al
  • Nussbaumer-Streit B ,
  • Hamel C , et al
  • Garritty C ,
  • Tricco AC ,
  • Smith M , et al
  • Gartlehner G ,
  • Devane D , et al
  • NHS Scotland
  • Campbell F ,
  • Flemming K , et al
  • Glenton C ,
  • Lubarsky S ,
  • Varpio L , et al
  • Meskell P ,
  • Glenton C , et al
  • Houghton C ,
  • Delaney H , et al
  • Beecher C ,
  • Maeso B , et al
  • McKenzie JE , et al
  • Harris JL ,
  • Cargo M , et al
  • Varley-Campbell J , et al
  • Downe S , et al
  • Shamseer L ,
  • Clarke M , et al
  • Nussbaumer-Streit B , et al
  • Finlayson KW ,
  • Lawrie TA , et al
  • Lewin S , et al
  • Frandsen TF ,
  • Gildberg FA ,
  • Tingleff EB
  • Mshelia S ,
  • Analo CV , et al
  • Husk K , et al
  • Carmona C ,
  • Carroll C ,
  • Ilott I , et al
  • Meehan B , et al
  • Munthe-Kaas H ,
  • Bohren MA ,
  • Munthe-Kaas HM ,
  • French DP ,
  • Flemming K ,
  • Garside R , et al
  • Shulman C , et al
  • Dixon-Woods M
  • Bright KS ,
  • Norris JM ,
  • Letourneau NL , et al
  • Sadjadi M ,
  • Mörschel KS ,
  • Petticrew M
  • France EF ,
  • Cunningham M ,
  • Ring N , et al
  • McInnes E , et al
  • Britten N ,
  • Garside R ,
  • Pope C , et al

Supplementary materials

Supplementary data.

This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

  • Data supplement 1

Contributors All authors (AB, IS, JN, CH, FC) have made substantial contributions to the conception and design of the guidance document. AB led on drafting the work and revising it critically for important intellectual content. All other authors (IS, JN, CH, FC) contributed to revisions of the document. All authors (AB, IS, JN, CH, FC) have given final approval of the version to be published. As members of the Cochrane Qualitative and Implementation Methods Group and/or the Cochrane Rapid Reviews Methods Group all authors (AB, IS, JN, CH, FC) agree to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

Competing interests AB is co-convenor of the Cochrane Qualitative and Implementation Methods Group. In the last 36 months, he received royalties from Systematic Approaches To a Successful Literature Review (Sage 3rd edition), honoraria from the Agency for Healthcare Research and Quality, and travel support from the WHO. JN is lead convenor of the Cochrane Qualitative and Implementation Methods Group. In the last 36 months, she has received honoraria from the Agency for Healthcare Research and Quality and travel support from the WHO. CH is co-convenor of the Cochrane Qualitative and Implementation Methods Group.

Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.

Provenance and peer review Not commissioned; internally peer reviewed.

Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.

Read the full text or download the PDF:

  • Open access
  • Published: 17 February 2024

The complexity of leadership in coproduction practices: a guiding framework based on a systematic literature review

  • Sofia Kjellström 1 , 2 ,
  • Sophie Sarre 2 &
  • Daniel Masterson 1  

BMC Health Services Research volume  24 , Article number:  219 ( 2024 ) Cite this article

102 Accesses

Metrics details

As coproduction in public services increases, understanding the role of leadership in this context is essential to the tasks of establishing relational partnerships and addressing power differentials among groups. The aims of this review are to explore models of coproduction leadership and the processes involved in leading coproduction as well as, based on that exploration, to develop a guiding framework for coproduction practices.

A systematic review that synthesizes the evidence reported by 73 papers related to coproduction of health and welfare.

Despite the fact that models of coleadership and collective leadership exhibit a better fit with the relational character of coproduction, the majority of the articles included in this review employed a leader-centric underlying theory. The practice of coproduction leadership is a complex activity pertaining to interactions among people, encompassing nine essential practices: initiating, power-sharing, training, supporting, establishing trust, communicating, networking, orchestration, and implementation.

Conclusions

This paper proposes a novel framework for coproduction leadership practices based on a systematic review of the literature and a set of reflective questions. This framework aims to help coproduction leaders and participants understand the complexity, diversity, and flexibility of coproduction leadership and to challenge and enhance their capacity to collaborate effectively.

Peer Review reports

Introduction

For more than 40 years, scholars and practitioners have sought to identify and understand various aspects of coproduction with the goal of improving services as well as equalizing (or at least reorganizing) power relations in service design and delivery [ 1 ]. More recently, such discussion has focused on the roles of leaders and leadership in coproduction, seeking to describe and assess the various types of leaders and leadership that might maximize the goals of coproduction processes and outcomes. Leaders can act to make coproduction, in all its forms, happen [ 2 , 3 ]. Leaders can enhance coproduction by providing resources, establishing inviting structures, and prioritizing the involvement of various stakeholders. Conversely, they can inhibit coproduction by perpetuating conservative administrative cultures, failing to provide training, or being reluctant to share power [ 3 ]. Coproduction relies on leadership at all levels, ranging from senior managers to local “champions” and including the citizens and third-sector organizations that participate in coproduction activities and practices.

This review presents a synthesis of research on the leadership of coproduction, which has been recognized for its scarcity [ 3 , 4 , 5 , 6 ]. The review provides new knowledge regarding the fact that coproduction leadership must become more deliberately (in)formed by collective leadership models. It also illustrates the multiplicity and complexity associated with coproduction leadership activities by outlining practices in which leaders must engage to ensure success. This review can inform a framework that offers guiding insights on which commissioners, evaluators, managers and leaders of coproduction can reflect as well as suggestions and directions for future research.

  • Coproduction

Coproduction is a broad concept that is associated with different meanings across a range of contexts [ 1 ]. Many definitions and uses of the term coproduction and codesign have been identified [ 7 ]. Throughout this paper, although we acknowledge the distinctions associated with the concepts and origins of the notion of codesign, we use the broad term coproduction to refer to some form of collaboration or partnership between service providers and service users or citizens. For this review, we follow the definitions provided by Osborne and Strokosch [ 8 ], who identified ‘ consumer coproduction’ as an inevitable component of value creation in interactions among service providers; ‘participatory coproduction’, in which context participation is deliberative and occurs at the strategic level of service design and planning; and ‘enhanced coproduction’, which represents a potential mechanism for transforming organizational processes and boundaries.

Power is inevitably central to coproduction. Schlappa and Ymani claimed that the coproduction process is “inherently negotiated, emergent and reliant on a range of actors who may have both common and contrasting motivations, and are able to exercise power, which in turn is moderated by the context in which these relations occur” [ 6 ]. This sensitivity to motivation, context and power is helpful for our understanding of leadership in coproduction.

Leadership models

Most conceptualizations of leadership have been based on the claim that leadership is a kind of inherent characteristic exhibited by human beings, such that leaders are depicted as heroes with unique traits, styles or behaviours [ 9 ]. However, research on leadership in coproduction is important in relation to an emerging body of research that focuses on the notion of “leadership in the plural” [ 10 ] or “collective leadership” [ 11 , 12 ]. These phrases act as umbrella terms that refer to overlapping concepts such as shared, collaborative, distributed, pooled and relational leadership. A core feature of these models is that leadership is not (only) viewed as a property of individuals and their behaviours but rather as a collective phenomenon that is distributed or shared among different people [ 10 ]. A distinction can be made between two types of collective leadership. Leadership can be shared in interpersonal relationships; for example, it can be pooled among duos or trios at the top of an organization, or shared leadership can be exercised within teams working on a project. This notion is based upon the assumption that people have different skills that complement each other. The second kind of collective leadership is a more radical version of this notion, according to which leadership emerges as a result of direction, alignment, and commitment within a group [ 11 ] or can be observed to reside within the system, for example, in the form of distributed leadership across interorganizational and intraorganizational boundaries and networks [ 10 , 12 ]. In cross-sectoral collaboration, leadership is distributed across time and space, which requires structures to guide how leadership is shared and organized. It has been argued that collective leadership is best suited to the analysis of coproduction practices [ 4 , 6 , 13 , 14 ].

It is important to note that distinctions have been made between management (planning, monitoring and controlling) and leadership (creating a vision, inspiring and changing) based on behaviours [ 15 ]. However, many authors have not made such a distinction, and the terms have frequently been used interchangeably. We therefore adopt the practice employed in the papers included in this review and use the terms leadership and leader as catch-all terms; we only use the words management or manager when the papers refer to job titles or ‘public management’.

Leadership models can be regarded as resembling a colour palette that offers a variety of choices, and similar to colours, some models fit a situation better than others. This paper investigates the use and fit of various leadership models for coproduction.

Leadership of coproduction research

Extant research on the leadership of coproduction has been described as “sparse” [ 4 ], a “neglected area” [ 5 ] and “overlooked” [ 3 , 6 ]. Despite a recent resurgence of interest in the potential of coproduction as a means of maintaining and improving the quality of health and social care, significant questions regarding how coproduction can and should be led in this context remain unanswered. Most reviews of coproduction have not addressed this issue [ 2 , 16 , 17 , 18 ]. Clarke et al.’s (2017) review identified the lack of managerial authority and leadership as a key barrier to the implementation of coproduced interventions but did not explore the implications of this finding for future practice. The review conducted by Bussu and Galanti (2018) stands alone in its focus on leadership, although the empirical cases explored by those authors were restricted to the context of local government in the UK. Recent empirical case studies that have explored leadership [ 13 , 14 , 15 , 19 ] have focused on public managers [ 3 , 5 , 14 ] or on identifying the consequences of different models of leadership. This review contributes to the literature by providing knowledge regarding how to make deliberate choices pertaining to coproduction leadership in terms of how it is conceptualized and shared and the activities that are necessary for leading coproduction.

Coproduction leadership practices

The leadership of coproduction poses a number of challenges. A proposed aim of coproduction is to drive change within services and in traditional state-citizen relationships by establishing equal and reciprocal relationships among professionals, the people using services, and their families and neighbours. This task requires a restructuring of health and welfare services to equalize power between providers and other stakeholders with an interest in the design and provision of these services. However, it has been suggested that coproduction runs the risk of reproducing existing inequalities in power rather than mitigating them since coproduction is inevitably saturated with unequal power relations that must be acknowledged but cannot be managed away [ 20 ].

In this paper, we present the findings of a systematic review of the literature on leadership in coproduction. The purpose of this review is to explore models of coproduction leadership and the practices involved in leading coproduction in the context of health and social care sectors [ 7 ]. The results are synthesized to develop a framework for actors who seek to commission, design, lead or evaluate coproduction processes. This framework emphasizes the need to make more deliberate choices regarding the underlying conceptualization of leadership and the ways in which such a conceptualization is related to the activities necessary for leading coproduction. Based on the framework, we also propose specific guiding questions for individuals involved in coproduction in practice and make suggestions for future research.

This systematic literature review is based on a study protocol on coproduction research in the context of health and social care sectors [ 21 ], and data were obtained from a published scoping review, where the full search strategy is provided [ 7 ]. The scoping review set out to identify ‘what is out there’ and to explore the definitions of the concepts of coproduction and codesign. In brief, the following search terms for the relevant concept (co-produc* OR coproduc* OR co-design* OR codesign*) and context (health OR social OR & “public service*” OR “public sector”) were used to query the following databases: CINAHL with Full Text (EBSCOHost), Cochrane Central Register of Controlled Trials (Wiley), MEDLINE (EBSCOhost), PsycINFO (ProQuest), PubMed (legacy), and Scopus (Elsevier). This paper focused on leadership. All titles and abstracts included in the scoping review ( n  = 979) were obtained and searched for leadership concepts (leader* OR manage*) ( n  = 415). These materials were reviewed independently by SK and SS using the following inclusion criterion: conceptual, empirical and reflection papers that included references to the management and/or leadership of coproduction. Study protocols were excluded because we wanted to capture lessons drawn from implementation, and conference papers were excluded because they lacked sufficient detail. Articles focusing on the context of individual-level coproduction (i.e., cases in which an individual client or patient was the focus of coproduction) were excluded, as we were interested in the leadership processes involved in collective coproduction. Conflicts were resolved through discussion and further consideration of disputed papers. This process led to the inclusion of 73 articles (Fig.  1 – PRISMA flow chart).

figure 1

PRISMA flow chart

The method used for this research was a systematic review with qualitative synthesis. The strength of this approach lies in its ability to complement research evidence with user and practitioner considerations [ 22 ]. In the process of examining the full texts of the papers, two researchers (SK and SS) extracted background data independently. To promote coproduction, four stakeholders were strategically selected through the personal networks of one of the authors, SK. These stakeholders exhibited diverse expertise in the leadership of coproduction. One was a leadership developer and family member of an individual with 24/7 care needs. Another was a physician. The third worked in peer support and had personal experience with mental health services. The fourth was a health care leader. Four key articles were chosen due to the diversity of leadership ideas they exhibit and the depth of the explicit text on leadership they provided. During the analysis by stakeholders, no themes were changed or refined; instead, the analysis confirmed the relevance of the initially identified themes, thus emphasizing the robustness of our findings based on a process that involved reading four key articles and identifying the perceived key implications for our research aim.

A qualitative synthesis unites the findings of individual studies in a different arrangement, thereby constructing new knowledge that is not apparent from the individual studies in isolation [ 23 ]. This fact is particularly evident in this review, since leadership was seldom the main focus of the included articles. Accordingly, we employed multiple pieces of information to construct a pattern. The process of synthesis started at a very broad level with the goal of understanding which aspects of leadership were addressed in the literature. This process then separated into two strands. One such strand focused on interpreting the data from the perspective of current leadership models, while the other focused on interpreting leadership practices – i.e., the activities and relationships that are part of the process of leading coproduction. We searched for themes both within and across individual articles, and our goal was interpretative rather than purely aggregative. This process resulted in three themes pertaining to coproduction leadership models and nine coproduction leadership practices. We present these findings together in the form of a framework because consideration of both leadership models and practices prompts better and more conscious choices, which can improve the quality of coproduction. Persons one and two from the stakeholder group also provided feedback on a draft of this paper, and their insights were integrated into this research.

Sample description

We included 73 papers (Additional file 1 ) dating from 1994 to 2019 (the year in which the initial search was performed). Most of these papers were empirical ( n  = 54), and more than half of them were case studies ( n  = 30). Fifteen articles were conceptual papers, and four were literature reviews. The setting or focus of the papers was predominantly on services ( n  = 66), while the remainder of the papers were on research ( n  = 4) or policy ( n  = 3). The papers drew on evidence collected from 13 countries, and the most common national setting was the UK ( n  = 29). Nine cross-national papers were also included. Issues related to leadership were rarely the focus of the papers.

Results: A coproduction leadership framework

The synthesis consists of three parts (roles, models and practices), which are combined to develop an overarching and integrative framework for essential issues pertaining to coproduction leadership [ 4 , 24 ].

People and roles

The way in which the leadership of coproduction has been conceptualized in the literature suggests that a range of actors are involved in the coproduction of health and wellbeing and that these actors can take on different leadership roles and functions. Service users, community members and community representatives can play a vital role in the task of deliberatively coproducing or even transforming services, as can third-sector organizations, external experts, politicians, mid-level facilitators, managers, and senior leaders.

It has been argued that it is important to involve leaders from diverse backgrounds who have personal experiential knowledge of public involvement to encourage involvement from a broader population [ 25 , 26 , 27 ]. Service users and community members play leadership roles in coproduction initiatives related to health or well-being. These roles involve shared decision-making and accountability at various levels, ranging from the personal to the systemic.

Senior leaders include formal representatives of organizations (executives, politicians, or formal managers) and formal or respected leaders of communities. They play an important role throughout this process. During the initiation stage, by implementing and sustaining the outcomes of coproduction, they play a crucial role in the provision of resources such as time, money, materials, and access to networks. In the interim stages, their commitment to coproduction, sponsorship, and engagement is vital.

Champions and ambassadors use their expertise and passion to drive coproduction efforts. In particular, "insider" champions can establish trust among participants and help service providers understand the importance of coproduction. These champions advocate for coproduction and actively support initiatives [ 28 , 29 , 30 , 31 ]. Ambassadors are individuals who have expertise and volunteer their time to train others or work with clients in coproduced services. They play a crucial role in the tasks of supporting and promoting coproduction [ 28 , 32 , 33 ].

Project leaders and facilitators are individuals who are responsible for guiding and supporting coproduction projects, thereby ensuring their smooth operation and collaborative nature. Project leaders are responsible for overall project management, including the setting of goals, objectives, and timelines. They play a pivotal role in ensuring that projects remain on track, and they facilitate accessible and transparent dialogue among stakeholders and ensure equal representation [ 34 , 35 ]. Facilitators focus on supporting the group involved in coproduction, maintaining respectful interactions, empowering service users and carers, and addressing any tensions that may arise during the collaborative process [ 36 , 37 ].

In summary, senior leaders sponsor and support coproduction. Champions and ambassadors are individuals who advocate for and support coproduction initiatives, while project leaders and facilitators are responsible for managing and guiding coproduction projects themselves, thereby ensuring effective collaboration among stakeholders. All of these roles can be played by people drawn from various backgrounds, including senior staff, health care professionals, experts in coproduction, researchers, citizens, or volunteers.

Three models of leadership in coproduction

These actors play different leadership roles, and leadership can be exercised by individuals or groups. Three leadership models have been proposed: leadership as enacted by individual leaders, coleadership and collective leadership.

Leadership by individual leaders

A leader-centric view has been the dominant interpretation of leadership in the field of coproduction. Many references were made to “senior leaders”. This term was used to describe formal representatives of organizations or services (senior managers, executives), formally appointed community leaders (policy-makers, local government leaders), or respected leaders of communities. Senior support was described as an important success factor in coproduction [ 37 , 38 , 39 , 40 , 41 , 42 , 43 , 44 , 45 ]. Other leadership roles included project leaders, facilitators, ambassadors, and champions – as described in the previous section.

Some papers referred to traits and characteristics exhibited by leaders that facilitate coproduction. These factors included innovativeness, personability, action orientation [ 46 ], courage [ 47 ], passion [ 32 , 46 ], and empathy [ 25 , 46 , 48 ]. “Strong leadership” was often mentioned, albeit without elaboration [ 49 , 50 , 51 , 52 , 53 , 54 , 55 ]. By implication, “strong leadership” appeared to include providing clear direction and guidance, having a clear vision [ 53 ], holding onto a vision [ 34 ], and keeping the vision alive for the team [ 43 ].

Other researchers noted a more collaborative and democratic leadership style that is characterized by listening, transparency, deliberation, and nurturing coproductive behaviours [ 27 , 30 , 48 ]. Senior leaders could use a “top-down” approach to promote user involvement. Alternatively, they could “learn to manage horizontally not top down; embrace ground up initiatives; [and] aim to empower partners” [ 32 , 45 , 51 ] and be “open to changes that would disturb traditional relationships and power disparities between service users and providers” [ 41 ]. Respondents to a survey of participants in a peer-led support network favoured a traditional directive model of leadership alongside a more facilitative and enabling style [ 56 ]. However, they found it challenging to transition to a more distributed and collective leadership approach.

Co-leadership

The terms “co-lead”, “co-leadership” and “dual leadership” refer to situations in which a formal leadership role is allocated to more than one person, in which context the relevant people may represent different institutions or different groups, e.g., different professional groups, researchers and service users/citizens, or teachers and students [ 28 , 31 , 40 , 41 , 57 , 58 ]. Coleads were defined as “individuals who led and made joint decisions” [ 59 ]. Some papers explored the leadership role of service users or community members in the coproduction of research related to health or wellbeing [ 35 , 60 , 61 ]. In these studies, areas of research were proposed by patients/community members, who then collaborated with academic researchers, thereby playing an equal or leading role. Coleadership was reported to result in shared learning.

Collective leadership

Few discernible differences among “ shared”, “distributed” and “collective” leadership were found in the papers included in this review. The approaches examined in this context were characterized by distributed roles and responsibilities in which different individuals’ skills and expertise were identified as best suited to the task at hand. Shared leadership depends on willingness on the part of leaders (implicitly non-community leaders) to be challenged and directed by community members rather than rigidly maintaining their previous conceptions of the issues and the appropriate means of addressing them [ 36 ].

Ward, De Brún, Beirne, Conway, Cunningham, English, Fitzsimons, Furlong, Kane and Kelly [ 62 ] referred to collective leadership as an emergent and dynamic team phenomenon. Other authors argued for a more structured approach to shared leadership [ 36 , 41 ] or distributed leadership [ 28 , 42 , 56 , 59 , 63 ]. Such an approach could involve allocating specific roles to service users, engaging them in a formal structure and/or enabling them to set an agenda [ 41 ], specifying shared roles and responsibilities [ 36 ], and/or providing dedicated support to lay “champions” in research studies [ 28 ]. Various benefits were attributed to collective leadership, such as empowering people to speak up [ 36 , 51 ] and feel engaged.

Nine practices associated with leading coproduction

We identified nine processes that encompass wide-ranging activities and interactions between individuals and groups with regard to leading the coproduction of health and wellbeing. As Farr noted, “Coproduction and codesign […] involves facilitating, managing and co-ordinating a complex set of psychological, social, cultural and institutional interactions” [ 64 ]. In some cases, these processes naturally align with certain actors—for instance, senior leaders play key roles in the tasks of initiating coproduction and implementing and sustaining its results—but other processes (championing coproduction, establishing trusting relationships, and ensuring good communication) are applicable to any and all participants in the coproduction process. Similarly, some of these practices occur at particular timepoints in a coproduction arc (namely, during the stages of initiation or implementation), while others can occur at any or all timepoints (i.e., during the assimilation stage or beyond). Deliberately considering the most suitable leadership model with regard to the aims and context of an initiative is useful at the start, but reflecting on the operation and appropriateness of the model is always salient.

Initiating coproduction

The initiation of coproduction entails recognizing the need for coproduction, dedicating resources, inviting and establishing relevant multi-stakeholder coproduction networks, and coproducing a vision and goals.

It has been argued that senior leaders act as gatekeepers for coproduction because they must recognize the need for it [ 45 ]. Senior leaders play a role in the task of determining the extent to which communities are given the opportunity to influence service design and integration [ 38 , 51 ]. Coproduction requires resources (principally time and money but also networks), which can be used to take advantage of other resources such as skills [ 29 , 31 , 34 , 40 ]. Senior leaders often control or provide access to such resources, which means that they are best positioned to initiate coproduction initiatives [ 41 , 65 ]. However, the findings of a cross-national study on the coproduction of policy showed that, in practice, senior leaders’ control over resources meant that they tended to define the means, methods and forms of participation [ 65 ].

In the task of establishing a conducive environment for coproduction, it is important to pay attention to which actors (organizations or individuals) are participating in the process [ 33 , 42 , 64 , 66 ] and to factors that may delimit those participants or their involvement [ 36 , 42 , 67 ]. Several papers emphasized the need to ensure that all stakeholders are involved from the outset [ 37 , 38 , 41 , 48 , 51 ]. In the initiation stages, a shared vision should be created [ 36 , 61 , 68 ], goals should be coproduced, and responsibilities should be clearly allocated [ 65 ]. Role clarity, ability, and motivation have been identified as determinants of coproductive behaviour, and leaders must implement arrangements to achieve these goals for coproducers [ 69 ].

Power sharing

It has been argued that coproduction leadership must attend to issues pertaining to power redistribution [ 60 , 61 , 63 , 64 ] and uphold the ideology of coproduction by promoting the values of democracy and transparency [ 30 , 32 , 70 , 71 , 72 , 73 , 74 ]. This process can occur at different levels.

At the macro system level, several cultural shifts have been implicated in the redistribution of power – a shift in current professional and stakeholder identities; more fluid, flattened and consensus-based ways of working; and a willingness to accommodate ‘messy’ issues [ 75 ]. The last of those issues was highlighted by Hopkins, Foster and Nikitin [ 29 , s 192], who suggested that coproduction requires service providers to “sit more easily with the unknown, to be comfortable in not having all the answers.” Similarly, “The challenge is that to be transformative, power must be shared with health service users. To do this entails building new relationships and fostering a new culture in health-care institutions that is supportive of participatory approaches” [ 42 , p 379].

At the meso level, several practices could be used to share power. Greenhalgh, Jackson, Shaw and Janamian [ 30 ] identified the importance of equitable decision-making practices and “evenly distributed power constellations.” This goal can be achieved, for instance, by ensuring that service users represent a majority on the project management committee or in codesign events with the goal of challenging dominant professional structures and discourses [ 37 ]. Other scholars called for clear roles and responsibilities [ 38 , 59 , 65 ]. Mulvale, Moll, Miatello, Robert, Larkin, Palmer, Powell, Gable and Girling [ 36 ] recommended the establishment of shared roles and responsibilities, the creation of a representative expert panel to resolve stalemates, and possibly the implementation of formal agreements regarding data and reporting. Importantly, however, Greenhalgh, Jackson, Shaw and Janamian [ 30 ] noted that governance structures and processes alone do not automatically overcome the subtle and inconspicuous uses of power. Farr [ 64 ] recommended the constant practice of critical reflection and dialogue and posed several questions for participants to consider: who is involved, what the interactions are like, how coproduction efforts are implemented within and across structures, and what changes are made.

Although sharing power has been described as an essential component in coproduction, the involvement of stakeholders does not necessarily entail empowerment [ 47 ], and case studies have demonstrated that service improvement initiatives that involve citizens or service users can be instrumental and effective with regard to improving services without enhancing or sharing power or political consciousness if stakeholders are invited but power is not shared [ 32 ]. Farr [ 64 ] noted that rather than coproduction being inherently emancipatory, coproduction and codesign processes can have either dominating or emancipatory effects [ 33 ], and the exclusion of vulnerable groups from coproduction has the potential to reinforce existing inequities [ 75 ].

Training and development for emerging leadership

The importance of appropriate training and mutual learning was noted in several papers [ 36 , 42 , 48 , 63 , 69 , 76 , 77 ]. Implicitly, training for professionals was framed in terms of training in the process of sharing power with service users or facilitating collaboration, whereas training for service users was framed as capacity-building in terms of collaboration and/or leadership. In one case study focusing on coproduced research, participants rejected the notion of “training” from academic researchers with the aim of avoiding suggesting that a certain level of “expertise” needed to be transferred [ 60 ].

Playing a leadership role can be empowering [ 51 , 71 ], but for some individuals, it can be overwhelming [ 71 ]. Leading coproduction requires practice and the development of skills and capacities [ 26 , 48 ]. In some initiatives, lay partners were initially involved in limited roles and gradually took on more responsible leadership tasks over time [ 28 , 42 , 78 ]. In addition, community members’ level of involvement was flexible—they could be participants or take on additional roles as volunteers, paid staff members or directors of organizations. This flexibility offered participants the opportunity to "begin sharing, as opposed to shouldering, the burden of involvement” [ 71 ].

The provision of support

Support is necessary throughout the coproduction process from its outset to the stages of implementation and sustainment [ 25 , 34 , 68 ]. Key dimensions of support include facilitating, advocating for, and championing coproduction. Project management is instrumental to the smooth operation and facilitation of coproduction [ 34 , 35 , 37 , 44 ]. Several facilitation activities are conducted by project leaders and facilitators [ 41 , 42 , 59 , 61 , 78 ]. These activities include holding onto a vision and keeping it alive for the team, ensuring that the project remains on track, and helping maintain momentum. In one codesign case study, facilitators helped people focus on quick wins with the goal of maintaining motivation and engagement; they "needed to support movement from inaction to action, by sifting through group ideas to fix a plan" [ 34 ]. Although these authors acknowledged that this approach may have limited coproduction, they argued that such initiatives would not be sustainable if they were perceived to be “unfeasible.”

Another key function entails advocating for and championing coproduction initiatives to ensure that the process remains ongoing [ 25 , 28 , 29 , 30 , 31 , 32 , 37 , 41 , 74 , 79 ]. Senior leaders play an important role in the task of championing coproduction, and their support has often been described as an important success factor [ 34 , 38 , 39 , 43 , 80 ]. However, effective champions could equally include health care professionals [ 37 ], experts in coproduction [ 51 ], researchers [ 35 , 60 , 61 ], volunteers [ 51 ] or other citizens [ 41 , 61 ]. Champions with lived experience can gain the confidence of their peers and help create understanding among service providers [ 28 , 36 ].

Establishing trusting relationships

Coproduction is essentially relational and requires concerted efforts to establish trusting relationships and a sense of commitment. The importance of trust among stakeholders in coproduction has been noted in several papers [ 28 , 30 , 36 , 37 , 38 , 46 , 48 , 64 , 74 , 81 , 82 ]. In the field of health research, it is difficult to secure funding for the process of establishing relationships and working in the context of partnerships during the early stages of development [ 25 ]. It can therefore be helpful to base recruitment for coproduction initiatives on pre-existing trusting relationships [ 36 ]. If such pre-existing trusting relationships do not exist, policy-makers and senior leaders play a role in the creation of frameworks that can facilitate the development of trust both among organizations and between organizations and citizens, such as political and bureaucratic commitment on the part of regional and local governments and the engagement of actors who play a “boundary-spanning” role in the relationships between service providers, non-government organizations and communities [ 38 ]. Trust is established based on clear responsibilities [ 38 ] and adherence to the principles of engagement in coproduction. In addition to these frameworks, individual leaders must develop trust through interactions with coproducers, using collaborative skills such as those pertaining to communication and listening [ 48 ]. In one case study, through the frank sharing of the organizational, financial, and governance challenges and opportunities faced by stakeholders, people reached a growing understanding and appreciation of each other’s positions, which engendered trust [ 30 ]. Mulvale, Moll, Miatello, Robert, Larkin, Palmer, Powell, Gable [ 36 ] highlighted the importance of understanding and responding to participants’ histories, contexts, and cultural differences.

Commitment can be viewed as more important than resources [ 59 ]. The commitment to and engagement in coproduction exhibited by an organization’s senior leaders demonstrate organizational commitment and lend credibility to coproduction initiatives [ 25 , 34 , 38 , 41 , 47 , 59 , 80 , 83 ]. On some occasions, coproduction initiatives are reported to senior leaders, while on other occasions, the senior leaders were part of the coproduction team. Senior leaders who adopt a more hands-on approach serve as role models [ 25 ], advocating for patient engagement and engendering commitment on the part of staff and patients [ 28 ]. In public health initiatives, buy-in from community leaders confers legitimacy on innovations, helps ensure community trust [ 61 , 78 ], increases the engagement of community members [ 78 ] and is key to a project’s success [ 83 ].

Communication

Communication is a key activity in coproduction, and leaders must establish an environment that is conducive to “epistemological tolerance” [ 47 ], such that different perspectives are valued and appreciated. Such environments facilitate dialogue among partners [ 28 , 30 , 35 , 51 ] and allow critical voices to be heard [ 42 ] . Open dialogue among stakeholders is a starting point for the task of identifying the sources of assumptions and stereotypes, which is itself a prerequisite for change in attitudes and practice [ 28 ]. Project leaders must also facilitate accessible and transparent dialogue and ensure the equal representation of all stakeholders, including those who are less able to communicate verbally [ 57 , 71 ]. Professional leaders are responsible for critically reviewing their professional norms, organizational/institutional processes and past and present policies and practices [ 55 , 75 ].

Dealing with multiple stakeholders, which is inevitably required in coproduction, requires addressing multiple perspectives in an attempt to bring them together. This task frequently involves a degree of conflict and peace negotiation [ 30 , 34 , 41 , 48 , 61 , 64 ]. Leaders should be alert to conflict and power dynamics [ 34 , 36 ]. It may be necessary for meeting chairs to encourage participants to move on from their familiar, entrenched positions to avoid descending into circular arguments and stalemates (Chisholm et al. 2018). This task could require the injection of a critical voice, as Greenhalgh explained:

“Meeting chairs were selected for their leadership qualities, ability to identify and rise above “groupthink” (bland consensus was explicitly discouraged), and commitment to ensuring that potential challenges to new ideas were identified and vigorously discussed. They set an important ethos of constructive criticism and creative innovation, with the patient experience as the central focus. They recognized that if properly handled, conflict was not merely healthy and constructive, but an essential process in achieving successful change in a complex adaptive system. ” [ 30 ]

Leaders must acknowledge the facts that discomfort can arise when more equitable relationships are established [ 61 ] and that challenges to professional identity [ 84 ] and the loss of control [ 72 ] are factors in this process.

Networking refers to the practice of establishing and maintaining relationships with various stakeholders both within and outside the coproduction initiative. Since coproduction involves working with different stakeholders in networks, several papers have discussed the vital mediating processes associated with this context.

“Bridging, brokering and boundary spanning roles have a key role in cross fertilization of ideas between groups, for generating new ideas and for increasing understanding and cooperation” [ 32 , 53 ].

In policy-making, it is helpful to develop coordination structures and processes such as cross-sector working groups and committees, intersector communication channels [ 65 ], and relationship and dialogue structures [ 42 ]. Community representatives can play a mediating role between individuals and public organizations and may alleviate professionals’ concerns regarding the transaction costs of coproduction in the planning and management of services [ 26 , 81 ]. However, these representatives may or may not use this power to amplify the voices of individual coproducers [ 81 ].

An important role of project leaders is that of the “broker” [ 32 , 85 ], who focuses on mediating among different stakeholders in an attempt to align their perspectives [ 26 ,  37 ,  72 , 86 ]. Another role focuses on spanning the boundaries across sites [ 50 ], between local service providers [ 68 ], or among local services, non-government organizations and the community [ 38 ]. Bovaird, drawing on a number of cases of coproduction, came to the following conclusion:

“ there is a need for a new type of public service professional: the coproduction development officer, who can help to overcome the reluctance of many professionals to share power with users and their communities and who can act internally in organizations (and partnerships) to broker new roles for coproduction between traditional service professionals, service managers, and the political decision-makers who shape the strategic direction of the service system.” [ 81 ]

Orchestration

This practice involves reflecting on and improving coproduction itself. It includes activities such as evaluating the effectiveness of coproduction efforts, assessing the impact of coproduction on outcomes, and making adjustments to improve the coproduction process. Several papers have addressed the roles of local government or public managers or health professionals in overseeing and (as we refer to this process) ‘orchestrating’ the networks involved in coproduction at the community or local government level [ 30 , 33 , 65 , 74 , 87 ]. Orchestration involves recruiting the appropriate actors as noted above as well as directing and coordinating activities, thereby ensuring that the whole is more than the sum of its parts. As part of their orchestration work, leaders play a role in the task of managing risk in service innovation [ 55 , 87 ] and must commit to self-reflexivity and a critical review of norms, policies and practices to alert themselves to any unintended negative consequences and strive to counteract them [ 55 ]. Sturmberg, Martin and O’Halloran [ 88 ] used the metaphor of ‘conducting’ to describe the function of leadership in health care – i.e., leading the orchestra through inspiration and empowerment rather than control, leading to the provision of feedback as the performance unfolds.

From a public service perspective, Powers and Thompson [ 69 ] argued that coproduction requires the leader (“usually a public official”) to mobilize the community on behalf of the public good, organize the provision of the good, create incentives, and supervise the enforcement of community norms. Sancino [ 74 ] argued that local governments play a ‘meta-coproduction role’ that requires them to maximize the coproduction and peer-production of community outcomes by taking into account community contributions and deciding which services should be commissioned or decommissioned (a point that was also made by Wilson [ 87 ]) and to promote coproduction and peer-production in such a way as to promote the coproduction of outcomes that have been decided through a democratic process. In this way, he argued,

"the local government becomes the pivot of different kinds of relationships and networks made up of different actors who collectively assume the responsibility for implementing an overall strategic plan of the community beyond their specific roles and interests." [ 74 ]

Sancino [ 74 ] attempted to draw out the leadership implications of this situation, arguing that rather than focusing on service delivery, public managers must create appropriate conditions for such meta-coproduction. This task entails a directing role based on framing shared scenarios for change in the community through sense-making; an activator role based on activating, mobilizing and consolidating the social capital of the community to promote diffused public leadership; a convenor role based on serving as a meta-manager in the process of self-organizing the knowledge, resources and competencies pertaining to the community in question; and an empowering role based on creating conditions in which peer production and coproduction can be combined to create the corresponding added value (i.e., higher levels of community outcomes) [ 74 ]. This practice essentially focuses on self-assessment and continuous improvement within the coproduction framework.

Implementation

It has been argued that coproduction in services [ 30 , 79 ] or policy-making [ 65 ] may improve implementation. The role of leadership in supporting the implementation of the outcomes of coproduction is essential [ 37 , 41 , 49 , 52 , 64 , 65 , 85 , 86 ]. Leaders can argue for the legitimacy of coproduced innovations [ 89 ] and implement mechanisms aimed at acting on the issues thus raised and continuing to promote patients’ involvement [ 28 , 41 ]. Implementing the outcomes of coproduction relies on outcome-focused leadership [ 30 ]. The results of coproduction initiatives must be transformed into strategic plans and policies [ 41 ], and patient perspectives must be translated into actionable quality improvement initiatives [ 49 ]. Conversely, implementation can be blocked by leaders who fail to respond to the results of coproduction initiatives or who implement policies or procedures that are poorly aligned with the recommendations arising from coproduction [ 30 , 41 ]. It should also be acknowledged that not all demands thus generated can always be met [ 61 ]. Failures of implementation run the risk of stakeholder disillusionment; thus, the management of expectations is important.

A framework for coproduction leadership

When coproduction is initiated, it is possible to consider the actors involved and to imagine various forms of coproduction. In the design process, it is possible to make a deliberate choice with regard to the most appropriate model of leadership, and depending on the leadership model selected (leader-centric, coleadership, or collective leadership), different leadership practices emerge. The nine leadership practices identified can be enacted by different people and in different ways. The leadership of coproduction that thus emerges is shaped by issues such as the model of coproduction, the stakeholders involved, participants’ motivations and the context of coproduction. A main concern lies in the need to design project structures and work practices that are aligned and that enable leadership to emerge. We thus created a table (Table  1 ) that illustrates potential reflective questions in this context.

This discussion highlights and problematizes the two main findings of this systematic review, namely, the need to deliberately consider underlying models of leadership and the complex character of leading coproduction.

The need for the deliberate use of leadership in the plural

A focus on leader-centric approaches and the quality of leaders has characterized public leadership research [ 90 ]. Such a focus is echoed in our findings on coproduction leadership, first with regard to the prominence of senior leaders and, to a lesser extent, facilitators. Politicians were rarely identified in the papers included in our review despite representing some of the main actors identified in a previous review [ 4 , 91 ]. Second, many papers referenced the need for “strong” leaders, and the skills and behaviours of individual leaders were noted. As other researchers have found, despite the focus of this field on relationships and interactions, its emphasis has frequently remained on the individual leader and their ability to engage and inspire followers [ 13 ]. Furthermore, even in papers that emphasized ‘coleadership’ or ‘collective leadership,’ the focus remained on public managers, service managers and facilitators. Very little evidence has been reported concerning individual service users’ or citizens’ leadership of (as distinct from involvement in ) coproduction. Although the involvement of community leaders was reported to play a role in project success, no articles explored this issue.

However, some important exceptions should be noted. For example, some studies exhibited a preference for mixed models, employing both a directive approach (particularly in the beginning) and a more facilitative and distributed leadership approach [ 56 ]. Rycroft-Malone, Burton, Wilkinson, Harvey, McCormack, Baker, Dopson, Graham, Staniszewska and Thompson [ 53 ] concluded that consideration should be given to models that combine hierarchical, directive structures with distributed facilitative forms of leadership.

One explanation for this rather narrow view of leadership is that despite the rapidly increasing number of publications in the general field of coproduction [ 7 , 18 ], empirical studies have still lacked depth with regard to investigations of the leadership of this process. Most empirical studies included in this review mentioned leadership only in passing or derived some conclusions regarding leadership from case studies focusing on other aspects of the coproduction process.

Another explanation for this situation is that although coproduction focuses on partnership, in most cases, senior leaders have control over resources and the power to define the means, methods, extent and forms of participation [ 65 ]. Even shared leadership models seem to rely on traditional leaders’ willingness to share power [ 10 ], as leaders are the actors who invite, facilitate, and support the participation of coleaders. However, some signs of change towards a broader view should be noted. Recent publications have theorized the leadership of coproduction and included case studies that have demonstrated leadership to be a social, collective and relational phenomenon that emerges as a property of interactions among individuals in given contexts [ 13 , 19 ].

The complexity of coproduction leadership practices

Our findings indicate that the leadership of coproduction practices entails challenging and complex tasks. Complexity emerges in cases in which many parts are interrelated in multiple ways. Different kinds of leadership activities may be necessary depending on the stakeholders involved [ 92 ], the context [ 13 ], and the mode, level, and phase of coproduction [ 93 ]. A complexity perspective based on systems thinking is therefore useful [ 13 , 19 ]. All actors involved in coproduction are potential leaders, but for that potential to be realized, the coproduction initiative and its leadership must be framed and comprehended in a more plural way. A recent study on systems thinking and complex adaptive thinking as means of initiating coproduction advocated a collective leadership approach [ 19 ].

Our findings highlight the need for a complex way of making meaning of leadership throughout the coproduction process, such as the ability to be flexible due to circumstances and employ both strong leadership and more facilitative approaches when necessary. Leaders must also promote the values of democracy, transparency and the redistribution of power among stakeholders throughout the process [ 64 , 94 ]. These practices and tasks are complex, which must be matched by an inner mental complexity [ 95 , 96 ]. Several practices identified in this research, such as genuinely valuing diverse perspectives, promoting mutual transformative power sharing and welcoming conflicts, require a complex mode of meaning-making that results from psychological development. These issues warrant further exploration. Future studies featuring a thoughtful choice of leadership and complexity models as well as a broader methodological repertoire are thus necessary (see Table  2 for an overview).

Methodological strengths and limitations

A strength of this review lies in its integration of research on the sparse and overlooked issue of leadership in coproduction. Our search strategy, which involved using the key words manag* and lead*, may have excluded some relevant papers. To verify that this approach did not represent an excessively blunt exclusion criterion, we checked 10% of the articles that were excluded based on this criterion. All of these articles would also have been excluded for failing to include any exploration of the management or leadership of coproduction. We therefore determined that this exclusion criterion was justifiable. Many papers did not have an explicit focus on leadership; however, by synthesizing the data, all data were treated as reflections that jointly created a larger pattern, similar to a kaleidoscope. The exclusion of non-peer-reviewed papers is likely to have led to the exclusion of coproduced outputs, which may have offered important insights into the leadership of coproduction, particularly with regard to the experiences of service users and citizens playing leadership roles. In the reporting of this review, the PRISMA guidelines were followed (Additional file 2 ). It should be noted that the lack of reporting bias assessment and certainty assessment represents a limitation of this study.

Future research

Future research (see Table  2 ) should focus on under-represented roles, such as those of politicians and community leaders, and explore emerging collective leadership models based on real-time observational studies. It should also investigate the balance between strong and shared leadership by using qualitative and participatory research methods. Incorporating systems thinking and relevant leadership models can offer new perspectives on collective leadership practices.

Practical implications

This paper explored coproduction leadership practices and revealed that they require a deliberate and plural understanding of leadership roles and tasks. We proposed a framework for coproduction leadership that takes into account the actors involved, the models of leadership, and the leadership practices that emerge in different contexts and during different phases of coproduction. We also provided a set of reflective considerations that can help all actors involved in this process make more deliberate choices regarding the parties involved, leadership models of coproduction, and practices (Table  1 ).

Our systematic review revealed some gaps in the literature on coproduction leadership, such as the lack of attention to the mental complexity of coproduction leaders, the under-representation of service users and citizens as leaders, and the need for more empirical studies that use appropriate models and methods to capture the complexity of coproduction leadership. We suggest that future research should address these gaps, thus contributing to the advancement of coproduction theory and practice.

Our framework also has some practical implications for coproduction leaders and participants. At the start of coproduction process, all people, particularly leaders, must learn more about different models of leadership and how power is shared. Throughout this process, flexibility is necessary because leadership constellations change over time; they emerge and fade away, thus implying different underlying leadership models. A multitude of practices must be implemented throughout the coproduction process. People in leader roles must be aware of their personal strengths and limitations, not only with the goal of sharing leadership but also with the aim of establishing partnerships with others who have competence in certain practices, such as facilitation or addressing conflicts. Reflecting upon the guiding questions can also help illustrate the extent to which power and leadership are being shared. In conclusion, to create more equal power relations over time, we must challenge our current practices and work deliberately to enhance the capacity of individuals and groups to effectively engage in coproduction leadership.

Osborne SP, Radnor Z, Strokosch K. Co-production and the co-creation of value in public services: a suitable case for treatment? Public Manag Rev. 2016;18(5):639–53.

Article   Google Scholar  

Voorberg WH, Bekkers VJJM, Tummers LG. A Systematic Review of Co-Creation and Co-Production: Embarking on the social innovation journey. Public Manag Rev. 2015;17(9):1333–57.

Gassner D, Gofen A. Coproduction investments: Street-level management perspective on coproduction. Cogent Business Management. 2019;6(1):1617023.

Bussu S, Tullia Galanti M. Facilitating coproduction: the role of leadership in coproduction initiatives in the UK. Policy Society. 2018;37(3):347–67.

Brown PR, Head BW. Navigating tensions in co-production: A missing link in leadership for public value. Public Administration (London). 2019;97(2):250–63.

Schlappa H, Imani Y: Who is in the lead? New perspectives on leading service co-production. In: Co-production and co-creation. edn.: Routledge; 2018: 99–108.

Masterson D. Mapping definitions of co-production and co-design in health and social care:systematic scoping review providing lessons for the future. Health Expect. 2022;25(3):902–13.

Article   PubMed   PubMed Central   Google Scholar  

Osborne SP, Strokosch K. It takes Two to Tango? Understanding the C o-production of Public Services by Integrating the Services Management and Public Administration Perspectives. Br J Manag. 2013;24:S31–47.

Parry KW, Bryman A: Leadership in organizations. In: The SAGE Handbook of Organization Studies. edn. Edited by Clegg SR, Hardy C, Lawrence TB, Nord WR. London: SAGE; 2006: 446–468.

Denis J-L, Langley A, Sergi V. Leadership in the plural. Acad Manag Ann. 2012;6(1):211–83.

Drath WH, McCauley CD, Palus CJ, Van Velsor E, O’Connor PMG, McGuire JB. Direction, alignment, commitment: Toward a more integrative ontology of leadership. Leadersh Q. 2008;19(6):635–53.

Ospina SM, Foldy EG, Fairhurst GT, Jackson B. Collective dimensions of leadership: Connecting theory and method. Human Relations. 2020;73(4):441–63.

Schlappa H, Imani Y, Nishino T. Relational leadership: an analytical lens for the exploration of co-production. In: Loeffler E, Bovaird T, editors. The Palgrave Handbook of Co-Production of Public Services and Outcomes. edn.: Springer; 2021. p. 471–90.

Loeffler E. Co-production of public services and outcomes. 1st ed. In: The Palgrave Handbook of Co-Production of Public Services and Outcomes. Cham, Switzerland: Palgrave Macmillan; 2021. p. 387–408. https://link.springer.com/book/10.1007/978-3-030-55509-2 .

Kotter JP. Force for change: how leadership differs from management: Simon and Schuster; 2008.

Palumbo R. Contextualizing co-production of health care: a systematic literature review. Intern J Public Sector Management. 2016;29(1):72–90.

Palumbo R, Manesh MF: Travelling along the public service co-production road: a bibliometric analysis and interpretive review. Public management review 2021:1–37.

Fusco F, Marsilio M, Guglielmetti C. Co-production in health policy and management: a comprehensive bibliometric review. BMC Health Serv Res. 2020;20(1):504–504.

Docherty K: Exploring Collective Leadership and Co-Production: An Empirical Study. In: Processual Perspectives on the Co-Production Turn in Public Sector Organizations. edn.: IGI Global; 2021: 130–155.

Turnhout E, Metze T, Wyborn C, Klenk N, Louder E. The politics of co-production: participation, power, and transformation. Current Opinion Environmental Sustainability. 2020;42:15–21.

Kjellström S, Areskoug-Josefsson K, Andersson Gäre B, Andersson A-C, Ockander M, Käll J, McGrath J, Donetto S, Robert G. Exploring, measuring and enhancing the coproduction of health and well-being at the national, regional and local levels through comparative case studies in Sweden and England: the “Samskapa” research programme protocol. BMJ Open. 2019;9(7):e029723–e029723.

Grant MJ, Booth A. A typology of reviews: an analysis of 14 review types and associated methodologies. Health Info Libr J. 2009;26(2):91–108.

Article   PubMed   Google Scholar  

Denyer D, Tranfield D: Producing a Systematic Review. In: The Sage handbook of organizational research methods. edn. Edited by Buchanan D, Bryman A: Sage Publications Ltd; 2009.

Sicilia M, Sancino A, Nabatchi T, Guarini E. Facilitating co-production in public services: management implications from a systematic literature review. Public Money Management. 2019;39(4):233–40.

Staniszewska S, Denegri S, Matthews R, Minogue V. Reviewing progress in public involvement in NIHR research: developing and implementing a new vision for the future. BMJ Open. 2018;8(7):e017124.

Byrne L, Stratford A, Davidson L. The global need for lived experience leadership. Psychiatr Rehabil J. 2018;41(1):76.

Nicol E, Sang B. A co-productive health leadership model to support the liberation of the NHS. J R Soc Med. 2011;104(2):64–8.

Hafford-Letchfield T, Simpson P, Willis PB, Almack K. Developing inclusive residential care for older lesbian, gay, bisexual and trans (LGBT) people: An evaluation of the Care Home Challenge action research project. Health Soc Care Community. 2018;26(2):e312–20.

Hopkins L, Foster A, Nikitin L. The process of establishing Discovery College in Melbourne. Ment Health Soc Incl. 2018;22(4):187–94.

Greenhalgh T, Jackson C, Shaw S, Janamian T. Achieving research impact through co-creation in community-based health services: literature review and case study. Milbank Q. 2016;94(2):392–429.

Green S, Beveridge E, Evans L, Trite J, Jayacodi S, Evered R, Parker C, Polledri L, Tabb E, Green J. Implementing guidelines on physical health in the acute mental health setting: a quality improvement approach. Int J Ment Heal Syst. 2018;12(1):1–9.

Poocharoen O-o. Ting B: Collaboration, co-production, networks: Convergence of theories. Public Manag Rev. 2015;17(4):587–614.

Topp SM, Sharma A, Chileshe C, Magwende G, Henostroza G, Moonga CN. The health system accountability impact of prison health committees in Zambia. International journal for equity in health. 2018;17(1):74–74.

Chisholm L, Holttum S, Springham N. Processes in an experience-based co-design project with family carers in community mental health. SAGE Open. 2018;8(4):2158244018809220.

Mader LB, Harris T, Kläger S, Wilkinson IB, Hiemstra TF. Inverting the patient involvement paradigm: defining patient led research. Res Involv Engagem. 2018;4(1):1–7.

Mulvale G, Moll S, Miatello A, Robert G, Larkin M, Palmer VJ, Powell A, Gable C, Girling M. Codesigning health and other public services with vulnerable and disadvantaged populations: Insights from an international collaboration. Health Expect. 2019;22(3):284–97.

Larkin M, Boden ZV, Newton E. On the brink of genuinely collaborative care: experience-based co-design in mental health. Qual Health Res. 2015;25(11):1463–76.

Farooqi SA. Co-production: what makes co-production work? Evidence from Pakistan. Int J Public Sect Manag. 2016;29(4):381–95.

Murphy L, Wells JS, Lachman P, Bergin M. A quality improvement initiative in community mental health in the Republic of Ireland. Health Sci J. 2015;9(1):1.

Google Scholar  

Redwood S, Brangan E, Leach V, Horwood J, Donovan JL. Integration of research and practice to improve public health and healthcare delivery through a collaborative’Health Integration Team’model-a qualitative investigation. BMC Health Serv Res. 2016;16(1):1–13.

Bombard Y, Baker GR, Orlando E, Fancott C, Bhatia P, Casalino S, Onate K, Denis J-L, Pomey M-P. Engaging patients to improve quality of care: a systematic review. Implement Sci. 2018;13(1):1–22.

Marston C, Hinton R, Kean S, Baral S, Ahuja A, Costello A, Portela A. Community participation for transformative action on women’s, children’s and adolescents’ health. Bull World Health Organ. 2016;94(5):376–82.

Burhouse A, Lea C, Ray S, Bailey H, Davies R, Harding H, Howard R, Jordan S, Menzies N, White S. Preventing cerebral palsy in preterm labour: a multiorganisational quality improvement approach to the adoption and spread of magnesium sulphate for neuroprotection. BMJ open quality. 2017;6(2):e000189.

Burhouse A, Rowland M, Niman HM, Abraham D, Collins E, Matthews H, Denney J, Ryland H. Coaching for recovery: a quality improvement project in mental healthcare. BMJ Open Quality. 2015;4(1):u206576–w202641.

Millenson ML, DiGioia AM III, Greenhouse PK, Swieskowski D. Turning patient-centeredness from ideal to real: lessons from 2 success stories. J Ambul Care Manage. 2013;36(4):319–34.

Yokota F, Biyani M, Islam R, Ahmed A, Nishikitani M, Kikuchi K, Nohara Y, Nakashima N. Lessons learned from Co-Design and Co-Production in a portable health clinic research project in Jaipur district, India (2016–2018). Sustainability. 2018;10(11):4148.

Cooke J, Langley J, Wolstenholme D, Hampshaw S. Seeing" the difference: the importance of visibility and action as a mark of" authenticity" in co-production: comment on" collaboration and co-production of knowledge in healthcare: opportunities and challenges. Int J Health Policy Manag. 2017;6(6):345.

Sicilia M, Guarini E, Sancino A, Andreani M, Ruffini R. Public services management and co-production in multi-level governance settings. Int Rev Adm Sci. 2016;82(1):8–27.

Bak K, Moody L, Wheeler SM, Gilbert J. Patient and Staff Engagement in Health System Improvement: A Qualitative Evaluation of the Experience-Based Co-design Approach in Canada. Healthcare Quarterly (Toronto, Ont). 2018;21(2):24–9.

Bagot KL, Cadilhac DA, Kim J, Vu M, Savage M, Bolitho L, Howlett G, Rabl J, Dewey HM, Hand PJ. Transitioning from a single-site pilot project to a state-wide regional telehealth service: The experience from the Victorian Stroke Telemedicine programme. J Telemed Telecare. 2017;23(10):850–5.

Bruce G, Wistow G, Kramer R. Connected Care Re-visited: Hartlepool and Beyond. J Integr Care. 2011;19(2):13–21.

Hogan MJ, Johnston H, Broome B, McMoreland C, Walsh J, Smale B, Duggan J, Andriessen J, Leyden KM, Domegan C. Consulting with citizens in the design of wellbeing measures and policies: lessons from a systems science application. Soc Indic Res. 2015;123(3):857–77.

Rycroft-Malone J, Burton CR, Wilkinson J, Harvey G, McCormack B, Baker R, Dopson S, Graham ID, Staniszewska S, Thompson C. Collective action for implementation: a realist evaluation of organisational collaboration in healthcare. Implement Sci. 2015;11(1):1–17.

Sorrentino M, Guglielmetti C, Gilardi S, Marsilio M. Health care services and the coproduction puzzle: filling in the blanks. Administration Society. 2017;49(10):1424–49.

Williams BN, Kang S-C, Johnson J. (Co)-contamination as the dark side of co-production: Public value failures in co-production processes. Public Manag Rev. 2016;18(5):692–717.

Gillard S, Foster R, Turner K. Evaluating the Prosper peer-led peer support network: a participatory, coproduced evaluation. Ment Health Soc Incl. 2016;20(2):80–92.

Greenwood DA, Litchman ML, Ng AH, Gee PM, Young HM, Ferrer M, Ferrer J, Memering CE, Eichorst B, Scibilia R. Development of the intercultural diabetes online community research council: codesign and social media processes. J Diabetes Sci Technol. 2019;13(2):176–86.

Lindsay C, Pearson S, Batty E, Cullen AM, Eadson W. Co-production as a route to employability: Lessons from services with lone parents. Public Administration. 2018;96(2):318–32.

Bell T, Vat LE, McGavin C, Keller M, Getchell L, Rychtera A, Fernandez N. Co-building a patient-oriented research curriculum in Canada. Res Involv Engagem. 2019;5(1):1–13.

Cox N, Clayson A, Webb L. A safe place to reflect on the meaning of recovery: a recovery community co-productive approach using multimedia interviewing technology. Drugs Alcohol Today. 2016;16(1):4–15.

Haynes E, Marawili M, Marika BM, Mitchell AG, Phillips J, Bessarab D, Walker R, Cook J, Ralph AP. Community-based participatory action research on rheumatic heart disease in an Australian Aboriginal homeland: Evaluation of the ‘On track watch’project. Eval Program Plann. 2019;74:38–53.

Ward ME, De Brún A, Beirne D, Conway C, Cunningham U, English A, Fitzsimons J, Furlong E, Kane Y, Kelly A. Using co-design to develop a collective leadership intervention for healthcare teams to improve safety culture. Int J Environ Res Public Health. 2018;15(6):1182.

McGregor J, Repper J, Brown H. “The college is so different from anything I have done”. A study of the characteristics of Nottingham Recovery College. J Mental Health Training, Education Practice. 2014;9(1):3–15.

Farr M. Power dynamics and collaborative mechanisms in co-production and co-design processes. Crit Soc Policy. 2018;38(4):623–44.

Hämäläinen R-M, Aro AR, Lau CJ, Rus D, Cori L, Syed AM. Cross-sector cooperation in health-enhancing physical activity policymaking: more potential than achievements? Health Res Policy Syst. 2016;14(1):33–33.

Farmer J, Taylor J, Stewart E, Kenny A. Citizen participation in health services co-production: a roadmap for navigating participation types and outcomes. Aust J Prim Health. 2017;23(6):509–15.

Morton M, Paice E. Co-Production at the Strategic Level: Co-Designing an Integrated Care System with Lay Partners in North West London. England Int J Integr Care. 2016;16(2):2–2.

Jeffs L, Merkley J, Sinno M, Thomson N, Peladeau N, Richardson S. Engaging Stakeholders to Co-design an Academic Practice Strategic Plan in an Integrated Health System: The Key Roles of the Nurse Executive and Planning Team. Nurs Adm Q. 2019;43(2):186–92.

Powers KJ, Thompson F. Managing Coprovision: Using Expectancy Theory to Overcome the Free-Rider Problem. J Public Adm Res Theory. 1994;4(2):179–96.

Andrews R, Brewer GA. Social capital, management capacity and public service performance: Evidence from the US states. Public Manag Rev. 2013;15(1):19–42.

Budge G, Mitchell A, Rampling T, Down P, Collective B. “It kind of fosters a culture of interdependence”: A participatory appraisal study exploring participants’ experiences of the democratic processes of a peer-led organisation. J Community Applied Social Psychology. 2019;29(3):178–92.

Farmer J, Currie M, Kenny A, Munoz S-A. An exploration of the longer-term impacts of community participation in rural health services design. Soc Sci Med. 2015;141:64–71.

Ersoy A. The spread of coproduction: How the concept reached the northernmost city in the UK. Local Econ. 2016;31(3):410–23.

Article   MathSciNet   Google Scholar  

Sancino A. The meta co-production of community outcomes: Towards a citizens’ capabilities approach. International J Voluntary Nonprofit Organizations. 2016;27(1):409–24.

Nies H. Communities as co-producers in integrated care. Int J Integr Care. 2014;(14):1–4.

Terp M, Laursen BS, Jørgensen R, Mainz J, Bjørnes CD. A room for design: Through participatory design young adults with schizophrenia become strong collaborators: A Room for Design. Int J Ment Health Nurs. 2016;25(6):496–506.

Seid M, Dellal G, Peterson LE, Provost L, Gloor PA, Fore DL, Margolis PA. Co-designing a Collaborative Chronic Care Network (C3N) for inflammatory bowel disease: development of methods. JMIR Hum Factors. 2018;5(1):e8083.

Schaaf M, Topp SM, Ngulube M. From favours to entitlements: community voice and action and health service quality in Zambia. Health Policy Plan. 2017;32(6):847–59.

Oertzen A-S, Odekerken-Schröder G, Brax SA, Mager B. Co-creating services—conceptual clarification, forms and outcomes. J Serv Manag. 2018;29(4):641–79.

Nimegeer A, Farmer J, West C, Currie M. Addressing the problem of rural community engagement in healthcare service design. Health Place. 2011;17(4):1004–6.

Bovaird T. Beyond engagement and participation: User and community coproduction of public services. Public Adm Rev. 2007;67(5):846–60.

Macaulay B. Considering social enterprise involvement in the commissioning of health services in Shetland. Local Econ. 2016;31(5):650–9.

Miller A, Young EL, Tye V, Cody R, Muscat M, Saunders V, Smith ML, Judd JA, Speare R. A community-directed integrated Strongyloides control program in Queensland, Australia. Trop Med Infect Dis. 2018;3(2):48.

Walsh M, Kittler MG, Mahal D. Towards a new paradigm of healthcare: Addressing challenges to professional identities through Community Operational Research. Eur J Oper Res. 2018;268(3):1125–33.

Windrum P. Third sector organizations and the co-production of health innovations. Manag Decis. 2014;5(6):1046–56.

Farmer J, Carlisle K, Dickson-Swift V, Teasdale S, Kenny A, Taylor J, Croker F, Marini K, Gussy M. Applying social innovation theory to examine how community co-designed health services develop: using a case study approach and mixed methods. BMC Health Serv Res. 2018;18(1):1–12.

Wilson G. Co-Production and Self-Care: New Approaches to Managing Community Care Services for Older People1. Social Policy Administration. 1994;28(3):236–50.

Sturmberg JP, Martin CM, O’Halloran D. Music in the Park. An integrating metaphor for the emerging primary (health) care system: Music in the Park. J Eval Clin Pract. 2010;16(3):409–14.

Vennik FD, van de Bovenkamp HM, Putters K, Grit KJ. Co-production in healthcare: rhetoric and practice. Int Rev Adm Sci. 2016;82(1):150–68.

Crosby BC, Bryson JM. Why leadership of public leadership research matters: and what to do about it. Public Manag Rev. 2018;20(9):1265–86.

Clarke D, Jones F, Harris R, Robert G. What outcomes are associated with developing and implementing co-produced interventions in acute healthcare settings? A rapid evidence synthesis. BMJ Open. 2017;7(7):e014650.

Gallan AS, Jarvis CB, Brown SW, Bitner MJ. Customer positivity and participation in services: an empirical test in a health care context. J Acad Mark Sci. 2013;41(3):338–56.

Nabatchi T, Sancino A, Sicilia M. Varieties of Participation in Public Services: The Who, When, and What of Coproduction. Public Adm Rev. 2017;77(5):766–76.

Larsen T, Karlsen JE, Sagvaag H: Keys to unlocking service provider engagement in constrained co-production partnerships. Action Research 2020:1476750320925862.

Kegan R, Laskow Lahey L. How the way we talk can change the way we work : seven languages for transformation. San Francisco: Jossey-Bass Pfeiffer; Wiley; 2001.

Hill R. Thinking like a round table leader: how mental complexity enables leaders to succeed in a complex environment. Journal of Character and Leadership Development. 2021;8(1):116–30.

Download references

Acknowledgements

The authors wish to thank Forte, the Swedish Research Council for Health, Working Life and Welfare. In particular, we would like to thank Mary McCall for valuable help.

Open access funding provided by Jönköping University. The study of Samskapa, a coproduction research programme, received funding from Forte, the Swedish Research Council for Health, Working Life and Welfare, under grant agreement no. 2018–01431.

Author information

Authors and affiliations.

The Jönköping Academy for Improvement of Health and Welfare, School of Health and Welfare, Jönköping University, Barnarpsgatan 39, Jönköping, Sweden

Sofia Kjellström & Daniel Masterson

Florence Nightingale Faculty of Nursing, Midwifery & Palliative Care, King’s College London, London, UK

Sofia Kjellström & Sophie Sarre

You can also search for this author in PubMed   Google Scholar

Contributions

SK and SS performed the data extraction, qualitative synthesis and drafted the manuscript and Table 1 . SK finalized the manuscript. D.M. screened the data from a previous scoping review, provided the search strategy (Additional file 1 : Appendix 1) and constructed the Prisma flowchart. SS compiled sample description in Additional file 2 : Appendix 2. All authors reviewed and approved the manuscript and agreed to be accountable for all aspects of the work.

Corresponding author

Correspondence to Sofia Kjellström .

Ethics declarations

Ethics approval and consent to participate.

Not applicable; according to Swedish law research, ethical approval is not needed for research that does not involve human persons.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1: appendix 1. .

Description of included papers. 

Additional file 2: Appendix 2. 

PRISMA_2020_checklist - Mangement review.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Kjellström, S., Sarre, S. & Masterson, D. The complexity of leadership in coproduction practices: a guiding framework based on a systematic literature review. BMC Health Serv Res 24 , 219 (2024). https://doi.org/10.1186/s12913-024-10549-4

Download citation

Received : 11 April 2023

Accepted : 03 January 2024

Published : 17 February 2024

DOI : https://doi.org/10.1186/s12913-024-10549-4

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

  • Health and welfare

BMC Health Services Research

ISSN: 1472-6963

what is a qualitative systematic literature review

U.S. flag

An official website of the United States government

The .gov means it's official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you're on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

  • Publications
  • Account settings
  • Browse Titles

NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

Galdas P, Darwin Z, Fell J, et al. A systematic review and metaethnography to identify how effective, cost-effective, accessible and acceptable self-management support interventions are for men with long-term conditions (SELF-MAN). Southampton (UK): NIHR Journals Library; 2015 Aug. (Health Services and Delivery Research, No. 3.34.)

Cover of A systematic review and metaethnography to identify how effective, cost-effective, accessible and acceptable self-management support interventions are for men with long-term conditions (SELF-MAN)

A systematic review and metaethnography to identify how effective, cost-effective, accessible and acceptable self-management support interventions are for men with long-term conditions (SELF-MAN).

Chapter 3 qualitative review methods.

The objective of the qualitative metaethnography was to systematically identify experiences of, and perceptions of, interventions or specific activities aimed at supporting or promoting self-management of LTCs among men of differing age, ethnicity and socioeconomic background.

A summary of the methods used in the metaethnography is provided in Appendix 3 , using the enhancing transparency in reporting the synthesis of qualitative research (ENTREQ) reporting standards for qualitative evidence synthesis, developed by Tong et al. 93

The evidence synthesis was conducted using a metaethnography approach originally described by Noblit and Hare. 94 This approach was chosen because of its emphasis on conceptual development and generating new insights (i.e. being interpretive rather than integrative 94 ) and because it is compatible with synthesising all types of qualitative research. 95

Metaethnography involves seven stages: getting started, deciding what is relevant, reading the studies, determining how studies are related to each other, translating studies into each other, synthesising translations and expressing the synthesis; 94 these seven, often overlapping, stages are depicted in Figure 7 .

Seven steps of metaethnography.

  • Step 1: getting started

The first stage involved identifying a ‘worthy’ research question and one that could be addressed through qualitative evidence synthesis. 94 This stage took place in developing the original funding application for the current review and its justification is presented in Chapter 1 .

  • Step 2: deciding what is relevant

The second stage, ‘deciding what is relevant’, was viewed as comprising the search strategy, inclusion criteria and quality appraisal, consistent with the experiences of Atkins et al. 96 These are presented next, before steps 3–7 are described in the section Data extraction strategy and data analysis .

  • Search methods

Search strategy

A comprehensive electronic search strategy ( Appendix 4 ) was developed in liaison with information specialists. It sought to identify all available studies, rather than using purposive sampling to identify all available concepts. Five electronic databases were searched in July 2013 [Cumulative Index to Nursing and Allied Health Literature (CINAHL), EMBASE, MEDLINE, PsycINFO and Social Science Citation Index].

Because of challenges with methodological indexing of qualitative research, 97 the electronic search was complemented by checking reference lists, and using an adapted strategy published elsewhere 98 that includes ‘thesaurus terms’ (keywords indexed in electronic databases, e.g. ‘Qualitative Research’), ‘free text terms’ (commonly used research methodology terms searched for in the titles, abstracts and keywords) and ‘broad-based terms’ (i.e. the broad free-text terms ‘qualitative’, ‘findings’ and ‘interview$’ and the thesaurus term ‘Interviews’). Terms relating to gender were combined with other terms to narrow the search and increase the precision of the strategy (e.g. ‘men’, ‘male’, ‘masculine$’, ‘gender’, ‘sex difference$’, ‘sex factors’).

Study selection: study screening methods and inclusion criteria

Records were initially screened by one reviewer (ZD) on the basis of the title and abstract. Decisions were recorded in EndNote X7.0.2 (Thomson Reuters, CA, USA), a reference management database. All articles identified as potentially eligible for inclusion were obtained in full. Attempts were made to identify and obtain published findings for unpublished literature that was otherwise eligible, for example doctoral theses or conference proceedings.

The full-text literature was screened independently by two reviewers (ZD and PG) using the inclusion criteria listed in Table 5 . Studies that explored the experiences of men alone, or included a clear and explicit comparison between men and women, were included. Studies which focused on self-management experiences of people with LTCs more generally (i.e. did not consider experiences of, or perceptions of, a self-management support intervention or activity) were excluded. The approach to screening was inclusive; for example, studies where the qualitative findings were limited (e.g. Iredale et al. , 99 Ramachandra et al. , 100 Smith et al. 101 ) and mixed-sex studies with limited findings on gender comparisons (e.g. Barlow et al. 102 , 103 ) were retained in case they contributed to the synthesis.

TABLE 5

Screening criteria: qualitative

  • Classification of self-management interventions and support activities in the qualitative evidence synthesis

The original study protocol sought to code self-management interventions and support activities using the most up-to-date version of the taxonomy of BCT. 104 – 106 As in the quantitative review (see Chapter 2 , Coding interventions for analysis ), we found that the level of detail reported on self-management interventions or activities in the qualitative literature was limited in detail, precision and consistency, making coding with the BCT taxonomy unfeasible.

Most of the qualitative literature did not focus on behaviour change per se or seek to address men’s views and experiences of behaviour change techniques; for example, some papers were concerned with the dynamics of social support groups, or the use of other self-management support and information. The BCT taxonomy is applicable to only studies that are judged as targeting behaviour change; we were therefore limited to ‘lifestyle’ and ‘psychological’ studies. Only a minority of the studies ( n  = 13) provided sufficient information on interventions to allow even rudimentary coding with the BCT taxonomy, and these are presented in Appendix 5 . Issues around application of the BCT taxonomy are returned to in the discussion chapter (see Chapter 6 ).

The lack of detail reported in the qualitative literature also made it unfeasible to classify interventions using the system developed for the quantitative review. Whereas the quantitative review concerned trials of specific interventions, approximately half of the studies in the qualitative review 99 , 101 , 107 – 130 included more than one intervention or activity (e.g. ‘any cancer support group’).

We therefore developed a broad system for classifying interventions and support activities that offered a pragmatic way to group studies and make the analysis process more manageable. The categories are shown in Table 6 .

TABLE 6

Categories and descriptions of self-management interventions and support activities in the qualitative evidence synthesis

  • Quality assessment strategy

The purpose of quality appraisal in the review was to provide descriptive information on the quality of the included studies rather than as a basis for inclusion. We considered that studies of weaker quality either would not contribute or would contribute only minimally to the final synthesis. 94 , 131 We therefore chose not to use design-specific appraisal tools (which the original protocol stated we would) because we placed emphasis on conceptual contribution, which did not require a detailed design-specific appraisal of methodological quality. With that in mind, we used the Critical Appraisal Skills Programme (CASP) tool. 132

The CASP tool comprises 10 checklist-style questions (see Appendix 6 ) for assessing the quality of various domains (including aims, design, methods, data analysis, interpretation, findings and value of the research). Because of the checklist nature of the CASP tool, we developed some additional questions informed by other metaethnography studies 96 , 131 that enabled us to extract and record more detailed narrative summaries of the main strengths, limitations and concerns of each study (see Appendix 7 ).

The CASP tool was used in the light of the experiences reported by other researchers who recommended that, despite rather low inter-rater agreement, such an approach ‘encourag[es] the reviewers to read the papers carefully and systematically, and serves as a reminder to treat the papers as data for the synthesis’ (p. 44). 131

Its focus is on procedural aspects of the conduct of the research rather than the insights offered. 133 The quality appraisal (which focused on methodological quality) did not form part of the inclusion criteria because, as recognised by Campbell et al. , 131 it is conceptual quality that is most important for evidence synthesis and it is the process of synthesis that judges the ‘worth’ of studies, with conceptually limited studies making a limited contribution. 94 Additionally, it is acknowledged that agreement is often slight, with low reproducibility. 131 , 133 Appraisal was conducted by two reviewers independently (ZD and PG), with discrepancies resolved through discussion.

Search outcome

The electronic search strategy identified 6330 unique references. Screening based on title and abstract identified 149 papers for full-text screening. Dual screening of these full-text articles identified 34 studies (reported in 38 papers) to be included in the review. Reasons for excluding the remaining 111 articles are shown in Table 7 .

TABLE 7

Reasons for exclusion of full-text articles

Inter-rater agreement on the decision to include was 88.6%. The majority of disagreements ( n  = 17) concerned the definition of self-management intervention or activity. Having discussed the 17 disagreements, we agreed that five studies on which there was disagreement would be included. 100 , 103 , 110 , 116 , 134

An additional four studies were identified through reference checks and efforts to locate published literature linked to unpublished work identified through the electronic search. 111 , 112 , 135 , 136 An additional two papers (women only), although individually ineligible, were located as ‘linked papers’ for two of the original 34 studies, 114 , 120 giving a total of 38 studies (reported in 44 papers), as shown in Figure 8 .

Preferred Reporting Items for Systematic Reviews and Meta-Analyses flow diagram for the qualitative review.

  • Data extraction strategy and data analysis

The lead reviewer (ZD) extracted all papers using data extraction forms previously tested and refined through a pilot study of four papers. All study details (including aim, participant details, methodology, method of data collection and analysis) were extracted into Microsoft Excel ® version 14 (Microsoft Corporation, Redmond, WA, USA) and checked by a second reviewer (PG). Extraction and analysis of study findings was undertaken by a group of coreviewers within the research team (ZD, PG, LK, CB, KM, KH) and followed steps 3–7 of the metaethnography process described by Noblit and Hare. 94 Despite being numbered sequentially, these phases do not occur in a linear process. 94

Step 3: reading the studies

The metaethnography process involved three levels of constructs, as described by Schutz 137 and operationalised by Atkins et al. : 96

  • first-order: participant quotes and participant observations, while recognising that in secondary analysis these represent the participants’ views as selected by the study authors in evidencing their second-order constructs
  • second-order: study authors’ themes/concepts and interpretations, also described by Noblit and Hare 94 as ‘metaphors’
  • third-order: our ‘interpretations of interpretations of interpretations’ (p. 35), 94 based on our analysis of the first-order and second-order constructs extracted from the studies.

Each paper was read in full and copied verbatim into NVivo version 10 (QSR International, Warrington, UK) for line-by-line coding by the lead reviewer. Coding involved repeated reading and line-by-line categorising of first-order and second-order constructs, using participants’ and authors’ words wherever possible, and reading for possible third-order constructs.

Third-order constructs were developed by building second-order constructs into broader categories and themes in a framework which was revised iteratively using the hierarchical functions of the NVivo software (i.e. using ‘parent’ and ‘child’ nodes).

Rather than simply being a synthesis of the second-order constructs, third-order-constructs were derived inductively from the extracted data; this was an interpretive process that was not limited to interpretations offered by the original authors of included studies.

Coding by coreviewers (i.e. other members of the research team) was idiosyncratic but commonly involved working with printed papers, noting key ‘metaphors’ (themes, concepts and ideas) in the margins and highlighting first-order and second-order evidence that supported the coreviewers’ interpretations. The lead reviewer, ZD, met with each coreviewer to discuss/debrief coding decisions and ensure the credibility (i.e. the congruence of coding decisions with the original author interpretations) of the overall analytical process.

Step 4: determining how the studies are related

To offer a ‘way in’ to the synthesis, we adopted a similar approach to that of Campbell et al. : 131 initially grouping studies by the broad categories of self-management intervention and support activity shown in Table 6 . Each coreviewer was allocated one or more category of studies to analyse. The lead reviewer then read each category of studies in the following order: face-to-face group support, online support, online information, information, psychological, lifestyle and ‘various’; within this, she read the studies in alphabetical order of first author rather than nominating ‘key’ papers. All included papers were analysed, rather than reading until saturation of concepts.

The lead reviewer and coreviewer independently completed matrices to report the second-order constructs and emerging third-order constructs for each paper (which for the lead reviewer were based on a more comprehensive line-by-line coding using NVivo). This facilitated the juxtaposing of metaphors and/or constructs alongside each other, leading to initial assumptions about relationships between studies.

Step 5: translating studies into one another

A defining element of metaethnography is the ‘translation’ of studies into each other, whereby metaphors, together with their inter-relationships, are compared across studies. Facilitated by discussions using the matrices of second- and third-order constructs, we translated studies firstly within types of support activity and then, secondly, across types.

The lead reviewer initially developed the constructs in relation to face-to-face support (the largest category of studies) and read other categories of studies with reference to this, using a constant comparison approach to identify and refine concepts. The ‘models’ function in NVivo was used to depict relationships between third-order constructs; this helped to develop the line-of-argument synthesis, which is discussed next.

Step 6: synthesising translations

Studies can be synthesised in three ways: 94

  • reciprocal translation, where the findings are directly comparable
  • refutational translation, where the findings are in opposition
  • a line-of-argument synthesis, where both similarities and contradictions are found and translations are encompassed in one overarching interpretation that aims to discover a whole among the set of parts, uncovering aspects that may be hidden in individual studies.

Because we found similarities and contradictions, we developed a line-of-argument synthesis (rather than reciprocal or refutational translation) that encompassed four key concepts, each of which was based around a set of third-order constructs.

Step 7: expressing the synthesis

The output of the synthesis, that is communicating our third-order concepts and overarching line-of-argument synthesis, is described by Noblit and Hare 94 as ‘expressing the synthesis’ (p. 29). They state that ‘the worth of any synthesis is in its comprehensibility to some audience’ (p. 82), 94 emphasising the importance of communicating the synthesis effectively, being mindful of the intended audience and using concepts and language that are meaningful (and understandable). We worked to make the synthesis comprehensible by discussion with coreviewers and, critically, through involvement of the patient and public involvement (PPI) group. The synthesis is presented in Chapter 5 and will also be expressed through other dissemination activities, for example the SELF-MAN symposium ( www.self-man.com ), mini-manuals and journal publications.

We undertook several steps to enhance the rigour of our analysis. Authors’ themes and interpretations (second-order constructs) were independently extracted by two reviewers, each of whom additionally suggested their own interpretations of the study findings (third-order constructs).

We were influenced by a recent Health Technology Assessment metaethnography which found multiple reviewers offered ‘broad similarities in interpretation, but differences of detail’ (p. x). 131 We therefore treated the lead reviewer’s analyses as the ‘master copy’ and compared these with the coreviewers’ extractions and interpretations. Peer debriefing meetings were held between the lead reviewer and each coreviewer to discuss matrices of second-order and third-order constructs which facilitated the consideration of alternative interpretations.

The third-order constructs and line-of-argument synthesis were further refined at a full-day meeting (January 2014) attended by the lead qualitative reviewer and wider team of five coreviewers involved in coding, extraction, analysis and interpretation (PG, KH, LK, KM, CB).

We identified the need to be reflexive about our interpretations and recognised potential sources of influence on our interpretations; for example, two reviewers (PG, KH) identified having a ‘constructions of masculinity’ lens, and we agreed to focus the line-of-argument synthesis on interpretations offered by authors of studies being synthesised, rather than framing our interpretations around constructions of masculinity. We considered it a strength that the six reviewers involved reflected a wide range of backgrounds and perspectives. Although PPI colleagues were not involved in the coding process, the line-of-argument synthesis and four key concepts were discussed with the PPI group to ensure credibility.

  • Public and patient involvement

The SELF-MAN research team worked with a specially constituted public and patient advisory group comprising men living with one or more LTCs who were involved in either running or attending a LTC support group in the north of England. Members were recruited via the research team’s existing networks. Stakeholders’ support groups were all condition-specific – arthritis ( n  = 1), diabetes ( n  = 1), heart failure ( n  = 2) and Parkinson’s disease ( n  = 1) – although some men lived with multiple LTCs. All stakeholders attended a welcome meeting prior to the commencement of the study to prepare them for the involvement in the research, and were provided with ongoing support and guidance by the chief investigator throughout the research process. Members were reimbursed for travel, expenses and time throughout the duration of the project (in line with current INVOLVE recommendations 138 ).

The overarching aims of PPI in the project were, first, to help ensure that the review findings spoke to the self-management needs and priorities of men with LTCs, and, second, to ensure the development of appropriate outputs that would have benefit and relevance for service users. A recognised limitation of our group was that stakeholder representation was drawn from face-to-face group-based support interventions.

The stakeholder group met on three half-days over the course of the 12-month project. On each occasion, the group provided positive affirmation that the project was being conducted in accordance with its stated objectives. In the first two meetings, the group offered feedback and advice to the investigative team on preliminary and emerging analysis of the qualitative data throughout the research process: specifically, the development of third-order constructs and the line-of-argument synthesis. Responding to their input, we made revisions to some of our interpretations, particularly in relation to the importance of physical aspects of environments in which interventions took place. The group’s input also highlighted the need for future research to address depression as a common and often overlooked comorbidity in men (see Chapter 7 , Recommendations for future research ), and that they welcomed recommendations for sustainability of support groups and improving communication within groups. When considering the key outcomes to be assessed in the quantitative review, stakeholders also recommended that emphasis should be placed on quality-of-life outcome measures when considering whether or not a self-management support intervention is effective.

In the final meeting, the stakeholder group provided detailed recommendations for the content of the Self-Manual: Man’s Guide to Better Self-Management of Long Term Conditions (not yet available). It advised that the guide should be rephrased from ‘how to’ self-manage to ‘how to better ’ self-manage because men may view themselves as already self-managing and therefore not identify with the former.

Six or seven stakeholders attended each meeting. The female partner of one of the men attended and contributed to discussions at each meeting. Members of the group each received reimbursement of travel expenses and a £150 honorarium for each meeting they attended. In the final meeting, the stakeholders provided feedback on their involvement in the research process overall, focusing on what was done well and what could be improved. Feedback indicated that most stakeholders had a positive experience, particularly valuing the opportunity to have their ‘voices heard’ and make a potential impact on future service delivery. Recommendations for improvements mostly centred on ensuring prompt reimbursement of expenses incurred in attending the meetings.

Included under terms of UK Non-commercial Government License .

  • Cite this Page Galdas P, Darwin Z, Fell J, et al. A systematic review and metaethnography to identify how effective, cost-effective, accessible and acceptable self-management support interventions are for men with long-term conditions (SELF-MAN). Southampton (UK): NIHR Journals Library; 2015 Aug. (Health Services and Delivery Research, No. 3.34.) Chapter 3, Qualitative review methods.
  • PDF version of this title (3.9M)

In this Page

Other titles in this collection.

  • Health Services and Delivery Research

Recent Activity

  • Qualitative review methods - A systematic review and metaethnography to identify... Qualitative review methods - A systematic review and metaethnography to identify how effective, cost-effective, accessible and acceptable self-management support interventions are for men with long-term conditions (SELF-MAN)

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

Connect with NLM

National Library of Medicine 8600 Rockville Pike Bethesda, MD 20894

Web Policies FOIA HHS Vulnerability Disclosure

Help Accessibility Careers

statistics

VIDEO

  1. Systematic literature review

  2. Introduction to Systematic Literature Review by Dr. K. G. Priyashantha

  3. Research Methods: Writing a Literature Review

  4. SYSTEMATIC AND LITERATURE REVIEWS

  5. Systematic Literature Review, by Prof. Ranjit Singh, IIIT Allahabad

  6. Systematic Literature Review Paper presentation

COMMENTS

  1. Qualitative systematic reviews: their importance for our understanding of research relevant to pain

    A qualitative systematic review brings together research on a topic, systematically searching for research evidence from primary qualitative studies and drawing the findings together.

  2. Systematic Review

    A systematic review is a type of review that uses repeatable methods to find, select, and synthesize all available evidence. It answers a clearly formulated research question and explicitly states the methods used to arrive at the answer. Example: Systematic review

  3. Guidance on Conducting a Systematic Literature Review

    In this article, through a systematic search on the methodology of literature review, we categorize a typology of literature reviews, discuss steps in conducting a systematic literature review, and provide suggestions on how to enhance rigor in literature reviews in planning education and research. Introduction

  4. A Guide to Writing a Qualitative Systematic Review Protocol to Enhance

    Qualitative systematic reviews should be based on well planned, peer reviewed protocols to enhance the trustworthiness of results and thus their usefulness in clinical practice. Protocols should outline, in detail, the processes which will be used to undertake the review, including key search terms, …

  5. How-to conduct a systematic literature review: A quick guide for

    A Systematic Literature Review (SLR) is a research methodology to collect, identify, and critically analyze the available research studies (e.g., articles, conference proceedings, books, dissertations) through a systematic procedure . An SLR updates the reader with current literature about a subject .

  6. How to Do a Systematic Review: A Best Practice Guide for Conducting and

    We argue that systematic reviews are a key methodology for clarifying whether and how research findings replicate and for explaining possible inconsistencies, and we call for researchers to conduct systematic reviews to help elucidate whether there is a replication crisis.

  7. PDF Systematic Literature Reviews: an Introduction

    Systematic literature reviews (SRs) are a way of synthesising scientific evidence to answer a particular research question in a way that is transparent and reproducible, while seeking to include all published evidence on the topic and appraising the quality of this evidence.

  8. How to Undertake an Impactful Literature Review: Understanding Review

    The systematic literature review (SLR) is one of the important review methodologies which is increasingly becoming popular to synthesize literature in any discipline in general and management in particular. In this article, we explain the SLR methodology and provide guidelines for performing and documenting these studies. ... Qualitative Health ...

  9. Research Guides: Systematic Reviews: Types of Literature Reviews

    Qualitative, narrative synthesis. Thematic analysis, may include conceptual models. Rapid review. Assessment of what is already known about a policy or practice issue, by using systematic review methods to search and critically appraise existing research. Completeness of searching determined by time constraints.

  10. Guidelines for writing a systematic review

    Contemporary issuesGuidelines for writing a systematic review. A key feature of any academic activity is to have a sufficient understanding of the subject area under investigation and thus an awareness of previous research. Undertaking a literature review with an analysis of the results on a specific issue is required to demonstrate sufficient ...

  11. Getting started

    What is a literature review? Definition: A literature review is a systematic examination and synthesis of existing scholarly research on a specific topic or subject. Purpose: It serves to provide a comprehensive overview of the current state of knowledge within a particular field. Analysis: Involves critically evaluating and summarizing key findings, methodologies, and debates found in ...

  12. How to Do a Systematic Review: A Best Practice Guide for Conducting and

    A systematic review is a special type of literature review that confers added advantages. It is "a review of a clearly formulated question that uses systematic and explicit methods to identify, select, and critically appraise relevant research, and to collect and analyze data from the studies that are included in the review" ( Cochrane ...

  13. LibGuides: Systematic Reviews: Introduction to Systematic Reviews

    "A systematic review is a summary of the medical literature that uses explicit and reproducible methods to systematically search, critically appraise, and synthesize on a specific issue. It synthesizes the results of multiple primary studies related to each other by using strategies that reduce biases and random errors." Gopalakrishnan S, Ganeshkumar P. Systematic Reviews and Meta-analysis ...

  14. PDF Qualitative systematic reviews

    Qualitative systematic review: aggregates, integrates, and/or interprets the findings from (usually) qualitative studies (Sandelowski & Barroso, 2007) What are the effects of high-stakes testing on curriculum? Overview of the Qualitative Systematic Review Process Protocol Research Question(s) Inclusion/ Exclusion Criteria Search Strategy

  15. Qualitative and mixed methods in systematic reviews

    Developing the range of methods to address different questions for review creates a challenge in describing and understanding such methods. There are many names and brands for the new methods which may or may not withstand the changes of historical time, but another way to comprehend the changes and new developments is to consider the dimensions on which the approaches to review differ [7, 8].

  16. Literature review as a research methodology: An ...

    In addition, a literature review is an excellent way of synthesizing research findings to show evidence on a meta-level and to uncover areas in which more research is needed, which is a critical component of creating theoretical frameworks and building conceptual models.

  17. Why Qualitative Research Needs More and Better Systematic Review

    Undertaking a qualitative systematic review provides a vital means to know and tune into the past conversation in your topic area that allows the researcher to position themselves and their work substantively, ontologically, theoretically, and methodologically in this landscape.

  18. How to Conduct a Systematic Review: A Narrative Literature Review

    Systematic reviews, according to Wright, et al., are defined as a "review of the evidence on a clearly formulated question that uses systematic and explicit methods to identify, select and critically appraise relevant primary research, and to extract and analyze data from the studies that are included in the review" [ 3 ].

  19. Qualitative Data Analysis in Systematic Reviews

    A qualitative systematic review aggregates integrates and interprets data from qualitative studies, which is collected through observation, interviews, and verbal interactions. Included studies may also use other qualitative methodologies of data collection in the relevant literature. The use of qualitative systematic reviews analyzes the ...

  20. (PDF) Systematic Literature Reviews: An Introduction

    Systematic literature reviews (SRs) are a way of synthesising scientific evidence to answer a particular research question in a way that is transparent and reproducible, while seeking to include ...

  21. Systematic and other reviews: criteria and complexities

    A systematic review follows explicit methodology to answer a well-defined research question by searching the literature comprehensively, evaluating the quantity and quality of research evidence rigorously, and analyzing the evidence to synthesize an answer to the research question.

  22. Qualitative systematic reviews: their importance for our ...

    A rigorous qualitative systematic review can also uncover new understandings, often helping illuminate 'why' and can help build theory. Such a review can answer the question 'What is it like to have chronic pain?' This article presents the different stages of meta-ethnography, which is the most common methodology used for qualitative systematic ...

  23. 'It depends': what 86 systematic reviews tell us about what strategies

    This review updates and extends our previous review of systematic reviews of interventions designed to implement research evidence into clinical practice. To identify potentially relevant peer-reviewed research papers, we developed a comprehensive systematic literature search strategy based on the terms used in the Grimshaw et al. [ 9 ] and ...

  24. Are Systematic Reviews Qualitative or Quantitative

    A systematic review is a type of review in which authors assess the evidence from a large number of studies to answer a specifically formulated clinical question. To know more about how to write a systematic review, click this link. A systematic review is designed to be transparent and replicable.

  25. Comparing Integrative and Systematic Literature Reviews

    There are two approaches to a systematic literature review: qualitative and quantitative. A qualitative systematic review is driven by a systematic process using a guiding logic or an explicit review method. Cho and Egan (2009) used Garrard's Matrix Method (2007) to systematically access and examine 50 empirical studies on action learning.

  26. Defining the process to literature searching in systematic reviews: a

    Background. Systematic literature searching is recognised as a critical component of the systematic review process. It involves a systematic search for studies and aims for a transparent report of study identification, leaving readers clear about what was done to identify studies, and how the findings of the review are situated in the relevant evidence.

  27. Rapid reviews methods series: guidance on rapid qualitative evidence

    This paper forms part of a series of methodological guidance from the Cochrane Rapid Reviews Methods Group and addresses rapid qualitative evidence syntheses (QESs), which use modified systematic, transparent and reproducible methodsu to accelerate the synthesis of qualitative evidence when faced with resource constraints. This guidance covers the review process as it relates to synthesis of ...

  28. The complexity of leadership in coproduction practices: a guiding

    This systematic literature review is based on a study protocol on coproduction research in the context of health and social care sectors [], and data were obtained from a published scoping review, where the full search strategy is provided [].The scoping review set out to identify 'what is out there' and to explore the definitions of the concepts of coproduction and codesign.

  29. Qualitative review methods

    Qualitative review methods - A systematic review and metaethnography to identify how effective, cost-effective, accessible and acceptable self-management support interventions are for men with long-term conditions (SELF-MAN) - NCBI Bookshelf