pubrica academy logo

Why is it important to do a literature review in research?

Scientific Communication in Healthcare industry

The importance of scientific communication in the healthcare industry

importance and role of biostatistics in clinical research, biostatistics in public health, biostatistics in pharmacy, biostatistics in nursing,biostatistics in clinical trials,clinical biostatistics

The Importance and Role of Biostatistics in Clinical Research

 “A substantive, thorough, sophisticated literature review is a precondition for doing substantive, thorough, sophisticated research”. Boote and Baile 2005

Authors of manuscripts treat writing a literature review as a routine work or a mere formality. But a seasoned one knows the purpose and importance of a well-written literature review.  Since it is one of the basic needs for researches at any level, they have to be done vigilantly. Only then the reader will know that the basics of research have not been neglected.

Importance of Literature Review In Research

The aim of any literature review is to summarize and synthesize the arguments and ideas of existing knowledge in a particular field without adding any new contributions.   Being built on existing knowledge they help the researcher to even turn the wheels of the topic of research.  It is possible only with profound knowledge of what is wrong in the existing findings in detail to overpower them.  For other researches, the literature review gives the direction to be headed for its success. 

The common perception of literature review and reality:

As per the common belief, literature reviews are only a summary of the sources related to the research. And many authors of scientific manuscripts believe that they are only surveys of what are the researches are done on the chosen topic.  But on the contrary, it uses published information from pertinent and relevant sources like

  • Scholarly books
  • Scientific papers
  • Latest studies in the field
  • Established school of thoughts
  • Relevant articles from renowned scientific journals

and many more for a field of study or theory or a particular problem to do the following:

  • Summarize into a brief account of all information
  • Synthesize the information by restructuring and reorganizing
  • Critical evaluation of a concept or a school of thought or ideas
  • Familiarize the authors to the extent of knowledge in the particular field
  • Encapsulate
  • Compare & contrast

By doing the above on the relevant information, it provides the reader of the scientific manuscript with the following for a better understanding of it:

  • It establishes the authors’  in-depth understanding and knowledge of their field subject
  • It gives the background of the research
  • Portrays the scientific manuscript plan of examining the research result
  • Illuminates on how the knowledge has changed within the field
  • Highlights what has already been done in a particular field
  • Information of the generally accepted facts, emerging and current state of the topic of research
  • Identifies the research gap that is still unexplored or under-researched fields
  • Demonstrates how the research fits within a larger field of study
  • Provides an overview of the sources explored during the research of a particular topic

Importance of literature review in research:

The importance of literature review in scientific manuscripts can be condensed into an analytical feature to enable the multifold reach of its significance.  It adds value to the legitimacy of the research in many ways:

  • Provides the interpretation of existing literature in light of updated developments in the field to help in establishing the consistency in knowledge and relevancy of existing materials
  • It helps in calculating the impact of the latest information in the field by mapping their progress of knowledge.
  • It brings out the dialects of contradictions between various thoughts within the field to establish facts
  • The research gaps scrutinized initially are further explored to establish the latest facts of theories to add value to the field
  • Indicates the current research place in the schema of a particular field
  • Provides information for relevancy and coherency to check the research
  • Apart from elucidating the continuance of knowledge, it also points out areas that require further investigation and thus aid as a starting point of any future research
  • Justifies the research and sets up the research question
  • Sets up a theoretical framework comprising the concepts and theories of the research upon which its success can be judged
  • Helps to adopt a more appropriate methodology for the research by examining the strengths and weaknesses of existing research in the same field
  • Increases the significance of the results by comparing it with the existing literature
  • Provides a point of reference by writing the findings in the scientific manuscript
  • Helps to get the due credit from the audience for having done the fact-finding and fact-checking mission in the scientific manuscripts
  • The more the reference of relevant sources of it could increase more of its trustworthiness with the readers
  • Helps to prevent plagiarism by tailoring and uniquely tweaking the scientific manuscript not to repeat other’s original idea
  • By preventing plagiarism , it saves the scientific manuscript from rejection and thus also saves a lot of time and money
  • Helps to evaluate, condense and synthesize gist in the author’s own words to sharpen the research focus
  • Helps to compare and contrast to  show the originality and uniqueness of the research than that of the existing other researches
  • Rationalizes the need for conducting the particular research in a specified field
  • Helps to collect data accurately for allowing any new methodology of research than the existing ones
  • Enables the readers of the manuscript to answer the following questions of its readers for its better chances for publication
  • What do the researchers know?
  • What do they not know?
  • Is the scientific manuscript reliable and trustworthy?
  • What are the knowledge gaps of the researcher?

22. It helps the readers to identify the following for further reading of the scientific manuscript:

  • What has been already established, discredited and accepted in the particular field of research
  • Areas of controversy and conflicts among different schools of thought
  • Unsolved problems and issues in the connected field of research
  • The emerging trends and approaches
  • How the research extends, builds upon and leaves behind from the previous research

A profound literature review with many relevant sources of reference will enhance the chances of the scientific manuscript publication in renowned and reputed scientific journals .

References:

http://www.math.montana.edu/jobo/phdprep/documents/phd6.pdf

journal Publishing services  |  Scientific Editing Services  |  Medical Writing Services  |  scientific research writing service  |  Scientific communication services

Related Topics:

Meta Analysis

Scientific Research Paper Writing

Medical Research Paper Writing

Scientific Communication in healthcare

pubrica academy

pubrica academy

Related posts.

importance of review related literature in research study

Statistical analyses of case-control studies

importance of review related literature in research study

PUB - Selecting material (e.g. excipient, active pharmaceutical ingredient) for drug development

Selecting material (e.g. excipient, active pharmaceutical ingredient, packaging material) for drug development

importance of review related literature in research study

PUB - Health Economics of Data Modeling

Health economics in clinical trials

Comments are closed.

News alert: UC Berkeley has announced its next university librarian

Secondary menu

  • Log in to your Library account
  • Hours and Maps
  • Connect from Off Campus
  • UC Berkeley Home

Search form

Conducting a literature review: why do a literature review, why do a literature review.

  • How To Find "The Literature"
  • Found it -- Now What?

Besides the obvious reason for students -- because it is assigned! -- a literature review helps you explore the research that has come before you, to see how your research question has (or has not) already been addressed.

You identify:

  • core research in the field
  • experts in the subject area
  • methodology you may want to use (or avoid)
  • gaps in knowledge -- or where your research would fit in

It Also Helps You:

  • Publish and share your findings
  • Justify requests for grants and other funding
  • Identify best practices to inform practice
  • Set wider context for a program evaluation
  • Compile information to support community organizing

Great brief overview, from NCSU

Want To Know More?

Cover Art

  • Next: How To Find "The Literature" >>
  • Last Updated: Dec 8, 2023 10:11 AM
  • URL: https://guides.lib.berkeley.edu/litreview

Harvey Cushing/John Hay Whitney Medical Library

  • Collections
  • Research Help

YSN Doctoral Programs: Steps in Conducting a Literature Review

  • Biomedical Databases
  • Global (Public Health) Databases
  • Soc. Sci., History, and Law Databases
  • Grey Literature
  • Trials Registers
  • Data and Statistics
  • Public Policy
  • Google Tips
  • Recommended Books
  • Steps in Conducting a Literature Review

What is a literature review?

A literature review is an integrated analysis -- not just a summary-- of scholarly writings and other relevant evidence related directly to your research question.  That is, it represents a synthesis of the evidence that provides background information on your topic and shows a association between the evidence and your research question.

A literature review may be a stand alone work or the introduction to a larger research paper, depending on the assignment.  Rely heavily on the guidelines your instructor has given you.

Why is it important?

A literature review is important because it:

  • Explains the background of research on a topic.
  • Demonstrates why a topic is significant to a subject area.
  • Discovers relationships between research studies/ideas.
  • Identifies major themes, concepts, and researchers on a topic.
  • Identifies critical gaps and points of disagreement.
  • Discusses further research questions that logically come out of the previous studies.

APA7 Style resources

Cover Art

APA Style Blog - for those harder to find answers

1. Choose a topic. Define your research question.

Your literature review should be guided by your central research question.  The literature represents background and research developments related to a specific research question, interpreted and analyzed by you in a synthesized way.

  • Make sure your research question is not too broad or too narrow.  Is it manageable?
  • Begin writing down terms that are related to your question. These will be useful for searches later.
  • If you have the opportunity, discuss your topic with your professor and your class mates.

2. Decide on the scope of your review

How many studies do you need to look at? How comprehensive should it be? How many years should it cover? 

  • This may depend on your assignment.  How many sources does the assignment require?

3. Select the databases you will use to conduct your searches.

Make a list of the databases you will search. 

Where to find databases:

  • use the tabs on this guide
  • Find other databases in the Nursing Information Resources web page
  • More on the Medical Library web page
  • ... and more on the Yale University Library web page

4. Conduct your searches to find the evidence. Keep track of your searches.

  • Use the key words in your question, as well as synonyms for those words, as terms in your search. Use the database tutorials for help.
  • Save the searches in the databases. This saves time when you want to redo, or modify, the searches. It is also helpful to use as a guide is the searches are not finding any useful results.
  • Review the abstracts of research studies carefully. This will save you time.
  • Use the bibliographies and references of research studies you find to locate others.
  • Check with your professor, or a subject expert in the field, if you are missing any key works in the field.
  • Ask your librarian for help at any time.
  • Use a citation manager, such as EndNote as the repository for your citations. See the EndNote tutorials for help.

Review the literature

Some questions to help you analyze the research:

  • What was the research question of the study you are reviewing? What were the authors trying to discover?
  • Was the research funded by a source that could influence the findings?
  • What were the research methodologies? Analyze its literature review, the samples and variables used, the results, and the conclusions.
  • Does the research seem to be complete? Could it have been conducted more soundly? What further questions does it raise?
  • If there are conflicting studies, why do you think that is?
  • How are the authors viewed in the field? Has this study been cited? If so, how has it been analyzed?

Tips: 

  • Review the abstracts carefully.  
  • Keep careful notes so that you may track your thought processes during the research process.
  • Create a matrix of the studies for easy analysis, and synthesis, across all of the studies.
  • << Previous: Recommended Books
  • Last Updated: Jan 4, 2024 10:52 AM
  • URL: https://guides.library.yale.edu/YSNDoctoral

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

  • Knowledge Base

Methodology

  • How to Write a Literature Review | Guide, Examples, & Templates

How to Write a Literature Review | Guide, Examples, & Templates

Published on January 2, 2023 by Shona McCombes . Revised on September 11, 2023.

What is a literature review? A literature review is a survey of scholarly sources on a specific topic. It provides an overview of current knowledge, allowing you to identify relevant theories, methods, and gaps in the existing research that you can later apply to your paper, thesis, or dissertation topic .

There are five key steps to writing a literature review:

  • Search for relevant literature
  • Evaluate sources
  • Identify themes, debates, and gaps
  • Outline the structure
  • Write your literature review

A good literature review doesn’t just summarize sources—it analyzes, synthesizes , and critically evaluates to give a clear picture of the state of knowledge on the subject.

Instantly correct all language mistakes in your text

Upload your document to correct all your mistakes in minutes

upload-your-document-ai-proofreader

Table of contents

What is the purpose of a literature review, examples of literature reviews, step 1 – search for relevant literature, step 2 – evaluate and select sources, step 3 – identify themes, debates, and gaps, step 4 – outline your literature review’s structure, step 5 – write your literature review, free lecture slides, other interesting articles, frequently asked questions, introduction.

  • Quick Run-through
  • Step 1 & 2

When you write a thesis , dissertation , or research paper , you will likely have to conduct a literature review to situate your research within existing knowledge. The literature review gives you a chance to:

  • Demonstrate your familiarity with the topic and its scholarly context
  • Develop a theoretical framework and methodology for your research
  • Position your work in relation to other researchers and theorists
  • Show how your research addresses a gap or contributes to a debate
  • Evaluate the current state of research and demonstrate your knowledge of the scholarly debates around your topic.

Writing literature reviews is a particularly important skill if you want to apply for graduate school or pursue a career in research. We’ve written a step-by-step guide that you can follow below.

Literature review guide

Here's why students love Scribbr's proofreading services

Discover proofreading & editing

Writing literature reviews can be quite challenging! A good starting point could be to look at some examples, depending on what kind of literature review you’d like to write.

  • Example literature review #1: “Why Do People Migrate? A Review of the Theoretical Literature” ( Theoretical literature review about the development of economic migration theory from the 1950s to today.)
  • Example literature review #2: “Literature review as a research methodology: An overview and guidelines” ( Methodological literature review about interdisciplinary knowledge acquisition and production.)
  • Example literature review #3: “The Use of Technology in English Language Learning: A Literature Review” ( Thematic literature review about the effects of technology on language acquisition.)
  • Example literature review #4: “Learners’ Listening Comprehension Difficulties in English Language Learning: A Literature Review” ( Chronological literature review about how the concept of listening skills has changed over time.)

You can also check out our templates with literature review examples and sample outlines at the links below.

Download Word doc Download Google doc

Before you begin searching for literature, you need a clearly defined topic .

If you are writing the literature review section of a dissertation or research paper, you will search for literature related to your research problem and questions .

Make a list of keywords

Start by creating a list of keywords related to your research question. Include each of the key concepts or variables you’re interested in, and list any synonyms and related terms. You can add to this list as you discover new keywords in the process of your literature search.

  • Social media, Facebook, Instagram, Twitter, Snapchat, TikTok
  • Body image, self-perception, self-esteem, mental health
  • Generation Z, teenagers, adolescents, youth

Search for relevant sources

Use your keywords to begin searching for sources. Some useful databases to search for journals and articles include:

  • Your university’s library catalogue
  • Google Scholar
  • Project Muse (humanities and social sciences)
  • Medline (life sciences and biomedicine)
  • EconLit (economics)
  • Inspec (physics, engineering and computer science)

You can also use boolean operators to help narrow down your search.

Make sure to read the abstract to find out whether an article is relevant to your question. When you find a useful book or article, you can check the bibliography to find other relevant sources.

You likely won’t be able to read absolutely everything that has been written on your topic, so it will be necessary to evaluate which sources are most relevant to your research question.

For each publication, ask yourself:

  • What question or problem is the author addressing?
  • What are the key concepts and how are they defined?
  • What are the key theories, models, and methods?
  • Does the research use established frameworks or take an innovative approach?
  • What are the results and conclusions of the study?
  • How does the publication relate to other literature in the field? Does it confirm, add to, or challenge established knowledge?
  • What are the strengths and weaknesses of the research?

Make sure the sources you use are credible , and make sure you read any landmark studies and major theories in your field of research.

You can use our template to summarize and evaluate sources you’re thinking about using. Click on either button below to download.

Take notes and cite your sources

As you read, you should also begin the writing process. Take notes that you can later incorporate into the text of your literature review.

It is important to keep track of your sources with citations to avoid plagiarism . It can be helpful to make an annotated bibliography , where you compile full citation information and write a paragraph of summary and analysis for each source. This helps you remember what you read and saves time later in the process.

The only proofreading tool specialized in correcting academic writing - try for free!

The academic proofreading tool has been trained on 1000s of academic texts and by native English editors. Making it the most accurate and reliable proofreading tool for students.

importance of review related literature in research study

Try for free

To begin organizing your literature review’s argument and structure, be sure you understand the connections and relationships between the sources you’ve read. Based on your reading and notes, you can look for:

  • Trends and patterns (in theory, method or results): do certain approaches become more or less popular over time?
  • Themes: what questions or concepts recur across the literature?
  • Debates, conflicts and contradictions: where do sources disagree?
  • Pivotal publications: are there any influential theories or studies that changed the direction of the field?
  • Gaps: what is missing from the literature? Are there weaknesses that need to be addressed?

This step will help you work out the structure of your literature review and (if applicable) show how your own research will contribute to existing knowledge.

  • Most research has focused on young women.
  • There is an increasing interest in the visual aspects of social media.
  • But there is still a lack of robust research on highly visual platforms like Instagram and Snapchat—this is a gap that you could address in your own research.

There are various approaches to organizing the body of a literature review. Depending on the length of your literature review, you can combine several of these strategies (for example, your overall structure might be thematic, but each theme is discussed chronologically).

Chronological

The simplest approach is to trace the development of the topic over time. However, if you choose this strategy, be careful to avoid simply listing and summarizing sources in order.

Try to analyze patterns, turning points and key debates that have shaped the direction of the field. Give your interpretation of how and why certain developments occurred.

If you have found some recurring central themes, you can organize your literature review into subsections that address different aspects of the topic.

For example, if you are reviewing literature about inequalities in migrant health outcomes, key themes might include healthcare policy, language barriers, cultural attitudes, legal status, and economic access.

Methodological

If you draw your sources from different disciplines or fields that use a variety of research methods , you might want to compare the results and conclusions that emerge from different approaches. For example:

  • Look at what results have emerged in qualitative versus quantitative research
  • Discuss how the topic has been approached by empirical versus theoretical scholarship
  • Divide the literature into sociological, historical, and cultural sources

Theoretical

A literature review is often the foundation for a theoretical framework . You can use it to discuss various theories, models, and definitions of key concepts.

You might argue for the relevance of a specific theoretical approach, or combine various theoretical concepts to create a framework for your research.

Like any other academic text , your literature review should have an introduction , a main body, and a conclusion . What you include in each depends on the objective of your literature review.

The introduction should clearly establish the focus and purpose of the literature review.

Depending on the length of your literature review, you might want to divide the body into subsections. You can use a subheading for each theme, time period, or methodological approach.

As you write, you can follow these tips:

  • Summarize and synthesize: give an overview of the main points of each source and combine them into a coherent whole
  • Analyze and interpret: don’t just paraphrase other researchers — add your own interpretations where possible, discussing the significance of findings in relation to the literature as a whole
  • Critically evaluate: mention the strengths and weaknesses of your sources
  • Write in well-structured paragraphs: use transition words and topic sentences to draw connections, comparisons and contrasts

In the conclusion, you should summarize the key findings you have taken from the literature and emphasize their significance.

When you’ve finished writing and revising your literature review, don’t forget to proofread thoroughly before submitting. Not a language expert? Check out Scribbr’s professional proofreading services !

This article has been adapted into lecture slides that you can use to teach your students about writing a literature review.

Scribbr slides are free to use, customize, and distribute for educational purposes.

Open Google Slides Download PowerPoint

If you want to know more about the research process , methodology , research bias , or statistics , make sure to check out some of our other articles with explanations and examples.

  • Sampling methods
  • Simple random sampling
  • Stratified sampling
  • Cluster sampling
  • Likert scales
  • Reproducibility

 Statistics

  • Null hypothesis
  • Statistical power
  • Probability distribution
  • Effect size
  • Poisson distribution

Research bias

  • Optimism bias
  • Cognitive bias
  • Implicit bias
  • Hawthorne effect
  • Anchoring bias
  • Explicit bias

A literature review is a survey of scholarly sources (such as books, journal articles, and theses) related to a specific topic or research question .

It is often written as part of a thesis, dissertation , or research paper , in order to situate your work in relation to existing knowledge.

There are several reasons to conduct a literature review at the beginning of a research project:

  • To familiarize yourself with the current state of knowledge on your topic
  • To ensure that you’re not just repeating what others have already done
  • To identify gaps in knowledge and unresolved problems that your research can address
  • To develop your theoretical framework and methodology
  • To provide an overview of the key findings and debates on the topic

Writing the literature review shows your reader how your work relates to existing research and what new insights it will contribute.

The literature review usually comes near the beginning of your thesis or dissertation . After the introduction , it grounds your research in a scholarly field and leads directly to your theoretical framework or methodology .

A literature review is a survey of credible sources on a topic, often used in dissertations , theses, and research papers . Literature reviews give an overview of knowledge on a subject, helping you identify relevant theories and methods, as well as gaps in existing research. Literature reviews are set up similarly to other  academic texts , with an introduction , a main body, and a conclusion .

An  annotated bibliography is a list of  source references that has a short description (called an annotation ) for each of the sources. It is often assigned as part of the research process for a  paper .  

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

McCombes, S. (2023, September 11). How to Write a Literature Review | Guide, Examples, & Templates. Scribbr. Retrieved February 26, 2024, from https://www.scribbr.com/dissertation/literature-review/

Is this article helpful?

Shona McCombes

Shona McCombes

Other students also liked, what is a theoretical framework | guide to organizing, what is a research methodology | steps & tips, how to write a research proposal | examples & templates, what is your plagiarism score.

Elsevier QRcode Wechat

  • Research Process

Literature Review in Research Writing

  • 4 minute read
  • 419.1K views

Table of Contents

Research on research? If you find this idea rather peculiar, know that nowadays, with the huge amount of information produced daily all around the world, it is becoming more and more difficult to keep up to date with all of it. In addition to the sheer amount of research, there is also its origin. We are witnessing the economic and intellectual emergence of countries like China, Brazil, Turkey, and United Arab Emirates, for example, that are producing scholarly literature in their own languages. So, apart from the effort of gathering information, there must also be translators prepared to unify all of it in a single language to be the object of the literature survey. At Elsevier, our team of translators is ready to support researchers by delivering high-quality scientific translations , in several languages, to serve their research – no matter the topic.

What is a literature review?

A literature review is a study – or, more accurately, a survey – involving scholarly material, with the aim to discuss published information about a specific topic or research question. Therefore, to write a literature review, it is compulsory that you are a real expert in the object of study. The results and findings will be published and made available to the public, namely scientists working in the same area of research.

How to Write a Literature Review

First of all, don’t forget that writing a literature review is a great responsibility. It’s a document that is expected to be highly reliable, especially concerning its sources and findings. You have to feel intellectually comfortable in the area of study and highly proficient in the target language; misconceptions and errors do not have a place in a document as important as a literature review. In fact, you might want to consider text editing services, like those offered at Elsevier, to make sure your literature is following the highest standards of text quality. You want to make sure your literature review is memorable by its novelty and quality rather than language errors.

Writing a literature review requires expertise but also organization. We cannot teach you about your topic of research, but we can provide a few steps to guide you through conducting a literature review:

  • Choose your topic or research question: It should not be too comprehensive or too limited. You have to complete your task within a feasible time frame.
  • Set the scope: Define boundaries concerning the number of sources, time frame to be covered, geographical area, etc.
  • Decide which databases you will use for your searches: In order to search the best viable sources for your literature review, use highly regarded, comprehensive databases to get a big picture of the literature related to your topic.
  • Search, search, and search: Now you’ll start to investigate the research on your topic. It’s critical that you keep track of all the sources. Start by looking at research abstracts in detail to see if their respective studies relate to or are useful for your own work. Next, search for bibliographies and references that can help you broaden your list of resources. Choose the most relevant literature and remember to keep notes of their bibliographic references to be used later on.
  • Review all the literature, appraising carefully it’s content: After reading the study’s abstract, pay attention to the rest of the content of the articles you deem the “most relevant.” Identify methodologies, the most important questions they address, if they are well-designed and executed, and if they are cited enough, etc.

If it’s the first time you’ve published a literature review, note that it is important to follow a special structure. Just like in a thesis, for example, it is expected that you have an introduction – giving the general idea of the central topic and organizational pattern – a body – which contains the actual discussion of the sources – and finally the conclusion or recommendations – where you bring forward whatever you have drawn from the reviewed literature. The conclusion may even suggest there are no agreeable findings and that the discussion should be continued.

Why are literature reviews important?

Literature reviews constantly feed new research, that constantly feeds literature reviews…and we could go on and on. The fact is, one acts like a force over the other and this is what makes science, as a global discipline, constantly develop and evolve. As a scientist, writing a literature review can be very beneficial to your career, and set you apart from the expert elite in your field of interest. But it also can be an overwhelming task, so don’t hesitate in contacting Elsevier for text editing services, either for profound edition or just a last revision. We guarantee the very highest standards. You can also save time by letting us suggest and make the necessary amendments to your manuscript, so that it fits the structural pattern of a literature review. Who knows how many worldwide researchers you will impact with your next perfectly written literature review.

Know more: How to Find a Gap in Research .

Language Editing Services by Elsevier Author Services:

What is a research gap

What is a Research Gap

Know the diferent types of Scientific articles

  • Manuscript Preparation

Types of Scientific Articles

You may also like.

what is a descriptive research design

Descriptive Research Design and Its Myriad Uses

Doctor doing a Biomedical Research Paper

Five Common Mistakes to Avoid When Writing a Biomedical Research Paper

importance of review related literature in research study

Making Technical Writing in Environmental Engineering Accessible

Risks of AI-assisted Academic Writing

To Err is Not Human: The Dangers of AI-assisted Academic Writing

Importance-of-Data-Collection

When Data Speak, Listen: Importance of Data Collection and Analysis Methods

choosing the Right Research Methodology

Choosing the Right Research Methodology: A Guide for Researchers

Why is data validation important in research

Why is data validation important in research?

Writing a good review article

Writing a good review article

Input your search keywords and press Enter.

  • UConn Library
  • Literature Review: The What, Why and How-to Guide
  • Introduction

Literature Review: The What, Why and How-to Guide — Introduction

  • Getting Started
  • How to Pick a Topic
  • Strategies to Find Sources
  • Evaluating Sources & Lit. Reviews
  • Tips for Writing Literature Reviews
  • Writing Literature Review: Useful Sites
  • Citation Resources
  • Other Academic Writings

What are Literature Reviews?

So, what is a literature review? "A literature review is an account of what has been published on a topic by accredited scholars and researchers. In writing the literature review, your purpose is to convey to your reader what knowledge and ideas have been established on a topic, and what their strengths and weaknesses are. As a piece of writing, the literature review must be defined by a guiding concept (e.g., your research objective, the problem or issue you are discussing, or your argumentative thesis). It is not just a descriptive list of the material available, or a set of summaries." Taylor, D.  The literature review: A few tips on conducting it . University of Toronto Health Sciences Writing Centre.

Goals of Literature Reviews

What are the goals of creating a Literature Review?  A literature could be written to accomplish different aims:

  • To develop a theory or evaluate an existing theory
  • To summarize the historical or existing state of a research topic
  • Identify a problem in a field of research 

Baumeister, R. F., & Leary, M. R. (1997). Writing narrative literature reviews .  Review of General Psychology , 1 (3), 311-320.

What kinds of sources require a Literature Review?

  • A research paper assigned in a course
  • A thesis or dissertation
  • A grant proposal
  • An article intended for publication in a journal

All these instances require you to collect what has been written about your research topic so that you can demonstrate how your own research sheds new light on the topic.

Types of Literature Reviews

What kinds of literature reviews are written?

Narrative review: The purpose of this type of review is to describe the current state of the research on a specific topic/research and to offer a critical analysis of the literature reviewed. Studies are grouped by research/theoretical categories, and themes and trends, strengths and weakness, and gaps are identified. The review ends with a conclusion section which summarizes the findings regarding the state of the research of the specific study, the gaps identify and if applicable, explains how the author's research will address gaps identify in the review and expand the knowledge on the topic reviewed.

  • Example : Predictors and Outcomes of U.S. Quality Maternity Leave: A Review and Conceptual Framework:  10.1177/08948453211037398  

Systematic review : "The authors of a systematic review use a specific procedure to search the research literature, select the studies to include in their review, and critically evaluate the studies they find." (p. 139). Nelson, L. K. (2013). Research in Communication Sciences and Disorders . Plural Publishing.

  • Example : The effect of leave policies on increasing fertility: a systematic review:  10.1057/s41599-022-01270-w

Meta-analysis : "Meta-analysis is a method of reviewing research findings in a quantitative fashion by transforming the data from individual studies into what is called an effect size and then pooling and analyzing this information. The basic goal in meta-analysis is to explain why different outcomes have occurred in different studies." (p. 197). Roberts, M. C., & Ilardi, S. S. (2003). Handbook of Research Methods in Clinical Psychology . Blackwell Publishing.

  • Example : Employment Instability and Fertility in Europe: A Meta-Analysis:  10.1215/00703370-9164737

Meta-synthesis : "Qualitative meta-synthesis is a type of qualitative study that uses as data the findings from other qualitative studies linked by the same or related topic." (p.312). Zimmer, L. (2006). Qualitative meta-synthesis: A question of dialoguing with texts .  Journal of Advanced Nursing , 53 (3), 311-318.

  • Example : Women’s perspectives on career successes and barriers: A qualitative meta-synthesis:  10.1177/05390184221113735

Literature Reviews in the Health Sciences

  • UConn Health subject guide on systematic reviews Explanation of the different review types used in health sciences literature as well as tools to help you find the right review type
  • << Previous: Getting Started
  • Next: How to Pick a Topic >>
  • Last Updated: Sep 21, 2022 2:16 PM
  • URL: https://guides.lib.uconn.edu/literaturereview

Creative Commons

Research Methods

  • Getting Started
  • Literature Review Research
  • Research Design
  • Research Design By Discipline
  • SAGE Research Methods
  • Teaching with SAGE Research Methods

Literature Review

  • What is a Literature Review?
  • What is NOT a Literature Review?
  • Purposes of a Literature Review
  • Types of Literature Reviews
  • Literature Reviews vs. Systematic Reviews
  • Systematic vs. Meta-Analysis

Literature Review  is a comprehensive survey of the works published in a particular field of study or line of research, usually over a specific period of time, in the form of an in-depth, critical bibliographic essay or annotated list in which attention is drawn to the most significant works.

Also, we can define a literature review as the collected body of scholarly works related to a topic:

  • Summarizes and analyzes previous research relevant to a topic
  • Includes scholarly books and articles published in academic journals
  • Can be an specific scholarly paper or a section in a research paper

The objective of a Literature Review is to find previous published scholarly works relevant to an specific topic

  • Help gather ideas or information
  • Keep up to date in current trends and findings
  • Help develop new questions

A literature review is important because it:

  • Explains the background of research on a topic.
  • Demonstrates why a topic is significant to a subject area.
  • Helps focus your own research questions or problems
  • Discovers relationships between research studies/ideas.
  • Suggests unexplored ideas or populations
  • Identifies major themes, concepts, and researchers on a topic.
  • Tests assumptions; may help counter preconceived ideas and remove unconscious bias.
  • Identifies critical gaps, points of disagreement, or potentially flawed methodology or theoretical approaches.
  • Indicates potential directions for future research.

All content in this section is from Literature Review Research from Old Dominion University 

Keep in mind the following, a literature review is NOT:

Not an essay 

Not an annotated bibliography  in which you summarize each article that you have reviewed.  A literature review goes beyond basic summarizing to focus on the critical analysis of the reviewed works and their relationship to your research question.

Not a research paper   where you select resources to support one side of an issue versus another.  A lit review should explain and consider all sides of an argument in order to avoid bias, and areas of agreement and disagreement should be highlighted.

A literature review serves several purposes. For example, it

  • provides thorough knowledge of previous studies; introduces seminal works.
  • helps focus one’s own research topic.
  • identifies a conceptual framework for one’s own research questions or problems; indicates potential directions for future research.
  • suggests previously unused or underused methodologies, designs, quantitative and qualitative strategies.
  • identifies gaps in previous studies; identifies flawed methodologies and/or theoretical approaches; avoids replication of mistakes.
  • helps the researcher avoid repetition of earlier research.
  • suggests unexplored populations.
  • determines whether past studies agree or disagree; identifies controversy in the literature.
  • tests assumptions; may help counter preconceived ideas and remove unconscious bias.

As Kennedy (2007) notes*, it is important to think of knowledge in a given field as consisting of three layers. First, there are the primary studies that researchers conduct and publish. Second are the reviews of those studies that summarize and offer new interpretations built from and often extending beyond the original studies. Third, there are the perceptions, conclusions, opinion, and interpretations that are shared informally that become part of the lore of field. In composing a literature review, it is important to note that it is often this third layer of knowledge that is cited as "true" even though it often has only a loose relationship to the primary studies and secondary literature reviews.

Given this, while literature reviews are designed to provide an overview and synthesis of pertinent sources you have explored, there are several approaches to how they can be done, depending upon the type of analysis underpinning your study. Listed below are definitions of types of literature reviews:

Argumentative Review      This form examines literature selectively in order to support or refute an argument, deeply imbedded assumption, or philosophical problem already established in the literature. The purpose is to develop a body of literature that establishes a contrarian viewpoint. Given the value-laden nature of some social science research [e.g., educational reform; immigration control], argumentative approaches to analyzing the literature can be a legitimate and important form of discourse. However, note that they can also introduce problems of bias when they are used to to make summary claims of the sort found in systematic reviews.

Integrative Review      Considered a form of research that reviews, critiques, and synthesizes representative literature on a topic in an integrated way such that new frameworks and perspectives on the topic are generated. The body of literature includes all studies that address related or identical hypotheses. A well-done integrative review meets the same standards as primary research in regard to clarity, rigor, and replication.

Historical Review      Few things rest in isolation from historical precedent. Historical reviews are focused on examining research throughout a period of time, often starting with the first time an issue, concept, theory, phenomena emerged in the literature, then tracing its evolution within the scholarship of a discipline. The purpose is to place research in a historical context to show familiarity with state-of-the-art developments and to identify the likely directions for future research.

Methodological Review      A review does not always focus on what someone said [content], but how they said it [method of analysis]. This approach provides a framework of understanding at different levels (i.e. those of theory, substantive fields, research approaches and data collection and analysis techniques), enables researchers to draw on a wide variety of knowledge ranging from the conceptual level to practical documents for use in fieldwork in the areas of ontological and epistemological consideration, quantitative and qualitative integration, sampling, interviewing, data collection and data analysis, and helps highlight many ethical issues which we should be aware of and consider as we go through our study.

Systematic Review      This form consists of an overview of existing evidence pertinent to a clearly formulated research question, which uses pre-specified and standardized methods to identify and critically appraise relevant research, and to collect, report, and analyse data from the studies that are included in the review. Typically it focuses on a very specific empirical question, often posed in a cause-and-effect form, such as "To what extent does A contribute to B?"

Theoretical Review      The purpose of this form is to concretely examine the corpus of theory that has accumulated in regard to an issue, concept, theory, phenomena. The theoretical literature review help establish what theories already exist, the relationships between them, to what degree the existing theories have been investigated, and to develop new hypotheses to be tested. Often this form is used to help establish a lack of appropriate theories or reveal that current theories are inadequate for explaining new or emerging research problems. The unit of analysis can focus on a theoretical concept or a whole theory or framework.

* Kennedy, Mary M. "Defining a Literature."  Educational Researcher  36 (April 2007): 139-147.

All content in this section is from The Literature Review created by Dr. Robert Larabee USC

Robinson, P. and Lowe, J. (2015),  Literature reviews vs systematic reviews.  Australian and New Zealand Journal of Public Health, 39: 103-103. doi: 10.1111/1753-6405.12393

importance of review related literature in research study

What's in the name? The difference between a Systematic Review and a Literature Review, and why it matters . By Lynn Kysh from University of Southern California

importance of review related literature in research study

Systematic review or meta-analysis?

A  systematic review  answers a defined research question by collecting and summarizing all empirical evidence that fits pre-specified eligibility criteria.

A  meta-analysis  is the use of statistical methods to summarize the results of these studies.

Systematic reviews, just like other research articles, can be of varying quality. They are a significant piece of work (the Centre for Reviews and Dissemination at York estimates that a team will take 9-24 months), and to be useful to other researchers and practitioners they should have:

  • clearly stated objectives with pre-defined eligibility criteria for studies
  • explicit, reproducible methodology
  • a systematic search that attempts to identify all studies
  • assessment of the validity of the findings of the included studies (e.g. risk of bias)
  • systematic presentation, and synthesis, of the characteristics and findings of the included studies

Not all systematic reviews contain meta-analysis. 

Meta-analysis is the use of statistical methods to summarize the results of independent studies. By combining information from all relevant studies, meta-analysis can provide more precise estimates of the effects of health care than those derived from the individual studies included within a review.  More information on meta-analyses can be found in  Cochrane Handbook, Chapter 9 .

A meta-analysis goes beyond critique and integration and conducts secondary statistical analysis on the outcomes of similar studies.  It is a systematic review that uses quantitative methods to synthesize and summarize the results.

An advantage of a meta-analysis is the ability to be completely objective in evaluating research findings.  Not all topics, however, have sufficient research evidence to allow a meta-analysis to be conducted.  In that case, an integrative review is an appropriate strategy. 

Some of the content in this section is from Systematic reviews and meta-analyses: step by step guide created by Kate McAllister.

  • << Previous: Getting Started
  • Next: Research Design >>
  • Last Updated: Aug 21, 2023 4:07 PM
  • URL: https://guides.lib.udel.edu/researchmethods

Usc Upstate Library Home

Literature Review: Purpose of a Literature Review

  • Literature Review
  • Purpose of a Literature Review
  • Work in Progress
  • Compiling & Writing
  • Books, Articles, & Web Pages
  • Types of Literature Reviews
  • Departmental Differences
  • Citation Styles & Plagiarism
  • Know the Difference! Systematic Review vs. Literature Review

The purpose of a literature review is to:

  • Provide a foundation of knowledge on a topic
  • Identify areas of prior scholarship to prevent duplication and give credit to other researchers
  • Identify inconstancies: gaps in research, conflicts in previous studies, open questions left from other research
  • Identify the need for additional research (justifying your research)
  • Identify the relationship of works in the context of their contribution to the topic and other works
  • Place your own research within the context of existing literature, making a case for why further study is needed.

Videos & Tutorials

VIDEO: What is the role of a literature review in research? What's it mean to "review" the literature? Get the big picture of what to expect as part of the process. This video is published under a Creative Commons 3.0 BY-NC-SA US license. License, credits, and contact information can be found here: https://www.lib.ncsu.edu/tutorials/litreview/

Elements in a Literature Review

  • Elements in a Literature Review txt of infographic
  • << Previous: Literature Review
  • Next: Searching >>
  • Last Updated: Oct 19, 2023 12:07 PM
  • URL: https://uscupstate.libguides.com/Literature_Review
  • Library databases
  • Library website

Library Guide to Capstone Literature Reviews: Role of the Literature Review

The role of the literature review.

Your literature review gives readers an understanding of the scholarly research on your topic.

In your literature review you will:

  • demonstrate that you are a well-informed scholar with expertise and knowledge in the field by giving an overview of the current state of the literature
  • find a gap in the literature, or address a business or professional issue, depending on your doctoral study program; the literature review will illustrate how your research contributes to the scholarly conversation
  • provide a synthesis of the issues, trends, and concepts surrounding your research

importance of review related literature in research study

Be aware that the literature review is an iterative process. As you read and write initial drafts, you will find new threads and complementary themes, at which point you will return to search, find out about these new themes, and incorporate them into your review.

The purpose of this guide is to help you through the literature review process. Take some time to look over the resources in order to become familiar with them. The tabs on the left side of this page have additional information.

Short video: Research for the Literature Review

Short Video: Research for the Literature Review

(4 min 10 sec) Recorded August 2019 Transcript 

Literature review as a dinner party

To think about the role of the literature review, consider this analogy:  pretend that you throw a dinner party for the other researchers working in your topic area. First, you’d need to develop a guest list.

  • The guests of honor would be early researchers or theorists; their work likely inspired subsequent studies, ideas, or controversies that the current researchers pursue.
  • Then, think about the important current researchers to invite. Which guests might agree with each other?  Which others might provide useful counterpoints?
  • You likely won’t be able to include everyone on the guest list, so you may need to choose carefully so that you don’t leave important figures out. 
  • Alternatively, if there aren’t many researchers working in your topic area, then your guest list will need to include people working in other, related areas, who can still contribute to the conversation.

After the party, you describe the evening to a friend. You’ll summarize the evening’s conversation. Perhaps one guest made a comment that sparked a conversation, and then you describe who responded and how the topic evolved. There are other conversations to share, too. This is how you synthesize the themes and developments that you find in your research. Thinking about your literature research this way will help you to present your dinner party (and your literature review) in a lively and engaging way.

Short video: Empirical research

Video: How to locate and identify empirical research for your literature review

(6 min 16 sec) Recorded May 2020 Transcript 

Here are some useful resources from the Writing Center, the Office of Research and Doctoral Services, and other departments within the Office of Academic Support. Take some time to look at what is available to help you with your capstone/dissertation.

  • Familiarize yourself with Walden support
  • Doctoral Capstone Resources website
  • Capstone writing resources
  • Office of Student Research Administration
  • Office of Research and Doctoral Services
  • Visit the Writing Center

You can watch recorded webinars on the literature review in our Library Webinar Archives .

  • Next Page: Scope
  • Office of Student Disability Services

Walden Resources

Departments.

  • Academic Residencies
  • Academic Skills
  • Career Planning and Development
  • Customer Care Team
  • Field Experience
  • Military Services
  • Student Success Advising
  • Writing Skills

Centers and Offices

  • Center for Social Change
  • Office of Academic Support and Instructional Services
  • Office of Degree Acceleration
  • Office of Student Affairs

Student Resources

  • Doctoral Writing Assessment
  • Form & Style Review
  • Quick Answers
  • ScholarWorks
  • SKIL Courses and Workshops
  • Walden Bookstore
  • Walden Catalog & Student Handbook
  • Student Safety/Title IX
  • Legal & Consumer Information
  • Website Terms and Conditions
  • Cookie Policy
  • Accessibility
  • Accreditation
  • State Authorization
  • Net Price Calculator
  • Contact Walden

Walden University is a member of Adtalem Global Education, Inc. www.adtalem.com Walden University is certified to operate by SCHEV © 2024 Walden University LLC. All rights reserved.

  • Systematic review
  • Open access
  • Published: 26 February 2024

Implementation strategies in suicide prevention: a scoping review

  • Jason I. Chen   ORCID: orcid.org/0000-0001-6490-164X 1 , 2 ,
  • Brandon Roth 1 , 2 , 3 ,
  • Steven K. Dobscha 1 , 2 &
  • Julie C. Lowery 4  

Implementation Science volume  19 , Article number:  20 ( 2024 ) Cite this article

2 Altmetric

Metrics details

Implementation strategies can be a vital leveraging point for enhancing the implementation and dissemination of evidence-based suicide prevention interventions and programming. However, much remains unknown about which implementation strategies are commonly used and effective for supporting suicide prevention efforts.

In light of the limited available literature, a scoping review was conducted to evaluate implementation strategies present in current suicide prevention studies. We identified studies that were published between 2013 and 2022 that focused on suicide prevention and incorporated at least one implementation strategy. Studies were coded by two independent coders who showed strong inter-rater reliability. Data were synthesized using descriptive statistics and a narrative synthesis of findings.

Overall, we found that studies most commonly utilized strategies related to iterative evaluation, training, and education. The majority of studies did not include direct measurement of suicide behavior outcomes, and there were few studies that directly tested implementation strategy effectiveness.

Implementation science strategies remain an important component for improving suicide prevention and intervention implementation. Future research should consider the incorporation of more type 3 hybrid designs as well as increased systematic documentation of implementation strategies.

Trial registration

 < de-identified > 

Peer Review reports

Contributions to the literature

Implementation science strategies are an important aspect of supporting the dissemination and implementation of suicide prevention interventions/programming.

There have been limited comprehensive literature reviews characterizing implementation strategies in suicide prevention.

Several implementation strategies were seen as more common (training and education, iterative evaluation), but there were notable gaps for those involving financial and provider support (e.g., cost sharing, financial incentives).

Future research should consider clearer documentation of implementation strategies, more regular measurement of suicide behavior outcomes (e.g., within type 1 and type 2 hybrid studies), and direct testing of implementation strategies to inform the broader suicide prevention field.

Suicide remains a leading cause of death worldwide [ 1 ]. Although suicide rates have decreased in certain regions of the world, rates within the USA have remained elevated over the past 20 years and have continued to rise across demographic groups [ 1 ]. The Socioecological Model of Suicide Prevention posits that suicide risk is multi-factorial and impacted by factors ranging from the individual level (e.g., mental health symptoms, financial challenges) through to the societal level (e.g., health policy, stigma) [ 2 ]. Accordingly, suicide prevention and intervention programming has been developed to address risk across these levels. For example, one such multicomponent intervention approach with demonstrated effectiveness was developed through the Garrett Lee Smith Memorial Act program funded by the Substance Abuse and Mental Health Services Administration [ 3 ]. This program supports multi-component state and tribal suicide prevention initiatives to address not only those with known risk but also increase the capacity of systems to identify and support those at risk [ 3 ]. Unsurprisingly, multi-component prevention programs carry an inherent level of complexity requiring multiple strategies for implementation support. Indeed, research shows this program is effective in decreasing suicide deaths over multiple years with increased effectiveness with more years of active implementation support, highlighting the importance of implementation strategies for suicide prevention efforts [ 4 ].

Systematic reviews have identified several promising interventions for decreasing suicide attempts and deaths [ 5 , 6 , 7 ]. However, there remains limited adoption of these interventions as well as significant variability in effectiveness, which may be secondary to implementation challenges. A recent review identifies several implementation barriers that impact suicide prevention programming, including but not limited to high levels of complexity and cost as well as insufficient tailoring to patient needs [ 8 ]. It is, however, unknown which implementation strategies may be most helpful for addressing these needs to enhance the reach and effectiveness of promising suicide prevention programming.

In light of the need to better understand the types of implementation strategies that may enhance suicide prevention efforts, a recent systematic review attempted to describe implementation strategies used in complex interventions and determined use of such strategies was inconsistent [ 9 ]. However, this review focused only on complex suicide prevention interventions (i.e., those which had more than two components operating at different levels of intervention [e.g., individual, community]) and excluded studies focused on implementing only one intervention component (e.g., only suicide screening or suicide safety planning). However, single-component studies are common among quality improvement and implementation research projects. Its limited scope may have underrepresented the breadth of suicide prevention programming. The current scoping review expands upon this work by exploring current implementation strategies used across a broader range of suicide prevention interventions and programs.

The protocol for this scoping review was prospectively published online on PROSPERO (< de-identified >). A completed Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR) Checklist for this manuscript is available in Additional file  1 . The research questions for this review were (1) what are the current implementation strategies being used for promoting suicide prevention programming as described in the literature (see “ Eligibility criteria ” section for further information)?; (2) how effective are these implementation strategies for promoting the use of suicide prevention programming?; and (3) What organizational factors may moderate the effectiveness of these implementation strategies? We were unable to evaluate research questions 2 and 3 due to a low volume of eligible studies and underreporting of necessary information (e.g., explicit descriptions of barriers and facilitators, site- and setting-specific information; issues identified in previous literature and discussed below) [ 9 , 10 , 11 ]. Additional protocol modifications, described below where applicable, included conducting two additional literature searches, suspending the USA-only eligibility criterion, implementing collaborative full-text screening, and electing to explore the studies’ usage of best practices instead of conducting a standardized quality assessment. We made these modifications to increase the inclusivity of our sample and to address challenges with the limited information present in both abstracts and full-text manuscripts.

The search strategy (see Additional file  1 ) was developed in collaboration with a health sciences education and research librarian following an initial review of relevant articles (e.g., [ 12 ]). The strategy was designed to cover a broad range of topics related to suicide prevention implementation research (e.g., program development, quality improvement). Articles were obtained by searching PubMed, Scopus, PsycInfo, and the EBSCO Psychology and Behavioral Sciences Collection. The search was initially conducted in October 2019. Two additional searches were conducted in June 2021 and October 2022 due to a low volume of eligible articles from the first search.

Eligibility criteria

To be included in the review, articles were required to have been published between January 1, 2013, and October 25, 2022 (date of the final search), be written in English, describe the implementation of a suicide prevention or intervention program (i.e., not a theory or concept paper), and describe the use of at least one implementation strategy as defined by the Consolidated Framework for Implementation Research (CFIR) [ 13 ]. Randomized controlled trials that focused only on establishing the initial effectiveness of an intervention (and not its implementation), clinical case studies, editorials, opinion pieces, newspaper articles, and other forms of popular media were excluded. During the first round of screening, reviewers decided to include studies conducted outside of the USA due to the low number of eligible studies.

Study selection

After the removal of duplicates, two reviewers collaboratively screened the full texts of all articles for inclusion in the review. Full-text screening was used due to the limited ability to identify the use of implementation strategies from titles, abstracts, and keywords. As the use of at least one implementation strategy was required for inclusion, full-text screening was conducted collaboratively to prevent false negatives. Incongruence between reviewers was resolved by joint consensus.

Data extraction and synthesis

The following study characteristics were initially extracted: author(s), publication year, population(s), intervention/program type, and intervention and implementation outcome(s) assessed. Data extraction was carried out primarily by one reviewer (BR) and checked for accuracy by the other (JC). Following the coding of two training studies [ 14 , 15 ] to establish initial reliability, both reviewers coded implementation strategies from each article independently using a spreadsheet tool. A round of coding was conducted after each of the three literature searches. Discrepancies were resolved by joint consensus. Subsequently, reviewers collaboratively explored adherence to study conduct and reporting best practices based on the extant literature (e.g., clarity of implementation activities, assessment of implementation strategy fidelity) [ 11 ]. This protocol modification was utilized in lieu of planned quality assessment tools [ 16 ] to better fit the included studies and the implementation science context as well as the limited information available within included studies (e.g., many quality assessment domains could not be coded due to lack of information). The hybrid effectiveness-implementation study type was also determined via joint consensus at this stage based on standardized definitions from the literature [ 17 , 18 ].

During implementation strategy coding, singular implementation activities that involved more than one implementation strategy were allowed to count toward all applicable strategies. CFIR implementation strategy definitions were often more granular than common narrative descriptions of study activities. For example, it was uncommon for any study to develop educational materials without distributing them. Utilizing this approach, we also sought to avoid underrepresenting strategies that commonly co-occur.

To facilitate data synthesis, reporting, and interpretation, implementation strategies were clustered based on prior publications from the Expert Recommendations for Implementing Change (ERIC) study [ 19 , 20 ] (see Table  1 ). Clusters ranged in size from containing 3 to 17 total strategies. Revised cluster assignments (e.g., unassigned strategies, a new cluster focused on messaging-based strategies) were developed based on joint consensus.

Following initial deduplication, full texts of 174 articles were screened. Thirty-two studies were included in the review following full-text screening [ 12 , 14 , 15 , 21 , 22 , 23 , 24 , 25 , 26 , 27 , 28 , 29 , 30 , 31 , 32 , 33 , 34 , 35 , 36 , 37 , 38 , 39 , 40 , 41 , 42 , 43 , 44 , 45 , 46 , 47 , 48 , 49 ]. The most common reason for exclusion was the absence of any reported implementation activities (e.g., no intervention implemented; see Fig.  1 ). Study characteristics are provided in Table  2 . Most studies were conducted in the USA ( n  = 26) and were single-site (i.e., implementation took place in a single organizational unit, such as one clinic; n  = 23). Multi-site studies ranged from 3 to 65 sites. Half ( n  = 16) of the included studies described the implementation of suicide risk screening and/or risk identification, such as in settings that did not previously have such protocols. Half of the included studies utilized a hybrid effectiveness-implementation design, testing both an intervention’s effectiveness and its implementation with at least one implementation strategy [ 17 , 18 ]. Of those, most ( n  = 9) were coded as type 1 hybrid effectiveness-implementation studies (i.e., focused mostly on an intervention’s effectiveness while also exploring its implementation). There were five type 2 studies (focused roughly equally on implementation and effectiveness) and two type 3 studies (focused mostly on formally testing implementation strategies while also exploring effectiveness).

figure 1

Study selection flow diagram

Intervention and implementation outcomes were not regularly distinguished by authors among the included studies. Some outcomes appeared to serve both roles depending on an intervention’s scope. For example, if training is being conducted to screen for suicide risk, training is the implementation strategy, screening is the clinical intervention, and the screening rate can be considered an implementation outcome (e.g., provider adoption) as well as a secondary intervention outcome (with patient-level suicidality the primary outcome). As such, outcomes were categorized as either intervention or implementation outcomes based on content domains to avoid misrepresenting how outcomes were used by the authors in practice.

General organizational factors outcomes (e.g., intervention adoption, costs, fidelity, leadership support) were most common ( n  = 21), followed by education- and training-related outcomes (e.g., knowledge, awareness, attitudes; n  = 19). Studies also commonly reported effectiveness outcomes such as risk identification outcomes (e.g., screening rates; n  = 17) and follow-up care outcomes (e.g., referral rates, appointments, psychiatric medication usage; n  = 15). The least commonly measured were outcomes related to suicidal behavior (e.g., suicide attempts, deaths; n  = 7) and feedback from patients ( n  = 4). Three studies provided narrative reflections on implementation processes without structured quantitative or qualitative measurement of outcomes.

Most articles adhered to at least some study conduct and reporting best practices described in the extant literature [ 11 ]. For example, most studies included some definition of their implementation outcomes (e.g., a new definition or some reference to the extant literature) and included at least some quantitative or qualitative measurement of their outcomes with clear specification of data sources (e.g., clinician feedback, electronic health record integration).

Several gaps were identified in the reporting of implementation activities. For example, several studies did not include clear implementation processes and data collection timelines (i.e., detailed enough to discern the order of events and support replication). Of the 9 multi-site studies, only Luci et al. [ 33 ] provided information on setting-level variations in the implementation process and disaggregated data by setting. Additionally, only three of the 32 included studies reported fidelity to at least one of their implementation strategies [ 24 , 25 , 34 ]. Overall, implementation strategies were not regularly referred to as implementation strategies (with or without citation of the ERIC framework) and were not regularly distinguished from intervention activities.

Use of implementation strategies

Percent agreement for independent implementation strategy coding was good (see Table  2 ).

Table  3 provides definitions, cluster assignments, and observed frequencies for all implementation strategies (i.e., the raw number of times each strategy was consensus-coded across all studies). Seventeen of the ERIC implementation strategies were not identified among the included studies. Among implementation strategies that were utilized, each was utilized 5.11 times on average (SD = 5.09) across studies suggesting studies on average employed multiple implementation strategies. ‘Purposefully reexamining the implementation’, a strategy focused on monitoring implementation progress to inform ongoing quality improvement, was most common ( n  = 20). Figure  2 shows the raw utilization of each of the individual strategies included in each cluster (i.e., sum of all individual strategy frequencies within a cluster). Strategies from the ‘train and educate stakeholders’ cluster (e.g., ‘conduct educational meetings’, ‘develop educational materials’) were utilized most often ( n  = 109). Relative to the number of strategies in each cluster (i.e., total strategy utilizations divided by cluster size), the evaluative and iterative strategies cluster (e.g., ‘purposefully reexamine the implementation’, ‘conduct local needs assessment’) cluster was the most popular.

figure 2

Total utilizations of strategies from each cluster across studies. Legend: Cluster sizes (number of strategies included in a cluster) are shown next to cluster names. Cluster sizes and utilization counts add to more than the total strategies and utilizations due to strategies assigned to more than one cluster (see Additional file  2 )

Figure  3 shows the count of studies that utilized at least one strategy from each cluster. The ‘train and educate stakeholders’ ( n  = 28) and ‘use evaluative and iterative strategies’ ( n  = 28) clusters were the most broadly used by this metric. Conversely, the ‘support clinicians’ ( n  = 6) and ‘utilize financial strategies’ (n  = 4) clusters were used in the fewest studies. Reviewers identified the use of 10.63 implementation strategies per study on average (SD = 6.07; see Table  4 for counts per study, Additional file  2 ). On average, studies utilized strategies from 4.97 of the 10 strategy clusters (SD = 1.82; see Additional file  2 ). These results are partially attributable to frequently co-occurring strategies and strategies that belonged to more than one cluster, respectively. For example, studies that developed and evaluated a training described utilizing multiple implementation strategies that were inherent to implementing suicide prevention training (e.g., identifying barriers and facilitators, developing education materials, and making training dynamic).

figure 3

Total studies utilizing at least one strategy from each cluster

Overall, our review identified several current patterns in the use of implementation strategies in suicide prevention as well as several gaps in the literature. Consistent with past reviews [ 9 , 10 ] few manuscripts clearly delineated or described implementation strategies in a comprehensive manner (e.g., implementation details were spread across different sections of the paper, details were limited). On average, we captured fewer strategies than those reported by Rudd and colleagues who were able to identify additional strategies through surveying authorship teams [ 9 ]. However, the most common strategy clusters noted by Rudd and colleagues were consistent with our findings. It is possible that authors were unaware they were utilizing implementation strategies and thus could not describe them in detail. As the majority of papers reviewed were not published in implementation science-oriented journals, authors may have also limited the inclusion of detailed implementation strategy information to accommodate the journal audience.

The ‘train and education stakeholders’ cluster of strategies and the ‘use evaluative and iterative strategies’ cluster were the most broadly utilized—all but four studies utilized at least one strategy from this cluster (Fig.  3 ). Strategies from this cluster (e.g., ‘conduct educational meetings’, ‘distribute educational materials’) were the most frequently utilized overall (Fig.  2 ). Similarly, education- and training-related outcomes (e.g., knowledge, awareness, attitudes), were the second most common outcome domain. This is unsurprising as the majority of suicide prevention interventions and programs focus on promoting awareness and skill-building among stakeholders [ 6 ]. However, fewer studies utilized strategies for supporting active, sustained learning (e.g., ‘provide clinical supervision’, ‘create an online learning collaborative’, ‘make training dynamic’). Additionally, 13 of the 28 studies that utilized at least one training or education strategy did not utilize any strategies from the ‘provide interactive assistance’ cluster (e.g., providing ongoing support). This is of concern, as past research shows that increased knowledge and skills from suicide training initiatives are not sustained long-term, which may, in turn, decrease the overall effectiveness of suicide prevention programming over time [ 4 , 50 ].

The ‘use evaluative and iterative strategies’ cluster was also commonly reported within our sample. This is congruent with the core principles of implementation science focused on understanding and adapting to organizational contexts to enhance the adoption and maintenance of evidence-based strategies [ 51 ]. The most commonly utilized strategy from this cluster was to ‘purposefully reexamine the implementation’—a critical aspect of quality improvement emphasized across several relevant frameworks and models (e.g., Plan-Do-Study-Act [ 52 ]). Interestingly, ‘identification of early adopters’ was among the least commonly used strategies within this cluster, which may have been secondary to the limited number of studies with multiple sites in our sample.

The ‘support clinicians’ (e.g., resource sharing agreements to support clinics) and ‘utilize financial strategies’ (e.g., financial disincentives) clusters were also among the least utilized in our sample. As these strategies often require financial resources, it is possible they are more difficult to implement in light of financial challenges among healthcare systems [ 53 ]. In addition, several recent commentaries have raised concerns regarding the impact of the COVID-19 pandemic on the financial resources of hospitals, which may further limit the ability to utilize implementation strategies requiring funding [ 54 , 55 ]. More popular than these strategies were those aimed toward making use of existing resources, such as those from the ‘change infrastructure’ cluster (e.g., ‘assess and redesign workflow’, ‘change record systems’) to support implementation.

Few studies reported suicide behavior outcomes. While several studies were only focused on implementation, types 1 and 2 hybrid studies remain interested in an intervention's effectiveness while exploring or formally testing its implementation and can offer vital information for informing future dissemination and implementation [ 17 , 18 ]. A broader range of suicide-related outcomes would better enable such studies to evaluate whether promising interventions remain effective in practice, a key advantage of hybrid study designs. Funding agencies may wish to encourage the incorporation of Type I hybrid study procedures (e.g., qualitative inquiry on barriers and facilitators post-implementation of interventions) to ensure research studies collect sufficient information regarding implementation processes to increase future adoption and uptake of findings. Additionally, past literature has highlighted tailoring to patient needs as a key facilitator in the implementation of suicide prevention interventions [ 8 ]. However, outcomes involving feedback from patients were the least common in our sample. Similarly, the ‘engage consumers’ strategy cluster was among the least popular clusters (see Figs.  2 and 3 ).

Limited systematic reporting of implementation strategies and their corresponding outcomes, as well as a lack of type 3 hybrid studies (focused on formal implementation testing), limited our ability to explore the relative effectiveness of individual implementation strategies for improving suicide prevention programming (Research Question 2). Our literature search only captured studies within a 10-year period due to a desire to report on the most recent research available and excluded pertinent studies with more systematic reporting of implementation strategies outside this time period. Additionally, there was an overall low volume of multi-site studies. Among them, information necessary to explore organizational factors that could moderate the effectiveness of implementation strategies was mostly absent or unclear (e.g., specific barriers and facilitators, site-specific procedures, disaggregated data; Research Question 3).

It is possible that this information, as well as the breadth of implementation strategies, was underreported in the text of the reviewed manuscripts. Similar challenges have been reported by other reviews focused on narrower sets of suicide prevention studies [ 9 , 10 ]. Rudd and colleagues found that direct outreach to authors was required to get a more comprehensive understanding of implementation science strategies present in a given study [ 10 ]. Future manuscripts may wish to utilize existing reporting guidelines, such as the Standards for Reporting Implementation Studies (StaRI) checklist in combination with frameworks that guided this review (e.g., CFIR), to help ensure implementation strategies are appropriately documented to inform the broader field [ 56 ].

Implementation science remains an important and promising area of research for increasing sustainable adoption and deployment of evidence-based suicide prevention interventions and programming. Although we identified commonly used implementation science strategies and current gaps in the literature, our review was limited by the inconsistent reporting of implementation strategies within our sample. Future implementation science studies in suicide prevention should consider clearer, systematic documentation of implementation strategies utilized and associated outcomes to better inform the broader suicide prevention field. For example, journals accepting manuscripts on the implementation of suicide prevention programming may encourage the use of a common lexicon of implementation science terms or provide explicit reporting requirements. In addition, direct testing of implementation strategies through type 3 hybrid studies remains necessary to enhance the effectiveness of implementation and dissemination of suicide prevention programming.

Availability of data and materials

The datasets used and analyzed during the current study are available from the corresponding author on reasonable request.

Abbreviations

Consolidated Framework for Implementation Research

Expert Recommendations for Implementing Change

World Health Organization. Suicide world wide in 2019: Global health estimates. 2021. https://www.who.int/publications/i/item/9789240026643

Cramer RJ, Kapusta ND. A Social-Ecological Framework of Theory, Assessment, and Prevention of Suicide. Front Psychol. 2017 Oct;8(1756). https://www.frontiersin.org/article/ https://doi.org/10.3389/fpsyg.2017.01756

Goldston DB, Walrath CM, McKeon R, Puddy RW, Lubell KM, Potter LB, et al. The Garrett Lee Smith memorial suicide prevention program. Suicide Life Threat Behav. 2010;40(3):245–56. https://doi.org/10.1521/suli.2010.40.3.245 .

Article   PubMed   PubMed Central   Google Scholar  

Godoy Garraza L, Kuiper N, Goldston D, McKeon R, Walrath C. Long-term impact of the Garrett Lee Smith Youth Suicide Prevention Program on youth suicide mortality, 2006–2015. J Child Psychol Psychiatry. 2019;60(10):1142–7. https://doi.org/10.1111/jcpp.13058 .

Article   PubMed   Google Scholar  

Calati R, Courtet P. Is psychotherapy effective for reducing suicide attempt and non-suicidal self-injury rates? Meta-analysis and meta-regression of literature data. J Psychiatr Res. 2016;1(79):8–20. https://doi.org/10.1016/j.jpsychires.2016.04.003 .

Article   Google Scholar  

Hofstra E, van Nieuwenhuizen C, Bakker M, Özgül D, Elfeddali I, de Jong SJ, et al. Effectiveness of suicide prevention interventions: A systematic review and meta-analysis. Gen Hosp Psychiatry. 2020;1(63):127–40. https://doi.org/10.1016/j.genhosppsych.2019.04.011 .

Peterson K, Parsons N, Vela K, Denneson LM, Dobscha KS. Compendium: Systematic Reviews on Suicide Prevention Topics. Washington, DC: Health Services Research and Development Service, Office of Research and Development; 2019. Report No.: VA ESP Project #09–199. https://www.hsrd.research.va.gov/centers/core/SPRINT-Compendium-Reviews.pdf

Kasal A, Táborská R, Juríková L, Grabenhofer-Eggerth A, Pichler M, Gruber B, et al. Facilitators and barriers to implementation of suicide prevention interventions: Scoping review. Camb Prisms Glob Ment Health. 2023;10:e15. https://doi.org/10.10172/gmh.2023.9 . (2023/03/13 ed).

Krishnamoorthy S, Mathieu S, Armstrong G, Ross V, Francis J, Reifels L, et al. Utilisation and application of implementation science in complex suicide prevention interventions: A systematic review. J Affect Disord. 2023;1(330):57–73. https://doi.org/10.1016/j.jad.2023.02.140 .

Rudd BN, Davis M, Doupnik S, Ordorica C, Marcus SC, Beidas RS. Implementation strategies used and reported in brief suicide prevention intervention studies. JAMA Psychiatry. 2022;79(8):829–31. https://doi.org/10.1001/jamapsychiatry.2022.1462 .

Lengnick-Hall R, Gerke DR, Proctor EK, Bunger AC, Phillips RJ, Martin JK, et al. Six practical recommendations for improved implementation outcomes reporting. Implement Sci. 2022;17(1):16. https://doi.org/10.1186/s13012-021-01183-3 .

Roaten K, Johnson C, Genzel R, Khan F, North CS. Development and Implementation of a Universal Suicide Risk Screening Program in a Safety-Net Hospital System. Jt Comm J Qual Patient Saf. 2018;44(1):4–11. https://doi.org/10.1016/j.jcjq.2017.07.006 .

Powell BJ, Waltz TJ, Chinman MJ, Damschroder LJ, Smith JL, Matthieu MM, et al. A refined compilation of implementation strategies: results from the Expert Recommendations for Implementing Change (ERIC) project. Implement Sci. 2015;10:21. https://doi.org/10.1186/s13012-015-0209-1 .

Donald M, Dower J, Bush R. Evaluation of a suicide prevention training program for mental health services staff. Community Ment Health J. 2013;49(1):86–94. https://doi.org/10.1007/s10597-012-9489-y .

Chugani CD. Dialectical behavior therapy in college counseling centers: practical applications and theoretical considerations. 2017; http://libproxy.chapman.edu/login?url=https://search.ebscohost.com/login.aspx?direct=true&AuthType=ip,uid&db=psyh&AN=2016-47712-095&site=eds-live

National Heart, Lung, and Blood Institute. Study Quality Assessment Tools. 2023. https://www.nhlbi.nih.gov/health-topics/study-quality-assessment-tools Cited 2023 Sep 3

Curran GM, Bauer M, Mittman B, Pyne JM, Stetler C. Effectiveness-implementation hybrid designs: combining elements of clinical effectiveness and implementation research to enhance public health impact. Med Care. 2012;50(3):217–26. https://doi.org/10.1097/MLR.0b013e3182408812 .

Landes SJ, McBain SA, Curran GM. An introduction to effectiveness-implementation hybrid designs. Psychiatry Res. 2019;280:112513. https://doi.org/10.1016/j.psychres.2019.112513 .

Waltz TJ, Powell BJ, Matthieu MM, Damschroder LJ, Chinman MJ, Smith JL, et al. Use of concept mapping to characterize relationships among implementation strategies and assess their feasibility and importance: results from the Expert Recommendations for Implementing Change (ERIC) study. Implement Sci. 2015;10(1):109. https://doi.org/10.1186/s13012-015-0295-0 .

Perry CK, Damschroder LJ, Hemler JR, Woodson TT, Ono SS, Cohen DJ. Specifying and comparing implementation strategies across seven large implementation interventions: a practical application of theory. Implement Sci. 2019;14(1):32. https://doi.org/10.1186/s13012-019-0876-4 .

Belhumeur J, Butts E, Michael KD, Zieglowsky S, Decoteau D, Four Bear D, et al. Adapting crisis intervention protocols: rural and tribal voices from Montana. In: Jameson JP, editor., et al., Handbook of Rural School Mental Health. 1st ed. Springer; 2017. p. 307–21. https://doi.org/10.1007/978-3-319-64735-7_20 .

Chapter   Google Scholar  

Blake C. Depression screening implementation: quality improvement project in a primary care clinic for first responders. Workplace Health Saf. 2022. https://doi.org/10.1177/21650799221119147 .

Bose J, Zeno R, Warren B, Sinnott LT, Fitzgerald EA. Implementation of universal adolescent depression screening: quality improvement outcomes. J Pediatr Health Care. 2021;35(3):270–7. https://doi.org/10.1016/j.pedhc.2020.08.004 . (2021/02/15 ed).

Boudreaux ED, Haskins BL, Larkin C, Pelletier L, Johnson SA, Stanley B, et al. Emergency department safety assessment and follow-up evaluation 2: an implementation trial to improve suicide prevention. Contemp Clin Trials. 2020;95:106075 (2020/06/23 ed).

Boudreaux ED, Larkin C, Sefair AV, Mick E, Clements K, Pelletier L, et al. Studying the implementation of zero suicide in a large health system: challenges, adaptations, and lessons learned. Contemp Clin Trials Commun. 2022;30:100999. https://doi.org/10.1016/j.cct.2020.106075 .

Cramer RJ, Judah MR, Badger NL, Holley AM, Judd S, Peterson M, et al. Suicide on college campuses: a public health framework and case illustration. J Am Coll Health. 2022;70(1):1–8. https://doi.org/10.1080/07448481.2020.1739053 . (2020/03/25 ed).

Day SC, Day G, Keller M, Touchett H, Amspoker AB, Martin L, et al. Personalized implementation of video telehealth for rural veterans(PIVOT-R). mHealth. 2021;7:24. https://doi.org/10.21037/mhealth.2020.03.02 .

Garner MS, Kunkel DE. Quality improvement of pastoral care for major depression in the community of an African American Religious Organization. Issues Ment Health Nurs. 2020;41(7):568–73. https://doi.org/10.1080/01612840.2019.1701155 .

Horowitz LM, Bridge JA, Tipton MV, Abernathy T, Mournet AM, Snyder DJ, et al. Implementing suicide risk screening in a pediatric primary care setting: from research to practice. Acad Pediatr. 2022;22(2):217–26. https://doi.org/10.1016/j.acap.2021.10.012 .

Kabatchnick R. Training nursing staff to recognize and respond to suicidal ideation in a nursing home. 2018. https://cdr.lib.unc.edu/concern/dissertations/5999n445d?locale=en

Lai CCS, Law YW, Shum AKY, Ip FWL, Yip PSF. A community-based response to a suicide cluster: A Hong Kong experience. Crisis. 2020;41(3):163–71. https://doi.org/10.1027/0227-5910/a000616 .

Landes SJ, Jegley SM, Kirchner JE, Areno JP, Pitcock JA, Abraham TH, et al. Adapting Caring Contacts for Veterans in a Department of Veterans Affairs Emergency Department: Results From a Type 2 Hybrid Effectiveness-Implementation Pilot Study. Front Psychiatry. 2021;12:746805. https://doi.org/10.3389/fpsyt.2021.746805 .

Luci K, Simons K, Hagemann L, Jacobs ML, Bower ES, Eichorst MK, et al. SAVE-CLC: an intervention to reduce suicide risk in older veterans following discharge from VA nursing facilities. Clin Gerontol. 2020;43(1):118–25. https://doi.org/10.1080/07317115.2019.1666444 . (2019/09/17 ed)

Marshall E, York J, Magruder K, Yeager D, Knapp R, De Santis ML, et al. Implementation of online suicide-specific training for VA providers. Acad Psychiatry. 2014;38(5):566–74. https://doi.org/10.1007/s40596-014-0039-5 . (2014/02/25ed).

McManus JQ. School nurses identifying at-risk adolescents for depression. Gd Canyon Univ ProQuest Diss Publ. 2021; https://www.proquest.com/openview/038681fc92427985a9ecd30fd73841b1/1?pq-origsite=gscholar&cbl=18750&diss=y

Mokkenstorm J, Franx G, Gilissen R, Kerkhof A, Smit JH. Suicide prevention guideline implementation in specialist mental healthcare institutions in the Netherlands. Int J Environ Res Public Health. 2018;15(5):910. https://doi.org/10.3390/ijerph15050910 .

Mueller KL, Naganathan S, Griffey RT. Counseling on Access to Lethal Means-Emergency Department (CALM-ED) a quality improvement program for firearm injury prevention. West J Emerg Med. 2020;20;21(5):1123–30. https://doi.org/10.5811/westjem.2020.5.46952 . (2020/09/25 ed).

Noelck M, Velazquez-Campbell M, Austin JP. A quality improvement initiative to reduce safety events among adolescents hospitalized after a suicide attempt. Hosp Pediatr. 2019;9(5):365–72. https://doi.org/10.1542/hpeds.2018-0218 . (2019/04/07 ed).

Powell N, Dalton H, Perkins D, Considine R, Hughes S, Osborne S, et al. Our healthy clarence: a community-driven wellbeing initiative. Int J Environ Res Public Health. 2019;16(19):3691. https://doi.org/10.3390/ijerph16193691 .

Riblet NB, Varela M, Ashby W, Zubkoff L, Shiner B, Pogue J, et al. Spreading a strategy to prevent suicide after psychiatric hospitalization: results of a quality improvement spread initiative. Jt Comm J Qual Patient Saf. 2022;48(10):503–12. https://doi.org/10.1016/j.jcjq.2022.02.009 .

Rudd BN, George JM, Snyder SE, Whyte M, Cliggitt L, Weyler R, et al. Harnessing quality improvement and implementation science to support the implementation of suicide prevention practices in juvenile detention. Psychotherapy. 2022;59(2):150–6. https://doi.org/10.1037/pst0000377 .

Ryan K, Tindall C, Strudwick G. Enhancing Key Competencies of health professionals in the assessment and care of adults at risk of suicide through education and technology. Clin Nurse Spec. 2017;31(5):268–75. https://doi.org/10.1097/nur.0000000000000322 . (2017/08/15 ed).

Siau CS, Wee LH, Ibrahim N, Visvalingam U, Yeap LLL, Wahab S. Gatekeeper suicide training’s effectiveness among malaysian hospital health professionals: a control group study with a three-month follow-up. J Contin Educ Health Prof. 2018;38(4):227–34. https://doi.org/10.1097/ceh.0000000000000213 . (2018/07/24 ed).

Snyder DJ, Jordan BA, Aizvera J, Innis M, Mayberry H, Raju M, et al. From pilot to practice: implementation of a suicide risk screening program in hospitalized medical patients. Jt Comm J Qual Patient Saf. 2020;46(7):417–26. https://doi.org/10.1016/j.jcjq.2020.04.011 . (2020/06/01 ed).

Sullivant SA, Brookstein D, Camerer M, Benson J, Connelly M, Lantos J, et al. Implementing universal suicide risk screening in a pediatric hospital. Jt Comm J Qual Patient Saf. 2021;47(8):496–502. https://doi.org/10.1016/j.jcjq.2021.05.001 . (2021/06/15 ed).

Tennant J. Implementation of the signs of suicide prevention program with 9th grade students in a public school setting. 2017; https://researchrepository.wvu.edu/etd/6782/

Vaughan B. Implementation and evaluation of the P4 suicide screening tool among sexual assault nurse examiners: a suicide prevention and intervention strategy. 2019; https://www.proquest.com/openview/fe6c57e3aa478424e76679074c354d09/1.pdf?pq-origsite=gscholar&cbl=18750&diss=y

Wright-Berryman J, Hudnall G, Bledsoe C, Lloyd M. Suicide concern reporting among Utah youths served by a school-based peer-to-peer prevention program. Child Sch. 2019;41(1):35–44. https://doi.org/10.1093/cs/cdy026 .

Yeung K, Richards J, Goemer E, Lozano P, Lapham G, Williams E, et al. Costs of using evidence-based implementation strategies for behavioral health integration in a large primary care system. Health Serv Res. 2020;55(6):913–23. https://doi.org/10.1111/1475-6773.13592 . (2020/12/02 ed).

Holmes G, Clacy A, Hermens DF, Lagopoulos J. The long-term efficacy of suicide prevention gatekeeper training: a systematic review. Arch Suicide Res. 2021;25(2):177–207. https://doi.org/10.1080/13811118.2019.1690608 .

Handley MA, Gorukanti A, Cattamanchi A. Strategies for implementing implementation science: a methodological overview. Emerg Med J. 2016;33(9):660–4. https://doi.org/10.1136/emermed-2015-205461 . (2016/02/20 ed).

Langley G, Moen R, Nolan K, Nolan T, Norman C, Provost L. The improvement guide: a practical approach to enhancing organizational performance. 2nd ed. San Francisco, CA: Jossey-Bass Publishers; 2009.

Google Scholar  

Bai G, Yehia F, Chen W, Anderson GF. Varying trends in the financial viability of US rural hospitals, 2011–17. Health Aff (Millwood). 2020;39(6):942–8. https://doi.org/10.1377/hlthaff.2019.01545 .

Khullar D, Bond AM, Schpero WL. COVID-19 and the Financial Health of US Hospitals. JAMA. 2020;323(21):2127–8. https://doi.org/10.1001/jama.2020.6269 .

Article   CAS   PubMed   Google Scholar  

Barnett Michael L, Mehrotra A, Landon Bruce E. Covid-19 and the upcoming financial crisis in health care. NEJM Catal Non-Issue Content. 2022 Apr 29. https://doi.org/10.1056/CAT.20.0153

Pinnock H, Barwick M, Carpenter CR, Eldridge S, Grandes G, Griffiths CJ, et al. Standards for Reporting Implementation Studies (StaRI) Statement. BMJ. 2017;6(356):i6795. https://doi.org/10.1136/bmj.i6795 .

Download references

Acknowledgements

We would like to express our appreciation to the following individuals for their support with this manuscript: Basia Delawska-Elliot, MLS for providing technical support in the development of the literature search strategy; Devan Kansagara, MD, MCR for providing initial feedback early in the development of this manuscript; the VA Partnerships in Implementation and Evaluation (PIE) Lab for early conceptual feedback; Riley Murphy, BA for helping with data collection and organization for this manuscript.

This project was funded by a VA Health Services Research & Development Career Development Award (CDA 18–185; PI: Chen). This material is the result of work supported with resources and the use of facilities at the VA Portland Health Care System, Portland, OR. The views expressed in this article are those of the authors and do not necessarily reflect the position or policy of the Department of Veterans Affairs or the United States government.

Author information

Authors and affiliations.

Center to Improve Veteran Involvement in Care (CIVIC), VA Portland Health Care System, U.S. Department of Veterans Affairs (VA), Portland, OR, USA

Jason I. Chen, Brandon Roth & Steven K. Dobscha

Department of Psychiatry, Oregon Health & Science University (OHSU), Portland, OR, USA

Portland VA Research Foundation, Portland, OR, USA

Brandon Roth

Center for Clinical Management Research (CCMR), VA Ann Arbor Healthcare System, Ann Arbor, MI, USA

Julie C. Lowery

You can also search for this author in PubMed   Google Scholar

Contributions

JIC contributed to the conception, design, data collection, analysis, interpretation, and writing of this manuscript. BR contributed to data collection, analysis, interpretation, and writing. SKD contributed to the conception design, interpretation, drafting, and revising of the manuscript. JCL contributed to the conception, design, data collection, analysis, interpretation, and writing of this manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Jason I. Chen .

Ethics declarations

Ethics approval and consent to participate.

Not applicable.

Consent for publication

Competing interests.

The authors report having no competing interests in relation to this manuscript.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1..

Contains the Preferred Reporting Items for Systematic reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR) Checklist and this review’s full search strategy.

Additional file 2.

Contains two worksheets. 1. “Coded Strategies by Study” provides the raw, consensus data from implementation strategy coding. 2. “Cluster Assignments” specifies the applicable strategy cluster(s) for each implementation strategy and whether this review changed the cluster assignment (e.g., new assignment, reassignment) from the original assignments in the literature.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Chen, J.I., Roth, B., Dobscha, S.K. et al. Implementation strategies in suicide prevention: a scoping review. Implementation Sci 19 , 20 (2024). https://doi.org/10.1186/s13012-024-01350-2

Download citation

Received : 26 October 2023

Accepted : 09 February 2024

Published : 26 February 2024

DOI : https://doi.org/10.1186/s13012-024-01350-2

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Implementation Science

ISSN: 1748-5908

  • Submission enquiries: Access here and click Contact Us
  • General enquiries: [email protected]

importance of review related literature in research study

  • Open access
  • Published: 24 February 2024

Physical activity improves stress load, recovery, and academic performance-related parameters among university students: a longitudinal study on daily level

  • Monika Teuber 1 ,
  • Daniel Leyhr 1 , 2 &
  • Gorden Sudeck 1 , 3  

BMC Public Health volume  24 , Article number:  598 ( 2024 ) Cite this article

274 Accesses

17 Altmetric

Metrics details

Physical activity has been proven to be beneficial for physical and psychological health as well as for academic achievement. However, especially university students are insufficiently physically active because of difficulties in time management regarding study, work, and social demands. As they are at a crucial life stage, it is of interest how physical activity affects university students' stress load and recovery as well as their academic performance.

Student´s behavior during home studying in times of COVID-19 was examined longitudinally on a daily basis during a ten-day study period ( N  = 57, aged M  = 23.5 years, SD  = 2.8, studying between the 1st to 13th semester ( M  = 5.8, SD  = 4.1)). Two-level regression models were conducted to predict daily variations in stress load, recovery and perceived academic performance depending on leisure-time physical activity and short physical activity breaks during studying periods. Parameters of the individual home studying behavior were also taken into account as covariates.

While physical activity breaks only positively affect stress load (functional stress b = 0.032, p  < 0.01) and perceived academic performance (b = 0.121, p  < 0.001), leisure-time physical activity affects parameters of stress load (functional stress: b = 0.003, p  < 0.001, dysfunctional stress: b = -0.002, p  < 0.01), recovery experience (b = -0.003, p  < 0.001) and perceived academic performance (b = 0.012, p  < 0.001). Home study behavior regarding the number of breaks and longest stretch of time also shows associations with recovery experience and perceived academic performance.

Conclusions

Study results confirm the importance of different physical activities for university students` stress load, recovery experience and perceived academic performance in home studying periods. Universities should promote physical activity to keep their students healthy and capable of performing well in academic study: On the one hand, they can offer opportunities to be physically active in leisure time. On the other hand, they can support physical activity breaks during the learning process and in the immediate location of study.

Peer Review reports

Introduction

Physical activity (PA) takes a particularly key position in health promotion and prevention. It reduces risks for several diseases, overweight, and all-cause mortality [ 1 ] and is beneficial for physical, psychological and social health [ 2 , 3 , 4 , 5 ] as well as for academic achievement [ 6 , 7 ]. However, PA levels decrease from childhood through adolescence and into adulthood [ 8 , 9 , 10 ]. Especially university students are insufficiently physically active according to health-oriented PA guidelines [ 11 ] because of academic workloads as well as difficulties in time management regarding study, work, and social demands [ 12 ]. Due to their independence and increasing self-responsibility, university students are at a crucial life stage. In this essential and still educational stage of the students´ development, it is important to study their PA behavior. Furthermore, PA as health behavior represents one influencing factor which is considered in the analytical framework of the impact of health and health behaviors on educational outcomes which was developed by the authors Suhrcke and de Paz Nieves [ 13 , 14 ]. In light of this, the present study examines how PA affects university students' academic situations.

Along with the promotion of PA, the reduction of sedentary behavior has also become a crucial part of modern health promotion and prevention strategies. Spending too much time sitting increases many health risks, including the risk of obesity [ 15 ], diabetes [ 16 ] and other chronic diseases [ 15 ], damage to muscular balances, bone metabolism and musculoskeletal system [ 17 ] and even early death [ 15 ]. University students are a population that has shown the greatest increase in sedentary behavior over the last two decades [ 18 ]. In Germany, they show the highest percentage of sitting time among all working professional groups [ 19 ]. Long times sitting in classes, self-study learning, and through smartphone use, all of which are connected to the university setting and its associated behaviors, might be the cause of this [ 20 , 21 ]. This goes along with technological advances which allow students to study in the comfort of their own homes without changing locations [ 22 ].

To counter a sedentary lifestyle, PA is crucial. In addition to its physical health advantages, PA is essential for coping with the intellectual and stress-related demands of academic life. PA shows positive associations with stress load and academic performance. It is positively associated with learning and educational success [ 6 ] and even shows stress-regulatory potential [ 23 ]. In contrast, sedentary behavior is associated with lower cognitive performance [ 24 ]. Moreover, theoretical derivations show that too much sitting could have a negative impact on brain health and diminish the positive effects of PA [ 16 ]. Given the theoretical background of the stressor detachment model [ 25 ] and the cybernetic approach to stress management in the workplace [ 26 ], PA can promote recovery experience, it can enhance academic performance, and it is a way to reduce the impact of study-related stressors on strain. Load-related stress response can be bilateral: On the one hand, it can be functional if it is beneficial to help cope with the study demands. On the other hand, it can be dysfunctional if it puts a strain on personal resources and can lead to load-related states of strain [ 27 ]. Thus, both, the promotion of PA and reduction of sedentary behavior are important for stress load, recovery, and performance in student life, which can be of particular importance for students in an academic context.

A simple but (presumably) effective way to integrate PA and reduce sedentary behavior in student life are short PA breaks. Due to the exercises' simplicity and short duration, students can perform them wherever they are — together in a lecture or alone at home. Short PA breaks could prevent an accumulation of negative stressors during the day and can help with prolonged sitting as well as inactivity. Especially in the university setting, evidence of the positive effects of PA breaks exists for self-perceived physical and psychological well-being of the university students [ 28 ]. PA breaks buffer university students’ perceived stress [ 29 ] and show positive impacts on recovery need [ 30 ] and better mood ratings [ 31 , 32 ]. In addition, there is evidence for reduction in tension [ 30 ], overall muscular discomfort [ 33 ], daytime sleepiness or fatigue [ 33 , 34 ] and increase in vigor [ 34 ] and experienced energy [ 30 ]. This is in line with cognitive, affective, behavioral, and biological effects of PA, all categorized as palliative-regenerative coping strategies, which addresses the consequences of stress-generating appraisal processes aiming to alleviate these consequences (palliative) or restore the baseline of the relevant reaction parameter (regenerative) [ 35 , 36 ]. This is achieved by, for example, reducing stress-induced cortisol release or tension through physical activity (reaction reduction) [ 35 ]. Such mechanisms are also in accordance with the previously mentioned stressor detachment model [ 25 ]. Lastly, there is a health-strengthening effect that impacts the entire stress-coping-health process, relying on the compensatory effects of PA which is in accordance to the stress-buffering effect of exercise [ 37 ]. Health, in turn, effects educational outcomes [ 13 , 14 ]. Therefore, stress regulating effects are also accompanied with the before mentioned analytical framework of the impact of health and health behaviors on educational outcomes [ 13 , 14 ].

Focusing on the effects of PA, this study is guided by an inquiry into how PA affects university students' stress load and recovery as well as their perceived academic performance. For that reason, the student´s behavior during home studying in times of COVID-19 is examined, a time in which reinforced prolonged sitting, inactivity, and a negative stress load response was at a high [ 38 , 39 , 40 , 41 , 42 ]. Looking separately on the relation of PA with different parameters based on the mentioned evidence, we assume that PA has a positive impact on stress load, recovery, and perceived academic performance-related parameters. Furthermore, a side effect of the home study behavior on the mentioned parameters is assumed regarding the accumulation of negative stressors during home studying. These associations are presented in Fig.  1 and summarized in the following hypotheses:

figure 1

Overview of the assumed effects and investigated hypotheses of physical activity (PA) behavior on variables of stress load and recovery and perceived academic performance-related parameters

Hypothesis 1 (path 1): Given that stress load always occurs as a duality—beneficial if it is functional for coping, or exhausting if it puts a strain on personal resources [ 27 ] – we consider two variables for stress load: functional stress and dysfunctional stress. In order to reduce the length of the daily surveys, we focused the measure of recovery only on the most obvious and accessible component of recovery experience, namely psychological detachment. PA (whether performed in leisure-time or during PA breaks) encourages functional stress and reduce dysfunctional stress (1.A) and has a positive effect on recovery experience through psychological detachment (1.B).

Hypothesis 2 (path 2): The academic performance-related parameters attention difficulties and study ability are positively influenced by PA (whether done in leisure-time or during PA breaks). We have chosen to assess attention difficulties for a cognitive parameter because poor control over the stream of occurring stimuli have been associated with impairment in executive functions or academic failure [ 43 , 44 , 45 , 46 ]. Furthermore, we have assessed the study ability to refer to the self-perceived feeling of functionality regarding the demands of students. PA reduces self-reported attention difficulties (2.A) and improves perceived study ability, indicating that a student feels capable of performing well in academic study (2.B).

Hypothesis 3: We assume that a longer time spent on studying at home (so called home studying) could result in higher accumulation of stressors throughout the day which could elicit immediate stress responses, while breaks in general could reduce the influence of work-related stressors on strain and well-being [ 47 , 48 ]. Therefore, the following covariates are considered for secondary effects:

the daily longest stretch of time without a break spent on home studying

the daily number of breaks during home studying

Study setting

The study was carried out during the COVID-19 pandemic containment phase. It took place in the middle of the lecture period between 25th of November and 4th of December 2020. Student life was characterized by home studying and digital learning. A so called “digital semester” was in effect at the University of Tübingen when the study took place. Hence, courses were mainly taught online (e.g., live or via a recorded lecture). Other events and actions at the university were not permitted. As such, the university sports department closed in-person sports activities. For leisure time in general, there were contact restrictions (social distancing), the performance of sports activities in groups was not permitted, and sports facilities were closed.

Thus, the university sports department of the University of Tübingen launched various online sports courses and the student health management introduced an opportunity for a new digital form of PA breaks. This opportunity provided PA breaks via videos with guided physical exercises and health-promoting explanations for a PA break for everyday home studying: the so called “Bewegungssnack digital” [in English “exercise snack digital” (ESD)] [ 49 ]. The ESD videos took 5–7 min and were categorized into three thematic foci: activation, relaxation, and coordination. Exercises were demonstrated by one or two student exercise leaders, accompanied by textual descriptions of the relevant execution features of each exercise.

Participants

Participants were recruited within the framework of an intervention study, which was conducted to investigate whether a digital nudging intervention has a beneficial effect on taking PA breaks during home study periods [ 49 ]. Students at the University of Tübingen which counts 27,532 enrolled students were approached for participation through a variety of digital means: via an email sent to those who registered for ESD course on the homepage of the university sports department and to all students via the university email distribution list; via advertisement on social media of the university sports department (Facebook, Instagram, YouTube, homepage). Five tablets, two smart watches, and one iPad were raffled off to participants who engaged actively during the full study period in an effort to motivate them to stick with it to the end. In any case, participants knew that the study was voluntary and that they would not suffer any personal disadvantages should they opt out. There was a written informed consent prompt together with a prompt for the approval of the data protection regulations immediately within the first questionnaire (T0) presented in a mandatory selection field. Positive ethical approval for the study was given by the first author´s institution´s ethics committee of the faculty of the University of Tübingen.

Participants ( N  = 57) who completed the daily surveys on at least half of the days of the study period, were included in the sample (male = 6, female = 47, diverse = 1, not stated = 3). As not all subjects provided data on all ten study days, the total number of observations was between 468 and 540, depending on the variable under study (see Table  1 ). The average number of observations per subject was around eight. Their age was between 18 and 32 years ( M  = 23.52, SD  = 2.81) and they were studying between the 1st to 13th semester ( M  = 5.76, SD  = 4.11) within the following major courses of study: mathematical-scientific majors (34.0%), social science majors (22.6%), philosophical majors (18.9%), medicine (13.2%), theology (5.7%), economics (3.8%), or law (1.9%). 20.4% of the students had on-site classroom teaching on university campus for at least one day a week despite the mandated digital semester, as there were exceptions for special forms of teaching.

Design and procedures

To examine these hypothesized associations, a longitudinal study design with daily surveys was chosen following the suggestion of the day-level study of Feuerhahn et al. (2014) and also of Sonnentag (2001) measuring recovery potential of (exercise) activities during leisure time [ 50 , 51 ]. Considering that there are also differences between people at the beginning of the study period, initial base-line value variables respective to the outcomes measured before the study period were considered as independent covariates. Therefore, the well-being at baseline serves as a control for stress load (2.A), the psychological detachment at baseline serves as a control for daily psychological detachment (2.B), the perception of study demands serves as a control for self-reported attention difficulties (1.A), and the perceived study ability at baseline serves as a control for daily study ability (2.B).

Subjects were asked to continue with their normal home study routine and additionally perform ESD at any time in their daily routine. Data were collected one to two days before (T0) as well as daily during the ten-day study period (Wednesday to Friday). The daily surveys (t 1 -t 10 ) were sent by email at 7 p.m. every evening. Each day, subjects were asked to answer questions about their home studying behavior, study related requirements, recovery experience from study tasks, attention, and PA, including ESD participation. The surveys were conducted online using the UNIPARK software and were recorded and analyzed anonymously.

Measures and covariates

In total, five outcome variables, two independent variables, and seven covariates were included in different analyses: three variables were used for stress load and recovery parameters, two variables for academic performance-related parameters, two variables for PA behavior, two variables for study behavior, four variables for outcome specific baseline values and one variable for age.

Outcome variables

Stress load & recovery parameters (hypothesis 1).

Stress load was included in the analysis with two variables: functional stress and dysfunctional stress. Followingly, a questionnaire containing a word list of adjectives for the recording of emotions and stress during work (called “Erfassung von Emotionen und Beanspruchung “ in German, also known as EEB [ 52 ]) was used. It is an instrument which were developed and validated in the context of occupational health promotion. The items are based on mental-workload research and the assessment of the stress potential of work organization [ 52 ]. Within the questionnaire, four mental and motivational stress items were combined to form a functional stress scale (energetic, willing to perform, attentive, focused) (α = 0.89) and four negative emotional and physical stress items were combined to form dysfunctional stress scale (nervous, physically tensioned, excited, physically unwell) (α = 0.71). Participants rated the items according to how they felt about home studying in general on the following scale (adjustment from “work” to “home studying”): hardly, somewhat, to some extent, fairly, strongly, very strongly, exceptionally.

Recovery experience was measured via psychological detachment. Therefore, the dimension “detachment” of the Recovery Experience Questionnaire (RECQ [ 53 ]) was adjusted to home studying. The introductory question was "How did you experience your free time (including short breaks between learning) during home studying today?". Students responded to four statements based on the extent to which they agreed or disagreed (not at all true, somewhat true, moderately true, mostly true, completely true). The statements covered subjects such as forgetting about studying, not thinking about studying, detachment from studying, and keeping a distance from student tasks. The four items were combined into a score for psychological detachment (α = 0.94).

Academic performance-related parameters (hypothesis 2)

Attention was assessed via the subscale “difficulty maintaining focused attention performance” of the “Attention and Performance Self-Assessment” (ASPA, AP-F2 [ 54 ]). It contains nine items with statements about disturbing situations regarding concentration (e.g. “Even a small noise from the environment could disturb me while reading.”). Participants had to answer how often such situations happened to them on a given day on the following scale: never, rarely, sometimes, often, always. The nine items were combined into the AP-F2 score (α = 0.87).

The perceived study ability was assessed using the study ability index (SAI [ 55 ]). The study ability index captures the current state of perceived functioning in studying. It is based on the Work Ability Index by Hasselhorn and Freude ([ 56 ]) and consists of an adjusted short scale of three adapted items in the context of studying. Firstly, (a) the perceived academic performance was asked after in comparison to the best study-related academic performance ever achieved (from 0 = completely unable to function to 10 = currently best functioning). Secondly, the other two items were aimed at assessing current study-related performance in relation to (b) study tasks that have to be mastered cognitively and (c) the psychological demands of studying. Both items were answered on a five-point Likert scale (1 = very poor, 2 = rather poor, 3 = moderate, 4 = rather good, 5 = very good). A sum index, the SAI, was formed which can indicate values between 2 and 20, with higher values corresponding to higher assessed functioning in studies (α = 0.86). In a previous study it already showed satisfying reliability (α = 0.72) [ 55 ].

Independent variables

Pa behavior.

Two indicators for PA behavior were included via self-reports: the time spent on ESD and the time spent on leisure-time PA (LTPA). Participants were asked the following overarching question daily: “How much time did you spend on physical activity today and in what context”. For the independent variable time spent on PA breaks, participants could answer the option “I participated in the Bewegungssnack digital” with the amount of time they spent on it (in minutes). To assess the time spent on LTPA besides PA breaks, participants could report their time for four different contexts of PA which comprised two forms: Firstly, structured supervised exercise was reported via time spent on (a) university sports courses and (b) other organized sports activities. Secondly, self-organized PA was indicated via (c) independent PA at home, such as a workout or other physically demanding activity such as cleaning or tidying up, as well as via (d) independent PA outside, like walking, cycling, jogging, a workout or something similar. Referring to the different domains of health enhancing PA [ 57 ], the reported minutes of these four types of PA were summed up to a total LTPA value. The total LTPA value was included in the analysis as a metric variable in minutes.

Covariates (hypothesis 3)

Regarding hypothesis 3 and home study behavior, the longest daily stretch of time without a break spent on home studying (in hours) and the daily number of breaks during home studying was assessed. Therein, participants had to answer the overarching question “How much time did you spend on your home studying today?” and give responses to the items: (1) longest stretch of time for home studying (without a break), and (2) number of short and long breaks you took during home studying.

In principle, efforts were made to control for potential confounders at the individual level (level 2) either by including the baseline measure (T0) of the respective variable or by including variables assessing related trait-like characteristics for respective outcomes. The reason why related trait-like characteristics were used for the outcomes was because brief assessments were used for daily surveys that were not concurrently employed in the baseline assessment. To enable the continued use of controlling for person-specific baseline characteristics in the analysis of daily associations, trait-like characteristics available from the baseline assessment were utilized as the best possible approximation.To sum up, four outcome specific baseline value variables were measured before the study period (at T0). The psychological detachment with the RECQ (α = 0.87) [ 53 ] was assessed at the beginning to monitor daily psychological detachment. Further, the SAI [ 55 ] was assessed at the beginning of the study period to monitor daily study ability. To monitor daily stress load, which in part measures mental stress aspects and negative emotional stress aspects, the well-being was assessed at the beginning using the WHO-Five Well-being Index (WHO-5 [ 58 ]). It is a one-dimensional self-report measure with five items. The index value is the sum of all items, with higher values indicating better well-being. As the well-being and stress load tolerance may linked with each other, this variable was assumed to be a good fit with the daily stress load indicating mental and emotional stress aspects. With respect to student life, daily academic performance-related attention was monitored with an instrument for the perception of study demands and resources (termed “Berliner Anforderungen Ressourcen-Inventar – Studierende” in German, the so-called BARI-S [ 59 ]). It contains eight items which capture overwork in studies, time pressure during studies, and the incompatibility of studies and private life. All together they form the BARI-S demand scale (α = 0.85) which was included in the analysis. As overwork and time pressure may result in attention difficulties (e.g. Elfering et al., 2013), this variable was assumed to have a good fit with academic performance-related attention [ 60 ]. Additionally, age in years at T0 was considered as a sociodemographic factor.

Statistical analysis

Since the study design provided ten measurement points for various people, the hierarchical structure of the nested data called for two-level analyses. Pre-analyses of Random-Intercept-Only models for each of the outcome variables (hypothesis 1 to 3) revealed an Intra-Class-Correlation ( ICC ) of at least 0.10 (range 0.26 – 0.64) and confirmed the necessity to perform multilevel analyses [ 61 ]. Specifically, the day-level variables belong to Level 1 (ESD time, LTPA time, longest stretch of time without a break spent on home studying, daily number of breaks during home studying). To analyze day-specific effects within the person, these variables were centered on the person mean (cw = centered within) [ 50 , 62 , 63 , 64 ]. This means that the analyses’ findings are based on a person’s deviations from their average values. The variables assessed at T0 belong to Level 2, which describe the person level (psychological detachment baseline, SAI baseline, well-being, study demands scale, age). These covariates on person level were centered around the grand mean [ 50 ] indicating that the analyses’ findings are based how far an individual deviates from the sample's mean values. As a result, the models’ intercept reflects the outcome value of an average student in the sample at his/her daily average behavior in PA and home study when all parameters are zero. For descriptive statistics SPSS 28.0.1.1 (IBM) and for inferential statistics R (version 4.1.2) were used. The hierarchical models were calculated using the package lme4 with the lmer-function in R in the following steps [ 65 ]. The Null Model was analyzed for all models first, with the corresponding intercept as the only predictor. Afterwards, all variables were entered. The regression coefficient estimates (”b”) were considered for statistical significance for the models and the respective BIC was provided.

In total, five regression models with ‘PA break time’ and ‘LTPA time’ as independent variables were computed due to the five measured outcomes of the present study. Three models belonged to hypothesis 1 and two models to hypothesis 2.

Hypothesis 1: To test hypothesis 1.A two outcome variables were chosen for two separate models: ‘functional stress’ and ‘dysfunctional stress’. Besides the PA behavior variables, the ‘number of breaks’, the ‘longest stretch of time without a break spent on home studying’, ‘age’, and the ‘well-being’ at the beginning of the study as corresponding baseline variable to the output variable were also included as independent variables in both models. The outcome variable ‘psychological detachment’ was utilized in conjunction with the aforementioned independent variables to test hypotheses 1.B, with one exception: psychological detachment at the start of the study was chosen as the corresponding baseline variable.

Hypothesis 2: To investigate hypothesis 2.A the outcome variable ‘attention difficulties’ was selected. Hypothesis 2.B was tested with the outcome variables ‘study ability’. Both models included both PA behavior variables as well as the ‘number of breaks’, the ‘longest stretch of time without a break spent on home studying’, ‘age’ and one corresponding baseline variable each: the ‘study demand scale’ at the start of the study for ‘attention difficulties’ and the ‘SAI’ at the beginning of the study for the daily ‘study ability’.

Hypothesis 3: In addition to both PA behavior variables, age and one baseline variable that matched the outcome variable, the covariates ‘daily longest stretch of time spent on home studying’ and ‘daily number of breaks during home studying’ were included in the models for all five outcome variables.

Handling missing data

The dataset had up to 18% missing values (most exhibit the variables ‘daily longest stretch of time without a break spent on home studying’ with 17.89% followed by ‘daily number of breaks during homes studying’ with 16.67%, and ‘functional / dysfunctional stress’ with 12.45%). Therefore, a sensitivity analysis was performed using the multiple imputation mice-package in the statistical program R [ 66 ], the package howManyImputation based on Von Hippel (2020, [ 67 ]), and the additional broom package [ 68 ]. The results of the models remained the same, with one exception for the Attention Difficulties Model: The daily longest stretch of time without a break spent on home studying showed a significant association (Table  1 in supplement). Due to this almost perfect consistency of results between analyses based on the dataset with missing data and those with imputed data alongside the lack of information provided by the packages for imputed datasets, we decided to stick with the main analysis including the missing data. Thus, in the following the results of the main analysis without imputations are presented.

Table 1 shows the descriptive statistics of the variables used in the analysis. An overview of the analysed models is presented in Table  2 .

Effects on stress load and recovery (hypothesis 1)

Hypothesis 1.A: The Model Functional Stress explained 13% of the variance by fixed factors (marginal R 2  = 0.13), and 52% by both fixed and random factors (conditional R 2  = 0.52). The time spent on ESD as well as the time spent on PA in leisure showed a positive significant influence on functional stress (b = 0.032, p  < 0.01). The same applied to LTPA (b = 0.003, p  < 0.001). The Model Dysfunctional Stress (marginal R 2  = 0.027, conditional R 2  = 0.647) showed only one significant result. The dysfunctional stress was only significantly negatively influenced by the time spent on LTPA (b = 0.002, p  < 0.01).

Hypothesis 1.B: With the Model Detachment, fixed factors contributed 18% of the explained variance and fixed and random factors 46% of the explained variance for psychological detachment. Only the amount of time spent on LTPA revealed a positive impact on psychological detachment (b = 0.003, p  < 0.001).

Effects on academic performance-related parameters (hypothesis 2)

Hypothesis 2.A: The Model Attention Difficulties showed 13% of the variance explained by fixed factors, and 51% explained by both fixed and random factors. It showed a significant negative association only for the time spent on LTPA (b = 0.003, p  < 0.001).

Hypothesis 2.B: The Model SAI showed 18% of the variance explained by fixed factors, and 39% explained by both fixed and random factors. There were significant positive associations for time spent on ESD (b = 0.121, p  < 0.001) and time spent on LTPA (b = 0.012, p  < 0.001). The same applied to LTPA (b = 0.012, p  < 0.001).

Effects of home study behavior (hypothesis 3)

Regarding the independent covariates for the outcome variables functional and dysfunctional stress, there were no significant results for the number of breaks during homes studying or the longest stretch of time without a break spent on home studying. Considering the outcome variable ‘psychological detachment’, there were significant results with negative impact for both study behavior variables: breaks during home studying (b = 0.058, p  < 0.01) and daily longest stretch of time without a break (b = 0.120, p  < 0.01). Evaluating the outcome variables ‘attention difficulties’, there were no significant results for the number of breaks during home studying or the longest stretch of time without a break spent on home studying. Testing the independent study behavior variables for the SAI, it increased with increasing number in daily breaks during homes studying relative to the person´s mean (b = 0.183, p  < 0.05). No significant effect was found for the longest stretch of time without a break spent on home studying ( p  = 0.07).

The baseline covariates of the models showed expected associations and thus confirmed their inclusion. The baseline variables well-being showed a significant impact on functional stress (b = 0.089, p  < 0.001), psychological detachment showed a positive effect on the daily output variables psychological detachment (b = 0.471, p  < 0.001), study demand scale showed a positive association on difficulties in attention (b = 0.240, p  < 0.01), and baseline SAI had a positive effect on the daily SAI (b = 0.335, p  < 0.001).

The present study theorized that PA breaks and LTPA positively influence the academic situation of university students. Therefore, impact on stress load (‘functional stress’ and ‘dysfunctional stress’) and ‘psychological detachment’ as well as academic performance-related parameters ‘self-reported attention difficulties’ and ‘perceived study ability’ was taken into account. The first and second hypotheses assumed that both PA breaks and LTPA are positively associated with the aforementioned parameters and were confirmed for LTPA for all parameters and for PA breaks for functional stress and perceived study ability. The third hypothesis assumed that home study behavior regarding the daily number of breaks during home studying and longest stretch of time without a break spent on home studying has side effects. Detected negative effects for both covariates on psychological detachment and positive effects for the daily number of breaks on perceived study ability were partly unexpected in their direction. These results emphasize the key position of PA in the context of modern health promotion especially for students in an academic context.

Regarding hypothesis 1 and the detected positive associations for stress load and recovery parameters with PA, the results are in accordance with the stress-regulatory potential of PA from the state of research [ 23 ]. For hypothesis 1.A, there is a positive influence of PA breaks and LTPA on functional stress and a negative influence of LTPA on dysfunctional stress. Given the bilateral role of stress load, the results indicate that PA breaks and LTPA are beneficial for coping with study demands, and may help to promote feelings of joy, pride, and learning progress [ 27 ]. This is in line with previous evidence that PA breaks in lectures can buffer university students’ perceived stress [ 29 ], lead to better mood ratings [ 29 , 31 ], and increase in motivation [ 28 , 69 ], vigor [ 34 ], energy [ 30 ], and self-perceived physical and psychological well-being [ 28 ]. Looking at dysfunctional stress, the result point that LTPA counteract load-related states of strain such as inner tension, irritability and nervous restlessness or feelings of boredom [ 27 ]. In contrast, short PA breaks during the day could not have enough impact in countering dysfunctional stress at the end of the day regarding the accumulation of negative stressors during home studying which might have occurred after the participant took PA breaks. Other studies have been able to show a reduction in tension [ 30 ] and general muscular discomfort [ 33 ] after PA breaks. However, this was measured as an immediate effect of PA breaks and not with general evening surveys. Blasche and colleagues [ 34 ] measured effects immediately and 20 min after different kind of breaks and found that PA breaks led to an additional short‐ and medium‐term increase in vigor while the relaxation break lead to an additional medium‐term decrease in fatigue compared to an unstructured open break. This is consistent with the results of the present study that an effect of PA breaks is only observed for functional stress and not for dysfunctional stress. Furthermore, there is evidence that long sitting during lectures leads to increased fatigue and lower concentration [ 31 , 70 ], which could be counteracted by PA breaks. For both types of stress loads, functional and dysfunctional stress, there is an influence of students´ well-being in this study. This shows that the stress load is affected by the way students have mentally felt over the last two weeks. The relevance of monitoring this seems important especially in the time of COVID-19 as, for example, 65.3% of the students of a cross-sectional online survey at an Australian university reported low to very low well-being during that time [ 71 ]. However, since PA and well-being can support functional stress load, they should be of the highest priority—not only as regards the pandemic, but also in general.

Looking at hypothesis 1.B; while there is a positive influence of LTPA on experienced psychological detachment, no significant influence for PA breaks was detected. The fact that only LTPA has a positive effect can be explained by the voluntary character of the activity [ 50 ]. The voluntary character ensures that stressors no longer affect the student and, thus, recovery as detachment can take place. Home studying is not present in leisure times, and thus detachment from study is easier. The PA break videos, on the other hand, were shot in a university setting, which would have made it more difficult to detach from study. In order to further understand how PA breaks affect recovery and whether there is a distinction between PA breaks and LTPA, future research should also consider other types of recovery (e.g. relaxation, mastery, and control). Additionally, different types of PA breaks, such as group PA breaks taken on-site versus video-based PA breaks, should be taken into account.

Considering the confirmed positive associations for academic performance-related parameters of hypothesis 2, the results are in accordance with the evidence of positive associations between PA and learning and educational success [ 6 ], as well as between PA breaks and better cognitive functioning [ 28 ]. Looking at the self-reported attention difficulties of hypothesis 2.A, only LTPA can counteract it. PA breaks showed no effects, contrary to the results of a study of Löffler and collegues (2011, [ 31 ]), in which acute effects of PA breaks could be found for higher attention and cognitive performance. Furthermore, the perception of study demands before the study periods has a positive impact on difficulties in attention. That means that overload in studies, time pressure during studies, and incompatibility of studies and private life leads to higher difficulties with attention in home studying. In these conditions, PA breaks might have been seen as interfering, resulting in the expected beneficial effects of exercise on attention and task-related participation behavior [ 72 , 73 ] therefore remaining undetected. With respect to the COVID-19 pandemic, accompanying education changes, and an increase in student´s worries [ 74 , 75 ], the perception of study demands could be affected. This suggests that especially in times of constraint and changes, it is important to promote PA in order to counteract attention difficulties. This also applies to post-pandemic phase.

Regarding the perceived academic performance of hypothesis 2.B, both PA breaks and LTPA have a positive effect on perceived study ability. This result confirms the positive short-term effects on cognition tasks [ 76 ]. It is also in line with the positive function of PA breaks in interrupting sedentary behavior and therefore counteracting the negative association between sitting behavior and lower cognitive performance [ 24 ]. Additionally, this result also fits with the previously mentioned positive relationship between LTPA and functional stress and between PA breaks and functional stress.

According to hypothesis 3, in relation to the mentioned stress load and recovery parameters, there are negative effects of the daily number of breaks during home studying and the longest stretch of time without a break spent on home studying on psychological detachment. As stressors result in negative activation, which impede psychological detachment from study during non-studying time [ 25 ], it was expected and confirmed that the longest stretch of time without a break spent on home studying has a negative effect on detachment. Initially unexpected, the number of breaks has a negative influence on psychological detachment, as breaks could prevent the accumulation of strain reactions. However, if the breaks had no recovery effect through successful detachment, the number might not have any influence on recovery via detachment. This is indicated by the PA breaks, which had no impact on psychological detachment. Since there are other ways to recover from stress besides psychological detachment, such as relaxation, mastery, and control [ 53 ], PA breaks must have had an additional impact in relation to the positive results for functional stress.

In relation to the mentioned academic performance-related parameters, only the number of breaks has a positive influence on the perceived study ability. This indicates that not only PA breaks but also breaks in general lead to better perceived functionality in studying. Paulus and colleagues (2021) found out that an increase in cognitive skills is not only attributed to PA breaks and standing breaks, but also to open breaks with no special instructions [ 28 ]. Either way, they found better improvement in self-perceived physical and psychological well-being of the university students with PA breaks than with open breaks. This is also reflected in the present study with the aforementioned positive effects of PA breaks on functional stress, which does not apply to the number of breaks.

Overall, it must be considered that the there is a more complex network of associations between the examined parameters. The hypothesized separate relation of PA with different parameters do not consider associations between parameters of stress load / recovery and academic performance although there might be a interdependency. Furthermore, moderation aspects were not examined. For example, PA could be a moderator which buffer negative effects of stress on the study ability [ 55 ]. Moreover, perceived study ability might moderate stress levels and academic performance. Further studies should try to approach and understand the different relationships between the parameters in its complexity.

Limitations

Certain limitations must be taken into account. Regarding the imbalanced design toward more female students in the sample (47 female versus 6 male), possible sampling bias cannot be excluded. Gender research on students' emotional states during COVID-19, when this study took place, or students´ acceptance of PA breaks is diverse and only partially supplied with inconsistent findings. For example, during the COVID-19 pandemic, some studies reported that female students were associated with lower well-being [ 71 ] or worse mental health trajectories [ 75 , 77 ]. Another study with a large sample of students from 62 countries reported that male students were more strongly affected by the pandemic because they were significantly less satisfied with their academic life [ 74 ]. However, Keating and colleges (2020) discovered that, despite the COVID-19 pandemic, females rated some aspects of PA breaks during lectures more positively than male students did. However, this was also based on a female slanted sample [ 78 ]. Further studies are needed to get more insights into gender bias.

Furthermore, the small sample size combined with up to 16% missing values comprises a significant short-coming. There were a lot of possibilities which could cause such missing data, like refused, forgotten or missed participation, technical problems, or deviation of the personal code for the questionnaire between survey times. Although the effects could be excluded by sensitive analysis due to missing data, the sample is still small. To generalize the findings, future replication studies are needed.

Additionally, PA breaks were only captured through participation in the ESD, the specially instructed PA break via video. Effects of other short PA breaks were not include in the study. However, participants were called to participate in ESD whenever possible, so the likelihood that they did take part in PA breaks in addition to the ESD could be ignored.

With respect to the baseline variables, it must be considered that two variables (stress load, attention difficulties) were adjusted not with their identical variable in T0, but with other conceptually associated variables (well-being index, BARI-S). Indeed, contrary to the assumption the well-being index does only show an association with functional stress, indicating that it does not control dysfunctional stress. Although the other three assumed associations were confirmed there might be a discrepancy between the daily measured variables and the variables measured in T0. Further studies should either proof the association between these used variables or measure the same variables in T0 for control the daily value of these variables.

Moreover, the measuring instruments comprised the self-assessed perception of the students and thus do not provide an objective information. This must be considered, especially for measuring cognitive and academic-performance-related measures. Here, existing objective tests, such as multiple choice exams after a video-taped lecture [ 72 ] might have also been used. Nevertheless, such methods were mostly used in a lab setting and do not reflect reality. Due to economic reasons and the natural learning environment, such procedures were not applied in this study. However, the circumstances of COVID-19 pandemic allowed a kind of lab setting in real life, as there were a lot of restrictions in daily life which limited the influence of other covariates. The study design provides a real natural home studying environment, producing results that are applicable to the healthy way that students learn in the real world. As this study took place under the conditions of COVID-19, new transformations in studying were also taken into account, as home studying and digital learning are increasingly part of everyday study.

However, the restrictions during the COVID-19 pandemic could result in a greater extent of leisure time per se. As the available leisure time in general was not measured on daily level, it is not possible to distinguish if the examined effects on the outcomes are purely attributable to PA. It is possible that being more physical active is the result of having a greater extent of leisure time and not that PA but the leisure time itself effected the examined outcomes. To address this issue in future studies, it is necessary to measure the proportion of PA in relation to the leisure time available.

Furthermore, due to the retrospective nature of the daily assessments of the variables, there may be overstated associations which must be taken into account. Anyway, the daily level of the study design provides advantages regarding the ability to observe changes in an individual's characteristics over the period of the study. This design made it possible to find out the necessity to analyze the hierarchical structure of the intraindividual data nested within the interindividual data. The performed multilevel analyses made it possible to reflect the outcome of an average student in the sample at his/her daily average behavior in PA and home study.

Conclusion and practical implications

The current findings confirm the importance of PA for university students` stress load, recovery experience, and academic performance-related parameters in home studying. Briefly summarized, it can be concluded that PA breaks positively affect stress load and perceived study ability. LTPA has a positive impact on stress load, recovery experience, and academic performance-related parameters regarding attention difficulties and perceived study ability. Following these results, universities should promote PA in both fashions in order to keep their students healthy and functioning: On the one hand, they should offer opportunities to be physically active in leisure time. This includes time, environment, and structural aspects. The university sport department, which offers sport courses and provides sport facilities on university campuses for students´ leisure time, is one good example. On the other hand, they should support PA breaks during the learning process and in the immediate location of study. This includes, for example, providing instructor videos for PA breaks to use while home studying, and furthermore having instructors to lead in-person PA breaks in on-site learning settings like universities´ libraries or even lectures and seminars. This not only promotes PA, but also reduces sedentary behavior and thereby reduces many other health risks. Further research should focus not only on the effect of PA behavior but also of sedentary behavior as well as the amount of leisure time per se. They should also try to implement objective measures for example on academic performance parameters and investigate different effect directions and possible moderation effects to get a deeper understanding of the complex network of associations in which PA plays a crucial role.

Availability of data and materials

The datasets used and analyzed during the current study are available from the corresponding author on reasonable request.

Abbreviations

Attention and Performance Self-Assessment

"Berliner Anforderungen Ressourcen-Inventar – Studierende" (instrument for the perception of study demands and resources)

Centered within

Grand centered

“Erfassung von Emotionen und Beanspruchung “ (questionnaire containing a word list of adjectives for the recording of emotions and stress during work)

Exercise snack digital (special physical activity break offer)

Intra-Class-Correlation

Leisure time physical activity

  • Physical activity

Recovery Experience Questionnaire

Study ability index

World Health Organization-Five Well-being index

Knight JA. Physical inactivity: associated diseases and disorders. Ann Clin Lab Sci. 2012;42(3):320–37.

PubMed   Google Scholar  

Kemel PN, Porter JE, Coombs N. Improving youth physical, mental and social health through physical activity: a systematic literature review. Health Promot J Austr. 2002;33(3):590–601.

Article   Google Scholar  

Gothe NP, Ehlers DK, Salerno EA, Fanning J, Kramer AF, McAuley E. Physical activity, sleep and quality of life in older adults: influence of physical, mental and social well-being. Behav Sleep Med. 2020;18(6):797–808.

Article   PubMed   Google Scholar  

Eime RM, Young JA, Harvey JT, Charity MJ, Payne WR. A systematic review of the psychological and social benefits of participation in sport for children and adolescents: informing development of a conceptual model of health through sport. Int J Behav Nutr Phys Act. 2013;10(1):1–21.

Google Scholar  

Iannotti RJ, Janssen I, Haug E, Kololo H, Annaheim B, Borraccino A. Interrelationships of adolescent physical activity, screen-based sedentary behaviour, and social and psychological health. Int J Public Health. 2009;54:191–8.

Dadaczynski K, Schiemann S. Welchen Einfluss haben körperliche Aktivität und Fitness im Kindes-und Jugendalter auf Bildungsoutcomes? German J Exerc Sport Res. 2015;4(45):190–9.

Kari JT, Pehkonen J, Hutri-Kähönen N, Raitakari OT, Tammelin TH. Longitudinal associations between physical activity and educational outcomes. Med Sci Sports  Exerc. 2017;49(11).

Grim M, Hortz B, Petosa R. Impact evaluation of a pilot web-based intervention to increase physical activity. Am J Health Promot. 2011;25(4):227–30.

Irwin JD. Prevalence of university students’ sufficient physical activity: A systematic review. Percept Mot Skills. 2004;98:927–43.

Kwan MY, Cairney J, Faulkner GE, Pullenayegum EE. Physical activity and other health-risk behaviors during the transition into early adulthood: a longitudinal cohort study. Am J Prev Med. 2012;42(1):14–20.

John JM, Gropper H, Thiel A. The role of critical life events in the talent development pathways of athletes and musicians: A systematic review. Psychol Sport Exerc. 2019;45.

Bopp M, Bopp C, Schuchert M. Active transportation to and on campus is associated with objectively measured fitness outcomes among college students. J Phys Act Health. 2015;12(3):418–23.

Dadaczynski K. Stand der Forschung zum Zusammenhang von Gesundheit und Bildung. Überblick und Implikationen für die schulische Gesundheitsförderung. Zeitschrift für Gesundheitspsychologie. 2012;20(3):141–53

Suhrcke M, de Paz NC. The impact of health and health behaviours on educational outcomes in high-income countries: a review of the evidence. Copenhagen: WHO Regional Offi ce for Europe; 2011.

Lynch BM, Owen N. Too much sitting and chronic disease risk: steps to move the science forward. Ann Intern Med. 2015;16(2):146–7.

Voss MW, Carr LJ, Clark R, Weng T. Revenge of the “sit” II: Does lifestyle impact neuronal and cognitive health through distinct mechanisms associated with sedentary behavior and physical activity? Ment Health Phys Act. 2014;7(1):9–24.

Huber G. Ist Sitzen eine tödliche Aktivität? B&G Bewegungstherapie und Gesundheitssport. 2014;30(01):13–6.

Peterson NE, Sirard JR, Kulbok PA, DeBoer MD, Erickson JM. Sedentary behavior and physical activity of young adult university students. Res Nurs Health. 2018;4(1):30–8.

Rupp R, Dold C, Bucksch J. Sitzzeitreduktion und Bewegungsaktivierung in der Hochschullehre – Entwicklung und Implementierung der Mehrebenen-Intervention Kopf-Stehen. Die Hochschullehre. 2019;5:525–42.

Ickes MJ, McMullen J, Pflug C, Westgate PM. Impact of a University-based Program on Obese College Students’ Physical Activity Behaviors, Attitudes, and Self-efficacy. Am J Health Educ. 2016;47(1):47–55.

Lepp A, Barkley JE, Karpinski AC. The relationship between cell phone use and academic performance in a sample of US college students. Sage Open. 2015;5(1).

Stapp AC, Prior LF. The Impact of Physically Active Brain Breaks on College Students’ Activity Levels and Perceptions. J Physic Activ Res. 2018;3(1):60–7.

Fuchs R, Klaperski S. Sportliche Aktivität und Stressregulation. In: Fuchs R, Schlicht W, editors. Sportaktivität und seelische Gesundheit Göttingen: Hogrefe; 2012. p. 100–21.

Falck RS, Davis JC, Liu-Ambrose T. What is the association between sedentary behaviour and cognitive function? A systematic review. Br J Sports Med. 2017;51(10):800–11.

Sonnentag S, Fritz C. Recovery from job stress: The stressor-detachment model as an integrative framework. J Organ Behav. 2015;36:72–103.

Edwards JR. A cybernetic theory of stress, coping, and well-being in organizations. Acad Manag Rev. 1992;17(2):238–74.

Wieland R. Status-Bericht: Psychische Gesundheit in der betrieblichen Gesundheitsförderung – eine arbeitspsychologische Perspektive. In: Nold H, Wenninger G, editors. Rückengesundheit und psychische Gesundheit. Rückengesundheit und psychische Gesundheit.: Asanger Verlag; 2013.

Paulus M, Kunkel J, Schmidt SCE, Bachert P, Wäsche H, Neumann R, et al. Standing breaks in lectures improve university students’ self-perceived physical, mental, and cognitive condition. Int J Environ Res Public Health. 2021;18.

Marschin V, Herbert C. A Short, Multimodal Activity Break Incorporated Into the Learning Context During the Covid-19 Pandemic: Effects of Physical Activity and Positive Expressive Writing on University Students’ Mental Health — Results and Recommendations From a Pilot Study. Front Psychol. 2021;12.

Gollner E, Savil M, Schnabel F, Braun C, Blasche G. Unterschiede in der Wirksamkeit von Kurzpausenaktivitäten im Vergleich von Bewegungspausen zu psychoregulativen Pausen bei kognitiver Belastung. Bewegungstherapie Gesundheitssport. 2019;35:134–43.

Löffler SN, Dominok E, von Haaren B, Schellhorn R, Gidion G. Aktivierung, Konzentration, Entspannung: Interventionsmöglichkeiten zur Förderung fitnessrelevanter Kompetenzen im Studium: KIT Scientific Publishing; 2011.

Marschin V, Herbert C. A short, multimodal activity break incorporated into the learning context during the Covid-19 pandemic: effects of physical activity and positive expressive writing on university students' mental health—results and recommendations from a pilot study. Front Psychol. 2021.

Kowalsky RJ, Farney TM, Hearon CM. Resistance Exercise Breaks Improve Ratings of Discomfort and Sleepiness in College Students. Res Q Exerc Sport. 2022;94(1):210–5.

Blasche G, Szabo B, Wagner-Menghin M, Ekmekcioglu C, Gollner E. Comparison of rest-break interventions during a mentally demanding task. Stress Health. 2018;34(5):629–38.

Article   PubMed   PubMed Central   Google Scholar  

Fuchs R, Klaperski S. Stressregulation durch Sport und Bewegung. In: Fuchs R, Gerber M, editors. Handbuch Stressregulation und Sport. Berlin: Springer; 2018. p. 205–26.

Chapter   Google Scholar  

Kaluza G, Renneberg B. Stressbewältigung. In: Bengel J, Jerusalem M, editors. HandbuchGesundheitspsychologie und medizinische Psychologie. Göttingen: Hogrefe; 2009. p. 265–72.

Klaperski S. Exercise, Stress and Health: The Stress-Buffering Effect of Exercise. In: Fuchs R, Gerber M, editors. Handbuch Stressregulation und Sport. Berlin: Springer; 2018. p. 227–50.

Lesser IA, Nienhuis CP. The impact of COVID-19 on physical activity behavior and well-being of Canadians. Int J Environ Res Public Health. 2020;17(11):3899.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Moore SA, Faulkner G, Rhodes RE, Brussoni M, Chulak-Bozzer T, Ferguson LJ, et al. Impact of the COVID-19 virus outbreak on movement and play behaviours of Canadian children and youth: a national survey. Int J Behav Nutr Phys Act. 2020;17(1):1–11.

Rodríguez-Larrad A, Mañas A, Labayen I, González-Gross M, Espin A, Aznar S, et al. Impact of COVID-19 confinement on physical activity and sedentary behaviour in Spanish university students: Role of gender. Int J Environ Res Public Health. 2021;18(2):369.

Stanton R, To QG, Khalesi S, Williams SL, Alley SJ, Thwaite TL, et al. Depression, anxiety and stress during COVID-19: associations with changes in physical activity, sleep, tobacco and alcohol use in Australian adults. Int J Environ Res Public Health. 2020;17(11):4065.

Zheng C, Huang WY, Sheridan S, Sit CHP, Chen XK, Wong SHS. COVID-19 pandemic brings a sedentary lifestyle in young adults: a cross-sectional and longitudinal study. Int J Environ Res Public Health. 2020;17(17):6035.

Commodari E. Attention Skills and Risk of Developing Learning Difficulties. Curr Psychol. 2012;31:17–34.

Commodari E, Guarnera M. Attention and reading skills. Percept Mot Skills. 2005;100:3753–86.

Raaijmakers MAJ, Smidts DP, Sergeant JA, Maassen GH, Posthumus JA, van Engeland H, et al. Executive functions in preschool children with aggressive behavior: impairments in inhibitory control. J Abnorm Child Psychol. 2008;36:1097–107.

Vellutino FR, Scanlon DM, Sipay ER, Small SG, Pratt A, Chen RS, et al. Cognitive profiles of difficulty to remediate and readily remediate poor readers: early intervention as a vehicle for distinguishing between cognitive and experiential deficits as basic of specific reading disability. J Educ Psychol. 1996;88:601–38.

Ilies R, Dimotakis N, De Pater IE. Psychological and physiological reactions to high workloads: Implications for well-being. Pers Psychol. 2010;63(2):407–36.

Rodell JB, Judge TA. Can “good” stressors spark “bad” behaviors? The mediating role of emotions in links of challenge and hindrance stressors with citizenship and counterproductive behaviors. J Appl Psychol. 2009;94(6).

Teuber M, Leyhr D, Moll J, Sudeck G. Nudging digital physical activity breaks for home studying of university students—A randomized controlled trial during the COVID-19 pandemic with daily activity measures. Front Sports Active Living. 2022;4.

Feuerhahn N, Sonnentag S, Woll A. Exercise after work, psychological mediators, and affect: A day-level study. Eur J Work Organ Psy. 2014;23(1):62–79.

Sonnentag S. Work, Recovery Activities, ans Individual Well-Being: A Diary Study. J Occup Health Psychol. 2001;6(3):196–210.

Article   CAS   PubMed   Google Scholar  

Wieland R. Gestaltung gesundheitsförderlicher Arbeitsbedingungen. In: Kleinbeck U, Schmidt K-H, editors. Arbeitspsychologie (Enzyklopädie der Psychologie, Serie Wirtschafts-, Organisations- und Arbeitspsychologie). 1. Göttingen: Hogrefe; 2010. p. 869–919.

Sonnentag S, Fritz C. The Recovery Experience Questionnaire: development and validation of a measure for assessing recuperation and unwinding from work. J Occup Health Psychol. 2007;12(3):204–21.

Bankstahl US, Görtelmeyer R. Measuring subjective complaints of attention and performance failures development and psychometric validation in tinnitus of the self-assessment scale APSA. Health and Quality of Life Outcomes. 2013;11(86).

Teuber M, Arzberger I, Sudeck G. Körperliche Aktivität, Gesundheit und Funktionsfähigkeit im Studium: Sportliche Freizeitaktivitäten und aktive Fortbewegung als Ressource im Studium? In: Göring A, Mayer J, Jetzke M, editors. Sport und Studienerfolg - Analysen zur Bedeutung sportlicher Aktivität im Setting Hochschule. Hochschulsport: Bildung und Wissenschaft, 4. Göttingen: Universitätsverlag Göttingen; 2020. p. 27–49.

Hasselhorn H-M, Freude G. Der Work-Ability-Index: ein Leitfaden In: Arbeitsmedizin BfAu, editor. Dortmund/Berlin/Dresden: Wirtschaftsverl. NW, Verlag für Neue Wissenschaft GmbH; 2007.

Rütten A, Pfeifer K. Nationale Empfehlungen für Bewegung und Bewegungsförderung. Köln: Bundeszentrale für Gesundheitliche Aufklärung (BZgA); 2017.

Brähler E, Mühlan H, Albani C, Schmidt S. Teststatistische Prüfung und Normierung der deutschen Versionen des EUROHIS-QOL Lebensqualität-Index und des WHO-5 Wohlbefindens-Index. Diagnostica. 2007;53(2):83–96.

Gusy B, Wörfel F, Lohmann K. Erschöpfung und Engagement im Studium. Zeitschrift für Gesundheitspsychologie. 2016;24(1):41–53.

Elfering A, Grebner S, de Tribolet-Hardy F. The long arm of time pressure at work: Cognitive failure and commuting near-accidents. Eur J Work Organ Psy. 2013;22(6):737–49.

Kreft IG, de Leeuw J. Introducing multilevel modeling. London: Sage; 1998.

Book   Google Scholar  

Bates D, Mächler M, Bolker BM, Walker SC. Fitting Linear Mixed-Effects Models Using lme4. J Stat Softw. 2015;67(1):1–48.

Hofmann DA, Gavin MB. Centering decisions in hierarchical linear models: Implications for research in organizations. J Manag. 1998;24(5):623–41.

Nezlek J. Diary Studies in Social and Personality Psychology: An Introduction With Some Recommendations and Suggestions. Social Psychological Bulletin. 2020;15(2).

Knapp G. Gemischte Modelle in R. Begleitskriptum zur Weiterbildung. In: Dortmund TU, editor. Braunschweig2019.

Van Buuren S, Groothuis-Oudshoorn K. mice: Multivariate imputation by chained equations in R. J Stat Softw. 2011;45:1–67.

von Hippel PT. How Many Imputations Do You Need? A Twostage Calculation Using a Quadratic Rule. Sociological Methods & Research. 2020;49(3):699–718.

Article   MathSciNet   Google Scholar  

Robinson D. broom: An R package for converting statistical analysis objects into tidy data frames. arXiv preprint arXiv:14123565. 2014.

Young-Jones A, McCain J, Hart B. Let’s Take a Break: The Impact of Physical Activity on Academic Motivation. Int J Teach Learn High Educ. 2022;33(3):110–8.

Barr-Anderson DJ, AuYoung M, Whitt-Glover MC, Glenn BA, Yancey AK. Integration of short bouts of physical activity into organizational routine: A systematic review of the literature. Am J Prev Med. 2011;40(1):76–93.

Dodd RH, Dadaczynski K, Okan O, McCaffery KJ, Pickles K. Psychological Wellbeing and Academic Experience of University Students in Australia during COVID-19. Int J Environ Res Public Health 2021;18.

Fenesi B, Lucibello K, Kim JA, Heisz JJ. Sweat so you don’t forget: exercise breaks during a university lecture increase on-task attention and learning. J Appl Res Mem Cogn. 2018;7(2):261–9.

Ruhland S, Lange KW. Effect of classroom-based physical activity interventions on attention and on-task behavior in schoolchildren: A systematic review. Sports Med Health Sci. 2021;3:125–33.

Aristovnik A, Keržič D, Ravšelj D, Tomaževič N, Umek L. Impacts of the COVID-19 Pandemic on Life of Higher Education Students: A Global Perspective. Sustainability. 2020;12(20).

Browning MHEM, Larson LR, Sharaievska I, Rigolon A, McAnirlin O, Mullenbach L, et al. Psychological impacts from COVID-19 among university students: Risk factors across seven states in the United States. PLoS ONE 2021;16(1).

Chang Y-K, Labban JD, Gapin JI, Etnier JL. The effects of acute exercise on cognitive performance: a meta-analysis. Brain Res. 2012;1453:87–101.

Elmer T, Mepham K, Stadtfeld C. Students under lockdown: Comparisons of students’ social networks and mental health before and during the COVID-19 crisis in Switzerland. PLoS ONE. 2020;15(7):e0236337.

Keating R, Ahern S, Bisgood L, Mernagh K, Nicolson GH, Barrett EM. Stand up, stand out. Feasibility of an active break targeting prolonged sitting in university students. J Am Coll Health. 2020;70(7).

Download references

Acknowledgements

We would like to thank Juliane Moll, research associate of the Student Health Management of University of Tübingen, for the support in the coordination and realization study. We would like to express our thanks also to Ingrid Arzberger, Head of University Sports at the University of Tübingen, for providing the resources and co-applying for the funding. We acknowledge support by Open Access Publishing Fund of University of Tübingen.

Open Access funding enabled and organized by Projekt DEAL. This research regarding the conduction of the study was funded by the Techniker Krankenkasse, health insurance fund.

Author information

Authors and affiliations.

Institute of Sports Science, Faculty of Economics and Social Sciences, University of Tübingen, Tübingen, Germany

Monika Teuber, Daniel Leyhr & Gorden Sudeck

Methods Center, Faculty of Economics and Social Sciences, University of Tübingen, Tübingen, Germany

Daniel Leyhr

Interfaculty Research Institute for Sports and Physical Activity, University of Tübingen, Tübingen, Germany

Gorden Sudeck

You can also search for this author in PubMed   Google Scholar

Contributions

M.T. and G.S. designed the study. M.T. coordinated and carried out participant recruitment and data collection. M.T. analyzed the data and M.T. and D.L. interpreted the data. M.T. drafted the initial version of the manuscript and prepared the figure and all tables. All authors contributed to reviewing and editing the manuscript and have read and agreed to the final version of the manuscript.

Corresponding author

Correspondence to Monika Teuber .

Ethics declarations

Ethics approval and consent to participate.

The study involves human participants and was reviewed and approved by the Ethics Committee of the Faculty of Social Sciences and Economics, University of Tübingen (ref. A2.54-127_kr). The participants provided their written informed consent to participate in this study. All methods were carried out in accordance with relevant guidelines and regulations.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Supplementary file 1., supplementary file 2., rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Teuber, M., Leyhr, D. & Sudeck, G. Physical activity improves stress load, recovery, and academic performance-related parameters among university students: a longitudinal study on daily level. BMC Public Health 24 , 598 (2024). https://doi.org/10.1186/s12889-024-18082-z

Download citation

Received : 30 June 2023

Accepted : 12 February 2024

Published : 24 February 2024

DOI : https://doi.org/10.1186/s12889-024-18082-z

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

  • Physical activity breaks
  • Stress load
  • Psychological detachment
  • Academic performance
  • Study ability
  • University students

BMC Public Health

ISSN: 1471-2458

importance of review related literature in research study

Volume 23 Supplement 1

Understanding Success: Multi-country implementation research in U5M reduction

  • Open access
  • Published: 28 February 2024

Reducing the equity gap in under-5 mortality through an innovative community health program in Ethiopia: an implementation research study

  • Laura Drown 1   na1 ,
  • Alemayehu Amberbir 2   na1 ,
  • Alula M. Teklu 3 ,
  • Meseret Zelalem 4 ,
  • Abreham Tariku 4 ,
  • Yared Tadesse 4 ,
  • Solomon Gebeyehu 4 ,
  • Yirdachew Semu 4 ,
  • Jovial Thomas Ntawukuriryayo 2 ,
  • Amelia VanderZanden   ORCID: orcid.org/0000-0001-6984-2132 2 ,
  • Agnes Binagwaho 2 &
  • Lisa R. Hirschhorn 5  

BMC Pediatrics volume  23 , Article number:  647 ( 2024 ) Cite this article

Metrics details

The Ethiopian government implemented a national community health program, the Health Extension Program (HEP), to provide community-based health services to address persisting access-related barriers to care using health extension workers (HEWs). We used implementation research to understand how Ethiopia leveraged the HEP to widely implement evidence-based interventions (EBIs) known to reduce under-5 mortality (U5M) and address health inequities.

This study was part of a six-country case study series using implementation research to understand how countries implemented EBIs between 2000–2015. Our mixed-methods research was informed by a hybrid implementation science framework using desk review of published and gray literature, analysis of existing data sources, and 11 key informant interviews. We used implementation of pneumococcal conjugate vaccine (PCV-10) and integrated community case management (iCCM) to illustrate Ethiopia’s ability to rapidly integrate interventions into existing systems at a national level through leveraging the HEP and other implementation strategies and contextual factors which influenced implementation outcomes.

Ethiopia implemented numerous EBIs known to address leading causes of U5M, leveraging the HEP as a platform for delivery to successfully introduce and scale new EBIs nationally. By 2014/15, estimated coverage of three doses of PCV-10 was at 76%, with high acceptability (nearly 100%) of vaccines in the community. Between 2000 and 2015, we found evidence of improved care-seeking; coverage of oral rehydration solution for treatment of diarrhea, a service included in iCCM, doubled over this period. HEWs made health services more accessible to rural and pastoralist communities, which account for over 80% of the population, with previously low access, a contextual factor that had been a barrier to high coverage of interventions.

Conclusions

Leveraging the HEP as a platform for service delivery allowed Ethiopia to successfully introduce and scale existing and new EBIs nationally, improving feasibility and reach of introduction and scale-up of interventions. Additional efforts are required to reduce the equity gap in coverage of EBIs including PCV-10 and iCCM among pastoralist and rural communities. As other countries continue to work towards reducing U5M, Ethiopia’s experience provides important lessons in effectively delivering key EBIs in the presence of challenging contextual factors.

With a population of more than 112 million, Ethiopia is currently the second most populous country in Africa [ 1 ]. About 79% of its population resides in rural areas where communities are typically sparsely distributed [ 2 ]. Health inequities are reflected in notable differences in health outcomes across ethnic-based regions and population groups. Under-5 mortality (U5M), an important area of focus in Ethiopia, decreased from 166 per 1,000 live births in 2000 to 67 in 2016, an impressive decline of about 59% [ 3 ]. However, subnational U5M rates show that this progress was not uniform across the country’s nine regional states and two administrative states. In 2016, U5M rates still ranged widely from 39 per 1,000 live births in Addis Ababa to 125 per 1,000 in the largely pastoralist region of Afar [ 3 ]. There were also differences between residence types, with U5M reported to be about 41% higher in rural areas of the country compared with urban ones [ 3 ]. These inequities mirror differences in coverage of many key health interventions and demonstrate that ensuring equitable access to health services for all is a major challenge in this large and diverse country [ 4 ].

In 1996, the Government of Ethiopia developed a 20-year plan under the Health Sector Development Program (HSDP), which was divided into four series of 5-year plans (HSDP-I – IV). The assessment of HSDP-I revealed that programs were not reaching the grassroots level, and called for mechanisms to improve access. The Ministry of Health (MOH) of Ethiopia invested in expansion and strengthening of its public health system to improve access to primary health care (PHC) [ 5 ]. This health system features three tiers – the primary, secondary, and tertiary levels (Fig.  1 ). In rural areas, the primary level includes the PHC unit, each of which consists of five health posts are under one health center, which report to one primary hospital in each kebele (Ethiopia’s lowest administrative level). This differs in rural areas, where the primary level simply consists of one health center per woreda or district. The secondary and tertiary levels consist of general and six specialized hospitals, respectively [ 6 ].

figure 1

Ethiopia’s three-tiered public health care system

In 2003/2004 the Health Extension Program (HEP) was started to improve coverage and it became part of HSDP-II [ 7 , 8 ]. This program was designed to provide community-based health services and address persisting barriers to care in rural areas [ 7 ]. Under the HEP, Ethiopia introduced a cadre of full-time, salaried, and primarily female health extension workers (HEWs), selected by the communities they serve. HEWs complete a one-year formal training after finishing high school, which prepares them to offer an initial package of 16 services at the community level, both at a new facility type called the health post and in homes [ 7 ]. This package comprise various EBIs including community-based growth monitoring and promotion, immunization services, maternal health interventions, integrated management of childhood illness (IMCI), promotion of breastfeeding, use of insecticide-treated bed nets, hygiene and sanitation promotion and community-based management of acute malnutrition. This broad package of services goes beyond the typical scope of community health workers in many countries to include services such as vaccination. In subsequent years following the HEP’s introduction, its scope expanded to improve access to key curative health services, including treatment of diseases like malaria, pneumonia, and diarrhea that represented major causes of death in the country [ 7 ]. By 2009/10, the government had trained and deployed 34,482 HEWs across all regions, with HEW-to-population ratios varying widely by region [ 8 ]. In addition it introduced part-time Health Development Army volunteers in 2011 to support HEWs and assist in health promotion activities [ 9 ].

Implementation research includes the study of how strategies are chosen and used to adopt and integrate evidence-based interventions (EBIs) into real-world settings. It further looks at the contextual factors which serve as facilitators or barriers to improve individual outcomes and population health [ 10 ]. This methodology is of particular importance in improving and sharing lessons from low- and middle-income countries (LMICs) for more effective implementation and improved progress in health outcomes [ 11 , 12 ].Existing literature has extensively described the role of the HEP in improving coverage of health services through increased access at the community level. For example a systematic review by Assefa et al. [ 13 ] found that the program helped Ethiopia achieve major improvements in several key areas such as communicable diseases, maternal and child health, sanitation and hygiene, community knowledge, and health care seeking [ 13 ]. Other studies have reported on the contribution of the HEP in increasing access to and utilization of health services, particularly for maternal and child health [ 14 , 15 , 16 ]. However, much of the current literature on EBIs is focused on effectiveness and coverage and does not explore what was done, what worked, and implementation strategies utilized.

This paper uses data collected as part of a multi-country implementation research case series to understand successes in addressing amenable U5M deaths through implementation of health system-delivered EBIs between 2000 and 2015. We used case study methodology to understand which implementation strategies Ethiopia utilized alongside leveraging the HEP to implement U5M-targeted health system-delivered EBIs and explore contextual factors that affected the success of the strategies in achieving implementation outcomes. Ethiopia implemented numerous EBIs known to address leading causes of U5M. We specifically used the implementation and role of the HEP in two EBIs, the pneumococcal conjugate vaccine (PCV) and integrated community case management (iCCM), to provide a deeper understanding of the processes and outcomes. The results provide transferable lessons for other countries working to accelerate their U5M reduction efforts by leveraging community-based care delivery while targeting inequity in EBI coverage.

This study drew from the larger six-country case study series utilizing implementation research methods to understand how countries implemented health-system delivered EBIs known to reduce amenable U5M. Details of the case study methods and our hybrid framework are reported elsewhere [ 17 ]. In brief, this mixed-methods implementation research was informed by a hybrid implementation science framework designed for the project to capture strategies, contextual factors, and implementation outcomes of EBI implementation for reducing U5M in LMICs (Fig.  2 ).This framework included expansion of the Exploration, Preparation, Implementation, Sustainment Framework to include an explicit Adaptation stage (EPIAS) [ 17 , 18 ].

figure 2

Implementation research framework for understanding evidence-based interventions to reduce under-5 mortality (Hirschhorn, L. R., et al., Gates Open Research, 2021;5,72)

Data sources

Desk review.

The study team undertook a review of available sources including peer-reviewed and grey literature and program documents focusing on the rates and progress of U5M and implementation of the EBIs known to reduce amenable U5M in countries (Table 1 ). Initial searches were performed through MEDLINE (PubMed) and Google Scholar using the search terms “child mortality” or “under-5 mortality” and Ethiopia. Further searches included specific EBIs, causes of death, or contextual factors as search terms (e.g. “insecticide-treated nets,” “malaria,” or “community health workers”). Initial desk research was completed by the Strategic Analysis, Research, & Training (START) Center at the University of Washington and the study team expanded on the review through an iterative process throughout the case study as additional data sources or information gaps were identified.

We extracted existing data from sources including the Ethiopian Demographic and Health Survey (DHS, 2001–2016), the World Bank, and WHO/UNICEF to measure changes in EBI coverage, U5M, and trends in health and development indicators related to U5M at the national and subnational levels between 2000 and 2015.

Key informant interviews

The study team conducted 11 key informant interviews in Ethiopia reflecting a broad range of experience and viewpoints. Key informants (KIs) were chosen based on the targeted EBIs and areas where additional insight was needed from the desk review. We prioritized KIs who were able to provide information on the EPIAS stages during the period of study. Potential KIs were approached by the in-country principal investigator (AT) by telephone or email. Key informants included current and former MOH employees responsible for high-level strategic direction of the ministry or specific disease or intervention areas, implementing partners, and individuals from multilateral or donor organizations who had managed partner-supported activities during the period of interest (2000–2015). We prioritized individuals active in this work during the study period but were also able to capture some experiences from the periods of 1995–1999 and 2017–2019. The interviews were conducted using a semi-structured interview guide based on the hybrid framework. All interviews were conducted in English and led by the project research associates or the in-country principal investigator, who were trained in qualitative interviewing. Interviews were audio recorded and transcribed prior to analysis.

Data analysis

We used a mixed-methods Quant-QUAL approach [ 18 ]. Descriptive statistics were used for EBI coverage at the national and subnational levels. We applied the framework to understand the implementation strategies, contextual factors serving as facilitators and barriers at the local, national, and global levels, and implementation outcomes from the literature. A priori codes for implementation strategies, outcomes, and contextual factors which were adapted and expanded as emerging codes were identified. The study team used the codes to analyze the transcripts and direct content analysis [ 19 ] was used to identify strategies, contextual factors, and implementation outcomes. Qualitative and quantitative data were triangulated to link the strategies, contextual factors, implementation outcomes, and coverage, and to emerge transferable lessons for other countries learning from EBI implementation in Ethiopia.

Role of health extension program

We found from the desk review and key informant interviews that for the majority of the targeted EBIs implemented in Ethiopia to reduce U5M the HEP was utilized as an important implementation strategy (Table 1 ). Ethiopia frequently leveraged the HEP as a platform for delivery of new services, with HEWs playing an important role as key implementers. Service delivery by HEWs increased coverage of many EBIs compared to implementation at health facilities alone. HEWs made health services more accessible to rural and pastoralist communities that previously had low access to health facilities and services, a contextual factor that served as a barrier to high coverage of health services and interventions. This strategy of leveraging the HEP improved feasibility and reach of introduction and enabled scale up of interventions in more resource-limited and rural settings in Ethiopia by using the strong, widespread platform. Utilization of the HEP also helped Ethiopia ensure acceptability of new interventions since service delivery involved HEWs already working in communities. The strategy of embedding new interventions into the HEP additionally improved the sustainability of new interventions by building upon an existing service delivery platform rather than relying on new, vertical ones.

In addition to direct service provision, HEP activities were designed to increase demand and uptake of maternal and child health services based at health facilities. Strategies included regular household visits, conducting community engagement and health promotion activities including sensitization to promote facility-based delivery, an EBI with very low coverage historically. Nationally, the Women Development Army cadre of volunteers were introduced into the HEP in 2011 to enhance the HEP’s ability to reach households. They also helped strengthen HEP activities and improve utilization of services through demand generation activities and linking community members and HEWs [ 9 ].

We identified several other implementation strategies to introduce and scale up EBIs to reduce U5M. The most common implementation strategies Ethiopia utilized (shown in Table 2 as used for PCV and iCCM) included: national policy and development planning, leveraging and coordinating strong donor and partner support, using data for evidence-based decision-making, and integration into or leveraging of existing programs or platforms.

The HEP’s role in the implementation of two EBIs, PCV and iCCM, is further described below. These two EBIs were introduced and successfully scaled to a national level during the study period to address major causes of U5M and are discussed to illustrate the role of the HEP in implementation and delivery of vaccines and curative care in Ethiopia.

Pneumococcal conjugate vaccine

The PCV-10 vaccine was introduced in late 2011 to prevent severe forms of pneumococcal disease such as pneumonia and meningitis that were previously estimated to account for up to 28% of all deaths among children under 5 in Ethiopia [ 20 ]. The work followed the implementation pathway of the EPIAS framework (Table 2 ). Exploration was followed by an intensive preparation stage typical of new vaccines in the country that included implementation strategies such as obtaining procurement support from Gavi, the Vaccine Alliance. This was leveraged through a new Pneumococcal Advance Market Commitment initiative (which served as a facilitating contextual factor by improving availability of funds targeting the new vaccine), comprehensive cascade training of health workers, and adoption of training manuals. During the preparation phase, HEWs also aided in implementation of community mobilization and advocacy activities, important implementation strategies to create demand for these vaccines [ 21 ]. The strategy of leveraging the HEP as a platform for service delivery allowed Ethiopia to successfully implement and scale PCV-10, supported by these other strategies. In addition to mass vaccination campaigns, a catch-up strategy to vaccinate all children under the age of 1 year was used, and the vaccine was provided through routine, free vaccination services at health facilities, including by HEWs at health posts. This provision of routine immunization services at health posts allowed new vaccines such as PCV-10 and others (such as rotavirus and pentavalent vaccines) to quickly achieve widespread implementation at a national scale and reach rural communities.

PCV was introduced throughout the country simultaneously and with high levels of general acceptability of vaccines in the community, according to interviewees in a 2015 study [ 22 ]. By 2014/15, WHO/UNICEF estimated coverage of all three doses of PCV-10 to be 76% [ 23 ]. Despite rollout in all regions and implementation strategies targeting national reach, geographic coverage of the vaccine was not equitable, with large regional differences [ 3 ]. According to a key informant, these inequities reflected barrier contextual factors in differences in health system strength and governance between the regions, as well as presence of pastoralist populations in regions such as Afar and Somali. These two largely pastoralist regions had the lowest coverage of three doses of PCV reported by the 2016 DHS – 23.6% and 22.9%, respectively [ 3 ]. We found that other contextual factors such as limited vaccine and resource availability were also hindering factors. Conversely, availability of donor resources, government investment in health, ongoing health systems strengthening efforts, and data availability and use were facilitating factors.

Integrated community case management of newborn and childhood illness

Another important EBI introduced during the study period was community-delivered integrated management of childhood illness (IMCI), or integrated community case management (iCCM). Ethiopia previously introduced the facility-based IMCI program to address several leading causes of death in children under 5 in an integrated manner at health centers. However, the government found that access to curative child health services remained lower than expected after scale-up. In response, the MOH began adding selected curative services to the scope of existing HEWs for a more community-based delivery strategy. Ethiopia’s national iCCM program formally launched in early 2010 within the HEP following pilot studies conducted by partners, an implementation strategy that generated local evidence on feasibility, and extensive preparation in collaboration with several partners. As part of iCCM, HEWs provide community-level management of malaria, diarrhea, pneumonia, malnutrition, and, later, essential newborn care and management of common neonatal problems including neonatal sepsis at health posts.

Similar to many other EBIs implemented in Ethiopia, the program utilized a strategy of phased introduction after the pilot testing to improve feasibility of expansion given Ethiopia’s geography, which was an important barrier contextual factor. ICCM was first implemented at a small scale in two of Ethiopia’s nine regions and eventually scaled to the entire country. Four agrarian regions were initially targeted (Amhara, Oromia, Southern Nations, Nationalities, and People's Region (SNNPR), and Tigray) due to their greater population density and strength of HEP implementation compared with other regions. It was later scaled up throughout the country, including in pastoralist regions after development of a contextualized implementation guide for pastoralist areas. By 2014, iCCM was implemented by almost 30,000 HEWs working at 14,500 health posts in eight regions, representing 86% of the country’s districts [ 24 ]. Notably, the program was not rolled out in the country’s urban areas or two city administrations, Addis Ababa and Dire Dawa.

Though implementation of iCCM had a significant impact on geographic access to curative child health services in rural areas of the country, we identified important challenges to the program that likely limited coverage, and ultimately the program’s benefits. One significant barrier contextual factor affecting coverage of services was health-seeking behavior in many areas of the country. Home visits by HEWs as part of iCCM enabled early identification of sick children and improved care-seeking behaviors; care-seeking for children with ARI symptoms and diarrhea increased by 48% and 112% respectively from 2000 to 2016. ICCM introduced evidence-based management of common childhood illnesses at the community level. However, uptake remained limited, with fewer than 50% of children under 5 with acute respiratory infection symptoms or diarrhea taken to any health facility, including a health post manned by HEWs, in 2016 [ 3 ]. The 2014 Ethiopia Service Provision Assessment Plus cross-sectional survey identified additional capacity issues at health posts affecting provision of iCCM [ 25 ]. While availability of curative services for children under 5 was very high at health posts, overall quality of care and adherence to iCCM guidelines was found to be poor. Treatment may have been affected by weak supply chain with low availability of essential medicines required for iCCM services – zinc for treatment of diarrhea and antibiotics were both available in fewer than half of health posts assessed [ 26 ]. In response to stock outs of essential medicines, the program later adapted its supply chain management to use a “pull” supply chain system, or integration into the Integrated Pharmaceutical Logistic System (IPLS), that improved access to drugs at the health post level. According to a key informant, “ this adaptation minimized drug wastage, improved access to drugs at the health post level, and improved equity.”

Despite these challenges, we found evidence of improved care-seeking and coverage and equity of treatment of the conditions by the iCCM program addressed following implementation (Table 3 ). While overall coverage increased across groups, some inequities remained. Though these services were offered for free at the health post and health center level, mitigating some financial barriers to care, children in the highest wealth quintile with diarrhea were more likely to be taken to a health facility (61%) than those in the lowest and second quintiles (40%) (Table 4 ). Oral rehydration solution (ORS) for treatment of diarrhea, a service included in iCCM, doubled over the same period. A notable increase in coverage of ORS in rural areas (Table 4 ) reflected overall improvement in equity. ORS use increased in all regions except Somali, where coverage declined. Despite improvement in most regions, regional differences in coverage persisted, ranging from 23.9% in Oromia to 52.5% in Benishangul-Gumuz in 2016 (Table 4 ) [ 3 ].

Implementation strategies and contextual factors

Ethiopia’s successful introduction and national scale-up of key EBIs to reduce U5M utilized various implementation strategies. One commonly used strategy was leveraging existing programs such as the HEP to implement new EBIs, an approach that improved feasibility, reach, and sustainability of new EBIs given the country’s limited resources and geography. Alongside this approach, Ethiopia used several other implementation strategies, including using data for evidence-based decision-making, leveraging donor and partner resources, introducing EBIs via a phased approach, and national policy and development planning. Implementation was also impacted by cross-cutting contextual factors that served as facilitators and barriers. In Ethiopia, we found the presence of a strong community health system, other health systems strengthening efforts, data availability and use, and availability of donor and implementing partner resources were contextual factors that facilitated U5M reduction efforts. We found that, meanwhile, other contextual factors such as geography, supply chain issues, and low utilization of health services served as barriers. For many EBIs, pastoralist culture was a hindering factor affecting coverage in regions including Afar, where the majority of the population is pastoralist and U5M rates are some of the highest in the country.

We found that most of the EBIs implemented in Ethiopia to reduce U5M between 2000 and 2015 leveraged the HEP as an implementation strategy, using the program as a platform for delivery of new services. Ethiopia also used a number of other strategies to successfully implement and scale up many of the existing and new EBIs, although gaps remained in coverage and in fidelity.

Prior studies have reported on the contribution of the HEP in increasing access and uptake of health services, including maternal and child health [ 13 , 14 , 15 , 16 ]. They have also found that the program helped Ethiopia achieve improvements in areas such as maternal and child health, communicable diseases, and sanitation and hygiene [ 13 ]. The contributions of this paper allow us to better understand some of the mechanisms through which this strategy worked and other implementation strategies that were used alongside the HEP to implement EBIs and contextual factors that affected their success. For example, implementation strategies such as use of data for evidence-based decision-making and leveraging donor and partner resources were beneficial in introducing and scaling up interventions. Implementation of PCV and iCCM were impacted, however, by contextual factors such as geography, supply chain challenges, and care-seeking behavior.

Similar to our findings on iCCM, a study on the assessment of success and challenges of HEP in Ethiopia between 2003 to 2018 found readiness and availability of services including staffing, equipment, and adequate supplies at the health post level where the HEP operates and service utilization were key barriers for the successful implementation various EBIs by HEWs [ 13 ].

Literature on CHW programs in other LMICs is abundant. As in Ethiopia, CHW programs in countries including Brazil, Nepal, and Iran have increased access to health services in rural areas and contributed to improved outcomes in these areas [ 27 , 28 , 29 , 30 ]. CHW programs often share the HEP’s challenges, such as those related to supply chain in Zambia and Pakistan or community care-seeking preferences in India [ 30 , 31 , 32 ]. Literature shows comparative strengths of the HEP that have likely contributed to the program’s success. For example, Accredited Social Health Activists (ASHAs) in India conduct activities like that of HEWs but are considered part-time volunteers. While HEWs are full time and salaried, ASHAs are reported to be dissatisfied with remuneration as they are only paid for a select few interventions in their wide scope [ 32 ]. CHW training in some countries such as Pakistan and Zimbabwe has been reported to be inadequate, though this is a strength in Ethiopia where HEWs receive more than a year of pre-service training [ 31 , 33 ].

Ethiopia provides a valuable example for other low-resource settings in its utilization of a community health program to improve access to both preventive and curative EBIs. Some countries have not yet adopted a similar approach of utilizing CHWs to provide very decentralized and comprehensive services at the community level. Ethiopia’s implementation strategies demonstrate the importance of building upon existing, widespread programs to introduce new EBIs more feasibly at a large scale or to further expand access and reach of existing ones. It has been well documented that the HEP enabled Ethiopia to improve critical health outcomes in areas of maternal and child health, communicable diseases, hygiene and sanitation, knowledge, and increasing health care-seeking behavior [ 13 ].

The experience of Ethiopia also demonstrates the importance of efforts to adapt strategies or introduce new ones to encourage uptake and ensure equity of coverage of services offered at the community level reflecting subnational variability in geography, culture, and other contextual factors. Establishment and expansion of the HEP was a key strategy to improve access to health services at the community level, particularly in Ethiopia’s vast rural and pastoralist areas with historically low access to health facilities. There is also evidence of pro-poor public spending on health in Ethiopia which supported the implementation of EBIs by the HEP [ 13 ]. The Ethiopian government allocated nearly 60% of health expenditures at health centers which supports the work of the HEP [ 34 ].

The challenges we identified in implementation were similar to contextual barriers described elsewhere. A systematic review of the HEP similarly highlighted the role of the HEP in improving maternal and child health services at the health center and community levels [ 13 ]. However, remaining challenges include capacity of health posts related to supplies, variation in performance of HEWs across geography, work overload, and contextual factors related to the wider health system issues [ 13 ]. For facility-based services, HEWs played a role in promoting care-seeking. A similar role was adopted by Health Development Army volunteers in 2011 as an adaptation to the HEP. Since uptake may be related to perceived quality of community-level care, ensuring high quality of care within community health programs should be a priority for countries leveraging them. Ethiopia used a system of supportive supervision during implementation of many EBIs to improve fidelity, a strategy that may be valuable in many settings. However, quality of care remains an area for improvement that became a priority area for the national Health Sector Transformation Plan [ 35 ]. Coverage of many EBIs, even ones such as ORS with expanded community-level access during the study period, still varies greatly by region, residence type, and wealth quintile in Ethiopia. This is an ongoing challenge that other countries are likely to face and should consider during planning and preparation of implementation.

Our study must be interpreted in light of its limitations. This case study is limited by the fact that publicly available data, reports, and publications often do not describe the implementation outcomes of interest at a subnational level, and data often are either reported at a national level or for a specific subnational population. This was a retrospective study that relied on recall for critical qualitative components, with the corresponding risk of recall bias.

As LMICs work towards further reducing child mortality, existing community health programs represent a valuable opportunity for introduction and expansion of new EBIs. Ethiopia utilized a strong, widespread community health program, the Health Extension Program, to implement new EBIs, including the pneumococcal vaccine and integrated community case management. Leveraging the HEP as a platform for service delivery allowed Ethiopia to successfully introduce and scale new and existing EBIs nationally. Further effort is required to reduce the equity gap in coverage of EBIs among pastoralist and rural communities. Ethiopia’s work in utilizing the HEP alongside other implementation strategies to effectively deliver key EBIs in the presence of potentially challenging contextual factors provides important lessons for other countries aiming to reduce under-5 mortality.

Availability of data and materials

The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.

Abbreviations

Acute Respiratory Infection

Community Health Worker

Accredited Social Health Activists

Demographic and Health Survey

Evidence-Based Intervention

Exploration, Preparation, Implementation, Adaptation, and Sustainment

Facility-Based Integrated Management of Childhood Illness

Health Care Worker

Health Extension Program

Health Extension Worker

Human Immunodeficiency Virus

Health Sector Development Programme

Integrated Community Case Management

Integrated Management of Childhood Illness

Key Informant

Low- and Middle-Income Country

Ministry of Health

Oral Rehydration Salts

Pneumococcal Conjugate Vaccine

Primary Health Care

Southern Nations, Nationalities, and Peoples' Region

Strategic Analysis, Research, & Training

Under-5 Mortality

University of Global Health Equity

United Nations Children's Fund

World Health Organization

Bank W. Urban population (% of total population) - Ethiopia. 2018. https://data.worldbank.org/indicator/SP.URB.TOTL.IN.ZS?locations=ET . Accessed 26 Apr 2021.

Bank W. Rural population (% of total population) – Ethiopia. 2018.  https://data.worldbank.org/indicator/SP.RUR.TOTL.ZS?locations=ET . Accessed 26 Apr 2021.

USAID. STATcompiler. https://www.statcompiler.com/en/ . Accessed 24 Mar2021.

Faye CM, Wehrmeister FC, Melesse DY, et al. Large and persistent subnational inequalities in reproductive, maternal, newborn and child health intervention coverage in sub-Saharan Africa. BMJ Glob Health. 2020;5(1):e002232.

Article   PubMed   PubMed Central   Google Scholar  

Teshome SB, Hoebink P. Aid, ownership, and coordination in the health sector in Ethiopia. Dev Stud Res. 2018;5(sup1):S40–55.

Article   Google Scholar  

Annis E, Ratcliffe H. Ethiopia. Primary Health Care Performance Initiative. https://improvingphc.org/ethiopia . Accessed 9 Sept 2022.

Admassu M CN, Hailu L, Jones T, Muther K, Panjabi R, Price M. Community Health Workers in Ethiopia. Exemplars in Global Health. 2020. Retrieved from https://www.exemplars.health/topics/community-health-workers/ethiopia . Accessed 26 Apr 2021.

Damtew ZA CC, Moges AS. The Health Extension Program of Ethiopia: Strengthening the community health system. Harvard Health Policy Review. 2016. Retrieved from: http://www.hhpronline.org/articles/2016/12/17/the-health-extension-program-of-ethiopia . Accessed 26 Apr 2021.

Wang H, Tesfaye R, Ramana GNV, Chekagn CT. Ethiopia Health Extension Program: An Institutionalized Community Approach for Universal Health Coverage. 2016.

Book   Google Scholar  

United States Department of Health and Human Services. Dissemination and Implementation Research in Health (R03). Available from: https://grants.nih.gov/grants/guide/pa-files/par-13-056.html . Accessed 27 Aug 2021.

Alonge O, Rodriguez DC, Brandes N, Geng E, Reveiz L, Peters DH. How is implementation research applied to advance health in low-income and middle-income countries? BMJ Glob Health. 2019;4(2):e001257.

Gimbel S, Mwanza M, Nisingizwe MP, Michel C, Hirschhorn L, Collaborative APP. Improving data quality across 3 sub-Saharan African countries using the Consolidated Framework for Implementation Research (CFIR): results from the African Health Initiative. BMC Health Serv Res. 2017;17(Suppl 3):828.

Assefa Y, Gelaw YA, Hill PS, Taye BW, Van Damme W. Community health extension program of Ethiopia, 2003–2018: successes and challenges toward universal coverage for primary healthcare services. Global Health. 2019;15(1):24.

Admassie A, Abebaw D, Woldemichael AD. Impact evaluation of the Ethiopian Health Services Extension Programme. J Dev Effectiveness. 2009;1(4):430–49.

Yitayal M, Berhane Y, Worku A, Kebede Y. The community-based Health Extension Program significantly improved contraceptive utilization in West Gojjam Zone. Ethiopia J Multidiscip Healthc. 2014;7:201–8.

Article   PubMed   Google Scholar  

Medhanyie A, Spigt M, Kifle Y, et al. The role of health extension workers in improving utilization of maternal health services in rural areas in Ethiopia: a cross sectional study. BMC Health Serv Res. 2012;12(1):352.

Hirschhorn LR, Frisch M, Ntawukuriryayo JT, et al. Development and application of a hybrid implementation research framework to understand success in reducing under-5 mortality in Rwanda. Gates Open Res. 2021;5:72.

Aarons GA, Hurlburt M, Horwitz SM. Advancing a conceptual model of evidence-based practice implementation in public service sectors. Adm Policy Ment Health. 2011;38(1):4–23.

Hsieh HF, Shannon SE. Three approaches to qualitative content analysis. Qual Health Res. 2005;15(9):1277–88.

MOH. Introducing pneumococcal conjugate vaccine in Ethiopia. Training Manual for Health Workers. Ministry of Health, Ethiopia. 2011. https://www.medbox.org/countries/introducing-pneumococcal-conjugatevaccine-in-ethiopia-training-manual-for-health-workers/preview . Accessed 29 May 2021.

GAVI. Proposal for NVS-Pneumo support: Ethiopia. Global Alliance for Vaccines and Immunization. https://www.gavi.org/news/document-library/proposal-nvs-pneumo-support-ethiopia . Accessed 5 Jan 2020.

Molla M, Burchett H, Mounier-Jack S, Belete H, Kitaw Y. New vaccine adoption and decision making in Ethiopia: qualitative study of national decision-making processes for the introduction of PCV 10. Ethiopian J Health Dev. 2017;29(1):17–22.

Google Scholar  

Ethiopia: WHO and UNICEF estimates of immunization coverage. WHO/UNICEF. 2019. https://www.who.int/immunization/monitoring_surveillance/data/eth.pdf . Accessed 3 Mar 2022.

Integrated Community Case Management- iCCM/ Story in Ethiopia. In: Evidence Review Symposium. Accra, Ghana; 2014.  https://www.exemplars.health/-/media/files/egh/resources/underfive-mortality/ethiopia/child-health-taskforce_integrated-community-case-management-in-ethiopia.pdf?la=en . Accessed 5 Jan 2020.

EPHI FMOH, ICF International. Key findings on Ethiopia Service Provision Assessment Plus (ESPA+) Survey 2014. Addis Ababa, Ethiopia: Ethiopian Public Health Institute, Federal Ministry of Health, ICF International; 2014.

Chandani Y, Andersson S, Heaton A, et al. Making products available among community health workers: Evidence for improving community health supply chains from Ethiopia, Malawi, and Rwanda. J Global Health. 2014;4(2):020405.

Rocha R, Soares RR. Evaluating the Impact of Community-Based Health Interventions: Evidence from Brazil’s Family Health Program. Health Econ. 2010. https://doi.org/10.1002/hec.1607 .

Pratap N. Technical consultation on the role of community based providers in improving maternal and neonatal health. Amsterdam, Netherlands: Community Health Workers Meeting; 2012.

Mehryar AH, Aghajanian A, Ahmad-Nia S, Mirzae M, Naghavi M. Primary health care system, narrowing of rural-urban gap in health indicators, and rural poverty reduction: the experience of Iran. XXV General Population Conference of the International Union for the Scientific Study of Population; 2005; Tours, France.

Worku Y, Shelley KD. Clinton Health Access Initiative. Lusaka, Zambia: Community Health Assistant Process Evaluation; 2013.

Global Health Workforce Alliance. Pakistan. 2012. Available at: http://www.who.int/workforcealliance/countries/pak/en/index.html . Accessed 18 Aug 2021.

Bajpai N, Dholakia RH. Improving the Performance of Accredited Social Health Activists in India: Working Paper No. 1. Mumbai, India: Columbia Global Centers, South Asia, Columbia University; 2011.

Zimbabwe MOHCW. The Zimbabwe Health Sector Investment Case (2010–2012): Accelerating Progress towards the Millennium Development Goals. Harare, Zimbabwe. 2010. Available at: http://www.unicef.org/esaro/Health_Investment_Case_Report1.pdf . Accessed 18 Aug 2021.

Hailu A, Gebreyes R, Norheim OF. Equity in public health spending in Ethiopia: a benefit incidence analysis. Health Policy and Planning. 2021;36(Supplement_1):i4–13.

Federal Democratic Republic of Ethiopia. Health Sector Transformation Plan 2015/16 – 2019/2020. Addis Ababa: Ministry of Health; 2015.

Download references

Acknowledgements

We would like to acknowledge and thank the key informants and other stakeholders in Ethiopia who provided essential information, historical perspectives and narratives, and feedback on our findings, ensuring we captured as accurate a reflection as possible on Ethiopia’s journey to reducing under-5 mortality.

About this supplement

This article has been published as part of BMC Pediatrics Volume 23 Supplement 1, 2023: Understanding Success: Multi-country implementation research in U5M reduction. The full contents of the supplement are available online at https://bmcpediatr.biomedcentral.com/articles/supplements/volume-23-supplement-1 .

This work was completed as part of a larger case study series funded by the Bill & Melinda Gates Foundation and Gates Ventures. These funding bodies were not directly involved in the development of this manuscript.

Author information

Laura Drown and Alemayehu Amberbir contributed equally to this work.

Authors and Affiliations

Division of Global Health Equity, Brigham and Women’s Hospital, Boston, MA, USA

Laura Drown

University of Global Health Equity, Kigali, Rwanda

Alemayehu Amberbir, Jovial Thomas Ntawukuriryayo, Amelia VanderZanden & Agnes Binagwaho

MERQ Consultancy PLC, Arbegnoch Street, Addis Ababa, Ethiopia

Alula M. Teklu

Minstry of Health, Addis Ababa, Ethiopia

Meseret Zelalem, Abreham Tariku, Yared Tadesse, Solomon Gebeyehu & Yirdachew Semu

Feinberg School of Medicine, Northwestern University, Chicago, IL, USA

Lisa R. Hirschhorn

You can also search for this author in PubMed   Google Scholar

Contributions

LD, AMT, AB, and LRH made substantial contributions to the design of this work. LD, AA, JTN, AMT, AB, and LRH contributed to the data analysis. LD, AA, AMT, MZ, AT, YT, SG, YS, JTN, AV, AB, and LRH interpreted the results to form the manuscript. LD, AA, AMT, MZ, AT, YT, SG, YS, JTN, AV, AB, and LRH contributed to writing and revision of the manuscript. All authors have approved the final version.

Corresponding author

Correspondence to Alemayehu Amberbir .

Ethics declarations

Ethics approval and consent to participate.

The study was reviewed by the Institutional Review Board in Ethiopia (Approval number PM23/281). All KIs provided informed consent before interviews were conducted. The overall project was also reviewed by the Rwanda National Ethics Committee and Northwestern University and determined to be non-human-subjects research.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Drown, L., Amberbir, A., Teklu, A.M. et al. Reducing the equity gap in under-5 mortality through an innovative community health program in Ethiopia: an implementation research study. BMC Pediatr 23 (Suppl 1), 647 (2024). https://doi.org/10.1186/s12887-023-04388-1

Download citation

Received : 21 April 2022

Accepted : 26 October 2023

Published : 28 February 2024

DOI : https://doi.org/10.1186/s12887-023-04388-1

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

  • Child mortality
  • Implementation research
  • Health extension program

BMC Pediatrics

ISSN: 1471-2431

importance of review related literature in research study

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • View all journals
  • My Account Login
  • Explore content
  • About the journal
  • Publish with us
  • Sign up for alerts
  • Open access
  • Published: 27 February 2024

Understanding inherent influencing factors to digital health adoption in general practices through a mixed-methods analysis

  • Lisa Weik   ORCID: orcid.org/0009-0002-2936-2948 1 ,
  • Leonard Fehring   ORCID: orcid.org/0000-0002-3322-3724 2 , 3 ,
  • Achim Mortsiefer 4 &
  • Sven Meister   ORCID: orcid.org/0000-0003-0522-986X 1 , 5  

npj Digital Medicine volume  7 , Article number:  47 ( 2024 ) Cite this article

Metrics details

  • Health services
  • Public health

Extensive research has shown the potential value of digital health solutions and highlighted the importance of clinicians’ adoption. As general practitioners (GPs) are patients’ first point of contact, understanding influencing factors to their digital health adoption is especially important to derive personalized practical recommendations. Using a mixed-methods approach, this study broadly identifies adoption barriers and potential improvement strategies in general practices, including the impact of GPs’ inherent characteristics – especially their personality – on digital health adoption. Results of our online survey with 216 GPs reveal moderate overall barriers on a 5-point Likert-type scale, with required workflow adjustments (M = 4.13, SD = 0.93), inadequate reimbursement (M = 4.02, SD = 1.02), and high training effort (M = 3.87, SD = 1.01) as substantial barriers. Improvement strategies are considered important overall, with respondents especially wishing for improved interoperability (M = 4.38, SD = 0.81), continued technical support (M = 4.33, SD = 0.91), and improved usability (M = 4.20, SD = 0.88). In our regression model, practice-related characteristics, the expected future digital health usage, GPs’ digital affinity, several personality traits, and digital maturity are significant predictors of the perceived strength of barriers. For the perceived importance of improvement strategies, only demographics and usage-related variables are significant predictors. This study provides strong evidence for the impact of GPs’ inherent characteristics on barriers and improvement strategies. Our findings highlight the need for comprehensive approaches integrating personal and emotional elements to make digitization in practices more engaging, tangible, and applicable.

Introduction

In the contemporary healthcare landscape, digital technologies have emerged as powerful tools, offering the potential to improve health outcomes 1 , 2 , reduce costs 3 , enhance patient care 4 , 5 , and improve the effectiveness and efficiency of healthcare delivery 3 , 6 , 7 . This spread of digital health solutions was further accelerated by the COVID-19 pandemic 8 . Despite the potential benefits of digital health solutions, their adoption and successful integration into healthcare organizations has been slow 9 , 10 and impeded by various barriers 11 , 12 . As the digitalization of healthcare continues to reshape medical practices, understanding and addressing perceived barriers among general practitioners (GPs) is paramount. In this context, extensive research has studied digital health adoption across various medical disciplines, healthcare settings, and technologies, ranging from remote consultations 13 , 14 to mHealth 15 , 16 , electronic medical records 17 , 18 , and remote monitoring 19 , 20 . Today, only a few studies considered a broader perspective on digital health adoption 21 , investigated potential strategies to improve adoption 22 , 23 , and studied potential influencing factors 24 .

Amid the digitalization of healthcare, GPs can choose various digital health solutions for their practice, ranging from video consultations and mobile health apps to digital appointment booking. As GPs are most patients’ primary point of contact 25 in European healthcare systems, they are thus at the center of providing comprehensive and continuous healthcare services 26 . Consequently, GPs’ adoption and effective utilization of digital health solutions significantly impact the integration of these technologies into routine clinical practice 12 and, hence, influence patient care. Moreover, GPs’ adoption of digital health solutions can enhance their job satisfaction and work-life balance 27 .

Therefore, understanding factors influencing the barriers perceived among GPs is vital to fostering the effective and sustainable adoption of digital health solutions in a rapidly evolving landscape. By digital health solutions, in this study, we mean digital tools, technologies, and services designed to improve healthcare, make it more efficient, and personalize it. This includes the use of digital services (e.g., video consultations, digital telephone assistance system, digital appointment booking, digital medical history, digital practice administration) and the use of connected medical devices and artificial intelligence (e.g., telemonitoring, decision support systems).

Through a mixed-methods research approach combining qualitative (i.e., literature review, expert interviews) and quantitative methodologies (i.e., online survey), we aim to identify adoption barriers and potential strategies for improvement in general practice settings more broadly and further evaluate their association with GPs’ inherent characteristics, especially their personality. As the research on influencing factors to digital health adoption in general practices is limited, we close this gap by providing a more nuanced understanding of inherent characteristics and their effect on digital health adoption among GPs. Understanding these inherent influencing factors enables the development of evidence-based, targeted strategies to address resistance and facilitate the successful integration of digital health solutions into clinical practice, whether through communication styles that resonate with different personality types or by providing additional support to individuals less comfortable with technological change. Tailoring interventions to the specific needs and characteristics of GPs enhances the effectiveness of digital health adoption strategies.

Adoption barriers and improvement strategies in general practices (literature review and expert interview results)

Our literature review and expert interviews aimed to identify and synthesize currently postulated adoption barriers and improvement strategies for digital health adoption more broadly and validate their relevance in general practice settings. We initially retrieved 1276 records in the literature search, of which we included 24 articles 13 , 15 , 17 , 18 , 19 , 22 , 23 , 28 , 29 , 30 , 31 , 32 , 33 , 34 , 35 , 36 , 37 , 38 , 39 , 40 , 41 , 42 , 43 , 44 .

The literature review identified technological, social, and organizational adoption barriers. More than 90% of included studies reported organizational adoption barriers 13 , 15 , 17 , 18 , 22 , 23 , 28 , 29 , 30 , 31 , 32 , 33 , 34 , 35 , 36 , 37 , 38 , 39 , 40 , 41 , 42 , 43 , 44 (23/24), with more than half reporting high workload 17 , 22 , 29 , 30 , 34 , 36 , 37 , 38 , 39 , 40 , 41 , 42 , 43 , 44 and a lack of time 13 , 15 , 17 , 18 , 23 , 28 , 29 , 31 , 32 , 33 , 34 , 36 , 40 , 42 (each 14/24; 58%) as predominant barriers. Another 88% of studies identified social adoption barriers 13 , 15 , 17 , 18 , 22 , 23 , 28 , 29 , 30 , 31 , 32 , 35 , 36 , 37 , 38 , 39 , 40 , 41 , 42 , 43 , 44 (21/24). Of these, physicians’ familiarity with digital health solutions 15 , 17 , 18 , 22 , 23 , 28 , 30 , 31 , 32 , 36 , 38 , 39 , 40 , 41 , 42 , 43 , 44 (17/24; 71%) was the most cited social barrier, followed by overall awareness 15 , 18 , 22 , 23 , 29 , 30 , 32 , 35 , 43 , 44 (10/24; 42%) and patient preferences 15 , 18 , 23 , 29 , 30 , 31 , 35 , 38 , 40 , 42 (10/24; 42%).

Our ten expert interviews with GPs confirmed and validated the relevance of all three categories of barriers and five categories of improvement strategy. Overall, the relevance of the three categories of barriers was consistently rated as high. In line with the high estimated relevance, all GPs mentioned technological barriers, especially regarding system reliability (10/10; 100%), usefulness (9/10; 90%), and technical support (9/10; 90%). Additionally, most GPs mentioned the familiarity and ability of practice staff (each 8/10; 80%), patients’ preferences and ability (8/10; 80%), a lack of reimbursement (9/10; 90%), a high workload and lack of time (each 9/10; 90%), and the socio-political context (9/10; 90%) as substantial adoption barriers. On average, GPs reported around 14 different barriers.

Looking into potential strategies to support and improve digital health adoption, we identified strategies in five categories in our literature review: development-related, awareness-related, knowledge-related, implementation-related, and policy-related strategies. Around two-thirds of studies identified strategies concerning the development of digital health solutions as potentially helpful to improve adoption 15 , 17 , 18 , 19 , 28 , 29 , 31 , 33 , 34 , 36 , 37 , 38 , 39 , 42 , 43 , 44 (16/24; 67%). Among these, the most frequently cited development-related strategies were improvements in the usefulness of digital health solutions 17 , 18 , 19 , 28 , 29 , 31 , 33 , 36 , 37 , 38 , 42 , 43 , 44 (13/24; 54%), followed by improvements in their usability 28 , 29 , 31 , 34 , 36 , 39 , 42 , 43 (8/24; 33%). All other categories were present in around half of the included studies, with the call for ongoing training 15 , 17 , 18 , 19 , 34 , 37 , 38 , 40 , 43 , 44 (10/24; 42%) and improved reimbursement 15 , 17 , 18 , 22 , 34 , 38 , 39 , 43 (8/24; 33%) as additionally vital improvement strategies.

Our expert interviews further highlighted that GPs considered development-related strategies particularly relevant: 80% of GPs would like to see improved usability of digital health solutions (8/10). In addition, GPs especially called for improvements in remuneration (8/10; 80%) and a simplification of political guidelines (9/10; 90%). Awareness-related strategies were rated as least relevant (6.2/10.0). Of these, GPs wished for further information on the functionalities and benefits of digital health solutions (each 7/10; 70%). Overall, GPs reported around 11 strategies. In our subsequent online survey, we only included items for barriers or strategies proposed by more than four articles or mentioned by more than one interviewee to ensure theoretical and expert consensus. Figure 1 shows the synthesized results.

figure 1

The figure shows categories and corresponding individual barriers (strategies) as well as their appearance in the literature review and expert interviews. n LR represents the number of studies identified in the literature review proposing the barrier (strategy); n EI shows the number of expert interviews in which the barrier (strategy) was mentioned. Light grey boxes with italic text show barriers (strategies) not included in the subsequent online survey. dhs digital health solutions.

Factors influencing adoption barriers and improvement strategies (online survey results)

To analyze factors that may influence adoption barriers and improvement strategies, our online survey focused on five areas of inherent characteristics: (i) demographics and practice-related characteristics, (ii) digital health usage, (iii) digital affinity, (iv) personality, and (v) digital maturity of the practice.

After data cleaning, quality, and privacy control, we included a broad sample of 216 German GPs with a diverse set of demographics (see Fig. 2 ).

figure 2

The figure shows assessed individual and practice-related characteristics of participating GPs.

Around half of respondents used digital health solutions daily (93/216; 43.1%), while almost a third did not use them at all (20/216; 9.3%) or rather seldomly (47/216; 21.8%). Most respondents further expected to rather or very likely use digital health solutions in the future (161/216; 74.5%).

Further, GPs perceived the work-related digital affinity of their medical assistants to be moderate (55/216; 25.5%) or relatively high (91/216; 42.1%) and had a relatively moderate affinity for technology interaction 45 themselves (M = 2.66, SD = 1.08).

Concerning personality 46 , respondents can be characterized as highly conscientious (M = 4.10, SD = 0.59) and open (M = 3.85, SD = 0.68), moderately extroverted (M = 3.64, SD = 0.80) and agreeable (M = 3.53, SD = 0.75), and mildly neurotic (M = 2.42, SD = 0.71). The digital maturity of their practices was moderate (M = 3.32, SD = 0.64).

Overall, respondents saw around 11 barriers (M = 11.12, SD = 6.01) and rated these as relatively moderate (M = 3.08, SD = 0.68). Among the three categories, organizational barriers were rated highest on average (M = 3.56, SD = 0.71), followed by technological (M = 2.93, SD = 0.76) and social (M = 2.76, SD = 0.79) barriers. For most individual barriers, scores were again moderate, with the highest rating for required workflow adjustments (M = 4.13, SD = 0.93), high costs and inadequate reimbursement (M = 4.02, SD = 1.02), and a high training and familiarization effort (M = 3.87, SD = 1.01) as the top three barriers (see Fig. 3 ).

figure 3

The figure shows items for adoption barriers per category, the respective sample size, descriptive statistics, and between-group comparison. The dot chart shows the mean value per item. Error bars represent +/− 2 standard errors. Cells with red framing show substantial differences between groups. %A Percentage of respondents agreeing to the statements and thus rating the respective barrier as relevant rating of (4) or (5); TB technological barriers; SB social barriers; OB organizational barriers; dhs digital health solutions.

On average, respondents perceived around 16 improvement strategies as important (M = 3.89, SD = 0.61). Policy-related (M = 4.00, SD = 0.81) and development-related strategies (M = 3.98, SD = 0.67) received the highest rating, followed by implementation-related (M = 3.90, SD = 0.78) and knowledge-related strategies (M = 3.85, SD = 0.81). Awareness-related strategies scored lowest but were also perceived as important (M = 3.70, SD = 0.74). Most individual strategies were similarly rated important (see Fig. 4 ): Respondents especially wished for improved interoperability (M = 4.38, SD = 0.81), continued technical support (M = 4.33, SD = 0.91), and improved usability (M = 4.20, SD = 0.88).

figure 4

The figure shows items for improvement strategies per category, the respective sample size, descriptive statistics, and between-group comparisons. The dot chart shows the mean value per item. Error bars represent +/−2 standard errors. Cells with red framing show substantial differences between groups. %A Percentage of respondents agreeing to the statement and thus rating the respective strategy as important rating of (4) or (5); DS development-related strategies; AS awareness-related strategies; KS knowledge-related strategies; IS implementation-related strategies; PS policy-related strategies; dhs digital health solutions.

We conducted separate univariate ANOVAs and post hoc tests, to assess differences in the number and strength of adoption barriers and the number and importance of improvement strategies given the several inherent characteristics considered (see Fig. 5 ).

figure 5

Both parts of the figure show the results for Welch ANOVAs (left) and Hochberg GT2 or Games-Howell post hoc tests for significant Welch ANOVAs in the order of appearance (right). The upper part reports results for the strength of barriers, the lower part reports results for the importance of strategies. Blue brackets represent significant comparisons. As gender is a dichotomous variable, we conducted a two-tailed t -test. The results show the t -statistic (in the column ‘Welch’s F’), its’ df, and P -value. MA digital affinity medical assistants’ digital affinity; ATI affinity for technology interaction; N neuroticism; DM digital maturity.

The strength of barriers differed based on gender, current and future use of digital health solutions, GPs’ level of affinity for technology interaction, the level of extraversion and neuroticism, and the level of digital maturity. Post hoc tests revealed that participants who were female (M = 3.16, SD = 0.64, Cohen’s d  = 0.25), never used digital health solutions (M = 3.42, SD = 0.64, p  = 0.029, Cohen’s d  = 0.73; Hochberg GT2 post hoc test), were very (M = 3.48, SD = 0.77, p  = 0.034, Cohen’s d  = 0.88; Hochberg GT2 post hoc test) or rather unlikely to use digital health solutions in the future (M = 3.61, SD = 0.59, p  < 0.001, Cohen’s d  = 1.12; Hochberg GT2 post hoc test), had a low level of affinity for technology interaction (M = 3.53, SD = 0.71), a low level of extraversion (M = 3.43, SD = 0.64, p  = 0.011, Cohen’s d  = 0.68; Hochberg GT2 post hoc test), a high (M = 3.63, SD = 0.60, p  < 0.001, Cohen’s d  = 1.04; Hochberg GT2 post hoc test) or moderate level of neuroticism (M = 3.21, SD = 0.61, p  = .004, Cohen’s d  = 0.44; Hochberg GT2 post hoc test), and a low (M = 3.43, SD = 0.56, p  = 0.002, Cohen’s d  = 0.90; Games-Howell post hoc test) or moderate level of digital maturity (M = 3.21, SD = 0.56, p  < 0.001, Cohen’s d  = 0.73; Games–Howell post hoc test) reported a higher strength of barriers compared to respondents who were male, used digital health solutions daily, were rather or very likely to use digital health solutions in the future, had a moderate or high level of affinity for technology interaction, a high level of extraversion, a low level of neuroticism, or a high level of digital maturity. Interestingly, male and female participants rated poor compatibility with work processes, a lack of reimbursement, high costs, and a high training effort as the most substantial adoption barriers (see Fig. 3 ).

We found a similar pattern for the number of barriers, except that there was no difference between GPs based on gender ( t (214) = −1.397, p  = 0.082; t -test), yet a significant difference based on the perceived digital affinity of medical assistants (Welch’s F (4, 21.51) = 3.433, p  = .003; Welch ANOVA). GPs who perceived their medical assistants to be somewhat not digitally affine (M = 13.60, SD = 5.13) reported significantly more adoption barriers adoption compared to respondents perceiving their medical assistants to be fully digitally affine (M = 9.42, SD = 5.87, p  = 0.053, Cohen’s d  = 0.77; Hochberg GT2 post hoc test).

Looking at the importance of improvement strategies, we found significant differences between GPs based on gender, professional experience, current usage and expected future digital health usage, the level of neuroticism, and the level of digital maturity (see Fig. 5 ). Post hoc tests revealed that respondents who were female (M = 3.97, SD = 0.55, p  = 0.017, Cohen’s d  = 0.29; t -test), used digital health solutions daily (M = 3.95, SD = 0.62, p  = 0.003, Cohen’s d  = 0.87; Hochberg GT2 post hoc test), monthly (M = 4.00, SD = 0.54, p  = 0.011, Cohen’s d  = 0.97; Hochberg GT2 post hoc test), or seldomly (M = 3.95, SD = 0.59, p  = .007, Cohen’s d  = 0.88; Hochberg GT2 post hoc test), and were very (M = 3.96, SD = 0.59, p  < 0.001, Cohen’s d  = 1.25; Hochberg GT2 post hoc test) or rather likely to use digital health solutions in the future (M = 3.95, SD = 0.45, p  = 0.002, Cohen’s d  = 1.49; Hochberg GT2 post hoc test), reported a higher importance of strategies, compared to respondents who were male, never used digital health solutions, and were very unlikely to use these in the future. Interestingly, female participants viewed continuous technical support, improved interoperability, and improved reimbursement as the most vital improvement strategies. In contrast, for male participants, it was an enhanced interoperability, improved usefulness, and improved usability (see Fig. 4 ). Overall, we found similar results for the number of improvement strategies, except that there was an additional significant difference based on respondents’ level of conscientiousness (Welch’s F (2, 3.34) = 11.988, p  = 0.030; Welch ANOVA).

In the next step, we conducted a linear hierarchical regression analysis to deepen our understanding of the association between adoption barriers, improvement strategies, and GPs’ inherent characteristics.

Looking at adoption barriers (see Table 1 ), demographics, practice-related characteristics, and digital health usage alone explained about 21.8% of the variance in the strength of barriers, reaching statistical significance of the model, F (21, 194) = 2.573, p  < 0.001 (F-test). When including digital affinity variables in model 3, the proportion of explained criterion variance increase by 8.8% to an overall R 2 of around 30.6% ( F (23, 192) = 3.684, p  < 0.001; F-test). Further including personality traits into our model led to an additional increase in R 2 of 10.3%. Finally, also including digital maturity led to an increase in R 2 of 3.6% to an overall R 2 of 44.5% ( F (29, 186) = 5.139, p  < 0.001; F-test). Thus, our model significantly improved at each stage of the hierarchical process. The same was true for the number of adoption barriers, with a final R 2 of 42.6% ( F (29, 186) = 4.762, p  = 0.005; F-test).

In our final regression model, eight variables were significantly associated with the strength of barriers (see Supplementary Table 1 and Supplementary Table 2 for a detailed overview of coefficients). The strength of barriers was significantly associated with the practice location, the practice type, the expected future use of digital health solutions, GPs’ affinity for technology interaction, their extraversion, neuroticism, and openness, and the digital maturity of the practice. Accordingly, practicing in cities with less than 5,000 inhabitants compared to cities with 100,001 to 500,000 inhabitants ( b  = −0.315, SE B  = 0.142, β  = −0.152, p  = 0.028; t -test) or cities with more than 500,000 inhabitants ( b  = −0.301, SE B  = 0.133, β  = −0.164, p  = 0.025; t -test), sharing practices ( b  = 0.498, SE B  = 0.189, β  = 0.155, p  = 0.009; t -test) compared to single practices, a lower expected likelihood of future usage ( b  = −0.151, SE B  = 0.051, β  = −0.281, p  = 0.003; t -test), a lower affinity for technology interaction ( b  = −0.159, SE B  = 0.042, β  = −0.254, p  < 0.001; t -test), lower extraversion ( b  = −0.109, SE B  = 0.055, β  = −0.129, p  = 0.048; t -test), higher neuroticism ( b  = 0.156, SE B  = 0.063, β  = 0.164, p  = 0.014; t -test) and openness ( b  = 0.134, SE B  = 0.062, β  = 0.135, p  = 0.031; t -test), and lower digital maturity ( b  = −0.247, SE B  = 0.071, β  = −0.236, p  < 0.001; t -test) were associated with a higher strength of barriers. We found similar results for the linear hierarchical regression model predicting the number of barriers, except that there was no substantial association with the practice type or extraversion.

Looking at the importance of improvement strategies (see Table 2 ), the model only including demographics and practice-related characteristics explained about 10.2% of the variance but did not reach statistical significance ( F (16, 199) = 1.407, p  = 0.141; F-test). Including digital health usage in our model yielded significant improvement in the proportion of explained criterion variance by 9.8%, leading to a total R 2 of 20.0% ( F (21, 194) = 2.305, p  = 0.002; F-test). Further including digital affinity, personality, or digital maturity as predictors did not significantly improve the model, although the respective regression models were significant (see Table 2 ). Thus, the regression model only including demographics, practice-related characteristics, and digital health usage best fit our data.

In this model (model 2), three variables were significantly associated with the importance of improvement strategies. We found a significant association with respondents’ professional experience, their current usage of digital health solutions, and their expected future usage. Having 1 to 5 years of professional experience compared to 21 to 30 ( b  = −0.524, SE B  = 0.210, β  = −0.381, p  = 0.013; t -test), using digital health solutions seldom ( b  = 0.458, SE B  = 0.165, β  = 0.308, p  = 0.006; t -test) or monthly ( b  = 0.430, SE B  = 0.208, β  = 0.221, p  = 0.040; t -test) compared to never, and a higher expected likelihood of future usage ( b  = 0.105, SE B  = 0.052, β  = 0.216, p  = 0.043; t -test) were associated with a higher importance. Again, results were similar for the linear hierarchical regression model of the number of improvement strategies, except that there was an additional significant association with respondents’ age: Being aged between 46 and 55 ( b  = 3.682, SE B  = 1.720, β  = 0.302, p  = 0.034; t -test) or older than 65 ( b  = 6.218, SE B  = 2.849, β  = .234, p  = 0.030; t -test) was significantly associated with a higher number of strategies.

Despite the high potential value of digital health solutions 1 , 2 , 3 , 6 , 7 , broad adoption and successful integration into healthcare organizations have been challenging 9 , 10 . This study systematically examined the influence of GPs’ personal and practice characteristics on adoption barriers and strategies to improve digital health adoption. In a linear hierarchical regression model, practice-related characteristics, the expected future digital health usage, GPs’ digital affinity, several personality traits, and digital maturity were significant predictors of the perceived strength of barriers. For the perceived importance of improvement strategies, demographics, and digital health usage-related variables were again significant predictors.

In line with previous research 11 , 16 , 22 , respondents saw multiple adoption barriers and rated these as rather moderate in our study. A recent systematic review 11 found that organizational adoption barriers were more prevalent than technological factors. Likewise, our study obtained the highest scores for organizational barriers, followed by technological and social barriers. This highlights the clear importance of organizational factors for digital health adoption that go beyond the pure technical features of the solutions themselves, contrasting with another previous review 16 . However, a comparison of both studies should be interpreted cautiously, given the rapidly evolving digital health landscape and technical improvements in tools and services accelerated by the COVID-19 pandemic 8 .

The main barriers identified in our study include poor compatibility with work processes, insufficient reimbursement and high costs, a required high training and familiarization effort, inadequate and indistinct regulations, guidelines, healthcare policies, and a workload-related lack of time. Most of these barriers are consistent with those identified in previous literature 11 , 16 , 17 , 22 . A recent systematic review 17 identified several influencing factors to adoption, highlighting a lack of interoperability that limits GPs’ ability to integrate digital health solutions flawlessly into existing workflows and exchange information with other healthcare providers as a strong barrier.

It is further not surprising that the high costs for implementing digital health solutions and a lack of reimbursement of corresponding services were proposed as core barriers in our study. Likewise, GPs mentioned financial incentives for digital health adoption as one of the most important improvement strategies. In line with this, a study on economic influencing factors for the acceptance of remote monitoring in Germany reported missing reimbursement arrangements, uncertain economic advantages, and missing business models as core barriers 20 . Interestingly, these findings contradict research highlighting the financial advantages of digital health solutions 3 but might be explained by the relatively low usage of digital health solutions in our sample. As pointed out by previous studies 20 , current users saw substantially greater financial benefits than non-users. Our finding of usage-related differences further seconds this: GPs using digital health solutions daily perceived barriers overall to be lower compared to GPs never using these.

It is promising that only 21% of respondents in our survey were afraid that using digital health solutions would hinder their communication with patients. Previous studies reported a potential disruption during visits due to the use of mHealth 47 . This finding is in line with a more recent systematic review, showing the impact of digital health solutions on patient-professional interaction was more often deemed a facilitator of digital health adoption, as digital health solutions could facilitate the relationship with patients by providing a new means of communication 16 .

Interestingly, GPs perceived insufficient technical infrastructure as a minor adoption barrier in our study. This contrasts with the results of our literature review and previous studies reporting a poor information technology infrastructure as a constant barrier 15 , 31 , 42 . The findings might be attributable to geographic differences in digital health requirements. In Germany, the Act on Secure Digital Communication and Applications in the Healthcare System (‘Gesetz für sichere digitale Kommunikation und Anwendungen im Gesundheitswesen’) 48 has established a legal framework for setting up the secure telematics infrastructure. Since then, various laws have advanced the digitalization of the German healthcare system, based on which, for example, general practices are required to use an electronic patient record (ePA), provide an electronic statement of fitness for work (eAU), and communicate via a uniform standard for the electronic transmission of healthcare-related documents (KIM). These standards have required practices to adopt a sufficient technical infrastructure to provide the services mentioned and providers to ensure the integrability of new digital health solutions.

To address these barriers and support the adoption of digital health solutions, GPs in our study explicitly wished for improvements in the interoperability of digital health solutions, continued technical support from providers, improvements in the usability and usefulness of digital health solutions, as well as financial incentives and simplifications in regulations for data protection. Most of these strategies are consistent with previous research 16 , 29 , 36 . Interestingly, a recent mixed-methods study on mHealth adoption in Germany 22 found additional information to be the most important measure. As we assessed various knowledge-related strategies, our results provide clarity as to which types of content are most critical for GPs: Respondents in our study perceived information about the digital health solutions offered as most helpful, followed by information about potential benefits for themselves and their practices, information about available reimbursement and financing models, and scientific evidence. These findings further underline the call for more research on medical evidence of the benefits of digital health adoption 15 , 22 , 40 , which is subsequently presented in a structured and transparent way and made publicly available via various channels, including medical newspapers, magazines, or conferences.

While extensive research has studied influencing factors to digital health adoption 17 , 21 , 24 , 49 , no study has investigated factors influencing adoption barriers or improvement strategies. According to our findings, the strength of barriers and the importance of strategies differed between GPs based on gender, with female participants perceiving barriers as higher and strategies as more important than their male colleagues. This is in line with studies reporting that being male was associated with using digital health technology 27 , 50 while being female was associated with lower digital health adoption 49 . Other studies found no gender-based differences for digital health adoption 24 or even a higher usage for female participants 51 .

A similarly inconclusive pattern of results can be obtained for age and professional experience. In our study both variables were significant predictors of the importance of strategies. Yet, we did not find a substantial association between age or experience and the strength of barriers. In line with our mixed findings, previous evidence is similarly inconclusive: while some studies found older physicians to be more likely to use digital health technology 27 , others observed the opposite to be true 11 , 24 , 50 , 51 and report that younger general practice staff with lower professional experience are more digitally competent and confident 21 . Given the mixed findings concerning age and gender 17 , there might, in fact, be no difference in digital health adoption based on gender or age at all, the effects be limited to certain digital health solutions only, or covariates substantially influencing the effects found.

Interestingly, practice location and practice type were significant predictors of the strength of adoption barriers, a finding that is consistent with previous studies 17 , 21 . A recent study on digital readiness in general practices found that rurality was associated with lower digital readiness 21 . Similarly, respondents practicing in urban areas perceived barriers to be significantly weaker in our study. This finding might be explained by the more pronounced population aging in rural than in urban areas 52 . As GPs in rural areas thus might have to deal with older populations, they might perceive digital health solutions to not be applicable to their patient populations. This is further underlined by studies proposing patients’ digital literacy as a key adoption barrier 15 , 31 , 35 , which is further consistent with our findings. Future research should therefore investigate measures to overcome this potential digital divide to support GPs in rural areas.

We further observed substantial differences in the number and strength of barriers as well as the number and importance of improvement strategies based on the current and expected future usage of digital health solutions. Countless studies have proposed a lack of experience and familiarity with digital health solutions to be a key barrier to adoption 11 , 16 , 17 , 23 , 36 , 38 , 41 , a finding that is further consistent with our literature review and expert interviews. As the expected future was as a strong predictor in our linear hierarchical regression model, it might be beneficial to provide GPs with information highlighting the importance of digital health solutions, the latest advancements, and outlooks in digital health. This is consistent with the perceived lack of information and need for further information highlighted in a recent study 22 and further seconded by our findings that GPs perceive various information as helpful for supporting digital health adoption.

Looking at digital affinity, GPs’ overall affinity for technology interaction was comparable to the general public in Germany 45 . Our literature review further highlighted that GPs’ familiarity with technologies in general and their digital literacy were perceived as facilitators or a lack thereof as a strong barrier to digital health adoption 11 , 16 , 17 . As the affinity for technology interaction provides a first indication of the actual use of technical systems in everyday settings 45 , this highlights the importance of digital skills for GPs to enable the efficient use and management of these in routine clinical practice 53 .

Concerning personality, our study found mixed results: Extraversion, neuroticism and openness were significant predictors of the perceived strength of barriers, while the perceived importance of improvement strategies only differed based on GPs’ level of neuroticism. Previous studies investigated the relationship between personality and digital health adoption 27 . Yet, no study has investigated the association between GPs’ personality traits and barriers or improvement strategies. Although studies have linked personality to technology adoption in general 54 and to patients’ continued app usage 55 this association seems to be rather weak for clinicians 27 . The specific results found in our study can be explained by looking at the associated personality traits: Extraversion can be characterized as being talkative, energetic, assertive, outgoing, and enthusiastic 56 . As the adoption of digital health solutions is largely driven by GPs themselves, their attitude is strongly linked to digital health adoption 11 , 16 . Thus, it is plausible that GPs with low levels of extraversion perceived barriers to be substantially stronger compared to respondents with higher levels of extraversion, and that this also holds in our linear hierarchical regression model. Consistent across barriers and improvement strategies, we observed differences between GPs based on their level of neuroticism. This, again, is in line with the characteristics of insecurity, anxiousness, and hostility associated with being neurotic 56 , which might as well translate to higher barriers to digital health adoption.

Interestingly, we obtained substantial differences in barriers and improvement strategies based on the practice’s digital maturity level. Digital maturity is a multifaceted construct that describes the digital status of healthcare facilities across various technological and organizational dimensions compared against a theoretical endpoint of maturity within the current digital health landscape 57 . Digitally mature practices have evolved along different dimensions and achieved a higher digital status. Consequently, the association found in our study is plausible, as digitally mature practices perceive lower barriers to digital health adoption. This is also consistent with studies linking prior experience with digital health solutions to higher adoption 11 , 16 . As digital maturity was not a significant predictor of the perceived importance of improvement strategies, this provides critical practical insights. The various improvement strategies identified in our study can be applied to practices regardless of their digital maturity. However, they should be tailored to demographics and practice-related characteristics that proved to be significant predictors and to associated barriers.

Based on the findings discussed above, various practical implications for providers, regulators, policymakers, and other healthcare stakeholders can be derived to support GPs in their digitalization efforts.

As the poor compatibility of digital health solutions with existing practice processes and workflows was a core adoption barrier in our study, there is a strong need for improvements in digital health solutions’ design, that calls providers of digital health solutions into action. Providers should pay paramount attention to ensuring a smooth integration with existing software and tools, high user friendliness, and continued technical support before and after implementation. This is underlined by studies highlighting that the design of digital health solutions is central for promoting patient access to digital health solutions and fostering patient adherence 22 , 58 . To further act on this proposal, regulators and policymakers could consider incentivizing providers of digital health solutions to address these technological barriers.

Based on our findings, ongoing training could further be a potential lever to satisfy the information needs discussed earlier. As GPs work in a profession with a typically high workload that has been additionally strained by the COVID-19 pandemic 59 , digitalization-related topics, including training on digital health solutions, would need to be performed outside of practice hours and thus might be perceived as an add-on to the actual medical work. To overcome this discrepancy, regulators, policymakers, and other healthcare stakeholders could consider providing incentives for training, for instance, by continuing medical education certifications, and by starting to raise awareness of the potential benefits of digitalization early on among medical students. Trainings should be centered around delivering skills concerning technology use and especially focused on digital health solutions, their benefits for practices and patients, and outlooks into future advancements. These efforts should not only be part of separate and dedicated trainings around digital health solutions but rather included into medical trainings. Such an integrated approach could discuss the use of digital health solutions in dedicated medical use cases, for example, regarding diabetes or asthma treatment which included elements of telehealth and remote monitoring. This combination of medical and digital aspects of care could be a fruitful approach to medical training, that is more interesting, applicable and tangible for GPs to experience potential benefits of digital health usage. In addition, recognizing the influence of personality traits on perceived barriers, policymakers could consider personalizing training programs to cater to the diverse characteristics of GPs to further enhance the effectiveness of training initiatives.

With one in two GPs in our study considering a heavy workload and lack of time as barriers to implementing digital health solutions, dedicated support is required. Following the operating model of GPs in the UK, where digitalization in general practices is oftentimes managed and taken care of by dedicated practice managers 60 , healthcare stakeholders could offer programs to non-medical staff to become dedicated digitalization officers focused on managing digitalization-related topics and consequently supporting and relieving GPs 61 .

In addition, regulators and policymakers should reconsider current reimbursement schemes for services related to digital health solutions and provide detailed information on financing models as well as financial benefits resulting from digital health adoption. This would allow GPs to identify suitable financing options for themselves that align with the economic goals of their practice and, in turn, alleviate perceived barriers around reimbursements and costs.

Recognizing GPs’ strong wish for improved interoperability in our study, policymakers could invest in initiatives that promote seamless integration of digital health solutions. Enhancing interoperability can streamline information exchange and improve the overall efficiency of healthcare delivery even beyond individual practices.

As our study has identified several characteristics inherent to GPs as substantial predictors of perceived adoption barriers, future approaches to supporting GPs with the integration of digital health solutions into clinical practice strategies should be tailored to these characteristics. Healthcare stakeholders such as the Association of Statutory Health Insurance Physicians (“Kassenärztliche Vereinigung”) could, for example, consider dedicated campaigns for GPs practicing in rural areas or in shared practices as these were more likely to perceive adoption barriers. Additionally, it could be worthwhile to develop interventions linked to our findings concerning personality. As extraversion and openness were associated with lower perceived barriers, stakeholders could develop interventions that aim to evoke emotions related to these personality traits, for example by choosing a gamified approach or allowing GPs to picture how their practice might look like in the future. Given that neuroticism was associated with higher perceived barriers, interventions and information campaigns should further convey confidence and a feeling of trust in the digital transformation process.

Although our study reveals important findings, it comes with several limitations. First, it must be noted that our research on the association of GPs’ inherent characteristics, barriers and strategies was exploratory and novice. This exploratory approach helps to identify key factors influencing digital health adoption among GPs and thus provides a foundation for future research endeavors. Although our study explores several potentially relevant variables in the context of digital health adoption in general practices, it may not exhaustively explore all relevant variables or factors influencing the phenomena. Similarly, we did not capture all adoption barriers or potential supporting measures identified in previous reviews 11 , 16 . As we based our assessment on a thorough literature review and expert interviews, we are confident that we covered a broad spectrum of relevant adoption barriers and potential improvement strategies in general practice settings. Compared to the single technology focus in previous studies 11 , our more comprehensive focus on digital health solutions allows us to draw a holistic picture while still being economical. Nevertheless, the findings should be replicated in future research to establish the robustness of our results.

Second, as there is a lack of evidence linking digital health adoption to healthcare quality 62 , overcoming the barriers and applying the strategies identified in our study does not necessarily lead to a higher quality of care. Although we assess adoption barriers and improvement strategies, we do not provide guidance on improving healthcare quality but rather highlight measures that healthcare stakeholders can utilize to support digitalization in general practice settings.

Third, as we focused on GPs in Germany, the results obtained might not translate to different geographies and healthcare systems. As our study’s findings align with previous literature across various countries 11 , 16 , 17 and the role of GPs is similar across European countries 26 , the results can be applied to European healthcare systems. However, we cannot claim validity in other countries with different healthcare systems. Future research could take a cross-country approach to validate our findings and uncover differences in barriers and improvement strategies based on geography.

Fourth, the main results of our study stem from an online survey. This might be associated with a bias towards a population with a higher electronic literacy, as is typical for web-based research. While the affinity for technology interaction in our sample was relatively moderate and comparable to a quota sample from the general public in larger German cities 45 , we are confident that a selection bias does not skew the results of our study. Nevertheless, our approach might have resulted in a tendency towards GPs with a higher interest in digital health-related topics or especially interested in voicing their wishes regarding digitalization in general practice settings.

Lastly, we asked GPs to self-assess their affinity for technology interaction and personality in our online survey. While the self-assessment utilized provides an economical and practical approach to capture GPs’ inherent traits, such self-assessment might be influenced by cognitive biases or social desirability, potentially limiting the reliability of the assessment. As the affinity for technology interaction was comparable to the German population 45 and we found a high Cronbach’s Alpha for the affinity for technology interaction in our study, we are confident that the results obtained are accurate. In addition, a recent study analyzing the psychometric properties of the scale used in our study highlights the reliability of the scale among several indices 63 . However, a longitudinal assessment would be needed to provide further confidence in the reliability of the assessment over time. Concerning personality, the scale used for the self-assessment of GPs’ personality traits in our study has already been shown to be relatively stable over time 46 , highlighting the reliability of the assessment.

By investigating various factors influencing adoption barriers and strategies for improvement, this study provides valuable insights into the personal, professional, and practice-related characteristics associated with the adoption of digital health solutions in general practice settings. GPs especially perceived organizational adoption barriers around poor workflow integrability, lack of reimbursement, and a high familiarization effort. We found practice-related characteristics, the expected future digital health usage, digital literacy, personality, and digital maturity being substantial predictors. To address these barriers and support the adoption of digital health solutions, GPs wish for several improvement strategies, especially concerning improved integrability and usability, technical support, and reliable training material. In conclusion, our findings highlight the need for approaches that not only cover pure information on digital health solutions but are integrated with more personal and emotional elements targeting the different inherent characteristics of GPs and, thus, making digitalization in practices more exciting, tangible, and applicable.

Study design

Data gathering and analysis for this study followed a mixed-methods approach using qualitative and quantitative methodologies. To identify relevant adoption barriers and improvement strategies in general practices, we first carried out a literature review in accordance with the PRISMA-ScR guidelines 64 . To validate the findings of our literature review and ensure their applicability to digital health solutions more broadly, we next conducted expert interviews with GPs based on the COREQ checklist 65 . Next, we created an online survey in accordance with the CHERRIES guideline 66 for internet surveys to assess adoption barriers, improvement strategies, and relevant characteristics inherent to GPs, and ultimately answer the following research question:

Which personal and practice-related characteristics, usage-related factors, and personality traits substantially influence adoption barriers and improvement strategies?

All steps of this research project were approved by the Ethics Committee of Witten/Herdecke University (Nr. S-242/2022).

Literature review

For our literature review, we followed the PRISMA-ScR guideline 64 and searched the PubMed and PsycINFO databases accordingly (see Supplementary Table 3 ). To identify potentially relevant citations in both databases, we developed a search string covering three categories of keywords combined with the Boolean OR operator: (1) adoption, (2) digital health, (3) barriers/improvement strategies. For a more targeted view of digital health adoption in general practices, we added MeSH terms concerning GPs to our search (details are provided in Supplementary Table 3 ).

We initially retrieved 1276 citations from the two databases. After removing duplicates, we narrowed our search to more recent articles published between 2018 and 2022 in either English or German. As the COVID-19 pandemic has accelerated the adoption of digital health, this approach aimed at capturing more recent evolvements. For the remaining citations, we carried out abstract screening in accordance with our pre-defined inclusion criteria, resulting in 96 potentially relevant articles being retained. To determine eligibility, we further conducted a full-text review based on our inclusion criteria, leading to 24 papers being included in the review after screening (see Fig. 6 for the detailed screening process). Following our inclusion criteria, we selected the 24 articles as they focused on clinician populations, digital health solutions, and general practice settings and addressed, measured, or reported factors impacting or promoting the adoption or use of digital health solutions.

figure 6

The flowchart shows the sequential screening process during the literature review. ‘Records removed for other reasons’ shows records removed based on language and publication date criteria.

For the abstract and full-text screening, we excluded articles if they (1) were not related to digital health to maintain the focus on digital health and ensure the relevance for our research question; (2) did not address barriers or improvement strategies to digital health adoption or usage (i.e., focused on general attitudes, experiences, or the development or evaluation of digital health) to narrow the scope of our study; (3) were majorly focused on non-clinician populations (i.e., nurses or patients) as GPs hold a pivotal role in the digitalization of general practices and, thus, capturing their perspectives is of utmost importance; (4) were focused on care areas other than general practice as we believe that decision processes for adopting digital health solutions and subsequently implementing these essentially differ between different healthcare settings; and (5) were not original, peer-reviewed, published full-text articles to ensure the reliability and quality of the included literature. All inclusion and exclusion criteria used in this study were aligned in an expert panel before screening. Evidence from the included studies was synthesized by extracting and grouping potentially relevant barriers per a framework proposed in a recent review 11 and improvement strategies based on the underlying adoption process steps. Thus, we utilized recent research findings to cluster individual barriers and strategies more practically and broadly.

Expert interviews

We aimed to validate the results of our literature review in qualitative expert interviews with GPs, ensuring the relevance and completeness of extracted barriers and improvement strategies to digital health adoption more broadly. Our expert interviews followed the COREQ checklist for qualitative research 65 (see Supplementary Table 4 ). We created a semi-structured interview guide based on the findings of our literature review to allow for flexibility yet achieve standardization of the interview procedure (see Supplementary Notes 1 for the full interview questionnaire). The questions were designed to capture GPs’ concerns and wishes for digital health adoption as well as their assessment of the relevance of the proposed categories of barriers and strategies. Next to open-ended questions on perceived adoption barriers and relevant improvement strategies, we asked GPs to assess the relevance of the categories of barriers and strategies uncovered in the literature review, i.e., social, organizational, and technological barriers as well as development-related, awareness-related, knowledge-related, implementation-related, and policy-relates strategies. Four topics were covered in detail: (1) experience with digital health solutions, (2) indicators of digital maturity, (3) barriers to digital health adoption, and (4) relevant strategies to improve digital health adoption. This study specifically focuses on the latter two topics, while the first will be part of a separate analysis.

Participants were primarily recruited through targeted sampling and snowballing of personal contacts and colleagues. Participants received information about the research design and topics to be covered in the interview, including a definition of digital health solutions. Participation was voluntary, and written informed consent was obtained from each participant before the interview. The interviews were then conducted virtually in a one-on-one setting, videotaped, and transcribed verbatim to enable further qualitative analysis.

Data saturation was achieved after ten interviews. Participants were 53 years old on average, had been GPs for 18 years, and worked in cities of about 100,000 inhabitants. Four GPs had a solo practice, five worked in a group practice, and one practiced in a medical care center. Interviews lasted 45 min on average.

Coding and qualitative analysis were performed using MAXQDA 2022 67 . To enable a comparison with the findings of our literature review, we developed our coding scheme for our content analysis 68 deductively based on those findings. To gain further insights, we also inductively inferred themes from the interview material when multiple interviewees brought up the same topic. Based on that, we determined the number of interviewees that indicated the specific barriers or improvement strategy. These results were then compared to the findings of our literature review to develop items for our subsequent online survey. In the survey, we only included items for adoption barriers or improvement strategies proposed by more than four articles or mentioned by more than one interviewee to ensure theoretical and expert consensus.

Online survey

We next conducted a cross-sectional survey investigating perceived adoption barriers, relevant improvement strategies, and inherent characteristics of GPs. The survey adhered to the CHERRIES checklist for internet surveys 66 (see Supplementary Table 5 for the completed CHERRIES; see Supplementary Notes 2 for a translated version of the survey questionnaire). The survey was divided into six sections: (1) demographics, practice-related characteristics, and digital health usage, (2) GPs’ affinity for technology interaction, (3) Big Five personality traits, (4) digital maturity of the practice, (5) perceived adoption barriers, and (6) relevant improvement strategies. This paper focuses on the findings of sections 1, 2, 3, 5, and 6, as we aimed to investigate the influence of GPs’ inherent characteristics on barriers and improvement strategies. The findings concerning digital maturity were covered in detail in another analysis 69 . Participants were informed of the research objectives, target population, length, and IRB approval on an introductory page. Information regarding data storage and security and the researchers involved were provided on the following page. Before continuing with the survey, participants had to provide informed consent. Afterwards, participants were given a definition of relevant concepts covered in the survey, i.e., digital maturity and digital health solutions.

We captured participants’ demographics and practice-related characteristics using single-choice questions. The current and expected future usage of digital health solutions, as well as the perceived digital affinity of medical assistants were assessed using 5-point Likert-type scales.

To measure GPs’ affinity for technology interaction, we used an established 9-item 6-point Likert-type scale 45 that captures a person’s tendency to actively participate in intense technology interaction. Using a 21-item German-language measure 46 , we evaluated GPs’ personality traits. The 5-point Likert-type scale evaluates the Big Five personality traits of extraversion, agreeableness, conscientiousness, neuroticism, and openness. Digital maturity was assessed using 28 items with a 5-point Likert-type scale developed in line with a recent systematic review 57 .

We developed the 26 items to assess adoption barriers based on the synthesis of our literature review and expert interview results. Participants were asked to rate their agreement with the items across technological, social, and organizational barriers on a 5-point Likert-type scale. Similarly, the 23 items for our assessment of improvement strategies were also developed based on previous results and captured using a 5-point Likert-type scale. Improvement strategies assessed covered development-related, awareness-related, knowledge-related, implementation-related, and policy-related strategies.

We pretested the survey questionnaire with 15 physicians working in ambulatory care settings to ensure clarity, comprehensiveness, usability, and technical functionality. Question wording and the introductory page were refined after the pre-test. The survey was then conducted between April and mid-August 2023 and took about 10 to 15 min to complete. Various recruitment channels were used to reach a broad sample of German GPs. These included interview participants, personal contacts, teaching practices, physician networks, research practice networks, and GP mailing lists. Participants were contacted via mail using publicly available mail addresses. As we conducted the survey in an open-access mode, anyone with an access link could participate, and we could not track which invited participants had started or completed the survey. We further did not provide incentives for participation.

We thoroughly cleaned the data obtained before performing statistical analyses (see Fig. 7 ). Following standard practice 70 , our data cleaning included removing responses without informed consent, incomplete responses, and duplicate responses. In the next step, we also removed responses that took very little time to complete 71 and those that displayed careless answer behavior over several survey pages 71 . We further eliminated any responses that did not adhere to our anonymity criterion to comply with data privacy. In total, 216 responses from the 373 people who initially clicked on the survey link are included in our analysis.

figure 7

The figure shows the sequential data cleaning approach, including the number of questionnaires excluded during each process step. As part of our data quality control procedures, we excluded respondents that showed straight-lining across more than two survey pages and thus in more than one item battery, i.e., that chose the very same answer option for all items in more than one item battery, as this might indicate careless responding as opposed to straight-lining due to respondents‘ actual views.

For our statistical analyses, we first computed the mean value for respondents’ affinity for technology interaction, their personality traits, their digital maturity, the three categories of adoption barriers, and the five categories of improvement strategies. To allow for an analysis of the influence of GPs’ inherent characteristics on barriers and improvement strategies, we further computed two overall outcome measures for both variables: One outcome measure represents the number of barriers (strategies) and was calculated as the sum of barriers (strategies) that received a score of 4 or higher on our 5-point Likert-type scale and were thus perceived as such. The second outcome measure was calculated as the average across barriers (strategies) and represents the perceived strength of barriers (the perceived importance of strategies). SPSS version 29.0 for Macintosh 72 was used for all statistical analyses.

We assessed the internal consistencies of the scale used using Cronbach’s Alpha 73 (see Table 3 ). Most internal consistencies can be considered as acceptable or good and are in line with previous research 45 , 46 . However, the internal consistency for conscientiousness was lower in our sample compared to the original study 46 , which might be due to the overall high conscientiousness and low variability of the score in our sample.

Given the several inherent variables pertinent to our study, we conducted independent univariate ANOVAs with 2-tailed significance ( p  < 0.05; Welch ANOVA) to compare differences in barriers and strategies. Welch’s F 74 was used as a robust measure for all ANOVAs because some of our variables did not follow a normal distribution, as demonstrated by Q-Q-Plots, Shapiro–Wilk’s test, and were heteroscedastic in some cases as indicated by Levene’s test. Were ANOVAs showed a significant omnibus difference ( p  < 0.05; Welch ANOVA), we looked at Hochberg’s GT2 (homogeneity of variance met) or Games-Howell (homogeneity of variance not met) as post hoc procedures 75 . In addition, we utilized Cohen’s d as a measure of effect size 76 . As we aimed to assess differences in barriers and strategies based on digital affinity and personality, we grouped participants into three categories (low, moderate, high) for each characteristic based on theoretical thirds of the underlying Likert-type scales. As this categorization does not cover the whole spectrum of the continuous underlying variable, it was only used in our ANOVAs as an initial indicator for differences in these variables between GPs. These differences were then analyzed more granularly in our regression model utilizing the continuous variables without categorization.

We conducted a linear hierarchical regression analysis to deepen our understanding of the association between barriers and strategies and the independent variables, further accounting for the continuity associated with personality and digital affinity-related variables. We chose a hierarchical approach for entering variables into our model to determine the influence of demographic and practice-related variables on barriers and strategies and to separate this from the influence of digital health usage, digital affinity, and personality. Potential multicollinearity of predictors was assessed following practical recommendations using VIF and tolerance values 75 . As all VIF values were below ten and tolerance values greater than 0.1, multicollinearity does not seem to flaw our analysis. In our sequential approach, the first stage incorporated demographics and practice-related characteristics, including gender, age, practice location size, professional experience, and practice type. The second stage introduced variables related to digital health usage – current and expected future usage. The third stage added digital affinity-related variables, encompassing the perceived digital affinity of medical assistants and GPs affinity for technology interaction. The fourth stage introduced the Big Five personality traits of extraversion, agreeableness, conscientiousness, neuroticism, and openness. In the final model, the digital maturity of the practice was included. This sequence followed prior research and theoretical reasoning, with variables analyzed in past research entering earlier in the model. In addition, the distinct blocks analyzed covered different categories of inherent variables, namely demographics, practice-related characteristics, digital health usage, digital affinity, and personality. As our analysis of the relationship between perceived barriers and strategies and inherent characteristics was novice, our analysis focused on main effects of individual predictor variables and explicitly refrained from analyzing and interpreting interaction effects.

Data availability

The data supporting this study’s findings are available from the corresponding author upon reasonable request.

Amarasingham, R., Plantinga, L., Diener-West, M., Gaskin, D. J. & Powe, N. R. Clinical information technologies and inpatient outcomes. Arch. Intern. Med. 169 , 108 (2009).

Article   PubMed   Google Scholar  

Martin, G. et al. Evaluating the impact of organisational digital maturity on clinical outcomes in secondary care in England. NPJ Digit. Med. 2 , 41 (2019).

Article   PubMed   PubMed Central   Google Scholar  

Chaudhry, B. et al. Systematic review: impact of health information technology on quality, efficiency, and costs of medical care. Ann. Intern. Med. 144 , 742 (2006).

Buntin, M. B., Burke, M. F., Hoaglin, M. C. & Blumenthal, D. The benefits of health information technology: a review of the recent literature shows predominantly positive results. Health Aff. 30 , 464–471 (2011).

Article   Google Scholar  

Campanella, P. et al. The impact of electronic health records on healthcare quality: a systematic review and meta-analysis. European J. Public Health 26 , 60–64 (2016).

Lingg, M. & Lütschg, V. Health system stakeholders’ perspective on the role of mobile health and its adoption in the swiss health system: qualitative study. JMIR Mhealth Uhealth 8 , e17315 (2020).

Poissant, L., Pereira, J., Tamblyn, R. & Kawasumi, Y. The impact of electronic health records on time efficiency of physicians and nurses: a systematic review. J. Am. Med. Inform. Assoc. 12 , 505–516 (2005).

Golinelli, D. et al. Adoption of digital technologies in health care during the COVID-19 pandemic: systematic review of early scientific literature. J. Med. Internet Res. 22 , e22280 (2020).

Choi, W. S., Park, J., Choi, J. Y. B. & Yang, J.-S. Stakeholders’ resistance to telemedicine with focus on physicians: utilizing the Delphi technique. J Telemed Telecare 25 , 378–385 (2019).

Greenhalgh, T. et al. Beyond adoption: a new framework for theorizing and evaluating nonadoption, abandonment, and challenges to the scale-up, spread, and sustainability of health and care technologies. J. Med. Internet Res. 19 , e367 (2017).

Jacob, C., Sanchez-Vazquez, A. & Ivory, C. Social, organizational, and technological factors impacting clinicians’ adoption of mobile health tools: systematic literature review. JMIR Mhealth Uhealth 8 , e15935 (2020).

Gagnon, M. P. et al. Systematic review of factors influencing the adoption of information and communication technologies by healthcare professionals. J. Med. Syst. 36 , 241–277 (2012).

Jetty, A., Moore, M. A., Coffman, M., Petterson, S. & Bazemore, A. Rural family physicians are twice as likely to use telehealth as urban family physicians. Telemed. e-Health 24 , 268–276 (2018).

Wanderås, M. R., Abildsnes, E., Thygesen, E. & Martinez, S. G. Video consultation in general practice: a scoping review on use, experiences, and clinical decisions. BMC Health Serv. Res. 23 , 316 (2023).

Byambasuren, O., Beller, E. & Glasziou, P. Current knowledge and adoption of mobile health apps among Australian general practitioners: survey study. JMIR Mhealth Uhealth 7 , e13199 (2019).

Gagnon, M. P., Ngangue, P., Payne-Gagnon, J. & Desmartis, M. M-Health adoption by healthcare professionals: a systematic review. J. Am. Med. Inform. Assoc. 23 , 212–220 (2016).

O’Donnell, A., Kaner, E., Shaw, C. & Haighton, C. Primary care physicians’ attitudes to the adoption of electronic medical records: a systematic review and evidence synthesis using the clinical adoption framework. BMC Med. Inform. Decis. Mak. 18 , 101 (2018).

Rahal, R. M., Mercer, J., Kuziemsky, C. & Yaya, S. Factors affecting the mature use of electronic medical records by primary care physicians: a systematic review. BMC Med. Inform. Decis. Mak. 21 , 67 (2021).

Iversen, T. & Ma, C. A. Technology adoption by primary care physicians. Health Econ 31 , 443–465 (2022).

Leppert, F. et al. Economic aspects as influencing factors for acceptance of remote monitoring by healthcare professionals in Germany. J. Int. Soc. Telemed. eHealth. 3 , e12 (2015).

Google Scholar  

Hammerton, M., Benson, T. & Sibley, A. Readiness for five digital technologies in general practice: perceptions of staff in one part of southern England. BMJ Open Qual 11 , e001865 (2022).

Dahlhausen, F. et al. Physicians’ attitudes toward prescribable mhealth apps and implications for adoption in Germany: mixed methods study. JMIR Mhealth Uhealth 9 , e33012 (2021).

Byambasuren, O., Beller, E., Hoffmann, T. & Glasziou, P. Barriers to and facilitators of the prescription of mHealth apps in Australian general practice: qualitative study. JMIR Mhealth Uhealth 8 , e17447 (2020).

Scott, A., Bai, T. & Zhang, Y. Association between telehealth use and general practitioner characteristics during COVID-19: findings from a nationally representative survey of Australian doctors. BMJ Open 11 , e046857 (2021).

EURACT & WONCA Europe. The European Definition of General Practice / Family Medicine - Short Version. https://www.woncaeurope.org/file/61a77842-76c2-45dd-a435-e0a8b875f30a/Definition%20EURACTshort%20version%20revised%202011.pdf (2011).

Kringos, D. S., Boerma, W., van der Zee, J. & Groenewegen, P. Europe’s strong primary care systems are linked to better population health but also to higher health spending. Health Aff. 32 , 686–694 (2013).

Zaresani, A. & Scott, A. Does digital health technology improve physicians’ job satisfaction and work-life balance? A cross-sectional national survey and regression analysis using an instrumental variable. BMJ Open 10 , e041690 (2020).

Krog, M. D. et al. Barriers and facilitators to using a web-based tool for diagnosis and monitoring of patients with depression: a qualitative study among Danish general practitioners. BMC Health Serv Res 18 , 503 (2018).

Poppe, L. et al. Process evaluation of an eHealth intervention implemented into general practice: general practitioners’ and patients’ views. Int. J. Environ. Res. Public Health 15 , 1475 (2018).

Breedvelt, J. J. et al. GPs’ attitudes towards digital technologies for depression: an online survey in primary care. Br. J. General Pract. 69 , e164–e170 (2019).

Lin, D., Papi, E. & McGregor, A. H. Exploring the clinical context of adopting an instrumented insole: a qualitative study of clinicians’ preferences in England. BMJ Open 9 , e023656 (2019).

Buhtz, C. et al. Receptiveness of GPs in the South Of Saxony-Anhalt, Germany to obtaining training on technical assistance systems for caregiving: a cross-sectional study. Clin. Interv. Aging 14 , 1649–1656 (2019).

Lim, H. M. et al. mHealth adoption among primary care physicians in Malaysia and its associated factors: a cross-sectional study. Fam Pract. 38 , 210–217 (2021).

Girdhari, R. et al. Electronic communication between family physicians and patients. Can. Family Phys. 67 , 39–46 (2021).

Muehlensiepen, F. et al. Acceptance of telerheumatology by rheumatologists and general practitioners in Germany: nationwide cross-sectional survey study. J. Med. Internet Res. 23 , e23742 (2021).

Jakobsen, P. R. et al. Identification of important factors affecting use of digital individualised coaching and treatment of Type 2 diabetes in general practice: a qualitative feasibility study. Int. J. Environ. Res. Public Health 18 , 3924 (2021).

Volpato, L., del Río Carral, M., Senn, N. & Santiago Delefosse, M. General practitioners’ perceptions of the use of wearable electronic health monitoring devices: qualitative analysis of risks and benefits. JMIR Mhealth Uhealth 9 , e23896 (2021).

Della Vecchia, C. et al. Willingness of French general practitioners to prescribe mHealth apps and devices: quantitative study. JMIR Mhealth Uhealth 10 , e28372 (2022).

Meurs, M., Keuper, J., Sankatsing, V., Batenburg, R. & van Tuyl, L. “Get used to the fact that some of the care is really going to take place in a different way”: general practitioners’ experiences with E-Health during the COVID-19 pandemic. Int. J. Environ. Res. Public Health 19 , 5120 (2022).

Article   CAS   PubMed   PubMed Central   Google Scholar  

Löbner, M. et al. What comes after the trial? An observational study of the real-world uptake of an E-mental health intervention by general practitioners to reduce depressive symptoms in their patients. Int. J. Environ. Res. Public Health 19 , 6203 (2022).

Fischer, S. et al. Einschätzung deutscher Hausärztinnen und Hausärzte zur integrierten Versorgung mittels Kommunikationstechnologien. MMW Fortschr Med 164 , 16–22 (2022).

Poon, Z. & Tan, N. C. A qualitative research study of primary care physicians’ views of telehealth in delivering postnatal care to women. BMC Primary Care 23 , 206 (2022).

Wangler, J. & Jansky, M. Welche Potenziale und Mehrwerte bieten DiGA für die hausärztliche Versorgung? – Ergebnisse einer Befragung von Hausärzt*innen in Deutschland. Bundesgesundheitsblatt Gesundheitsforschung Gesundheitsschutz 65 , 1334–1343 (2022).

Job, J., Nicholson, C., Calleja, Z., Jackson, C. & Donald, M. Implementing a general practitioner-to-general physician eConsult service (eConsultant) in Australia. BMC Health Serv. Res. 22 , 1278 (2022).

Franke, T., Attig, C. & Wessel, D. A personal resource for technology interaction: development and validation of the affinity for technology interaction (ATI) scale. Int. J. Hum. Comput. Interact 35 , 456–467 (2019).

Rammstedt, B. & John, O. P. Kurzversion des Big Five Inventory (BFI-K): Entwicklung und Validierung eines ökonomischen Inventars zur Erfassung der fünf Faktoren der Persönlichkeit. Diagnostica 51 , 195–206 (2005).

Sclafani, J., Tirrell, T. F. & Franko, O. I. Mobile tablet use among academic physicians and trainees. J. Med. Syst. 37 , 9903 (2013).

Bundesanzeiger Verlag. Gesetz Für Sichere Digitale Kommunikation Und Anwendungen Im Gesundheitswesen. Bundesgesetzblatt Jahrgang 2015 Teil I Nr. 54 ( https://www.bgbl.de/xaver/bgbl/text.xav?SID=&tf=xaver.component.Text_0&tocf=&qmf=&hlf=xaver.component.Hitlist_0&bk=bgbl&start=%2F%2F*%5B%40node_id%3D%27944185%27%5D&skin=pdf&tlevel=-2&nohist=1&sinst=3A147306 2015).

Poba-Nzaou, P., Uwizeyemungu, S. & Liu, X. Adoption and performance of complementary clinical information technologies: analysis of a survey of general practitioners. J. Med. Internet Res. 22 , e16300 (2020).

Djalali, S., Ursprung, N., Rosemann, T., Senn, O. & Tandjung, R. Undirected health IT implementation in ambulatory care favors paper-based workarounds and limits health data exchange. Int. J. Med. Inform. 84 , 920–932 (2015).

Holanda, A. A., do Carmo e Sá, H. L., Vieira, A. P. G. F. & Catrib, A. M. F. Use and satisfaction with electronic health record by primary care physicians in a health district in Brazil. J. Med. Syst. 36 , 3141–3149 (2012).

Goujon, A., Jacobs-Crisioni, C., Natale, F. & Lavalle, C. The Demographic Landscape of EU Territories - Challenges and Opportunities in Diversely Ageing Regions . https://doi.org/10.2760/658945 (2021).

Slevin, P. et al. Exploring the barriers and facilitators for the use of digital health technologies for the management of COPD: a qualitative study of clinician perceptions. QJM: Int. J. Med. https://doi.org/10.1093/qjmed/hcz241 (2019).

Devaraj, S., Easley, R. F. & Crant, J. M. How does personality matter? Relating the five-factor model to technology acceptance and use. Inform. Syst. Res. 19 , 93–105 (2008).

Su, J., Dugas, M., Guo, X. & Gao, G. Influence of personality on mHealth use in patients with diabetes: prospective pilot study. JMIR Mhealth Uhealth 8 , e17709 (2020).

McCrae, R. R. & Costa, P. T. Validation of the five-factor model of personality across instruments and observers. J. Pers Soc. Psychol. 52 , 81–90 (1987).

Article   CAS   PubMed   Google Scholar  

Duncan, R., Eden, R., Woods, L., Wong, I. & Sullivan, C. Synthesizing dimensions of digital maturity in hospitals: systematic review. J. Med. Internet Res. 24 , e32994 (2022).

Kelders, S. M., Kok, R. N., Ossebaard, H. C. & Van Gemert-Pijnen, J. E. Persuasive system design does matter: a systematic review of adherence to web-based interventions. J. Med. Internet Res. 14 , e152 (2012).

Schrimpf, A., Bleckwenn, M. & Braesigk, A. COVID-19 Continues to Burden General Practitioners: Impact on Workload, Provision of Care, and Intention to Leave. Healthcare 11 , 320 (2023).

Hanna, L., May, C. & Fairhurst, K. Non-face-to-face consultations and communications in primary care: the role and perspective of general practice managers in Scotland. J. Innov. Health Inform. 19 , 17–24 (2011).

KVWL. Digi-Managerin: Neue Fortbildung für nicht-ärztliches Praxispersonal. https://www.kvwl.de/themen-a-z/digi-managerin (2023).

Eden, R., Burton-Jones, A., Scott, I., Staib, A. & Sullivan, C. Effects of eHealth on hospital practice: synthesis of the current literature. Aust. Health Rev. 42 , 568–578 (2018).

Lezhnina, O. & Kismihók, G. A multi-method psychometric assessment of the affinity for technology interaction (ATI) scale. Comp. Hum. Behav. Rep. 1 , 100004 (2020).

Tricco, A. C. et al. PRISMA extension for scoping reviews (PRISMA-ScR): checklist and explanation. Ann. Intern. Med. 169 , 467–473 (2018).

Tong, A., Sainsbury, P. & Craig, J. Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int. J. Quality Health Care 19 , 349–357 (2007).

Eysenbach, G. Improving the quality of web surveys: the checklist for reporting results of internet E-Surveys (CHERRIES). J. Med. Internet Res. 6 , e34 (2004).

VERBI Software. MAXQDA 2022 . (2021).

Kuckartz, U. & Rädiker, S. Qualitative Inhaltsanalyse. Methoden, Praxis, Computerunterstützung . (Beltz Juventa, Weinheim, Basel, 2022).

Weik, L., Fehring, L., Mortsiefer, A. & Meister, S. Big 5 personality traits and individual- and practice-related characteristics as influencing factors of digital maturity in general practices: quantitative web-based survey study. J. Med. Internet Res. 26 , e52085 (2024).

Leiner, D. J. Too fast, too straight, too weird: non-reactive indicators for meaningless data in internet surveys. Surv. Res. Methods 13 , 229–248 (2019).

ADS   Google Scholar  

Bais, F., Schouten, B. & Toepoel, V. Investigating response patterns across surveys: do respondents show consistency in undesirable answer behaviour over multiple surveys? Bull. Sociol. Methodol. 147–148 , 150–168 (2020).

IBM Corp. IBM SPSS Statistics for Macintosh, Version 29.0. (2022).

Tavakol, M. & Dennick, R. Making sense of Cronbach’s alpha. Int. J. Med. Educ. 2 , 53–55 (2011).

Welch, B. L. On the comparison of several mean values: an alternative approach. Biometrika 38 , 330 (1951).

Article   MathSciNet   Google Scholar  

Field, A. Discovering Statistics Using IBM SPSS Statistics . (Sage Publications, London, 2018).

Cohen, J. Statistical Power Analysis for the Behavioral Sciences . (Routledge, New York, 2013).

Download references

Open Access funding enabled and organized by Projekt DEAL.

Author information

Authors and affiliations.

Health Care Informatics, Faculty of Health, School of Medicine, Witten/Herdecke University, Witten, Germany

Lisa Weik & Sven Meister

Helios University Hospital Wuppertal, Department of Gastroenterology, Witten/Herdecke University, Wuppertal, Germany

Leonard Fehring

Faculty of Health, School of Medicine, Witten/Herdecke University, Witten, Germany

General Practice II and Patient-Centredness in Primary Care, Institute of General Practice and Primary Care, Faculty of Health, School of Medicine, Witten/Herdecke University, Witten, Germany

Achim Mortsiefer

Department Healthcare, Fraunhofer Institute for Software and Systems Engineering ISST, Dortmund, Germany

Sven Meister

You can also search for this author in PubMed   Google Scholar

Contributions

L.W., L.F., and S.M. developed the overarching research question and study design. L.W. performed the literature review, conducted, transcribed, and qualitatively analyzed the expert interviews, and performed statistical analysis of the data obtained in the online survey. S.M., L.F., and A.M. supported the recruitment of participants for the interviews and the online survey. LW drafted the manuscript. S.M., L.F., and A.M. provided critical review. S.M. coordinated the project. All authors reviewed the final manuscript.

Corresponding author

Correspondence to Sven Meister .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary material, rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Weik, L., Fehring, L., Mortsiefer, A. et al. Understanding inherent influencing factors to digital health adoption in general practices through a mixed-methods analysis. npj Digit. Med. 7 , 47 (2024). https://doi.org/10.1038/s41746-024-01049-0

Download citation

Received : 18 October 2023

Accepted : 16 February 2024

Published : 27 February 2024

DOI : https://doi.org/10.1038/s41746-024-01049-0

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Quick links

  • Explore articles by subject
  • Guide to authors
  • Editorial policies

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

importance of review related literature in research study

U.S. flag

An official website of the United States government

The .gov means it's official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you're on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

  • Publications
  • Account settings
  • Browse Titles

NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

Lau F, Kuziemsky C, editors. Handbook of eHealth Evaluation: An Evidence-based Approach [Internet]. Victoria (BC): University of Victoria; 2017 Feb 27.

Cover of Handbook of eHealth Evaluation: An Evidence-based Approach

Handbook of eHealth Evaluation: An Evidence-based Approach [Internet].

Chapter 9 methods for literature reviews.

Guy Paré and Spyros Kitsiou .

9.1. Introduction

Literature reviews play a critical role in scholarship because science remains, first and foremost, a cumulative endeavour ( vom Brocke et al., 2009 ). As in any academic discipline, rigorous knowledge syntheses are becoming indispensable in keeping up with an exponentially growing eHealth literature, assisting practitioners, academics, and graduate students in finding, evaluating, and synthesizing the contents of many empirical and conceptual papers. Among other methods, literature reviews are essential for: (a) identifying what has been written on a subject or topic; (b) determining the extent to which a specific research area reveals any interpretable trends or patterns; (c) aggregating empirical findings related to a narrow research question to support evidence-based practice; (d) generating new frameworks and theories; and (e) identifying topics or questions requiring more investigation ( Paré, Trudel, Jaana, & Kitsiou, 2015 ).

Literature reviews can take two major forms. The most prevalent one is the “literature review” or “background” section within a journal paper or a chapter in a graduate thesis. This section synthesizes the extant literature and usually identifies the gaps in knowledge that the empirical study addresses ( Sylvester, Tate, & Johnstone, 2013 ). It may also provide a theoretical foundation for the proposed study, substantiate the presence of the research problem, justify the research as one that contributes something new to the cumulated knowledge, or validate the methods and approaches for the proposed study ( Hart, 1998 ; Levy & Ellis, 2006 ).

The second form of literature review, which is the focus of this chapter, constitutes an original and valuable work of research in and of itself ( Paré et al., 2015 ). Rather than providing a base for a researcher’s own work, it creates a solid starting point for all members of the community interested in a particular area or topic ( Mulrow, 1987 ). The so-called “review article” is a journal-length paper which has an overarching purpose to synthesize the literature in a field, without collecting or analyzing any primary data ( Green, Johnson, & Adams, 2006 ).

When appropriately conducted, review articles represent powerful information sources for practitioners looking for state-of-the art evidence to guide their decision-making and work practices ( Paré et al., 2015 ). Further, high-quality reviews become frequently cited pieces of work which researchers seek out as a first clear outline of the literature when undertaking empirical studies ( Cooper, 1988 ; Rowe, 2014 ). Scholars who track and gauge the impact of articles have found that review papers are cited and downloaded more often than any other type of published article ( Cronin, Ryan, & Coughlan, 2008 ; Montori, Wilczynski, Morgan, Haynes, & Hedges, 2003 ; Patsopoulos, Analatos, & Ioannidis, 2005 ). The reason for their popularity may be the fact that reading the review enables one to have an overview, if not a detailed knowledge of the area in question, as well as references to the most useful primary sources ( Cronin et al., 2008 ). Although they are not easy to conduct, the commitment to complete a review article provides a tremendous service to one’s academic community ( Paré et al., 2015 ; Petticrew & Roberts, 2006 ). Most, if not all, peer-reviewed journals in the fields of medical informatics publish review articles of some type.

The main objectives of this chapter are fourfold: (a) to provide an overview of the major steps and activities involved in conducting a stand-alone literature review; (b) to describe and contrast the different types of review articles that can contribute to the eHealth knowledge base; (c) to illustrate each review type with one or two examples from the eHealth literature; and (d) to provide a series of recommendations for prospective authors of review articles in this domain.

9.2. Overview of the Literature Review Process and Steps

As explained in Templier and Paré (2015) , there are six generic steps involved in conducting a review article:

  • formulating the research question(s) and objective(s),
  • searching the extant literature,
  • screening for inclusion,
  • assessing the quality of primary studies,
  • extracting data, and
  • analyzing data.

Although these steps are presented here in sequential order, one must keep in mind that the review process can be iterative and that many activities can be initiated during the planning stage and later refined during subsequent phases ( Finfgeld-Connett & Johnson, 2013 ; Kitchenham & Charters, 2007 ).

Formulating the research question(s) and objective(s): As a first step, members of the review team must appropriately justify the need for the review itself ( Petticrew & Roberts, 2006 ), identify the review’s main objective(s) ( Okoli & Schabram, 2010 ), and define the concepts or variables at the heart of their synthesis ( Cooper & Hedges, 2009 ; Webster & Watson, 2002 ). Importantly, they also need to articulate the research question(s) they propose to investigate ( Kitchenham & Charters, 2007 ). In this regard, we concur with Jesson, Matheson, and Lacey (2011) that clearly articulated research questions are key ingredients that guide the entire review methodology; they underscore the type of information that is needed, inform the search for and selection of relevant literature, and guide or orient the subsequent analysis. Searching the extant literature: The next step consists of searching the literature and making decisions about the suitability of material to be considered in the review ( Cooper, 1988 ). There exist three main coverage strategies. First, exhaustive coverage means an effort is made to be as comprehensive as possible in order to ensure that all relevant studies, published and unpublished, are included in the review and, thus, conclusions are based on this all-inclusive knowledge base. The second type of coverage consists of presenting materials that are representative of most other works in a given field or area. Often authors who adopt this strategy will search for relevant articles in a small number of top-tier journals in a field ( Paré et al., 2015 ). In the third strategy, the review team concentrates on prior works that have been central or pivotal to a particular topic. This may include empirical studies or conceptual papers that initiated a line of investigation, changed how problems or questions were framed, introduced new methods or concepts, or engendered important debate ( Cooper, 1988 ). Screening for inclusion: The following step consists of evaluating the applicability of the material identified in the preceding step ( Levy & Ellis, 2006 ; vom Brocke et al., 2009 ). Once a group of potential studies has been identified, members of the review team must screen them to determine their relevance ( Petticrew & Roberts, 2006 ). A set of predetermined rules provides a basis for including or excluding certain studies. This exercise requires a significant investment on the part of researchers, who must ensure enhanced objectivity and avoid biases or mistakes. As discussed later in this chapter, for certain types of reviews there must be at least two independent reviewers involved in the screening process and a procedure to resolve disagreements must also be in place ( Liberati et al., 2009 ; Shea et al., 2009 ). Assessing the quality of primary studies: In addition to screening material for inclusion, members of the review team may need to assess the scientific quality of the selected studies, that is, appraise the rigour of the research design and methods. Such formal assessment, which is usually conducted independently by at least two coders, helps members of the review team refine which studies to include in the final sample, determine whether or not the differences in quality may affect their conclusions, or guide how they analyze the data and interpret the findings ( Petticrew & Roberts, 2006 ). Ascribing quality scores to each primary study or considering through domain-based evaluations which study components have or have not been designed and executed appropriately makes it possible to reflect on the extent to which the selected study addresses possible biases and maximizes validity ( Shea et al., 2009 ). Extracting data: The following step involves gathering or extracting applicable information from each primary study included in the sample and deciding what is relevant to the problem of interest ( Cooper & Hedges, 2009 ). Indeed, the type of data that should be recorded mainly depends on the initial research questions ( Okoli & Schabram, 2010 ). However, important information may also be gathered about how, when, where and by whom the primary study was conducted, the research design and methods, or qualitative/quantitative results ( Cooper & Hedges, 2009 ). Analyzing and synthesizing data : As a final step, members of the review team must collate, summarize, aggregate, organize, and compare the evidence extracted from the included studies. The extracted data must be presented in a meaningful way that suggests a new contribution to the extant literature ( Jesson et al., 2011 ). Webster and Watson (2002) warn researchers that literature reviews should be much more than lists of papers and should provide a coherent lens to make sense of extant knowledge on a given topic. There exist several methods and techniques for synthesizing quantitative (e.g., frequency analysis, meta-analysis) and qualitative (e.g., grounded theory, narrative analysis, meta-ethnography) evidence ( Dixon-Woods, Agarwal, Jones, Young, & Sutton, 2005 ; Thomas & Harden, 2008 ).

9.3. Types of Review Articles and Brief Illustrations

EHealth researchers have at their disposal a number of approaches and methods for making sense out of existing literature, all with the purpose of casting current research findings into historical contexts or explaining contradictions that might exist among a set of primary research studies conducted on a particular topic. Our classification scheme is largely inspired from Paré and colleagues’ (2015) typology. Below we present and illustrate those review types that we feel are central to the growth and development of the eHealth domain.

9.3.1. Narrative Reviews

The narrative review is the “traditional” way of reviewing the extant literature and is skewed towards a qualitative interpretation of prior knowledge ( Sylvester et al., 2013 ). Put simply, a narrative review attempts to summarize or synthesize what has been written on a particular topic but does not seek generalization or cumulative knowledge from what is reviewed ( Davies, 2000 ; Green et al., 2006 ). Instead, the review team often undertakes the task of accumulating and synthesizing the literature to demonstrate the value of a particular point of view ( Baumeister & Leary, 1997 ). As such, reviewers may selectively ignore or limit the attention paid to certain studies in order to make a point. In this rather unsystematic approach, the selection of information from primary articles is subjective, lacks explicit criteria for inclusion and can lead to biased interpretations or inferences ( Green et al., 2006 ). There are several narrative reviews in the particular eHealth domain, as in all fields, which follow such an unstructured approach ( Silva et al., 2015 ; Paul et al., 2015 ).

Despite these criticisms, this type of review can be very useful in gathering together a volume of literature in a specific subject area and synthesizing it. As mentioned above, its primary purpose is to provide the reader with a comprehensive background for understanding current knowledge and highlighting the significance of new research ( Cronin et al., 2008 ). Faculty like to use narrative reviews in the classroom because they are often more up to date than textbooks, provide a single source for students to reference, and expose students to peer-reviewed literature ( Green et al., 2006 ). For researchers, narrative reviews can inspire research ideas by identifying gaps or inconsistencies in a body of knowledge, thus helping researchers to determine research questions or formulate hypotheses. Importantly, narrative reviews can also be used as educational articles to bring practitioners up to date with certain topics of issues ( Green et al., 2006 ).

Recently, there have been several efforts to introduce more rigour in narrative reviews that will elucidate common pitfalls and bring changes into their publication standards. Information systems researchers, among others, have contributed to advancing knowledge on how to structure a “traditional” review. For instance, Levy and Ellis (2006) proposed a generic framework for conducting such reviews. Their model follows the systematic data processing approach comprised of three steps, namely: (a) literature search and screening; (b) data extraction and analysis; and (c) writing the literature review. They provide detailed and very helpful instructions on how to conduct each step of the review process. As another methodological contribution, vom Brocke et al. (2009) offered a series of guidelines for conducting literature reviews, with a particular focus on how to search and extract the relevant body of knowledge. Last, Bandara, Miskon, and Fielt (2011) proposed a structured, predefined and tool-supported method to identify primary studies within a feasible scope, extract relevant content from identified articles, synthesize and analyze the findings, and effectively write and present the results of the literature review. We highly recommend that prospective authors of narrative reviews consult these useful sources before embarking on their work.

Darlow and Wen (2015) provide a good example of a highly structured narrative review in the eHealth field. These authors synthesized published articles that describe the development process of mobile health ( m-health ) interventions for patients’ cancer care self-management. As in most narrative reviews, the scope of the research questions being investigated is broad: (a) how development of these systems are carried out; (b) which methods are used to investigate these systems; and (c) what conclusions can be drawn as a result of the development of these systems. To provide clear answers to these questions, a literature search was conducted on six electronic databases and Google Scholar . The search was performed using several terms and free text words, combining them in an appropriate manner. Four inclusion and three exclusion criteria were utilized during the screening process. Both authors independently reviewed each of the identified articles to determine eligibility and extract study information. A flow diagram shows the number of studies identified, screened, and included or excluded at each stage of study selection. In terms of contributions, this review provides a series of practical recommendations for m-health intervention development.

9.3.2. Descriptive or Mapping Reviews

The primary goal of a descriptive review is to determine the extent to which a body of knowledge in a particular research topic reveals any interpretable pattern or trend with respect to pre-existing propositions, theories, methodologies or findings ( King & He, 2005 ; Paré et al., 2015 ). In contrast with narrative reviews, descriptive reviews follow a systematic and transparent procedure, including searching, screening and classifying studies ( Petersen, Vakkalanka, & Kuzniarz, 2015 ). Indeed, structured search methods are used to form a representative sample of a larger group of published works ( Paré et al., 2015 ). Further, authors of descriptive reviews extract from each study certain characteristics of interest, such as publication year, research methods, data collection techniques, and direction or strength of research outcomes (e.g., positive, negative, or non-significant) in the form of frequency analysis to produce quantitative results ( Sylvester et al., 2013 ). In essence, each study included in a descriptive review is treated as the unit of analysis and the published literature as a whole provides a database from which the authors attempt to identify any interpretable trends or draw overall conclusions about the merits of existing conceptualizations, propositions, methods or findings ( Paré et al., 2015 ). In doing so, a descriptive review may claim that its findings represent the state of the art in a particular domain ( King & He, 2005 ).

In the fields of health sciences and medical informatics, reviews that focus on examining the range, nature and evolution of a topic area are described by Anderson, Allen, Peckham, and Goodwin (2008) as mapping reviews . Like descriptive reviews, the research questions are generic and usually relate to publication patterns and trends. There is no preconceived plan to systematically review all of the literature although this can be done. Instead, researchers often present studies that are representative of most works published in a particular area and they consider a specific time frame to be mapped.

An example of this approach in the eHealth domain is offered by DeShazo, Lavallie, and Wolf (2009). The purpose of this descriptive or mapping review was to characterize publication trends in the medical informatics literature over a 20-year period (1987 to 2006). To achieve this ambitious objective, the authors performed a bibliometric analysis of medical informatics citations indexed in medline using publication trends, journal frequencies, impact factors, Medical Subject Headings (MeSH) term frequencies, and characteristics of citations. Findings revealed that there were over 77,000 medical informatics articles published during the covered period in numerous journals and that the average annual growth rate was 12%. The MeSH term analysis also suggested a strong interdisciplinary trend. Finally, average impact scores increased over time with two notable growth periods. Overall, patterns in research outputs that seem to characterize the historic trends and current components of the field of medical informatics suggest it may be a maturing discipline (DeShazo et al., 2009).

9.3.3. Scoping Reviews

Scoping reviews attempt to provide an initial indication of the potential size and nature of the extant literature on an emergent topic (Arksey & O’Malley, 2005; Daudt, van Mossel, & Scott, 2013 ; Levac, Colquhoun, & O’Brien, 2010). A scoping review may be conducted to examine the extent, range and nature of research activities in a particular area, determine the value of undertaking a full systematic review (discussed next), or identify research gaps in the extant literature ( Paré et al., 2015 ). In line with their main objective, scoping reviews usually conclude with the presentation of a detailed research agenda for future works along with potential implications for both practice and research.

Unlike narrative and descriptive reviews, the whole point of scoping the field is to be as comprehensive as possible, including grey literature (Arksey & O’Malley, 2005). Inclusion and exclusion criteria must be established to help researchers eliminate studies that are not aligned with the research questions. It is also recommended that at least two independent coders review abstracts yielded from the search strategy and then the full articles for study selection ( Daudt et al., 2013 ). The synthesized evidence from content or thematic analysis is relatively easy to present in tabular form (Arksey & O’Malley, 2005; Thomas & Harden, 2008 ).

One of the most highly cited scoping reviews in the eHealth domain was published by Archer, Fevrier-Thomas, Lokker, McKibbon, and Straus (2011) . These authors reviewed the existing literature on personal health record ( phr ) systems including design, functionality, implementation, applications, outcomes, and benefits. Seven databases were searched from 1985 to March 2010. Several search terms relating to phr s were used during this process. Two authors independently screened titles and abstracts to determine inclusion status. A second screen of full-text articles, again by two independent members of the research team, ensured that the studies described phr s. All in all, 130 articles met the criteria and their data were extracted manually into a database. The authors concluded that although there is a large amount of survey, observational, cohort/panel, and anecdotal evidence of phr benefits and satisfaction for patients, more research is needed to evaluate the results of phr implementations. Their in-depth analysis of the literature signalled that there is little solid evidence from randomized controlled trials or other studies through the use of phr s. Hence, they suggested that more research is needed that addresses the current lack of understanding of optimal functionality and usability of these systems, and how they can play a beneficial role in supporting patient self-management ( Archer et al., 2011 ).

9.3.4. Forms of Aggregative Reviews

Healthcare providers, practitioners, and policy-makers are nowadays overwhelmed with large volumes of information, including research-based evidence from numerous clinical trials and evaluation studies, assessing the effectiveness of health information technologies and interventions ( Ammenwerth & de Keizer, 2004 ; Deshazo et al., 2009 ). It is unrealistic to expect that all these disparate actors will have the time, skills, and necessary resources to identify the available evidence in the area of their expertise and consider it when making decisions. Systematic reviews that involve the rigorous application of scientific strategies aimed at limiting subjectivity and bias (i.e., systematic and random errors) can respond to this challenge.

Systematic reviews attempt to aggregate, appraise, and synthesize in a single source all empirical evidence that meet a set of previously specified eligibility criteria in order to answer a clearly formulated and often narrow research question on a particular topic of interest to support evidence-based practice ( Liberati et al., 2009 ). They adhere closely to explicit scientific principles ( Liberati et al., 2009 ) and rigorous methodological guidelines (Higgins & Green, 2008) aimed at reducing random and systematic errors that can lead to deviations from the truth in results or inferences. The use of explicit methods allows systematic reviews to aggregate a large body of research evidence, assess whether effects or relationships are in the same direction and of the same general magnitude, explain possible inconsistencies between study results, and determine the strength of the overall evidence for every outcome of interest based on the quality of included studies and the general consistency among them ( Cook, Mulrow, & Haynes, 1997 ). The main procedures of a systematic review involve:

  • Formulating a review question and developing a search strategy based on explicit inclusion criteria for the identification of eligible studies (usually described in the context of a detailed review protocol).
  • Searching for eligible studies using multiple databases and information sources, including grey literature sources, without any language restrictions.
  • Selecting studies, extracting data, and assessing risk of bias in a duplicate manner using two independent reviewers to avoid random or systematic errors in the process.
  • Analyzing data using quantitative or qualitative methods.
  • Presenting results in summary of findings tables.
  • Interpreting results and drawing conclusions.

Many systematic reviews, but not all, use statistical methods to combine the results of independent studies into a single quantitative estimate or summary effect size. Known as meta-analyses , these reviews use specific data extraction and statistical techniques (e.g., network, frequentist, or Bayesian meta-analyses) to calculate from each study by outcome of interest an effect size along with a confidence interval that reflects the degree of uncertainty behind the point estimate of effect ( Borenstein, Hedges, Higgins, & Rothstein, 2009 ; Deeks, Higgins, & Altman, 2008 ). Subsequently, they use fixed or random-effects analysis models to combine the results of the included studies, assess statistical heterogeneity, and calculate a weighted average of the effect estimates from the different studies, taking into account their sample sizes. The summary effect size is a value that reflects the average magnitude of the intervention effect for a particular outcome of interest or, more generally, the strength of a relationship between two variables across all studies included in the systematic review. By statistically combining data from multiple studies, meta-analyses can create more precise and reliable estimates of intervention effects than those derived from individual studies alone, when these are examined independently as discrete sources of information.

The review by Gurol-Urganci, de Jongh, Vodopivec-Jamsek, Atun, and Car (2013) on the effects of mobile phone messaging reminders for attendance at healthcare appointments is an illustrative example of a high-quality systematic review with meta-analysis. Missed appointments are a major cause of inefficiency in healthcare delivery with substantial monetary costs to health systems. These authors sought to assess whether mobile phone-based appointment reminders delivered through Short Message Service ( sms ) or Multimedia Messaging Service ( mms ) are effective in improving rates of patient attendance and reducing overall costs. To this end, they conducted a comprehensive search on multiple databases using highly sensitive search strategies without language or publication-type restrictions to identify all rct s that are eligible for inclusion. In order to minimize the risk of omitting eligible studies not captured by the original search, they supplemented all electronic searches with manual screening of trial registers and references contained in the included studies. Study selection, data extraction, and risk of bias assessments were performed inde­­pen­dently by two coders using standardized methods to ensure consistency and to eliminate potential errors. Findings from eight rct s involving 6,615 participants were pooled into meta-analyses to calculate the magnitude of effects that mobile text message reminders have on the rate of attendance at healthcare appointments compared to no reminders and phone call reminders.

Meta-analyses are regarded as powerful tools for deriving meaningful conclusions. However, there are situations in which it is neither reasonable nor appropriate to pool studies together using meta-analytic methods simply because there is extensive clinical heterogeneity between the included studies or variation in measurement tools, comparisons, or outcomes of interest. In these cases, systematic reviews can use qualitative synthesis methods such as vote counting, content analysis, classification schemes and tabulations, as an alternative approach to narratively synthesize the results of the independent studies included in the review. This form of review is known as qualitative systematic review.

A rigorous example of one such review in the eHealth domain is presented by Mickan, Atherton, Roberts, Heneghan, and Tilson (2014) on the use of handheld computers by healthcare professionals and their impact on access to information and clinical decision-making. In line with the methodological guide­lines for systematic reviews, these authors: (a) developed and registered with prospero ( www.crd.york.ac.uk/ prospero / ) an a priori review protocol; (b) conducted comprehensive searches for eligible studies using multiple databases and other supplementary strategies (e.g., forward searches); and (c) subsequently carried out study selection, data extraction, and risk of bias assessments in a duplicate manner to eliminate potential errors in the review process. Heterogeneity between the included studies in terms of reported outcomes and measures precluded the use of meta-analytic methods. To this end, the authors resorted to using narrative analysis and synthesis to describe the effectiveness of handheld computers on accessing information for clinical knowledge, adherence to safety and clinical quality guidelines, and diagnostic decision-making.

In recent years, the number of systematic reviews in the field of health informatics has increased considerably. Systematic reviews with discordant findings can cause great confusion and make it difficult for decision-makers to interpret the review-level evidence ( Moher, 2013 ). Therefore, there is a growing need for appraisal and synthesis of prior systematic reviews to ensure that decision-making is constantly informed by the best available accumulated evidence. Umbrella reviews , also known as overviews of systematic reviews, are tertiary types of evidence synthesis that aim to accomplish this; that is, they aim to compare and contrast findings from multiple systematic reviews and meta-analyses ( Becker & Oxman, 2008 ). Umbrella reviews generally adhere to the same principles and rigorous methodological guidelines used in systematic reviews. However, the unit of analysis in umbrella reviews is the systematic review rather than the primary study ( Becker & Oxman, 2008 ). Unlike systematic reviews that have a narrow focus of inquiry, umbrella reviews focus on broader research topics for which there are several potential interventions ( Smith, Devane, Begley, & Clarke, 2011 ). A recent umbrella review on the effects of home telemonitoring interventions for patients with heart failure critically appraised, compared, and synthesized evidence from 15 systematic reviews to investigate which types of home telemonitoring technologies and forms of interventions are more effective in reducing mortality and hospital admissions ( Kitsiou, Paré, & Jaana, 2015 ).

9.3.5. Realist Reviews

Realist reviews are theory-driven interpretative reviews developed to inform, enhance, or supplement conventional systematic reviews by making sense of heterogeneous evidence about complex interventions applied in diverse contexts in a way that informs policy decision-making ( Greenhalgh, Wong, Westhorp, & Pawson, 2011 ). They originated from criticisms of positivist systematic reviews which centre on their “simplistic” underlying assumptions ( Oates, 2011 ). As explained above, systematic reviews seek to identify causation. Such logic is appropriate for fields like medicine and education where findings of randomized controlled trials can be aggregated to see whether a new treatment or intervention does improve outcomes. However, many argue that it is not possible to establish such direct causal links between interventions and outcomes in fields such as social policy, management, and information systems where for any intervention there is unlikely to be a regular or consistent outcome ( Oates, 2011 ; Pawson, 2006 ; Rousseau, Manning, & Denyer, 2008 ).

To circumvent these limitations, Pawson, Greenhalgh, Harvey, and Walshe (2005) have proposed a new approach for synthesizing knowledge that seeks to unpack the mechanism of how “complex interventions” work in particular contexts. The basic research question — what works? — which is usually associated with systematic reviews changes to: what is it about this intervention that works, for whom, in what circumstances, in what respects and why? Realist reviews have no particular preference for either quantitative or qualitative evidence. As a theory-building approach, a realist review usually starts by articulating likely underlying mechanisms and then scrutinizes available evidence to find out whether and where these mechanisms are applicable ( Shepperd et al., 2009 ). Primary studies found in the extant literature are viewed as case studies which can test and modify the initial theories ( Rousseau et al., 2008 ).

The main objective pursued in the realist review conducted by Otte-Trojel, de Bont, Rundall, and van de Klundert (2014) was to examine how patient portals contribute to health service delivery and patient outcomes. The specific goals were to investigate how outcomes are produced and, most importantly, how variations in outcomes can be explained. The research team started with an exploratory review of background documents and research studies to identify ways in which patient portals may contribute to health service delivery and patient outcomes. The authors identified six main ways which represent “educated guesses” to be tested against the data in the evaluation studies. These studies were identified through a formal and systematic search in four databases between 2003 and 2013. Two members of the research team selected the articles using a pre-established list of inclusion and exclusion criteria and following a two-step procedure. The authors then extracted data from the selected articles and created several tables, one for each outcome category. They organized information to bring forward those mechanisms where patient portals contribute to outcomes and the variation in outcomes across different contexts.

9.3.6. Critical Reviews

Lastly, critical reviews aim to provide a critical evaluation and interpretive analysis of existing literature on a particular topic of interest to reveal strengths, weaknesses, contradictions, controversies, inconsistencies, and/or other important issues with respect to theories, hypotheses, research methods or results ( Baumeister & Leary, 1997 ; Kirkevold, 1997 ). Unlike other review types, critical reviews attempt to take a reflective account of the research that has been done in a particular area of interest, and assess its credibility by using appraisal instruments or critical interpretive methods. In this way, critical reviews attempt to constructively inform other scholars about the weaknesses of prior research and strengthen knowledge development by giving focus and direction to studies for further improvement ( Kirkevold, 1997 ).

Kitsiou, Paré, and Jaana (2013) provide an example of a critical review that assessed the methodological quality of prior systematic reviews of home telemonitoring studies for chronic patients. The authors conducted a comprehensive search on multiple databases to identify eligible reviews and subsequently used a validated instrument to conduct an in-depth quality appraisal. Results indicate that the majority of systematic reviews in this particular area suffer from important methodological flaws and biases that impair their internal validity and limit their usefulness for clinical and decision-making purposes. To this end, they provide a number of recommendations to strengthen knowledge development towards improving the design and execution of future reviews on home telemonitoring.

9.4. Summary

Table 9.1 outlines the main types of literature reviews that were described in the previous sub-sections and summarizes the main characteristics that distinguish one review type from another. It also includes key references to methodological guidelines and useful sources that can be used by eHealth scholars and researchers for planning and developing reviews.

Table 9.1. Typology of Literature Reviews (adapted from Paré et al., 2015).

Typology of Literature Reviews (adapted from Paré et al., 2015).

As shown in Table 9.1 , each review type addresses different kinds of research questions or objectives, which subsequently define and dictate the methods and approaches that need to be used to achieve the overarching goal(s) of the review. For example, in the case of narrative reviews, there is greater flexibility in searching and synthesizing articles ( Green et al., 2006 ). Researchers are often relatively free to use a diversity of approaches to search, identify, and select relevant scientific articles, describe their operational characteristics, present how the individual studies fit together, and formulate conclusions. On the other hand, systematic reviews are characterized by their high level of systematicity, rigour, and use of explicit methods, based on an “a priori” review plan that aims to minimize bias in the analysis and synthesis process (Higgins & Green, 2008). Some reviews are exploratory in nature (e.g., scoping/mapping reviews), whereas others may be conducted to discover patterns (e.g., descriptive reviews) or involve a synthesis approach that may include the critical analysis of prior research ( Paré et al., 2015 ). Hence, in order to select the most appropriate type of review, it is critical to know before embarking on a review project, why the research synthesis is conducted and what type of methods are best aligned with the pursued goals.

9.5. Concluding Remarks

In light of the increased use of evidence-based practice and research generating stronger evidence ( Grady et al., 2011 ; Lyden et al., 2013 ), review articles have become essential tools for summarizing, synthesizing, integrating or critically appraising prior knowledge in the eHealth field. As mentioned earlier, when rigorously conducted review articles represent powerful information sources for eHealth scholars and practitioners looking for state-of-the-art evidence. The typology of literature reviews we used herein will allow eHealth researchers, graduate students and practitioners to gain a better understanding of the similarities and differences between review types.

We must stress that this classification scheme does not privilege any specific type of review as being of higher quality than another ( Paré et al., 2015 ). As explained above, each type of review has its own strengths and limitations. Having said that, we realize that the methodological rigour of any review — be it qualitative, quantitative or mixed — is a critical aspect that should be considered seriously by prospective authors. In the present context, the notion of rigour refers to the reliability and validity of the review process described in section 9.2. For one thing, reliability is related to the reproducibility of the review process and steps, which is facilitated by a comprehensive documentation of the literature search process, extraction, coding and analysis performed in the review. Whether the search is comprehensive or not, whether it involves a methodical approach for data extraction and synthesis or not, it is important that the review documents in an explicit and transparent manner the steps and approach that were used in the process of its development. Next, validity characterizes the degree to which the review process was conducted appropriately. It goes beyond documentation and reflects decisions related to the selection of the sources, the search terms used, the period of time covered, the articles selected in the search, and the application of backward and forward searches ( vom Brocke et al., 2009 ). In short, the rigour of any review article is reflected by the explicitness of its methods (i.e., transparency) and the soundness of the approach used. We refer those interested in the concepts of rigour and quality to the work of Templier and Paré (2015) which offers a detailed set of methodological guidelines for conducting and evaluating various types of review articles.

To conclude, our main objective in this chapter was to demystify the various types of literature reviews that are central to the continuous development of the eHealth field. It is our hope that our descriptive account will serve as a valuable source for those conducting, evaluating or using reviews in this important and growing domain.

  • Ammenwerth E., de Keizer N. An inventory of evaluation studies of information technology in health care. Trends in evaluation research, 1982-2002. International Journal of Medical Informatics. 2004; 44 (1):44–56. [ PubMed : 15778794 ]
  • Anderson S., Allen P., Peckham S., Goodwin N. Asking the right questions: scoping studies in the commissioning of research on the organisation and delivery of health services. Health Research Policy and Systems. 2008; 6 (7):1–12. [ PMC free article : PMC2500008 ] [ PubMed : 18613961 ] [ CrossRef ]
  • Archer N., Fevrier-Thomas U., Lokker C., McKibbon K. A., Straus S.E. Personal health records: a scoping review. Journal of American Medical Informatics Association. 2011; 18 (4):515–522. [ PMC free article : PMC3128401 ] [ PubMed : 21672914 ]
  • Arksey H., O’Malley L. Scoping studies: towards a methodological framework. International Journal of Social Research Methodology. 2005; 8 (1):19–32.
  • A systematic, tool-supported method for conducting literature reviews in information systems. Paper presented at the Proceedings of the 19th European Conference on Information Systems ( ecis 2011); June 9 to 11; Helsinki, Finland. 2011.
  • Baumeister R. F., Leary M.R. Writing narrative literature reviews. Review of General Psychology. 1997; 1 (3):311–320.
  • Becker L. A., Oxman A.D. In: Cochrane handbook for systematic reviews of interventions. Higgins J. P. T., Green S., editors. Hoboken, nj : John Wiley & Sons, Ltd; 2008. Overviews of reviews; pp. 607–631.
  • Borenstein M., Hedges L., Higgins J., Rothstein H. Introduction to meta-analysis. Hoboken, nj : John Wiley & Sons Inc; 2009.
  • Cook D. J., Mulrow C. D., Haynes B. Systematic reviews: Synthesis of best evidence for clinical decisions. Annals of Internal Medicine. 1997; 126 (5):376–380. [ PubMed : 9054282 ]
  • Cooper H., Hedges L.V. In: The handbook of research synthesis and meta-analysis. 2nd ed. Cooper H., Hedges L. V., Valentine J. C., editors. New York: Russell Sage Foundation; 2009. Research synthesis as a scientific process; pp. 3–17.
  • Cooper H. M. Organizing knowledge syntheses: A taxonomy of literature reviews. Knowledge in Society. 1988; 1 (1):104–126.
  • Cronin P., Ryan F., Coughlan M. Undertaking a literature review: a step-by-step approach. British Journal of Nursing. 2008; 17 (1):38–43. [ PubMed : 18399395 ]
  • Darlow S., Wen K.Y. Development testing of mobile health interventions for cancer patient self-management: A review. Health Informatics Journal. 2015 (online before print). [ PubMed : 25916831 ] [ CrossRef ]
  • Daudt H. M., van Mossel C., Scott S.J. Enhancing the scoping study methodology: a large, inter-professional team’s experience with Arksey and O’Malley’s framework. bmc Medical Research Methodology. 2013; 13 :48. [ PMC free article : PMC3614526 ] [ PubMed : 23522333 ] [ CrossRef ]
  • Davies P. The relevance of systematic reviews to educational policy and practice. Oxford Review of Education. 2000; 26 (3-4):365–378.
  • Deeks J. J., Higgins J. P. T., Altman D.G. In: Cochrane handbook for systematic reviews of interventions. Higgins J. P. T., Green S., editors. Hoboken, nj : John Wiley & Sons, Ltd; 2008. Analysing data and undertaking meta-analyses; pp. 243–296.
  • Deshazo J. P., Lavallie D. L., Wolf F.M. Publication trends in the medical informatics literature: 20 years of “Medical Informatics” in mesh . bmc Medical Informatics and Decision Making. 2009; 9 :7. [ PMC free article : PMC2652453 ] [ PubMed : 19159472 ] [ CrossRef ]
  • Dixon-Woods M., Agarwal S., Jones D., Young B., Sutton A. Synthesising qualitative and quantitative evidence: a review of possible methods. Journal of Health Services Research and Policy. 2005; 10 (1):45–53. [ PubMed : 15667704 ]
  • Finfgeld-Connett D., Johnson E.D. Literature search strategies for conducting knowledge-building and theory-generating qualitative systematic reviews. Journal of Advanced Nursing. 2013; 69 (1):194–204. [ PMC free article : PMC3424349 ] [ PubMed : 22591030 ]
  • Grady B., Myers K. M., Nelson E. L., Belz N., Bennett L., Carnahan L. … Guidelines Working Group. Evidence-based practice for telemental health. Telemedicine Journal and E Health. 2011; 17 (2):131–148. [ PubMed : 21385026 ]
  • Green B. N., Johnson C. D., Adams A. Writing narrative literature reviews for peer-reviewed journals: secrets of the trade. Journal of Chiropractic Medicine. 2006; 5 (3):101–117. [ PMC free article : PMC2647067 ] [ PubMed : 19674681 ]
  • Greenhalgh T., Wong G., Westhorp G., Pawson R. Protocol–realist and meta-narrative evidence synthesis: evolving standards ( rameses ). bmc Medical Research Methodology. 2011; 11 :115. [ PMC free article : PMC3173389 ] [ PubMed : 21843376 ]
  • Gurol-Urganci I., de Jongh T., Vodopivec-Jamsek V., Atun R., Car J. Mobile phone messaging reminders for attendance at healthcare appointments. Cochrane Database System Review. 2013; 12 cd 007458. [ PMC free article : PMC6485985 ] [ PubMed : 24310741 ] [ CrossRef ]
  • Hart C. Doing a literature review: Releasing the social science research imagination. London: SAGE Publications; 1998.
  • Higgins J. P. T., Green S., editors. Cochrane handbook for systematic reviews of interventions: Cochrane book series. Hoboken, nj : Wiley-Blackwell; 2008.
  • Jesson J., Matheson L., Lacey F.M. Doing your literature review: traditional and systematic techniques. Los Angeles & London: SAGE Publications; 2011.
  • King W. R., He J. Understanding the role and methods of meta-analysis in IS research. Communications of the Association for Information Systems. 2005; 16 :1.
  • Kirkevold M. Integrative nursing research — an important strategy to further the development of nursing science and nursing practice. Journal of Advanced Nursing. 1997; 25 (5):977–984. [ PubMed : 9147203 ]
  • Kitchenham B., Charters S. ebse Technical Report Version 2.3. Keele & Durham. uk : Keele University & University of Durham; 2007. Guidelines for performing systematic literature reviews in software engineering.
  • Kitsiou S., Paré G., Jaana M. Systematic reviews and meta-analyses of home telemonitoring interventions for patients with chronic diseases: a critical assessment of their methodological quality. Journal of Medical Internet Research. 2013; 15 (7):e150. [ PMC free article : PMC3785977 ] [ PubMed : 23880072 ]
  • Kitsiou S., Paré G., Jaana M. Effects of home telemonitoring interventions on patients with chronic heart failure: an overview of systematic reviews. Journal of Medical Internet Research. 2015; 17 (3):e63. [ PMC free article : PMC4376138 ] [ PubMed : 25768664 ]
  • Levac D., Colquhoun H., O’Brien K. K. Scoping studies: advancing the methodology. Implementation Science. 2010; 5 (1):69. [ PMC free article : PMC2954944 ] [ PubMed : 20854677 ]
  • Levy Y., Ellis T.J. A systems approach to conduct an effective literature review in support of information systems research. Informing Science. 2006; 9 :181–211.
  • Liberati A., Altman D. G., Tetzlaff J., Mulrow C., Gøtzsche P. C., Ioannidis J. P. A. et al. Moher D. The prisma statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: Explanation and elaboration. Annals of Internal Medicine. 2009; 151 (4):W-65. [ PubMed : 19622512 ]
  • Lyden J. R., Zickmund S. L., Bhargava T. D., Bryce C. L., Conroy M. B., Fischer G. S. et al. McTigue K. M. Implementing health information technology in a patient-centered manner: Patient experiences with an online evidence-based lifestyle intervention. Journal for Healthcare Quality. 2013; 35 (5):47–57. [ PubMed : 24004039 ]
  • Mickan S., Atherton H., Roberts N. W., Heneghan C., Tilson J.K. Use of handheld computers in clinical practice: a systematic review. bmc Medical Informatics and Decision Making. 2014; 14 :56. [ PMC free article : PMC4099138 ] [ PubMed : 24998515 ]
  • Moher D. The problem of duplicate systematic reviews. British Medical Journal. 2013; 347 (5040) [ PubMed : 23945367 ] [ CrossRef ]
  • Montori V. M., Wilczynski N. L., Morgan D., Haynes R. B., Hedges T. Systematic reviews: a cross-sectional study of location and citation counts. bmc Medicine. 2003; 1 :2. [ PMC free article : PMC281591 ] [ PubMed : 14633274 ]
  • Mulrow C. D. The medical review article: state of the science. Annals of Internal Medicine. 1987; 106 (3):485–488. [ PubMed : 3813259 ] [ CrossRef ]
  • Evidence-based information systems: A decade later. Proceedings of the European Conference on Information Systems ; 2011. Retrieved from http://aisel ​.aisnet.org/cgi/viewcontent ​.cgi?article ​=1221&context ​=ecis2011 .
  • Okoli C., Schabram K. A guide to conducting a systematic literature review of information systems research. ssrn Electronic Journal. 2010
  • Otte-Trojel T., de Bont A., Rundall T. G., van de Klundert J. How outcomes are achieved through patient portals: a realist review. Journal of American Medical Informatics Association. 2014; 21 (4):751–757. [ PMC free article : PMC4078283 ] [ PubMed : 24503882 ]
  • Paré G., Trudel M.-C., Jaana M., Kitsiou S. Synthesizing information systems knowledge: A typology of literature reviews. Information & Management. 2015; 52 (2):183–199.
  • Patsopoulos N. A., Analatos A. A., Ioannidis J.P. A. Relative citation impact of various study designs in the health sciences. Journal of the American Medical Association. 2005; 293 (19):2362–2366. [ PubMed : 15900006 ]
  • Paul M. M., Greene C. M., Newton-Dame R., Thorpe L. E., Perlman S. E., McVeigh K. H., Gourevitch M.N. The state of population health surveillance using electronic health records: A narrative review. Population Health Management. 2015; 18 (3):209–216. [ PubMed : 25608033 ]
  • Pawson R. Evidence-based policy: a realist perspective. London: SAGE Publications; 2006.
  • Pawson R., Greenhalgh T., Harvey G., Walshe K. Realist review—a new method of systematic review designed for complex policy interventions. Journal of Health Services Research & Policy. 2005; 10 (Suppl 1):21–34. [ PubMed : 16053581 ]
  • Petersen K., Vakkalanka S., Kuzniarz L. Guidelines for conducting systematic mapping studies in software engineering: An update. Information and Software Technology. 2015; 64 :1–18.
  • Petticrew M., Roberts H. Systematic reviews in the social sciences: A practical guide. Malden, ma : Blackwell Publishing Co; 2006.
  • Rousseau D. M., Manning J., Denyer D. Evidence in management and organizational science: Assembling the field’s full weight of scientific knowledge through syntheses. The Academy of Management Annals. 2008; 2 (1):475–515.
  • Rowe F. What literature review is not: diversity, boundaries and recommendations. European Journal of Information Systems. 2014; 23 (3):241–255.
  • Shea B. J., Hamel C., Wells G. A., Bouter L. M., Kristjansson E., Grimshaw J. et al. Boers M. amstar is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. Journal of Clinical Epidemiology. 2009; 62 (10):1013–1020. [ PubMed : 19230606 ]
  • Shepperd S., Lewin S., Straus S., Clarke M., Eccles M. P., Fitzpatrick R. et al. Sheikh A. Can we systematically review studies that evaluate complex interventions? PLoS Medicine. 2009; 6 (8):e1000086. [ PMC free article : PMC2717209 ] [ PubMed : 19668360 ]
  • Silva B. M., Rodrigues J. J., de la Torre Díez I., López-Coronado M., Saleem K. Mobile-health: A review of current state in 2015. Journal of Biomedical Informatics. 2015; 56 :265–272. [ PubMed : 26071682 ]
  • Smith V., Devane D., Begley C., Clarke M. Methodology in conducting a systematic review of systematic reviews of healthcare interventions. bmc Medical Research Methodology. 2011; 11 (1):15. [ PMC free article : PMC3039637 ] [ PubMed : 21291558 ]
  • Sylvester A., Tate M., Johnstone D. Beyond synthesis: re-presenting heterogeneous research literature. Behaviour & Information Technology. 2013; 32 (12):1199–1215.
  • Templier M., Paré G. A framework for guiding and evaluating literature reviews. Communications of the Association for Information Systems. 2015; 37 (6):112–137.
  • Thomas J., Harden A. Methods for the thematic synthesis of qualitative research in systematic reviews. bmc Medical Research Methodology. 2008; 8 (1):45. [ PMC free article : PMC2478656 ] [ PubMed : 18616818 ]
  • Reconstructing the giant: on the importance of rigour in documenting the literature search process. Paper presented at the Proceedings of the 17th European Conference on Information Systems ( ecis 2009); Verona, Italy. 2009.
  • Webster J., Watson R.T. Analyzing the past to prepare for the future: Writing a literature review. Management Information Systems Quarterly. 2002; 26 (2):11.
  • Whitlock E. P., Lin J. S., Chou R., Shekelle P., Robinson K.A. Using existing systematic reviews in complex systematic reviews. Annals of Internal Medicine. 2008; 148 (10):776–782. [ PubMed : 18490690 ]

This publication is licensed under a Creative Commons License, Attribution-Noncommercial 4.0 International License (CC BY-NC 4.0): see https://creativecommons.org/licenses/by-nc/4.0/

  • Cite this Page Paré G, Kitsiou S. Chapter 9 Methods for Literature Reviews. In: Lau F, Kuziemsky C, editors. Handbook of eHealth Evaluation: An Evidence-based Approach [Internet]. Victoria (BC): University of Victoria; 2017 Feb 27.
  • PDF version of this title (4.5M)
  • Disable Glossary Links

In this Page

  • Introduction
  • Overview of the Literature Review Process and Steps
  • Types of Review Articles and Brief Illustrations
  • Concluding Remarks

Related information

  • PMC PubMed Central citations
  • PubMed Links to PubMed

Recent Activity

  • Chapter 9 Methods for Literature Reviews - Handbook of eHealth Evaluation: An Ev... Chapter 9 Methods for Literature Reviews - Handbook of eHealth Evaluation: An Evidence-based Approach

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

Connect with NLM

National Library of Medicine 8600 Rockville Pike Bethesda, MD 20894

Web Policies FOIA HHS Vulnerability Disclosure

Help Accessibility Careers

statistics

This paper is in the following e-collection/theme issue:

Published on 27.2.2024 in Vol 26 (2024)

This is a member publication of Imperial College London (Jisc)

Attrition in Conversational Agent–Delivered Mental Health Interventions: Systematic Review and Meta-Analysis

Authors of this article:

Author Orcid Image

  • Ahmad Ishqi Jabir 1, 2 , BSc, MSc   ; 
  • Xiaowen Lin 1 , BA   ; 
  • Laura Martinengo 1 , MD, PhD   ; 
  • Gemma Sharp 3 , PhD   ; 
  • Yin-Leng Theng 4 , PhD   ; 
  • Lorainne Tudor Car 1, 5 , MD, PhD  

1 Lee Kong Chian School of Medicine, Nanyang Technological University Singapore, Singapore, Singapore

2 Future Health Technologies, Singapore-ETH Centre, Campus for Research Excellence And Technological Enterprise, Singapore, Singapore

3 Department of Neuroscience, Monash University, Melbourne, Australia

4 Centre for Healthy and Sustainable Cities, Wee Kim Wee School of Communication and Information, Nanyang Technological University Singapore, Singapore, Singapore

5 Department of Primary Care and Public Health, School of Public Health, Imperial College London, London, United Kingdom

Corresponding Author:

Lorainne Tudor Car, MD, PhD

Lee Kong Chian School of Medicine

Nanyang Technological University Singapore

11 Mandalay Road, Level 18

Singapore, 308232

Phone: 65 69041258

Email: [email protected]

Background: Conversational agents (CAs) or chatbots are computer programs that mimic human conversation. They have the potential to improve access to mental health interventions through automated, scalable, and personalized delivery of psychotherapeutic content. However, digital health interventions, including those delivered by CAs, often have high attrition rates. Identifying the factors associated with attrition is critical to improving future clinical trials.

Objective: This review aims to estimate the overall and differential rates of attrition in CA-delivered mental health interventions (CA interventions), evaluate the impact of study design and intervention-related aspects on attrition, and describe study design features aimed at reducing or mitigating study attrition.

Methods: We searched PubMed, Embase (Ovid), PsycINFO (Ovid), Cochrane Central Register of Controlled Trials, and Web of Science, and conducted a gray literature search on Google Scholar in June 2022. We included randomized controlled trials that compared CA interventions against control groups and excluded studies that lasted for 1 session only and used Wizard of Oz interventions. We also assessed the risk of bias in the included studies using the Cochrane Risk of Bias Tool 2.0. Random-effects proportional meta-analysis was applied to calculate the pooled dropout rates in the intervention groups. Random-effects meta-analysis was used to compare the attrition rate in the intervention groups with that in the control groups. We used a narrative review to summarize the findings.

Results: The systematic search retrieved 4566 records from peer-reviewed databases and citation searches, of which 41 (0.90%) randomized controlled trials met the inclusion criteria. The meta-analytic overall attrition rate in the intervention group was 21.84% (95% CI 16.74%-27.36%; I 2 =94%). Short-term studies that lasted ≤8 weeks showed a lower attrition rate (18.05%, 95% CI 9.91%- 27.76%; I 2 =94.6%) than long-term studies that lasted >8 weeks (26.59%, 95% CI 20.09%-33.63%; I 2 =93.89%). Intervention group participants were more likely to attrit than control group participants for short-term (log odds ratio 1.22, 95% CI 0.99-1.50; I 2 =21.89%) and long-term studies (log odds ratio 1.33, 95% CI 1.08-1.65; I 2 =49.43%). Intervention-related characteristics associated with higher attrition include stand-alone CA interventions without human support, not having a symptom tracker feature, no visual representation of the CA, and comparing CA interventions with waitlist controls. No participant-level factor reliably predicted attrition.

Conclusions: Our results indicated that approximately one-fifth of the participants will drop out from CA interventions in short-term studies. High heterogeneities made it difficult to generalize the findings. Our results suggested that future CA interventions should adopt a blended design with human support, use symptom tracking, compare CA intervention groups against active controls rather than waitlist controls, and include a visual representation of the CA to reduce the attrition rate.

Trial Registration: PROSPERO International Prospective Register of Systematic Reviews CRD42022341415; https://www.crd.york.ac.uk/prospero/display_record.php?ID=CRD42022341415

Introduction

Description of the problem.

Mental health disorders are among the largest contributors to the global disease burden, affecting 1 in every 8 people, or 970 million people around the world [ 1 , 2 ]. However, access to evidence-based interventions for the prevention and treatment of mental health disorders is limited [ 3 , 4 ]. This is due to various factors such as a lack of mental health services and professionals, poor mental health literacy, fear of stigma, and low perceived need for treatment [ 5 - 10 ]. There is a need for scalable and accessible mental health services. Digital technologies such as smartphones or websites are increasingly being used for the delivery of mental health interventions and have the potential to improve access to mental health care. Digital mental health interventions allow for the scalable delivery of diverse therapeutic approaches such as cognitive behavioral therapy and mindfulness for the treatment of mental health conditions such as depression, anxiety, substance abuse, and eating disorders [ 11 - 16 ].

Description of the Intervention

Conversational agents (CAs) or chatbots are a more recent type of digital intervention, and they are becoming a popular method to deliver mental health interventions. CAs can be defined as computer algorithms designed to simulate human conversations textually or via speech through an interface [ 17 ]. CA-delivered mental health interventions (CA interventions) combine the delivery of psychotherapeutic content with an automated dialogue system that simulates the interaction between a mental health expert and the user [ 18 ]. These interventions provide an alternative avenue of psychotherapy to individuals who are not able to access mental health services owing to issues regarding time, location, or availability of resources [ 19 ]. CAs can also be a useful addition to traditional in-person therapy [ 20 , 21 ]. The presence of a CA can further contribute to improved therapeutic alliances with users to enhance adherence to the intervention [ 22 , 23 ]. Evidence for the efficacy of CAs in delivering mental health support is growing steadily. A recent meta-analysis showed that CA-delivered psychotherapy in adults significantly improved depressive symptoms with a medium effect size [ 19 ]. Providing self-guided therapy remotely via CAs may help address barriers to mental health access such as cost, long waiting time, and stigma [ 24 ]. Although the impact of mental health interventions delivered by CAs seems promising, studies evaluating such interventions also suggest high study attrition among participants [ 19 ]. Attrition or dropout occurs when participants do not complete the randomized controlled trial (RCT) assessments or complete the research protocol.

Digital health interventions typically report rapid and high attrition [ 13 , 25 ]. The overall attrition rate quantifies the level of attrition for the whole sample in a clinical trial, and the differential attrition rate refers to the level of attrition in the intervention group compared with that in the comparison group [ 26 ]. Attrition in clinical trials may introduce bias by modifying the random composition of the trial groups, limiting the generalizability of the study, and reducing the study power owing to reduced sample size [ 13 , 27 ]. To improve the quality of future clinical trials on CA interventions, there is a need to determine the attrition rates and the factors contributing to attrition in CA interventions.

Why Is It Important to Conduct This Review?

There is scant evidence on the possible factors associated with attrition in CA interventions for mental health and health care in general. The review conducted by Lim et al [ 19 ] on the effectiveness of CA interventions for depression and anxiety symptoms indicated that almost a fifth of the participants (19%) attrited throughout the trials without exploring factors associated with the attrition. This was comparable with other reviews on digital health and digital mental health interventions reporting attrition rates that ranged from 24.1% to 47.8% after adjusting for publication bias [ 13 , 28 ]. In general, factors shown to be associated with attrition in trials of digital health interventions include poor user experience, a lack of perceived value, and privacy concerns [ 28 , 29 ]; for example, studies on mental health apps reported technical issues and errors that might affect users’ overall experience [ 15 , 30 ]. Qualitative findings further suggested that factors such as a lack of human interactions in digital health interventions and users’ technological competence also played a role in participants’ attrition [ 31 ].

In addition, for smartphone-based mental health interventions, providing monetary compensation and reminders to engage were associated with significantly lower attrition rates [ 13 ]. Conversely, participants in the intervention condition were more likely to drop out than the waitlist participants [ 13 , 32 ]. These reviews focused only on smartphone-delivered interventions and included studies published before 2020, omitting several more recently published studies on CA interventions. To fully harness CA interventions, there is a need to better understand the factors associated with both overall attrition as well as differential attrition in these interventions.

To this end, we aimed to (1) estimate the overall and differential rates of attrition in CA interventions, (2) evaluate the impact of the study on design- and intervention-related aspects on the overall and differential attrition rates in CA interventions, and (3) map and describe study design features aimed at reducing or mitigating study attrition in the trials.

We performed a systematic review of attrition rates in RCTs of CA health interventions in line with the Cochrane gold standard methodology [ 33 ] and the meta-analysis approach outlined by Linardon and Fuller-Tyszkiewicz [ 13 ]. We reported this review in line with the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines [ 34 ]. The PRISMA checklist is included in Multimedia Appendix 1 .

Criteria for Study Selection

Types of studies.

Our inclusion criteria included RCTs, cluster RCTs, crossover RCTs, quasi-RCTs, and pilot RCTs in English. We decided to include these variations of RCTs because the field is still nascent, and findings from different forms of RCTs could be beneficial to understand the attrition rate in CA interventions. The publication types included peer-reviewed journals and conference papers that were published up to June 2022.

Types of Participants

Participants’ characteristics included healthy participants and participants with subclinical or clinically diagnosed mental health disorders such as depression, anxiety, attention-deficit/hyperactivity disorder, and bipolar disorder. Participants of any age were included so long as they personally interacted with the CA.

Types of Interventions

We included studies reporting a synchronous 2-way exchange with the participants via a CA. We excluded studies where the CA dialogues were wholly or partially delivered by human operators ( Wizard of Oz ) and studies with asynchronous response systems.

The interventions included either the delivery of psychotherapeutic content or those that provided training to improve mental well-being, reduced the symptoms of mental health conditions, or both. This included studies that aimed to reduce the symptoms of depression for clinical and subclinical populations or studies implementing mindfulness training for healthy adults. Detailed inclusion and exclusion criteria are outlined in Multimedia Appendix 2 [ 13 , 17 , 33 , 35 - 39 ].

Types of Outcome Measures

The primary outcome was the reported attrition number and the attrition rate calculated by the weighted risk of attrition of participants against the sample size of the studies for participants assigned to the CA intervention who then discontinued the study. This included the total attrition rate and the differential attrition rate in the intervention and comparison groups.

Search Methods for the Identification of Studies

The search strategy included index terms and keywords that describe CAs, such as “conversational agent,” “embodied conversational agent,” “virtual assistant,” and “virtual coach” ( Multimedia Appendix 3 ). The search strategy was previously developed for our scoping reviews [ 40 , 41 ] and was updated to include studies from 2020 to 2022 with the assistance of a medical librarian. We conducted the searches in the following databases on June 6, 2022: PubMed, Embase (Ovid), PsycINFO (Ovid), Cochrane Central Register of Controlled Trials, and Web of Science. In addition, we conducted a gray literature search on the first 200 entries from Google Scholar as suggested by the literature for the most optimal combination of databases [ 42 , 43 ]. We did not include any filter terms in the search. We also performed citation chasing by searching for all records citing ≥1 of the included studies (forward citation chasing) and all records referenced in the included studies (backward citation chasing) using Citationchaser [ 44 ]. The tables of excluded studies are presented in Multimedia Appendix 4 .

Data Collection and Analysis

Selection of studies.

On updating the searches from 2020 to 2022, we imported all identified references from the different electronic databases into a single file. The duplicated records were removed using revtool on R [ 35 ] and manually on Zotero (Corporation for Digital Scholarship). One reviewer (AIJ) performed the title and abstract screening using ASReview [ 36 ], an open-source machine learning software tool. The tool uses an active learning algorithm to actively sort and re-sort the records by prioritizing the most relevant records first based on the user’s inclusion and exclusion decisions. The title and abstract screening steps are detailed in Multimedia Appendix 2 .

Three reviewers (AIJ, XL, and LM) were engaged in the full-text review. One reviewer (AIJ) retrieved the full text of the studies, and 2 reviewers (AIJ and XL) assessed their eligibility independently and in parallel. Any disagreements were discussed and resolved between the reviewers and with a third reviewer (LM) acting as the arbiter. Studies that were identified in our previous reviews (up to 2020) [ 41 ] and met the inclusion criteria of this review were included based on discussions among the 3 reviewers (AIJ, XL, and LM).

Data Extraction and Management

Data were extracted using a data extraction form on Microsoft Excel. The data extraction form was piloted by 2 reviewers (AIJ and XL) on the same papers (5/41, 12%) and amended in line with the feedback. We also included additional fields as required from the data extraction form that we referenced from Linardon and Fuller-Tyszkiewicz [ 13 ]. Four reviewers (AIJ, XL, GS, and Nileshni Fernando) extracted the data independently and in parallel.

We extracted the year of publication; study design; the type of comparison group (active or inactive); the type of intervention; and details of the CAs, including the type of CA (rule based or artificial intelligence [AI] enhanced), the personality of the CA [ 17 ], the level of human support, the presence of a reminder mechanism, and the input and output modalities of the CA. In addition, we extracted information on the study duration, compensation paid to study participants, and any other mechanism included specifically to increase user engagement. Any disagreements among the reviewers were resolved by discussion.

Assessment of the Risk of Bias in Included Studies

Four reviewers (AIJ, XL, GS, and Nileshni Fernando) independently assessed the risk of bias in the included studies using the Cochrane Risk of Bias Tool 2.0 [ 33 ] and visualized using robvis [ 45 ]. The risk of bias assessment was piloted with 10 (24%) of the 41 studies for consistency and clarity of judgment by 2 reviewers (AIJ and XL). The steps involved in the assessment of the risk of bias are detailed in Multimedia Appendix 2 , and we have provided a summary, along with a table, in Multimedia Appendix 5 . We requested clarification or more data from the authors of 1 (2%) of the 41 studies but did not receive any response even after sending a reminder 2 weeks later. The assessment of publication bias was reported via funnel plots and the Egger test for publication bias [ 37 ].

Data Analysis

The meta-analysis was conducted based on the approach outlined by Linardon and Fuller-Tyszkiewicz [ 13 ] and the Cochrane Handbook for Systematic Reviews of Interventions (version 6.3) [ 33 ]. We defined attrition as the number of participants who dropped out of the study during the intervention period by not completing the postintervention assessment. We did not include the follow-up period [ 13 ]. For crossover design studies, we only included the data before the crossover following the aforementioned definition. The second part of the crossover was not considered as the follow-up period.

The study’s overall attrition rate was estimated by calculating the weighted pooled event rate using random-effect models based on a meta-proportional approach [ 33 ] using Freeman-Tukey double arcsine transformed proportion [ 38 ]. This indicated the relative risk of attrition against the sample size of the studies for participants assigned to the CA intervention group. The aim of this overall attrition analysis was to compute the overall rate of attrition in the intervention group after controlling for the different sample sizes in the included studies. Event rates were then converted to percentages of events per 100 participants and calculated separately for all included studies (short-term studies [≤8 wk from baseline] as well as longer-term studies [>8 wk from baseline]). We used short-term and long-term groupings to facilitate a comparison between our results and those of the previous study on attrition in smartphone-delivered interventions for mental health problems [ 13 ].

The differential attrition rate was calculated as the odds ratio (OR) of the likelihood to attrit between the CA intervention condition and the comparison condition. The aim of the differential attrition analysis was to understand the odds of attrition compared with the control group. The ORs were calculated using random-effect models separately for short-term and long-term studies weighted by their inverse variance. Studies with 0 events in both arms were weighted as 0, and a correction of 0.5 was added to the arm with 0 events as a continuity correction. A log OR of >1 indicated a higher likelihood of attrition in the CA intervention groups compared with the control groups. We also conducted subgroup analyses to explore the sources of heterogeneity in both overall and differential meta-analyses. The detailed meta-analysis procedure and subgroup analyses conducted are specified in Multimedia Appendix 2 . We also performed post hoc sensitivity analyses for subgroups with <5 studies because the estimate of tau-square might be imprecise [ 39 ]. In addition, we conducted exploratory analyses of all included studies regardless of the intervention duration using the same set of prespecified subgroup analyses on the overall and differential meta-analyses.

Meta-analysis was not conducted on the participant-level factors and the predictors of attrition owing to variability in reporting. We also identified additional factors significantly associated with attrition ( P <.05) in the included RCTs. We collated and narratively presented these factors associated with attrition as reported by the studies.

The updated search strategy retrieved 2228 records from peer-reviewed databases and 2319 from citation searching. After removing duplicates, of the total 4547 (2228+2319) records, 4030 (1877+2319[citation searching]-147[duplicates in citation searching]-19[records from other methods]) (88.63%) were screened on ASReview. Of these 4030 studies, 179 (4.4%) were then considered for full-text screening. We included 11 (6.1%) of the 179 studies identified from the full-text screening. We further identified 2 systematic reviews on CA intervention and included 14 studies that were not identified from our search strategy [ 19 , 46 ]. These studies used the Deprexis and electronic problem-solving treatment (ePST) interventions that did not explicitly identify themselves as CAs; for instance, both Deprexis and ePST described themselves as simulated dialogue that tailored their responses based on users’ input [ 47 , 48 ]. Subsequently, we followed up with an additional search on PubMed using “Deprexis OR ePST” as the search term and included 3 additional studies. We also included 13 studies identified in our previous review [ 41 ]. Thus, the total number of included studies is 41 (studies included in previous scoping review: n=13, 32%; new studies included from databases: n=11, 27%; and new studies included via other methods: n=17, 41%). Figure 1 presents the study selection process.

importance of review related literature in research study

Study Characteristics

Of the 41 studies included in this review, 22 (54%) were published before 2020 [ 15 , 47 - 67 ], and 19 (46%) were published in 2020 or later [ 14 , 68 - 85 ] ( Table 1 ). Most of the studies were conducted in the United States (13/41, 32%) [ 14 , 15 , 48 , 56 , 58 , 59 , 64 - 66 , 68 , 70 , 82 , 85 ] and Germany (13/41, 32%) [ 47 , 50 , 52 - 55 , 60 - 62 , 69 , 75 ], with some studies (2/41, 5%) conducted in multiple countries such as Switzerland and Germany [ 51 ] and Switzerland, Germany, and Austria [ 57 ]. Most of the interventions were short-term interventions and lasted ≤8 weeks (26/41, 63%) [ 14 , 15 , 48 , 49 , 52 , 56 , 58 , 62 - 68 , 70 - 74 , 76 - 81 , 84 ], whereas some of the studies (15/41, 37%) lasted >8 weeks [ 47 , 50 , 51 , 53 - 55 , 57 , 59 - 61 , 69 , 75 , 82 , 83 , 85 ].

a Conducted in both Switzerland and Germany.

b Conducted in Switzerland, Germany, and Austria.

c Ireland, Sweden, Italy, and the Netherlands.

d Japan, Ukraine, Argentina, New Zealand, China, and Russia.

e RCT: randomized controlled trial.

f Anxiety only, panic disorder, height phobia, gambling disorder, substance abuse, attention-deficit/hyperactivity disorder, irritable bowel syndrome, eating disorder, and personality disorder.

Psychoeducation and training were the focus of 24 (59%) of the 41 studies [ 47 - 57 , 59 - 62 , 64 , 66 , 69 , 70 , 75 , 77 - 79 , 84 ]. In almost half of the studies (18/41, 44%), participants were screened for mental health symptoms before the start of the study [ 14 , 50 , 52 - 56 , 59 , 62 , 63 , 66 , 68 , 72 - 74 , 80 , 82 , 83 ], and more than half of the studies (23/41, 56%) enrolled participants remotely using the web or the telephone [ 14 , 15 , 47 , 49 - 53 , 56 , 57 , 62 , 64 , 65 , 68 - 70 , 72 , 75 , 77 , 81 - 84 ]. More than one-third of the studies (17/41, 41%) focused on depression as the target disorder [ 47 , 48 , 50 - 56 , 58 - 62 , 67 , 69 , 83 ]. Of the 41 studies, 18 (44%) used waitlist control group participants [ 14 , 48 - 54 , 56 , 58 , 62 , 63 , 66 , 68 , 72 , 77 , 78 , 82 ], and 15 (37%) used an active control that included information or self-help (n=10, 67%) [ 15 , 60 , 65 , 71 , 74 - 76 , 80 , 81 , 83 ], alternative or comparable treatments such as stress-management cognitive behavioral therapy without a digital component (n=3, 20%) [ 73 , 84 , 86 ], or symptoms rating only (n=2, 13%) [ 64 , 70 ].

In the intervention group, of the 41 studies, 13 (32%) reported attrition between 0% and 10% [ 15 , 48 , 49 , 51 , 58 , 59 , 63 , 65 , 70 , 71 , 73 , 79 , 80 ], 6 (15%) reported attrition between 11% and 20% [ 14 , 57 , 67 , 74 , 78 , 83 ], 11 (27%) reported attrition between 21% and 30% [ 47 , 52 , 54 - 56 , 60 , 61 , 66 , 69 , 72 , 85 ], 2 (5%) reported attrition between 31% and 40% [ 53 , 64 ], 3 (7%) reported attrition between 41% and 50% [ 68 , 75 , 81 ], and 6 (15%) reported >50% attrition [ 50 , 62 , 76 , 77 , 82 , 84 ].

Risk-of-Bias Assessment

The most common risk of bias in the included studies was the bias in the measurement of the outcomes because they were all self-reported outcomes ( Multimedia Appendix 5 ). The second most common bias was due to the selection of the reported results because most of the studies (18/41, 44%) [ 15 , 48 - 51 , 58 , 59 , 62 , 64 , 66 , 70 , 71 , 74 , 77 , 80 , 81 , 83 , 84 ] did not cite the RCT protocol or statistical analysis plan. Most of the studies (29/41, 71%) reported an intention-to-treat analysis. Figure 2 shows the summary plot of the risk of bias.

importance of review related literature in research study

CA Characteristics

Most of the CAs were rule based (29/41, 71%), and 12 (29%) were AI enhanced using natural language processing or other AI-based algorithms ( Table 2 ). More than one-third of the studies (15/41, 37%) did not describe any specific visual representation of the CA. These were mainly studies that included Deprexis or Deprexis- based interventions (14/15, 93%) because they did not specifically identify themselves as CAs but used dialogue-based interventions. Of the 41 studies, 14 (34%) included an avatar or a static visual representation of the CA and 8 (20%) represented the CA using an embodied CA (ECA). With regard to the ECAs, of the 8 studies, 4 (50%) used relational agents, 3 (38%) used 3D-generated renders, and 2 (25%) used prerecorded videos. The CAs mostly presented a coach-like personality characterized by encouraging and nurturing personalities (19/41, 46%), followed by a factual personality characterized by being nonjudgmental and offering responses based on facts and observations (14/41, 34%). Of the 41 studies, in 5 (12%), the CA was designed to look like a physician or a health care professional, and in 3 (7%), the CA conversed with users using informal language characterized by exclamations, abbreviations, and emoticons in the dialogue (7%). More than one-third of the interventions were delivered via web-based applications (15/41, 37%), followed by those delivered by a stand-alone smartphone app (11/41, 27%).

a ECA: embodied CA.

b Slack, Facebook Messenger, or WeChat.

c AI: artificial intelligence.

Study Attrition Rates

The overall weighted attrition rate for the intervention groups in all included studies was 21.84% (95% CI 16.74%-27.36%; I 2 =94%), whereas the differential attrition rate differed from 0% (log OR 1.28, 95% CI 1.10-1.48; I 2 =34.6%), indicating that the participants who received CA interventions were more likely to attrit than the control group participants. Figure 3 [ 14 , 15 , 47 - 85 ] shows the attrition rates for all included studies.

importance of review related literature in research study

Short-Term Studies (≤8 Wk)

In the short-term studies, the overall weighted attrition rate in the intervention groups was 18.05% (95% CI 9.91%-27.76%), and there was evidence of high trial heterogeneity ( I 2 =94.6%, 95% CI 93.05%-95.74%). The high heterogeneity was due to high variations among the studies in terms of symptoms, types of interventions, and study populations. Of the 26 studies, 5 (19%) reported 0% attrition in the intervention group [ 48 , 49 , 70 , 73 , 86 ]. The lowest attrition rate was 6.12% (95% CI 1.48%-17.15%) [ 63 ], and the highest was 71.05% (95% CI 63.38%-77.69%) [ 77 ].

The differential attrition rate did not differ from 0% (log OR 1.22, 95% CI 0.99-1.50), indicating that the attrition rates were similar across the intervention and control groups.

The heterogeneity was low to moderate ( I 2 =21.89%, 95% CI 0%-54.6%). Of the 26 studies, 1 (4%) was identified as a potential outlier [ 15 ]. Removing this study from the model improved the I 2 value greatly ( I 2 =1.49%, 95% CI 0%-49.68%), and the differential attrition rate differed from 0% (log OR 1.27, 95% CI 1.04-1.54). This indicated that the attrition rate in the intervention group was larger than that in the control group after removing the outlying study. Multimedia Appendix 6 [ 14 , 15 , 47 - 85 ] shows the forest plot for the differential attrition meta-analysis [ 14 , 15 , 47 - 85 ].

Publication Bias for Short-Term Studies (≤8 Wk)

For the overall attrition rate, the Egger test was significant (intercept −4.70, 95% CI −8.12 to −1.28; P =.01), indicating possible publication bias. A closer look at the funnel plot showed missing studies toward the bottom right of the plot, which suggested possible publication bias for small sample–sized studies with high attrition rates ( Figure 4 A [ 14 , 15 , 48 , 49 , 52 , 56 , 58 , 62 - 68 , 70 - 74 , 76 - 81 , 84 ]). For the differential attrition rate, the funnel plot indicated evidence of plot symmetry, and the Egger test was not significant (intercept −4.85, 95% CI −1.56 to 0.58; P =.39; Figure 4 B [ 14 , 15 , 48 , 49 , 52 , 56 , 58 , 62 - 68 , 70 - 74 , 76 - 81 , 84 ]).

importance of review related literature in research study

Subgroup Analyses of the Attrition Rates in Short-Term Studies (≤8 Wk)

Subgroup analyses were conducted for both overall attrition rate ( Table 3 ) and differential attrition rate ( Multimedia Appendix 6 ) in the CA-intervention groups compared with the control groups.

a N/A: not applicable.

b RCT: randomized controlled trial.

c Anxiety only, panic disorder, height phobia, gambling disorder, substance abuse, attention-deficit/hyperactivity disorder, irritable bowel syndrome, eating disorder, and personality disorder.

d CBT: cognitive behavioral therapy.

e AI: artificial intelligence.

f ECA: embodied CA.

g Subgroup analyses were not significant after dropping subgroups with <5 studies.

For the overall attrition rate, there were significant differences in the attrition rates in short-term studies depending on the inclusion of mindfulness content ( χ 2 1 =5.1; P =.02). Interventions that included mindfulness content (n=12) showed a higher rate of attrition in the intervention group (30.24%, 95% CI 17.02%-42.27%) compared with interventions without mindfulness content (n=14; 8.66%, 95% CI 0.89%-21.2%). There were also significant differences depending on the population types, delivery channels, and types of disorders. However, these differences were not significant after excluding subgroups with <5 studies.

Subgroup analysis of the differential attrition rates showed that there were significantly different attrition rates in the intervention group compared with the control group depending on the subdurations ( χ 2 1 =5.8; P= .02). There were also significantly different attrition rates between study populations ( χ 2 2 =9.3; P =.009), and the types of controls ( χ 2 1 =4.7; P =.03). The relative risks of attrition for studies that lasted between 5 and 8 weeks were significantly different (n=9; log OR 1.61, 95% CI 1.22-2.13) compared with studies that lasted <5 weeks (n=17; log OR 0.99, 95% CI 0.75-1.31). Studies that recruited populations considered to be at risk (n=9) showed significantly higher attrition rates in the intervention group than in the control group (log OR 1.65, 95% CI 1.26-2.15) when compared with general populations (n=7; log OR 0.96, 95% CI 0.71-1.30) and clinical populations (n=3; log OR 0.47, 95% CI 0.13-1.66). The subgroup analysis was still significant when compared between the general population and the group considered to be at risk only ( χ 2 1 =6.9; P =.03). Finally, there were higher attrition rates in the intervention studies than in studies that used waitlist controls (n=11; log OR 1.52 95% CI 1.18-1.95) than those that used active controls (n=7; log OR 0.96, 95% CI 0.69-1.54). Only 1 (2%) of the 41 studies used treatment as usual as the control group [ 67 ]. No other comparisons were significant.

Long-Term Studies (>8 Wk)

The weighted attrition rate for the intervention group in long-term studies was 26.59% (95% CI 20.09%-33.63%), and there was evidence of high trial heterogeneity ( I 2 =93.89%, 95% CI 91.43%-95.64%). The lowest relative attrition rate was 6% (95% CI 1.44%-16.84%) [ 51 ], and the highest was 54.83% (95% CI 49.61%-59.95%) [ 77 ].

The differential attrition rate differed from 0% (log OR 1.33, 95% CI 1.08-1.65), indicating that the attrition rates in the intervention group were higher than those in the control group. The heterogeneity was moderate ( I 2 =49.43%, 95% CI 0.083%-72.11%). However, 1 (2%) of the 41 studies was identified as a potential outlier [ 50 ]. Removing this study from the model improved the I 2 value greatly ( I 2 =24.22%, 95% CI 0%-59.80%), and the differential attrition rate still differed from 0% (log OR 1.22, 95% CI 1.05-1.42); again, this indicated that the attrition rates in the intervention group were significantly larger than those in the control group even after removing the outlying study. The outlying study [ 50 ] used a weighted randomization method in which 80% of the participants were allocated to the intervention group. The subgroup analyses were conducted without the outlier because the study seemed to explain >20% of the heterogeneity in the model. However, sensitivity analyses conducted with and without the outlying study did not change the results of the subgroup analysis.

Publication Bias in Long-Term Studies (>8 Wk)

For the overall attrition rate, the funnel plot indicated evidence of plot asymmetry, but the Egger test was not significant (intercept −0.79, 95% CI −4.34 to 2.75; P =.67), suggesting a low likelihood of publication bias. For the differential attrition rate, the funnel plot indicated evidence of plot symmetry, and the Egger test was not significant (intercept 0.46, 95% CI −0.66 to 1.58; P =.43; Figure 5 [ 14 , 15 , 47 - 85 ]).

importance of review related literature in research study

Subgroup Analyses of the Attrition Rates in Long-Term Studies (>8 Wk)

For overall attrition, there were significant differences in the attrition rates in the intervention groups of long-term studies that had a blended design ( χ 2 1 =4.7; P =.03) and included symptom trackers or mood monitoring ( χ 2 1 =9.0; P =.003). Interventions that included blended designs (n=6) showed a significantly lower attrition rates (20.41%, 95% CI 15.49%-25.81%) than those without (n=9; 33.3%, 95% CI 23.01%-44.44%). Interventions that included symptom trackers (n=6) showed a significantly lower attrition rates (16.36%, 95% CI 10.32%-23.39%) than those without (n=9; 33.48%, 95% CI 24.91%-42.62%; Table 3 ). No other comparisons were significant.

The differential attrition rates were significantly different across studies that included mindfulness content compared with those without ( χ 2 1 =5.0; P =.03) and studies that targeted depression symptoms compared with those that targeted other mental health symptoms ( χ 2 1 =8.6; P =.003). Studies without mindfulness intervention showed higher attrition rates in the intervention groups than in the controls (n=10; log OR 1.56, 95% CI 1.25%-1.96%) compared with studies that included mindfulness content (n=4; log OR 1.11, 95% CI 1.05%-1.42%). Studies that targeted depression symptoms specifically showed relatively similar attrition rates in both intervention and control groups (n=10; log OR 1.09, 95% CI 0.96%-1.22%) compared with studies that targeted other mental health symptoms such as gambling disorder and substance abuse (n=4; log OR 1.63, 95% CI 1.28%-2.08%). No other comparisons were significant.

Exploratory Subgroup Analyses

The overall weighted attrition rate for the intervention group in all included studies was 21.84% (95% CI 16.74%-27.36%; I 2 =94%). Exploratory subgroup analyses using the prespecified subgroup analyses were conducted to explain the heterogeneity of the included studies regardless of the duration of the interventions. We have included the full findings in Multimedia Appendix 7 and only highlight findings that differed from our prespecified analysis here. For overall attrition, as in our prespecified analysis, there were significant differences in the attrition rates in the intervention groups depending on the inclusion of mindfulness content ( χ 2 1 =4.0; P =.05). However, we did not find significant differences in the inclusion of blended support compared with nonblended intervention and the inclusion of symptom tracker compared with intervention without symptom tracker. In addition, we found significant differences depending on the type of CA used ( χ 2 3 =13.1; P =.005), the CA delivery channel ( χ 2 4 =21.3; P <.001), and the study enrollment method ( χ 2 2 =7.4; P =.02). Studies that did not use any identifiable avatar reported the highest rate of attrition (n=15; 30%, 95% CI 23.44%-37.01%), followed by studies that did not specify the use of an avatar or a visual representation of the CA (n=14; 20.12%, 95% CI 7.29%-36.82%), studies that used a static avatar (n=4; 15.15%, 95% CI 1.79%-35.93%), and studies that used an ECA (n=8; 10.3%, 95% CI 4.29%-18.04%). Interventions that were delivered via messaging app (meaning “Slack, Facebook Messenger, or WeChat” based) showed the highest rate of attrition (n=7; 31.19%, 95% CI 10.68%-56.28%), followed by those delivered by web-based applications (n=15; 27.9%, 95% CI 22.35%-33.78%), and those delivered by stand-alone smartphone apps (n=11; 17.36%, 95% CI 6.54%-31.48%). CAs installed on a computer, a laptop computer, or a tablet computer showed the lowest rate of attrition (n=7; 5.61%, 95% CI 1.09%-12.3%). Finally, studies that offered remote onboarding only (n=23) showed a higher attrition rate (28.42%, 95% CI 21.30%-36.1%) than studies that offered an in-person onboarding process (n=16; 15.01%, 95% CI 8.46%-22.82%).

The differential attrition rate differed from 0% (log OR 1.28, 95% CI 1.10-1.48; I 2 =34.6%), indicating that the participants who received CA interventions were more likely to attrit than the control group participants.

For differential attrition, our findings mostly concurred with our prespecified analysis. Unlike in the prespecified analysis, there was a significant difference for studies that included symptom trackers ( χ 2 1 =5.0; P =.02). Studies that included symptom trackers (n=17) showed relatively lower attrition in the intervention group than in the control group (log OR 1.02, 95% CI 0.81-1.29) compared with studies without symptom trackers (n=18; log OR 1.44, 95% CI 1.19-1.74).

Additional Factors Associated With Attrition

Of the 41 studies, 16 (39%) assessed the association of different study features on participants’ attrition (short-term studies: n=8, 50%; long-term studies: n=8, 50%). We grouped the findings into two categories: (1) demographic predictors and (2) baseline measurement predictors or symptom severity.

The associations between participants’ demographics and attrition were assessed and reported in 10 (63%) of the 16 studies (short-term studies: n=4, 40%; long-term studies: n=6, 60%). Only 1 (25%) of the 4 short-term studies found age to be significantly associated with attrition. Participants who dropped out were found to be significantly younger than those who completed the whole intervention [ 62 ]. Other demographics-related factors assessed in these 10 studies were not associated with attrition, including sex, years of education, marital status, employment, actively receiving therapy or medication, and current diagnosis, for both short-term and long-term studies.

Of the 41 studies, 12 (29%) explored the association between baseline predictors or symptom severity and attrition (short-term studies: n=6, 50%; long-term studies: n=6, 50%). More severe baseline symptoms were associated with attrition for some of the short-term studies (2/6, 33%) but not for the long-term studies. Higher anxiety-related symptoms measured using the General Anxiety Disorder-7 questionnaire [ 62 ] and the Visceral Sensitivity Index [ 68 ] were significantly related to attrition. Other factors found to be associated with higher attrition included lower quality of life measured using the World Health Organization Quality of Life Brief Questionnaire [ 52 ], higher Fear of Food Questionnaire score [ 68 ], higher severity of gambling pathology measured using the Pathological Gambling Adaptation of the Yale-Brown Obsessive-Compulsive Scale [ 62 ], and lower self-esteem [ 81 ]. Of the 10 studies, 1 (10%) reported that participants who attrited significantly greater positive affect compared to those who completed the study using the Positive and Negative Affect Schedule [ 77 ].

Principal Findings

To our knowledge, this systematic review and meta-analysis is the first to examine attrition in RCTs of CA interventions for mental health. A total of 41 RCTs met the inclusion criteria. Our findings showed that approximately one-fifth of the participants (18.05%) dropped out in short-term studies, and approximately a quarter (26.59%) dropped out in long-term studies. Participants who received CA interventions were more likely to attrit than the control group participants for both long-term and short-term studies. Several study-level moderators were identified. For short-term studies, higher overall attrition rates were found in intervention groups that included mindfulness content and those that included participants from the general population. Compared with the control group participants, participants in the short-term CA interventions were also more likely to attrit in studies that lasted >1 month, those that included participants considered to be at risk, and studies in which intervention group participants were compared against waitlist controls as opposed to alternative active controls. For long-term studies, higher overall attrition rates were found in interventions that did not include human support and studies that did not include a symptom tracker. Exploratory subgroup analyses conducted on all included studies regardless of the study duration largely supported the analysis except for the inclusion of blended support. In addition, exploratory analyses found that studies that used an ECA, delivered via a computer or smartphones, and provided in-person enrollment options were associated with lower attrition rates.

Comparison With Prior Research

Overall attrition.

The overall attrition rates for short-term and long-term studies in our review are lower than the attrition estimates of short-term and long-term studies on smartphone-delivered interventions (24.1% and 35.5%, respectively) [ 13 ] and electronic mental health treatments for eating disorders (21%) [ 87 ]. Our findings are comparable with attrition rates in studies evaluating face-to-face mental health treatments such as interpersonal psychotherapy (20.6%) [ 88 ], individual psychotherapy for depressive disorders (19.9%) [ 89 ], and generalized anxiety disorders (17%) [ 90 ]. When focusing only on studies evaluating smartphone-delivered CAs in our review (n=11), we found that only 14.32% of the participants dropped out of the short-term studies and 17.36% dropped out of all included studies; these rates are lower than the estimated attrition rate previously reported for smartphone-delivered mental health interventions [ 13 ]. Taken together, the delivery channel may indirectly influence study attrition [ 13 ]. Although we found lower attrition in studies that were delivered via programs installed on a computer or a laptop computer compared with other delivery channels, these studies [ 48 , 58 , 59 , 63 , 79 ] were conducted in a laboratory setting compared with the participants’ environment, which might influence the retention rate.

Factors Associated With Attrition

CA interventions used as adjuvants of psychotherapy sessions or with human support may aid in retaining participants, particularly in long-term studies. A closer look at the long-term studies that included human support revealed that most of these studies (6/15, 40%) used the CA interventions as an adjunctive tool with no specific instruction given to the primary therapist on how to support participants’ journey through the CA interventions. This suggests that the presence of the therapist alone could suffice to retain participation in the study over a longer period. Although participants may be staying for the primary therapist and not engaging with the CA intervention, a study with both therapist-guided and unguided groups found no significant differences in the time spent with the CA intervention [ 51 ]. It is also possible that participants may have consulted their primary therapist about the CA interventions, which was not reported by the studies. A recent scoping review reported that many studies that included human support did not consistently report the type of support provided by humans [ 91 ]. This was similarly observed in our study because none of the included studies mentioned whether the therapist discussed the CA intervention during the participants’ routine sessions. Furthermore, it is also possible that participants within clinical settings are more likely to stay with the intervention, as was found in our results, unrelated to the blended support provided. Therefore, it is difficult to draw conclusions regarding the impact of using CA interventions adjunctively in terms of retaining participants within the study. However, this finding should be taken with caution because we found that there were no significant differences in the attrition rates between studies that included blended and nonblended approaches when we included all studies regardless of the study duration. This may be because fewer studies (9/41, 22%) included the blended approach overall.

In terms of the intervention content, short-term studies that included mindfulness content had higher overall attrition rates than those without mindfulness content. This was contrary to 2 previous meta-analyses that showed that the inclusion of the mindfulness component had no impact on attrition rates [ 13 , 28 ]. However, the attrition rate was similar to that in a systematic review of self-guided mindfulness interventions that reported an unweighted mean attrition rate of 31% (range 0%-79%) [ 92 ]. Future interventions may need to pay closer attention to participants’ engagement when including mindfulness content as part of a CA intervention. Factors such as using symptom trackers and personalized feedback may increase the engagement rate [ 93 ]. This is aligned with our findings and prior meta-analyses that suggest that including feedback may improve the retention rate [ 28 ].

Interestingly, our results also found relatively lower differential attrition rates in the intervention groups of long-term studies that included mindfulness compared with studies without. However, this finding was not replicated when we analyzed all studies regardless of the study duration. A recent review suggested that longer mindfulness practice sessions may be associated with the development of mindfulness skills [ 92 ]. Therefore, this result should be interpreted with caution because the relationship between the frequency and the duration of mindfulness practice sessions is still unclear [ 92 ].

Our study also found that including any form of visual representation of a CA may be associated with lower attrition rates compared with no visualization at all. This is aligned with many studies on the design of CAs that stressed the importance of design to create positive perceptions of the CA [ 94 ]. However, a recent scoping review reported that visual representation of the CA showed mixed and no association with subjective user experience [ 95 ]. A closer look at the studies included in this review suggested that most of them (35/41, 85%) lasted only 1 session [ 95 ]. It is possible that not having any CA visualization could affect user experience over time as alluded to by our findings. Future studies should explore the relationship between CA visual representation and user experience and study adherence over a longer duration.

In terms of the study sample population, sample populations considered to be at risk were more likely to attrit than samples from general and clinical sample populations. However, other meta-analyses of digital interventions for mental health issues found no difference in attrition rates across sample populations [ 13 , 28 ]. This finding is difficult to interpret because there could be multiple factors that may affect this relationship, such as symptom severity and other demographic factors not included in our analysis. Future studies may explore this relationship further to better understand this association.

Factors Not Associated With Attrition

Providing monetary incentives did not affect the attrition rates significantly. This finding is similar to that of a previous meta-analysis focusing on smartphone apps specifically for depressive symptoms [ 28 ] but contrary to that of a study focusing on smartphone interventions for mental health in general [ 13 ]. Time spent within the study may be a greater driver for attrition, as can be seen in the higher attrition rates for longer-term studies found in our results. However, research on the impact of monetary incentives on participants’ retention in digital health interventions is still in its infancy [ 96 ]. More needs to be done to understand how monetary incentives affect participants’ retention as well as effective engagement in the intervention.

Finally, several features such as providing reminders and the level of personalization provided by the CA also did not affect attrition rates. This is noteworthy because a personalized intervention that is responsive to users’ inputs is related to better engagement with the intervention [ 93 ]. There may be further nuances between attrition and effective engagement; for instance, factors that lower the risk of attrition might not be directly related to factors that promote study adherence [ 28 ].

Strengths and Limitations

We conducted a comprehensive literature search that included both peer-reviewed databases and gray literature sources; in addition, we conducted backward and forward citation searches. As this is a nascent area, we prioritized the sensitivity of our search to capture the various representations of CAs.

However, our study has some limitations. First, some unpublished literature presented at niche conferences and meetings may have been omitted. In addition, some studies might have escaped our search strategy, as evidenced by the inclusion of Deprexis , Deprexis- based interventions, and ePST intervention that did not explicitly mention terms related to conversational agent in the studies concerned. Second, the heterogeneity of the mental health conditions addressed by the CA intervention made it difficult to generalize the findings to a specific disorder. The recommendations provided here should be taken as a general suggestion to improve retention rates in CA interventions for mental health but not for a specific mental health disorder. Third, our results indicated possible publication bias in short-term studies. A closer look at the funnel plot suggested a lack of small sample–sized studies (n<20) with high attrition rates in the intervention groups. It is possible that these studies were not being published because they could be too experimental and small scaled. However, it is possible that the findings are reported elsewhere at niche conferences and internal sharing, which may have been omitted based on our search strategy. Fourth, our results may not directly translate into understanding factors that may increase engagement with the interventions. Although we recognize that engagement is interlinked with attrition [ 97 ], the lack of consensus and reporting of engagement metrics limits our understanding of this relationship. A recent review identified several patient-, intervention-, and system-level factors associated with engagement [ 98 ]. However, many of these associations were not consistent across different digital mental health interventions, and there was poor consistency in the reporting of the metrics. We echoed others’ recommendations to include and standardize the reporting of engagement metrics to better understand the indicators of nonuse attrition [ 41 , 93 , 97 , 98 ]. Our findings can inform future researchers of the potential factors for attrition in CA interventions. These may serve as a basis to make informed decisions on the sample size required or to conduct further studies on the specific mechanisms that may or may not motivate attrition. Fifth and last, our subgroup analyses for all studies regardless of the intervention duration were exploratory post hoc analyses and should be interpreted as such.

Implications and Recommendations

Several implications and recommendations emerged from our findings. First, researchers may want to account for the attrition of approximately one-third of the participants when designing RCTs involving CA interventions. This number may need to be further adjusted depending on the sample population, delivery modes, and comparison group used in the intervention to minimize the potential threats to the external and internal validity of studies that evaluate the efficacy of CA interventions for mental health. Second, researchers may want to consider including active controls in RCTs. Our results and the findings from other similar reviews on attrition in digital health research [ 13 , 28 ] suggested that comparing digital interventions with waitlist controls might not be the ideal way because participants in the comparison group may be more motivated to remain in the study than those in the intervention group [ 13 ]. Control interventions consisting of periodic mood assessments via an app or a nonconversational version of the app may be more appropriate for the assessment of CA effectiveness. Third, the inclusion of a visual representation of the CA may help create a more positive perception of the CA and reduce attrition. A recent review suggested that design considerations such as having a humanlike representation and having medical attire for the CA may be helpful to reduce attrition [ 95 ]. Fourth, CA interventions should be adjunctive to ongoing therapy sessions. Although the association between attrition and the use of blended support may be inconclusive, the use of CA interventions may further enrich participants’ experience between sessions and provide ongoing support to practice the skills learned during regular sessions. Fifth and last, clinicians interested in implementing CA interventions in their practice should be aware of the high attrition rate and should closely monitor patients’ progress within their practice.

Conclusions

According to our findings, at least one-fifth of the intervention group participants in RCTs of CA interventions will drop out of the trials within the intervention period. This attrition rate is comparable with that of face-to-face mental health interventions and less than that of other digital health interventions. Furthermore, intervention group participants were more likely to drop out than control group participants. Attrition was lower in shorter-term studies, studies that included participants considered to be at risk, and studies in which intervention group participants were compared against waitlist controls as opposed to alternative active controls. In addition, not including mindfulness content or symptom trackers was found to be associated with a smaller risk of attrition. Future studies may benefit from delivering CA interventions in a blended setting, with symptom screening; comparing the CA interventions against active controls such as symptom tracking only without the CA component; and including a visual representation of the CA to reduce attrition rates in the intervention group.

Acknowledgments

The authors would like to acknowledge Ms Yasmin Ally, Lee Kong Chian School of Medicine librarian, for her assistance in translating and executing the search strategy. The authors would also like to acknowledge Ms Nileshni Fernando for her assistance in the data extraction and risk-of-bias assessment. This research was supported by the Singapore Ministry of Education under the Singapore Ministry of Education Academic Research Fund Tier 1 (RG36/20). The research was conducted as part of the Future Health Technologies program, which was established collaboratively between ETH Zürich and the National Research Foundation, Singapore. This research was also supported by the National Research Foundation, Prime Minister’s Office, Singapore, under its Campus for Research Excellence and Technological Enterprise program.

Data Availability

This systematic review and meta-analysis does not contain primary data. The data sets generated and analyzed during this study are available from the corresponding author upon reasonable request. The data will be made available by emailing the corresponding author.

Authors' Contributions

LTC conceptualized the study and provided supervision at all steps of the research. LTC and AIJ designed the study. AIJ, XL, and GS extracted the data. AIJ conducted the analysis and wrote the original manuscript. LTC, LM, XL, GS, and YLT provided a critical review of the manuscript. All authors approved the final version of the manuscript and take accountability for all aspects of the work.

Conflicts of Interest

LTC is an associate editor of JMIR Medical Education. The other authors reported no conflict of interest.

PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) 2020 checklist.

Criteria for study selection and extended meta-analysis method.

Search strategy for PubMed.

Tables of excluded studies.

Risk-of-bias assessment of the included studies.

Differential attrition analysis for the short-term and long-term studies.

Extended results for the exploratory subgroup analysis.

  • GBD 2019 Mental Disorders Collaborators. Global, regional, and national burden of 12 mental disorders in 204 countries and territories, 1990-2019: a systematic analysis for the Global Burden of Disease Study 2019. Lancet Psychiatry. Feb 2022;9 (2):137-150. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Vigo D, Jones L, Atun R, Thornicroft G. The true global disease burden of mental illness: still elusive. Lancet Psychiatry. Feb 2022;9 (2):98-100. [ CrossRef ] [ Medline ]
  • Silverman AL, Teachman BA. The relationship between access to mental health resources and use of preferred effective mental health treatment. J Clin Psychol. Jun 2022;78 (6):1020-1045. [ CrossRef ] [ Medline ]
  • Thornicroft G, Chatterji S, Evans-Lacko S, Gruber M, Sampson N, Aguilar-Gaxiola S, et al. Undertreatment of people with major depressive disorder in 21 countries. Br J Psychiatry. Feb 2017;210 (2):119-124. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Chong SA, Abdin E, Vaingankar JA, Kwok KW, Subramaniam M. Where do people with mental disorders in Singapore go to for help? Ann Acad Med Singap. Apr 2012;41 (4):154-160. [ FREE Full text ] [ Medline ]
  • Gulliver A, Griffiths KM, Christensen H. Perceived barriers and facilitators to mental health help-seeking in young people: a systematic review. BMC Psychiatry. Dec 30, 2010;10:113. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Ku BS, Li J, Lally C, Compton MT, Druss BG. Associations between mental health shortage areas and county-level suicide rates among adults aged 25 and older in the USA, 2010 to 2018. Gen Hosp Psychiatry. May 2021;70:44-50. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Oladeji BD, Gureje O. Brain drain: a challenge to global mental health. BJPsych Int. Aug 2016;13 (3):61-63. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Clement S, Schauman O, Graham T, Maggioni F, Evans-Lacko S, Bezborodovs N, et al. What is the impact of mental health-related stigma on help-seeking? A systematic review of quantitative and qualitative studies. Psychol Med. Jan 2015;45 (1):11-27. [ CrossRef ] [ Medline ]
  • Weisel KK, Fuhrmann LM, Berking M, Baumeister H, Cuijpers P, Ebert DD. Standalone smartphone apps for mental health-a systematic review and meta-analysis. NPJ Digit Med. 2019;2:118. [ CrossRef ] [ Medline ]
  • Richards D, Richardson T. Computer-based psychological treatments for depression: a systematic review and meta-analysis. Clin Psychol Rev. Jun 2012;32 (4):329-342. [ CrossRef ] [ Medline ]
  • Linardon J, Hindle A, Brennan L. Dropout from cognitive-behavioral therapy for eating disorders: a meta-analysis of randomized, controlled trials. Int J Eat Disord. May 01, 2018;51 (5):381-391. [ CrossRef ] [ Medline ]
  • Linardon J, Fuller-Tyszkiewicz M. Attrition and adherence in smartphone-delivered interventions for mental health problems: a systematic and meta-analytic review. J Consult Clin Psychol. Jan 2020;88 (1):1-13. [ CrossRef ] [ Medline ]
  • Prochaska JJ, Vogel EA, Chieng A, Kendra M, Baiocchi M, Pajarito S, et al. A therapeutic relational agent for reducing problematic substance use (Woebot): development and usability study. J Med Internet Res. Mar 23, 2021;23 (3):e24850. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Fitzpatrick KK, Darcy A, Vierhile M. Delivering cognitive behavior therapy to young adults with symptoms of depression and anxiety using a fully automated conversational agent (Woebot): a randomized controlled trial. JMIR Ment Health. Jun 06, 2017;4 (2):e19. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Inkster B, Sarda S, Subramanian V. An empathy-driven, conversational artificial intelligence agent (Wysa) for digital mental well-being: real-world data evaluation mixed-methods study. JMIR Mhealth Uhealth. Nov 23, 2018;6 (11):e12106. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Tudor Car L, Dhinagaran DA, Kyaw BM, Kowatsch T, Joty S, Theng YL, et al. Conversational agents in health care: scoping review and conceptual analysis. J Med Internet Res. Aug 07, 2020;22 (8):e17158. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Bendig E, Erb B, Schulze-Thuesing L, Baumeister H. The next generation: chatbots in clinical psychology and psychotherapy to foster mental health – a scoping review. Verhaltenstherapie. Aug 20, 2019;32 (Suppl. 1):64-76. [ CrossRef ]
  • Lim SM, Shiau CW, Cheng LJ, Lau Y. Chatbot-delivered psychotherapy for adults with depressive and anxiety symptoms: a systematic review and meta-regression. Behav Ther. Mar 2022;53 (2):334-347. [ CrossRef ] [ Medline ]
  • Beilharz F, Sukunesan S, Rossell SL, Kulkarni J, Sharp G. Development of a positive body image chatbot (KIT) with young people and parents/carers: qualitative focus group study. J Med Internet Res. Jun 16, 2021;23 (6):e27807. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Salamanca-Sanabria A, Jabir AI, Lin X, Alattas A, Kocaballi AB, Lee J, et al. Exploring the perceptions of mHealth interventions for the prevention of common mental disorders in university students in Singapore: qualitative study. J Med Internet Res. Mar 20, 2023;25:e44542. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Scholten MR, Kelders SM, Van Gemert-Pijnen JE. Self-guided web-based interventions: scoping review on user needs and the potential of embodied conversational agents to address them. J Med Internet Res. Nov 16, 2017;19 (11):e383. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Nißen M, Rüegger D, Stieger M, Flückiger C, Allemand M, V Wangenheim F, et al. The effects of health care chatbot personas with different social roles on the client-chatbot bond and usage intentions: development of a design codebook and web-based study. J Med Internet Res. Apr 27, 2022;24 (4):e32630. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Miner AS, Milstein A, Hancock JT. Talking to machines about personal mental health problems. JAMA. Oct 03, 2017;318 (13):1217-1218. [ CrossRef ] [ Medline ]
  • Eysenbach G. The law of attrition. J Med Internet Res. Mar 31, 2005;7 (1):e11. [ CrossRef ] [ Medline ]
  • What Works Clearinghouse™ standards handbook, version 4.1. Institute of Education Sciences. URL: https://ies.ed.gov/ncee/wwc/Docs/referenceresources/WWC-Standards-Handbook-v4-1-508.pdf [accessed 2023-12-07]
  • Li P, Stuart EA, Allison DB. Multiple imputation: a flexible tool for handling missing data. JAMA. Nov 10, 2015;314 (18):1966-1967. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Torous J, Lipschitz J, Ng M, Firth J. Dropout rates in clinical trials of smartphone apps for depressive symptoms: a systematic review and meta-analysis. J Affect Disord. Feb 15, 2020;263:413-419. [ CrossRef ] [ Medline ]
  • Torous J, Nicholas J, Larsen ME, Firth J, Christensen H. Clinical review of user engagement with mental health smartphone apps: evidence, theory and improvements. Evid Based Ment Health. Aug 2018;21 (3):116-119. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Bendig E, Erb B, Meißner D, Bauereiß N, Baumeister H. Feasibility of a software agent providing a brief intervention for self-help to uplift psychological wellbeing ("SISU"). A single-group pretest-posttest trial investigating the potential of SISU to act as therapeutic agent. Internet Interv. Feb 24, 2021;24:100377. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Sanders GJ, Cooke C, Gately P. Exploring reasons for attrition among vulnerable and under-served sub-groups across an online integrated healthy lifestyles service during COVID-19. SAGE Open Med. Oct 22, 2021;9:20503121211054362. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Goldberg SB, Bolt DM, Davidson RJ. Data missing not at random in mobile health research: assessment of the problem and a case for sensitivity analyses. J Med Internet Res. Jun 15, 2021;23 (6):e26749. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Higgins JP, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, et al. Cochrane Handbook for Systematic Reviews of Interventions version 6.3. London, UK. The Cochrane Collaboration; 2022. .
  • Moher D, Liberati A, Tetzlaff J, Altman DG, PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. Jul 21, 2009;6 (7):e1000097. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Westgate MJ. revtools: an R package to support article screening for evidence synthesis. Res Synth Methods. Dec 2019;10 (4):606-614. [ CrossRef ] [ Medline ]
  • van de Schoot R, de Bruin J, Schram R, Zahedi P, de Boer J, Weijdema F, et al. An open source machine learning framework for efficient and transparent systematic reviews. Nat Mach Intell. Feb 01, 2021;3 (2):125-133. [ CrossRef ]
  • Egger M, Davey Smith G, Schneider M, Minder C. Bias in meta-analysis detected by a simple, graphical test. BMJ. Sep 13, 1997;315 (7109):629-634. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Harrer M, Cuijpers P, Furukawa TA, Ebert DD. Doing Meta-Analysis with R: A Hands-On Guide. Boca Raton, FL. CRC Press; Sep 14, 2021. .
  • Borenstein M, Hedges LV, Higgins JP, Rothstein HR. Introduction to Meta-Analysis. Hoboken, NJ. Wiley; Mar 12, 2009. .
  • Martinengo L, Jabir AI, Goh WW, Lo NY, Ho MH, Kowatsch T, et al. Conversational agents in health care: scoping review of their behavior change techniques and underpinning theory. J Med Internet Res. Oct 03, 2022;24 (10):e39243. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Jabir AI, Martinengo L, Lin X, Torous J, Subramaniam M, Tudor Car L. Evaluating conversational agents for mental health: scoping review of outcomes and outcome measurement instruments. J Med Internet Res. Apr 19, 2023;25:e44548. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Bramer WM, Rethlefsen ML, Kleijnen J, Franco OH. Optimal database combinations for literature searches in systematic reviews: a prospective exploratory study. Syst Rev. Dec 06, 2017;6 (1):245. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Haddaway NR, Collins AM, Coughlin D, Kirk S. The role of Google scholar in evidence reviews and its applicability to grey literature searching. PLoS One. Sep 17, 2015;10 (9):e0138237. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Haddaway NR, Grainger MJ, Gray CT. Citationchaser: a tool for transparent and efficient forward and backward citation chasing in systematic searching. Res Synth Methods. Jul 2022;13 (4):533-545. [ CrossRef ] [ Medline ]
  • McGuinness LA, Higgins JP. Risk-of-bias VISualization (robvis): an R package and Shiny web app for visualizing risk-of-bias assessments. Res Synth Methods. Jan 2021;12 (1):55-61. [ CrossRef ] [ Medline ]
  • Twomey C, O'Reilly G, Meyer B. Effectiveness of an individually-tailored computerised CBT programme (Deprexis) for depression: a meta-analysis. Psychiatry Res. Oct 2017;256:371-377. [ CrossRef ] [ Medline ]
  • Meyer B, Bierbrodt J, Schröder J, Berger T, Beevers CG, Weiss M, et al. Effects of an internet intervention (Deprexis) on severe depression symptoms: randomized controlled trial. Internet Interv. Mar 2015;2 (1):48-59. [ CrossRef ]
  • Cartreine JA, Locke SE, Buckey JC, Sandoval L, Hegel MT. Electronic problem-solving treatment: description and pilot study of an interactive media treatment for depression. JMIR Res Protoc. Sep 25, 2012;1 (2):e11. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Ly KH, Ly AM, Andersson G. A fully automated conversational agent for promoting mental well-being: a pilot RCT using mixed methods. Internet Interv. Dec 2017;10:39-46. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Meyer B, Berger T, Caspar F, Beevers CG, Andersson G, Weiss M. Effectiveness of a novel integrative online treatment for depression (Deprexis): randomized controlled trial. J Med Internet Res. May 11, 2009;11 (2):e15. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Berger T, Hämmerli K, Gubser N, Andersson G, Caspar F. Internet-based treatment of depression: a randomized controlled trial comparing guided with unguided self-help. Cogn Behav Ther. 2011;40 (4):251-266. [ CrossRef ] [ Medline ]
  • Moritz S, Schilling L, Hauschildt M, Schröder J, Treszl A. A randomized controlled trial of internet-based therapy in depression. Behav Res Ther. Aug 2012;50 (7-8):513-521. [ CrossRef ] [ Medline ]
  • Schröder J, Brückner K, Fischer A, Lindenau M, Köther U, Vettorazzi E, et al. Efficacy of a psychological online intervention for depression in people with epilepsy: a randomized controlled trial. Epilepsia. Dec 2014;55 (12):2069-2076. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Fischer A, Schröder J, Vettorazzi E, Wolf OT, Pöttgen J, Lau S, et al. An online programme to reduce depression in patients with multiple sclerosis: a randomised controlled trial. Lancet Psychiatry. Mar 2015;2 (3):217-223. [ CrossRef ] [ Medline ]
  • Klein JP, Berger T, Schröder J, Späth C, Meyer B, Caspar F, et al. Effects of a psychological internet intervention in the treatment of mild to moderate depressive symptoms: results of the EVIDENT study, a randomized controlled trial. Psychother Psychosom. 2016;85 (4):218-228. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Beevers CG, Pearson R, Hoffman JS, Foulser AA, Shumake J, Meyer B. Effectiveness of an internet intervention (Deprexis) for depression in a united states adult sample: a parallel-group pragmatic randomized controlled trial. J Consult Clin Psychol. Apr 2017;85 (4):367-380. [ CrossRef ] [ Medline ]
  • Berger T, Urech A, Krieger T, Stolz T, Schulz A, Vincent A, et al. Effects of a transdiagnostic unguided Internet intervention ('velibra') for anxiety disorders in primary care: results of a randomized controlled trial. Psychol Med. Jan 2017;47 (1):67-80. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Sandoval LR, Buckey JC, Ainslie R, Tombari M, Stone W, Hegel MT. Randomized controlled trial of a computerized interactive media-based problem solving treatment for depression. Behav Ther. May 2017;48 (3):413-425. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Shamekhi A, Bickmore T, Lestoquoy A, Gardiner P. Augmenting group medical visits with conversational agents for stress management behavior change. In: Proceedings of the 12th International Conference, PERSUASIVE 2017. 2017. Presented at: 12th International Conference, PERSUASIVE 2017; April 4–6, 2017, 2017; Amsterdam, The Netherlands. [ CrossRef ]
  • Zwerenz R, Becker J, Knickenberg RJ, Siepmann M, Hagen K, Beutel ME. Online self-help as an add-on to inpatient psychotherapy: efficacy of a new blended treatment approach. Psychother Psychosom. 2017;86 (6):341-350. [ CrossRef ] [ Medline ]
  • Berger T, Krieger T, Sude K, Meyer B, Maercker A. Evaluating an e-mental health program ("deprexis") as adjunctive treatment tool in psychotherapy for depression: results of a pragmatic randomized controlled trial. J Affect Disord. Feb 2018;227:455-462. [ CrossRef ] [ Medline ]
  • Bücker L, Bierbrodt J, Hand I, Wittekind C, Moritz S. Effects of a depression-focused internet intervention in slot machine gamblers: a randomized controlled trial. PLoS One. Jun 08, 2018;13 (6):e0198859. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Freeman D, Reeve S, Robinson A, Ehlers A, Clark D, Spanlang B, et al. Virtual reality in the assessment, understanding, and treatment of mental health disorders. Psychol Med. Oct 2017;47 (14):2393-2400. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Greer S, Ramo D, Chang YJ, Fu M, Moskowitz J, Haritatos J. Use of the chatbot "Vivibot" to deliver positive psychology skills and promote well-being among young people after cancer treatment: randomized controlled feasibility trial. JMIR Mhealth Uhealth. Oct 31, 2019;7 (10):e15018. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Fulmer R, Joerin A, Gentile B, Lakerink L, Rauws M. Using psychological artificial intelligence (Tess) to relieve symptoms of depression and anxiety: randomized controlled trial. JMIR Ment Health. Dec 13, 2018;5 (4):e64. [ CrossRef ] [ Medline ]
  • Sidner CL, Bickmore T, Nooraie B, Rich C, Ring L, Shayganfar M, et al. Creating new technologies for companionable agents to support isolated older adults. ACM Trans Interact Intell Syst. Jul 24, 2018;8 (3):1-27. [ CrossRef ]
  • Burton C, Szentagotai Tatar A, McKinstry B, Matheson C, Matu S, Moldovan R, et al. Pilot randomised controlled trial of Help4Mood, an embodied virtual agent-based system to support treatment of depression. J Telemed Telecare. Sep 2016;22 (6):348-355. [ CrossRef ] [ Medline ]
  • Hunt M, Miguez S, Dukas B, Onwude O, White S. Efficacy of Zemedy, a mobile digital therapeutic for the self-management of irritable bowel syndrome: crossover randomized controlled trial. JMIR Mhealth Uhealth. May 20, 2021;9 (5):e26152. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Gräfe V, Moritz S, Greiner W. Health economic evaluation of an internet intervention for depression (deprexis), a randomized controlled trial. Health Econ Rev. Jun 16, 2020;10 (1):19. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Narain J, Quach T, Davey M, Park HW, Breazeal C, Picard R. Promoting wellbeing with sunny, a chatbot that facilitates positive messages within social groups. In: Proceedings of the Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems. 2020. Presented at: CHI EA '20; April 25-30, 2020, 2020; Honolulu, HI. [ CrossRef ]
  • Oh J, Jang S, Kim H, Kim JJ. Efficacy of mobile app-based interactive cognitive behavioral therapy using a chatbot for panic disorder. Int J Med Inform. Aug 2020;140:104171. [ CrossRef ] [ Medline ]
  • So R, Furukawa TA, Matsushita S, Baba T, Matsuzaki T, Furuno S, et al. Unguided chatbot-delivered cognitive behavioural intervention for problem gamblers through messaging app: a randomised controlled trial. J Gambl Stud. Dec 2020;36 (4):1391-1407. [ CrossRef ] [ Medline ]
  • Danieli M, Ciulli T, Mousavi SM, Riccardi G. A conversational artificial intelligence agent for a mental health care app: evaluation study of its participatory design. JMIR Form Res. Dec 01, 2021;5 (12):e30053. [ CrossRef ] [ Medline ]
  • Jang S, Kim JJ, Kim SJ, Hong J, Kim S, Kim E. Mobile app-based chatbot to deliver cognitive behavioral therapy and psychoeducation for adults with attention deficit: a development and feasibility/usability study. Int J Med Inform. Jun 2021;150:104440. [ CrossRef ] [ Medline ]
  • Klein JP, Hauer-von Mauschwitz A, Berger T, Fassbinder E, Mayer J, Borgwardt S, et al. Effectiveness and safety of the adjunctive use of an internet-based self-management intervention for borderline personality disorder in addition to care as usual: results from a randomised controlled trial. BMJ Open. Sep 08, 2021;11 (9):e047771. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Klos MC, Escoredo M, Joerin A, Lemos VN, Rauws M, Bunge EL. Artificial intelligence-based chatbot for anxiety and depression in university students: pilot randomized controlled trial. JMIR Form Res. Aug 12, 2021;5 (8):e20678. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Lavelle J, Dunne N, Mulcahy HE, McHugh L. Chatbot-delivered cognitive defusion versus cognitive restructuring for negative self-referential thoughts: a pilot study. Psychol Rec. Aug 24, 2021;72 (2):247-261. [ CrossRef ]
  • Loveys K, Sagar M, Pickering I, Broadbent E. A digital human for delivering a remote loneliness and stress intervention to at-risk younger and older adults during the COVID-19 pandemic: randomized pilot trial. JMIR Ment Health. Nov 08, 2021;8 (11):e31586. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Park S, Thieme A, Han J, Lee S, Rhee W, Suh B. “I wrote as if I were telling a story to someone I knew.”: designing chatbot interactions for expressive writing in mental health. In: Proceedings of the 2021 ACM Designing Interactive Systems Conference. 2021. Presented at: DIS '21; June 28-July 2, 2021, 2021; Virtual Event. [ CrossRef ]
  • Romanovskyi O, Pidbutska N, Knysh A. Elomia chatbot: the effectiveness of artificial intelligence in the fight for mental health. In: Proceedings of the 5th International Conference on Computational Linguistics and Intelligent Systems. 2021. Presented at: COLINS-2021; April 22-23, 2021, 2021; Kharkiv, Ukraine. URL: https://ceur-ws.org/Vol-2870/paper89.pdf
  • Troitskaya O, Batkhina A. Mobile application for couple relationships: results of a pilot effectiveness study. Fam Process. Jun 14, 2022;61 (2):625-642. [ CrossRef ] [ Medline ]
  • Fitzsimmons-Craft EE, Chan WW, Smith AC, Firebaugh ML, Fowler LA, Topooco N, et al. Effectiveness of a chatbot for eating disorders prevention: a randomized clinical trial. Int J Eat Disord. Mar 2022;55 (3):343-353. [ CrossRef ] [ Medline ]
  • Liu H, Peng H, Song X, Xu C, Zhang M. Using AI chatbots to provide self-help depression interventions for university students: a randomized trial of effectiveness. Internet Interv. Mar 2022;27:100495. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Medeiros L, Bosse T, Gerritsen C. Can a chatbot comfort humans? Studying the impact of a supportive chatbot on users’ self-perceived stress. IEEE Trans Hum Mach Syst. Jun 2022;52 (3):343-353. [ CrossRef ]
  • Rubin A, Livingston NA, Brady J, Hocking E, Bickmore T, Sawdy M, et al. Computerized relational agent to deliver alcohol brief intervention and referral to treatment in primary care: a randomized clinical trial. J Gen Intern Med. Jan 2022;37 (1):70-77. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Park S, Choi J, Lee S, Oh C, Kim C, La S, et al. Designing a chatbot for a brief motivational interview on stress management: qualitative case study. J Med Internet Res. Apr 16, 2019;21 (4):e12231. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Linardon J, Shatte A, Messer M, Firth J, Fuller-Tyszkiewicz M. E-mental health interventions for the treatment and prevention of eating disorders: an updated systematic review and meta-analysis. J Consult Clin Psychol. Nov 2020;88 (11):994-1007. [ CrossRef ] [ Medline ]
  • Linardon J, Fitzsimmons-Craft EE, Brennan L, Barillaro M, Wilfley DE. Dropout from interpersonal psychotherapy for mental health disorders: a systematic review and meta-analysis. Psychother Res. Oct 2019;29 (7):870-881. [ CrossRef ] [ Medline ]
  • Cooper AA, Conklin LR. Dropout from individual psychotherapy for major depression: a meta-analysis of randomized clinical trials. Clin Psychol Rev. Aug 2015;40:57-65. [ CrossRef ] [ Medline ]
  • Gersh E, Hallford DJ, Rice SM, Kazantzis N, Gersh H, Gersh B, et al. Systematic review and meta-analysis of dropout rates in individual psychotherapy for generalized anxiety disorder. J Anxiety Disord. Dec 2017;52:25-33. [ CrossRef ] [ Medline ]
  • Bernstein EE, Weingarden H, Wolfe EC, Hall MD, Snorrason I, Wilhelm S. Human support in app-based cognitive behavioral therapies for emotional disorders: scoping review. J Med Internet Res. Apr 08, 2022;24 (4):e33307. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Taylor H, Strauss C, Cavanagh K. Can a little bit of mindfulness do you good? A systematic review and meta-analyses of unguided mindfulness-based self-help interventions. Clin Psychol Rev. Nov 2021;89:102078. [ CrossRef ] [ Medline ]
  • Jakob R, Harperink S, Rudolf AM, Fleisch E, Haug S, Mair JL, et al. Factors influencing adherence to mHealth apps for prevention or management of noncommunicable diseases: systematic review. J Med Internet Res. May 25, 2022;24 (5):e35371. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Ter Stal S, Sloots J, Ramlal A, Op den Akker H, Lenferink A, Tabak M. An embodied conversational agent in an eHealth self-management intervention for chronic obstructive pulmonary disease and chronic heart failure: exploratory study in a real-life setting. JMIR Hum Factors. Nov 04, 2021;8 (4):e24110. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Curtis RG, Bartel B, Ferguson T, Blake HT, Northcott C, Virgara R, et al. Improving user experience of virtual health assistants: scoping review. J Med Internet Res. Dec 21, 2021;23 (12):e31737. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Boucher EM, Ward HE, Mounts AC, Parks AC. Engagement in digital mental health interventions: can monetary incentives help? Front Psychol. Nov 18, 2021;12:746324. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Pham M, Do HM, Su Z, Bishop A, Sheng W. Negative emotion management using a smart shirt and a robot assistant. IEEE Robot Autom Lett. Apr 2021;6 (2):4040-4047. [ CrossRef ]
  • Lipschitz JM, Pike CK, Hogan TP, Murphy SA, Burdick KE. The engagement problem: a review of engagement with digital mental health interventions and recommendations for a path forward. Curr Treat Options Psych. Aug 25, 2023;10 (3):119-135. [ CrossRef ]

Abbreviations

Edited by T de Azevedo Cardoso; submitted 25.04.23; peer-reviewed by X B, Y Xi, E Kim, C Muñoz; comments to author 24.07.23; revised version received 21.09.23; accepted 04.12.23; published 27.02.24.

©Ahmad Ishqi Jabir, Xiaowen Lin, Laura Martinengo, Gemma Sharp, Yin-Leng Theng, Lorainne Tudor Car. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 27.02.2024.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.

IMAGES

  1. How To Write A Literature Review

    importance of review related literature in research study

  2. Importance of Literature Reviews & Writing Tips by IsEssay Writing

    importance of review related literature in research study

  3. Why is it important to do a literature review in research?

    importance of review related literature in research study

  4. 10 Easy Steps: How to Write a Literature Review in a Research Proposal

    importance of review related literature in research study

  5. Role Of Literature Review In Qualitative Research

    importance of review related literature in research study

  6. benefits of literature review for a research worker

    importance of review related literature in research study

VIDEO

  1. Research Methods: Writing a Literature Review

  2. Review of Related Literature and Studies Part 1

  3. Writing Research Proposal

  4. What is Literature Review?

  5. Doing Review Related Literature and Studies , and Conceptual Framework

  6. Importance of literature review in research 2024

COMMENTS

  1. What is the importance of a review of related literature in the study

    A review of related - and preferably recent - literature is meant to set your research in the context of what is currently known about the topic and to establish that what you have to offer is novel, something different from what has been already attempted.

  2. Why is it important to do a literature review in research?

    Importance of Literature Review in Research The aim of any literature review is to summarize and synthesize the arguments and ideas of existing knowledge in a particular field without adding any new contributions. Being built on existing knowledge they help the researcher to even turn the wheels of the topic of research.

  3. Conducting a Literature Review: Why Do A Literature Review?

    Besides the obvious reason for students -- because it is assigned! -- a literature review helps you explore the research that has come before you, to see how your research question has (or has not) already been addressed. You identify: core research in the field experts in the subject area methodology you may want to use (or avoid)

  4. The Literature Review: A Foundation for High-Quality Medical Education

    A literature review forms the basis for high-quality medical education research and helps maximize relevance, originality, generalizability, and impact. A literature review provides context, informs methodology, maximizes innovation, avoids duplicative research, and ensures that professional standards are met.

  5. Literature review as a research methodology: An ...

    In addition, a literature review is an excellent way of synthesizing research findings to show evidence on a meta-level and to uncover areas in which more research is needed, which is a critical component of creating theoretical frameworks and building conceptual models.

  6. Approaching literature review for academic purposes: The Literature

    A sophisticated literature review (LR) can result in a robust dissertation/thesis by scrutinizing the main problem examined by the academic study; anticipating research hypotheses, methods and results; and maintaining the interest of the audience in how the dissertation/thesis will provide solutions for the current gaps in a particular field.

  7. How does the review of related literature (RRL) help the ...

    A review of related literature (RRL) is important for obtaining an overview of the current knowledge on the topic. It provides the investigator with a framework on which to build an appropriate hypothesis. Further, an RRL guides the researcher in the direction of adding something new to the field without duplicating previous efforts.

  8. Steps in Conducting a Literature Review

    A literature review is an integrated analysis-- not just a summary-- of scholarly writings and other relevant evidence related directly to your research question. That is, it represents a synthesis of the evidence that provides background information on your topic and shows a association between the evidence and your research question.

  9. How to Write a Literature Review

    A literature review is a survey of scholarly sources on a specific topic. It provides an overview of current knowledge, allowing you to identify relevant theories, methods, and gaps in the existing research that you can later apply to your paper, thesis, or dissertation topic. There are five key steps to writing a literature review:

  10. Literature Reviews, Theoretical Frameworks, and Conceptual Frameworks

    The first element we discuss is a review of research (literature reviews), which highlights the need for a specific research question, study problem, or topic of investigation. ... In stating one's beliefs and understandings related to the study (e.g., students construct their knowledge, instructional materials support learning), an ...

  11. Writing a literature review

    Writing a literature review requires a range of skills to gather, sort, evaluate and summarise peer-reviewed published data into a relevant and informative unbiased narrative. Digital access to research papers, academic texts, review articles, reference databases and public data sets are all sources of information that are available to enrich ...

  12. Literature Review in Research Writing

    Why are literature reviews important? Research on research? If you find this idea rather peculiar, know that nowadays, with the huge amount of information produced daily all around the world, it is becoming more and more difficult to keep up to date with all of it. In addition to the sheer amount of research, there is also its origin.

  13. Literature Review: The What, Why and How-to Guide

    Narrative review: The purpose of this type of review is to describe the current state of the research on a specific topic/research and to offer a critical analysis of the literature reviewed. Studies are grouped by research/theoretical categories, and themes and trends, strengths and weakness, and gaps are identified.

  14. Literature Review Research

    Literature Review is a comprehensive survey of the works published in a particular field of study or line of research, usually over a specific period of time, in the form of an in-depth, critical bibliographic essay or annotated list in which attention is drawn to the most significant works.. Also, we can define a literature review as the collected body of scholarly works related to a topic:

  15. PDF Literature Review and Focusing the Research

    Literature reviews are important as research tools, especially in emerging areas, with populations that typically yield small samples (e.g., special education research ... Use of the literature review to plan and conduct a study requires that you critically evaluate the research that you read. This critical analysis can form the basis for your

  16. 5. The Literature Review

    A literature review may consist of simply a summary of key sources, but in the social sciences, a literature review usually has an organizational pattern and combines both summary and synthesis, often within specific conceptual categories.A summary is a recap of the important information of the source, but a synthesis is a re-organization, or a reshuffling, of that information in a way that ...

  17. Purpose of a Literature Review

    The purpose of a literature review is to: Provide a foundation of knowledge on a topic Identify areas of prior scholarship to prevent duplication and give credit to other researchers Identify inconstancies: gaps in research, conflicts in previous studies, open questions left from other research

  18. How to Undertake an Impactful Literature Review: Understanding Review

    Important aspects of a systematic literature review (SLR) include a structured method for conducting the study and significant transparency of the approaches used for summarizing the literature (Hiebl, 2023).The inspection of existing scientific literature is a valuable tool for (a) developing best practices and (b) resolving issues or controversies over a single study (Gupta et al., 2018).

  19. How to Write Review of Related Literature (RRL) in Research

    Tips on how to write a review of related literature in research. Given that you will probably need to produce a number of these at some point, here are a few general tips on how to write an effective review of related literature 2. Define your topic, audience, and purpose: You will be spending a lot of time with this review, so choose a topic ...

  20. Reviewing literature for research: Doing it the right way

    Review of research literature can be summarized into a seven step process: (i) Selecting research questions/purpose of the literature review (ii) Selecting your sources (iii) Choosing search terms (iv) Running your search (v) Applying practical screening criteria (vi) Applying methodological screening criteria/quality appraisal (vii) Synthesizin...

  21. Importance and Issues of Literature Review in Research

    (PDF) Importance and Issues of Literature Review in Research Importance and Issues of Literature Review in Research November 2020 DOI: Conference: 'Tackle a literature review'...

  22. Role of the Literature Review

    Your literature review gives readers an understanding of the scholarly research on your topic. In your literature review you will: demonstrate that you are a well-informed scholar with expertise and knowledge in the field by giving an overview of the current state of the literature

  23. PDF Literature Review: An Overview

    The major purpose of reviewing the literature is to determine what has already been done that relates to your topic, This knowledge not only prevents you from unintentionally duplicating another person's research, it also gives you the understanding and insight you need to place your topic within a logical frame.

  24. Full article: Engaging the front line: The important role of

    The first and possibly most important is collaboration between practitioners and academics to publish real world studies in academic literature (Foote et al., Citation 2023). Under this approach, a first step could be academics reviewing practitioners' completed project reports and together reformatting and submitting as case studies or ...

  25. Implementation strategies in suicide prevention: a scoping review

    Implementation strategies can be a vital leveraging point for enhancing the implementation and dissemination of evidence-based suicide prevention interventions and programming. However, much remains unknown about which implementation strategies are commonly used and effective for supporting suicide prevention efforts. In light of the limited available literature, a scoping review was conducted ...

  26. Physical activity improves stress load, recovery, and academic

    Physical activity has been proven to be beneficial for physical and psychological health as well as for academic achievement. However, especially university students are insufficiently physically active because of difficulties in time management regarding study, work, and social demands. As they are at a crucial life stage, it is of interest how physical activity affects university students ...

  27. Reducing the equity gap in under-5 mortality through an innovative

    Data sources Desk review. The study team undertook a review of available sources including peer-reviewed and grey literature and program documents focusing on the rates and progress of U5M and implementation of the EBIs known to reduce amenable U5M in countries (Table 1).Initial searches were performed through MEDLINE (PubMed) and Google Scholar using the search terms "child mortality" or ...

  28. Understanding inherent influencing factors to digital health ...

    Extensive research has shown the potential value of digital health solutions and highlighted the importance of clinicians' adoption. As general practitioners (GPs) are patients' first point of ...

  29. Chapter 9 Methods for Literature Reviews

    9.3. Types of Review Articles and Brief Illustrations. EHealth researchers have at their disposal a number of approaches and methods for making sense out of existing literature, all with the purpose of casting current research findings into historical contexts or explaining contradictions that might exist among a set of primary research studies conducted on a particular topic.

  30. Journal of Medical Internet Research

    Objective: This review aims to estimate the overall and differential rates of attrition in CA-delivered mental health interventions (CA interventions), evaluate the impact of study design and intervention-related aspects on attrition, and describe study design features aimed at reducing or mitigating study attrition.