Research Basics: Secondary Research | Literature Review
- What Is Research?
- Types of Research
- Secondary Research | Literature Review
- Developing Your Topic
- Primary vs. Secondary Sources
- Evaluating Sources
- Responsible Conduct of Research
What is Secondary Research?
Secondary research, also known as a literature review , preliminary research , historical research , background research , desk research , or library research , is research that analyzes or describes prior research. Rather than generating and analyzing new data, secondary research analyzes existing research results to establish the boundaries of knowledge on a topic, to identify trends or new practices, to test mathematical models or train machine learning systems, or to verify facts and figures. Secondary research is also used to justify the need for primary research as well as to justify and support other activities. For example, secondary research may be used to support a proposal to modernize a manufacturing plant, to justify the use of newly a developed treatment for cancer, to strengthen a business proposal, or to validate points made in a speech.
The following guides, published by the library, offer more information on how to do secondary research or a literature review:
- << Previous: Types of Research
- Next: Developing Your Topic >>
- Last Updated: Jul 28, 2023 3:41 PM
- URL: https://guides.library.iit.edu/research_basics
Literature Review Research
Literature review, types of literature reviews.
- Finding information
- Additional Resources
- Explains the background of research on a topic
- Demonstrates why a topic is significant to a subject area
- Helps focus your own research questions or problems
- Discovers relationships between research studies/ideas
- Suggests unexplored ideas or populations
- Identifies major themes, concepts, and researchers on a topic
- Tests assumptions; may help counter preconceived ideas and remove unconscious bias.
- Identifies critical gaps, points of disagreement, or potentially flawed methodology or theoretical approaches
Argumentative Review This form examines literature selectively in order to support or refute an argument, deeply imbedded assumption, or philosophical problem already established in the literature.
Integrative Review Considered a form of research that reviews, critiques, and synthesizes representative literature on a topic in an integrated way such that new frameworks and perspectives on the topic are generated. The body of literature includes all studies that address related or identical hypotheses. A well-done integrative review meets the same standards as primary research in regard to clarity, rigor, and replication.
Historical Review Historical reviews are focused on examining research throughout a period of time, often starting with the first time an issue, concept, theory, phenomena emerged in the literature, then tracing its evolution within the scholarship of a discipline. The purpose is to place research in a historical context to show familiarity with state-of-the-art developments and to identify the likely directions for future research.
Methodological Review This approach provides a framework of understanding at different levels (i.e. those of theory, substantive fields, research approaches and data collection and analysis techniques), enables researchers to draw on a wide variety of knowledge ranging from the conceptual level to practical documents for use in fieldwork.
Systematic Review Uses pre-specified and standardized methods to identify and critically appraise relevant research, and to collect, report, and analyze data from the studies that are included in the review. Typically it focuses on a very specific empirical question.
Examines the theory that has accumulated in regard to an issue, concept, theory, phenomena. Helps to establish what theories already exist, the relationships between them, to what degree the existing theories have been investigated, and to develop new hypotheses to be tested. Often this form is used to help establish a lack of appropriate theories or reveal that current theories are inadequate for explaining new or emerging research problems.
* Kennedy, Mary M. "Defining a Literature." Educational Researcher 36 (April 2007): 139-147.
All content in this section is from The Literature Review created by Dr. Robert Larabee USC
- Next: Finding information >>
- Last Updated: Oct 26, 2023 8:34 PM
- URL: https://guides.lib.odu.edu/literaturereview
- Getting Started
- Literature Review Research
- Research Design
- Research Design By Discipline
- SAGE Research Methods
- Teaching with SAGE Research Methods
- What is a Literature Review?
- What is NOT a Literature Review?
- Purposes of a Literature Review
- Types of Literature Reviews
- Literature Reviews vs. Systematic Reviews
- Systematic vs. Meta-Analysis
Literature Review is a comprehensive survey of the works published in a particular field of study or line of research, usually over a specific period of time, in the form of an in-depth, critical bibliographic essay or annotated list in which attention is drawn to the most significant works.
Also, we can define a literature review as the collected body of scholarly works related to a topic:
- Summarizes and analyzes previous research relevant to a topic
- Includes scholarly books and articles published in academic journals
- Can be an specific scholarly paper or a section in a research paper
The objective of a Literature Review is to find previous published scholarly works relevant to an specific topic
- Help gather ideas or information
- Keep up to date in current trends and findings
- Help develop new questions
A literature review is important because it:
- Explains the background of research on a topic.
- Demonstrates why a topic is significant to a subject area.
- Helps focus your own research questions or problems
- Discovers relationships between research studies/ideas.
- Suggests unexplored ideas or populations
- Identifies major themes, concepts, and researchers on a topic.
- Tests assumptions; may help counter preconceived ideas and remove unconscious bias.
- Identifies critical gaps, points of disagreement, or potentially flawed methodology or theoretical approaches.
- Indicates potential directions for future research.
All content in this section is from Literature Review Research from Old Dominion University
Keep in mind the following, a literature review is NOT:
Not an essay
Not an annotated bibliography in which you summarize each article that you have reviewed. A literature review goes beyond basic summarizing to focus on the critical analysis of the reviewed works and their relationship to your research question.
Not a research paper where you select resources to support one side of an issue versus another. A lit review should explain and consider all sides of an argument in order to avoid bias, and areas of agreement and disagreement should be highlighted.
A literature review serves several purposes. For example, it
- provides thorough knowledge of previous studies; introduces seminal works.
- helps focus one’s own research topic.
- identifies a conceptual framework for one’s own research questions or problems; indicates potential directions for future research.
- suggests previously unused or underused methodologies, designs, quantitative and qualitative strategies.
- identifies gaps in previous studies; identifies flawed methodologies and/or theoretical approaches; avoids replication of mistakes.
- helps the researcher avoid repetition of earlier research.
- suggests unexplored populations.
- determines whether past studies agree or disagree; identifies controversy in the literature.
- tests assumptions; may help counter preconceived ideas and remove unconscious bias.
As Kennedy (2007) notes*, it is important to think of knowledge in a given field as consisting of three layers. First, there are the primary studies that researchers conduct and publish. Second are the reviews of those studies that summarize and offer new interpretations built from and often extending beyond the original studies. Third, there are the perceptions, conclusions, opinion, and interpretations that are shared informally that become part of the lore of field. In composing a literature review, it is important to note that it is often this third layer of knowledge that is cited as "true" even though it often has only a loose relationship to the primary studies and secondary literature reviews.
Given this, while literature reviews are designed to provide an overview and synthesis of pertinent sources you have explored, there are several approaches to how they can be done, depending upon the type of analysis underpinning your study. Listed below are definitions of types of literature reviews:
Argumentative Review This form examines literature selectively in order to support or refute an argument, deeply imbedded assumption, or philosophical problem already established in the literature. The purpose is to develop a body of literature that establishes a contrarian viewpoint. Given the value-laden nature of some social science research [e.g., educational reform; immigration control], argumentative approaches to analyzing the literature can be a legitimate and important form of discourse. However, note that they can also introduce problems of bias when they are used to to make summary claims of the sort found in systematic reviews.
Integrative Review Considered a form of research that reviews, critiques, and synthesizes representative literature on a topic in an integrated way such that new frameworks and perspectives on the topic are generated. The body of literature includes all studies that address related or identical hypotheses. A well-done integrative review meets the same standards as primary research in regard to clarity, rigor, and replication.
Historical Review Few things rest in isolation from historical precedent. Historical reviews are focused on examining research throughout a period of time, often starting with the first time an issue, concept, theory, phenomena emerged in the literature, then tracing its evolution within the scholarship of a discipline. The purpose is to place research in a historical context to show familiarity with state-of-the-art developments and to identify the likely directions for future research.
Methodological Review A review does not always focus on what someone said [content], but how they said it [method of analysis]. This approach provides a framework of understanding at different levels (i.e. those of theory, substantive fields, research approaches and data collection and analysis techniques), enables researchers to draw on a wide variety of knowledge ranging from the conceptual level to practical documents for use in fieldwork in the areas of ontological and epistemological consideration, quantitative and qualitative integration, sampling, interviewing, data collection and data analysis, and helps highlight many ethical issues which we should be aware of and consider as we go through our study.
Systematic Review This form consists of an overview of existing evidence pertinent to a clearly formulated research question, which uses pre-specified and standardized methods to identify and critically appraise relevant research, and to collect, report, and analyse data from the studies that are included in the review. Typically it focuses on a very specific empirical question, often posed in a cause-and-effect form, such as "To what extent does A contribute to B?"
Theoretical Review The purpose of this form is to concretely examine the corpus of theory that has accumulated in regard to an issue, concept, theory, phenomena. The theoretical literature review help establish what theories already exist, the relationships between them, to what degree the existing theories have been investigated, and to develop new hypotheses to be tested. Often this form is used to help establish a lack of appropriate theories or reveal that current theories are inadequate for explaining new or emerging research problems. The unit of analysis can focus on a theoretical concept or a whole theory or framework.
* Kennedy, Mary M. "Defining a Literature." Educational Researcher 36 (April 2007): 139-147.
All content in this section is from The Literature Review created by Dr. Robert Larabee USC
Robinson, P. and Lowe, J. (2015), Literature reviews vs systematic reviews. Australian and New Zealand Journal of Public Health, 39: 103-103. doi: 10.1111/1753-6405.12393
What's in the name? The difference between a Systematic Review and a Literature Review, and why it matters . By Lynn Kysh from University of Southern California
Systematic review or meta-analysis?
A systematic review answers a defined research question by collecting and summarizing all empirical evidence that fits pre-specified eligibility criteria.
A meta-analysis is the use of statistical methods to summarize the results of these studies.
Systematic reviews, just like other research articles, can be of varying quality. They are a significant piece of work (the Centre for Reviews and Dissemination at York estimates that a team will take 9-24 months), and to be useful to other researchers and practitioners they should have:
- clearly stated objectives with pre-defined eligibility criteria for studies
- explicit, reproducible methodology
- a systematic search that attempts to identify all studies
- assessment of the validity of the findings of the included studies (e.g. risk of bias)
- systematic presentation, and synthesis, of the characteristics and findings of the included studies
Not all systematic reviews contain meta-analysis.
Meta-analysis is the use of statistical methods to summarize the results of independent studies. By combining information from all relevant studies, meta-analysis can provide more precise estimates of the effects of health care than those derived from the individual studies included within a review. More information on meta-analyses can be found in Cochrane Handbook, Chapter 9 .
A meta-analysis goes beyond critique and integration and conducts secondary statistical analysis on the outcomes of similar studies. It is a systematic review that uses quantitative methods to synthesize and summarize the results.
An advantage of a meta-analysis is the ability to be completely objective in evaluating research findings. Not all topics, however, have sufficient research evidence to allow a meta-analysis to be conducted. In that case, an integrative review is an appropriate strategy.
Some of the content in this section is from Systematic reviews and meta-analyses: step by step guide created by Kate McAllister.
- << Previous: Getting Started
- Next: Research Design >>
- Last Updated: Aug 21, 2023 4:07 PM
- URL: https://guides.lib.udel.edu/researchmethods
- Locations and Hours
- UCLA Library
- Research Guides
- Biomedical Library Guides
- Types of Literature Reviews
What Makes a Systematic Review Different from Other Types of Reviews?
- Planning Your Systematic Review
- Database Searching
- Creating the Search
- Search Filters & Hedges
- Grey Literature
- Managing & Appraising Results
- Further Resources
Reproduced from Grant, M. J. and Booth, A. (2009), A typology of reviews: an analysis of 14 review types and associated methodologies. Health Information & Libraries Journal, 26: 91–108. doi:10.1111/j.1471-1842.2009.00848.x
- << Previous: Home
- Next: Planning Your Systematic Review >>
- Last Updated: Sep 12, 2023 2:52 PM
- URL: https://guides.library.ucla.edu/systematicreviews
- Library Guides
The Literature Review
Primary and secondary sources, the literature review: primary and secondary sources.
- Searching the literature
- Grey literature
- Organising and analysing
- Systematic Reviews
- The Literature Review Toolbox
On this page
- Primary vs secondary sources: The differences explained
Can something be both a primary and secondary source?
Research for your literature review can be categorised as either primary or secondary in nature. The simplest definition of primary sources is either original information (such as survey data) or a first person account of an event (such as an interview transcript). Whereas secondary sources are any publshed or unpublished works that describe, summarise, analyse, evaluate, interpret or review primary source materials. Secondary sources can incorporate primary sources to support their arguments.
Ideally, good research should use a combination of both primary and secondary sources. For example, if a researcher were to investigate the introduction of a law and the impacts it had on a community, he/she might look at the transcripts of the parliamentary debates as well as the parliamentary commentary and news reporting surrounding the laws at the time.
Examples of primary and secondary sources
Primary vs secondary sources: The differences explained
Finding primary sources
- VU Special Collections - The Special Collections at Victoria University Library are a valuable research resource. The Collections have strong threads of radical literature, particularly Australian Communist literature, much of which is rare or unique. Women and urban planning also feature across the Collections. There are collections that give you a picture of the people who donated them like Ray Verrills, John McLaren, Sir Zelman Cowen, and Ruth & Maurie Crow. Other collections focus on Australia's neighbours – PNG and Timor-Leste.
- POLICY - Sharing the latest in policy knowledge and evidence, this database supports enhanced learning, collaboration and contribution.
- Indigenous Australia - The Indigenous Australia database represents the collections of the Aboriginal and Torres Strait Islander Commission Library.
- Australian Heritage Bibliography - Aboriginal and Torres Strait Islander Subset (AHB-ATSIS) - AHB is a bibliographic database that indexes and abstracts articles from published and unpublished material on Australia's natural and cultural environment. The AHB-ATSIS subset contains records that specifically relate to the Aboriginal and Torres Strait Islander peoples.include journal articles, unpublished reports, books, videos and conference proceedings from many different sources around Australia. Emphasis is placed on reports written or commissioned by government and non-government heritage agencies throughout the country.
- ATSIhealth - The Aboriginal and Torres Strait Islander Health Bibliography (ATSIhealth), compiled by Neil Thomson and Natalie Weissofner at the School of Indigenous Australian Studies, Kurongkurl Katitjin, Edith Cowan University, is a bibliographic database that indexes published and unpublished material on Australian Indigenous health. Source documents include theses, unpublished articles, government reports, conference papers, abstracts, book chapters, books, discussion and working papers, and statistical documents.
- National Archive of Australia - The National Archives of Australia holds the memory of our nation and keeps vital Australian Government records safe.
- National Library of Australia: Manuscripts - Manuscripts collection that is wide ranging and provides rich evidence of the lives and activities of Australians who have shaped our society.
- National Library of Australia: Printed ephemera - The National Library has been selectively collecting Australian printed ephemera since the early 1960s as a record of Australian life and social customs, popular culture, national events, and issues of national concern.
- National Library of Australia: Oral history and folklore - The Library’s Oral History and Folklore Collection dates back to the 1950’s and includes a rich and diverse collection of interviews and recordings with Australians from all walks of life.
- Historic Hansard - Commonwealth of Australia parliamentary debates presented in an easy-to-read format for historians and other lovers of political speech.
- The Old Bailey Online - A fully searchable edition of the largest body of texts detailing the lives of non-elite people ever published, containing 197,745 criminal trials held at London's central criminal court.
- British Library Sounds - Listen to a selection from the British Library’s extensive collections of unique sound recordings, which come from all over the world and cover the entire range of recorded sound: music, drama and literature, oral history, wildlife and environmental sounds.
Whether or not a source can be considered both primary and secondary, depends on the context. In some instances, material may act as a secondary source for one research area, and as a primary source for another. For example, Niccolò Machiavelli’s The Prince , published in 1513, is an important secondary source for any study of the various Renaissance princes in the Medici family; but the same book is also a primary source for the political thought that was characteristic of the sixteenth century because it reflects the attitudes of a person living in the 1500s.
Source: Craver, 1999, as cited in University of South Australia Library. (2021, Oct 6). Can something be a primary and secondary source?. University of South Australia Library. https://guides.library.unisa.edu.au/historycultural/sourcetypes
- << Previous: Overview
- Next: Searching the literature >>
- Last Updated: Aug 9, 2023 1:03 PM
- URL: https://libraryguides.vu.edu.au/the-literature-review
Understanding research and critical appraisal
What is secondary research?
Secondary research study designs.
- Primary research
- Critical appraisal of research papers
- Useful terminology
- Further reading and helpful resources
The aim of secondary research is to produce a more or less systematic appraisal and/or synthesis of the existing primary research on a topic. There are numerous types of reviews which aim to summarise or synthesise the evidence on a topic, but here we will focus on two: meta-analyses and systematic reviews.
For a fuller discussion of the range of review types, their features and uses, see: Sutton, A. et al . (2019) 'Meeting the review family: exploring review types and associated information retrieval requirements', Health Information and Libraries Journal , 36 (3), pp. 202-222. doi:10.1111/hir.12276
A meta-analysis is a statistical synthesis of the results from multiple individual studies, usually randomised controlled trials (RCTs),
Carrying out a meta-analysis of studies allows results from multiple studies looking at the effect of an intervention to be combined, allowing for greater precision in the estimation of effects, and clarity over the direction and size of an effect. A meta-analysis can provide more conclusive evidence for or against the effectiveness of an intervention than individual studies alone.
A good meta-analysis should always be based on a systematic review of studies, and requires some homogeneity of participants, settings, interventions and outcome measures in the studies included.
A systematic review is not simply a literature review. A systematic review is a study which aims to synthesise all of the available primary research on a specific topic. The first step in a systematic review is a thorough search of all appropriate sources, including subject related databases, clinical trial registers and grey literature, in order to identify all of the relevant evidence. These searches should ideally be carried out by a librarian or information specialist in the field, or by others with a similar level of expertise. The systematic review itself should be carried out by two or more researchers, as a means of reducing possible bias.
All identified studies are screened for inclusion or exclusion according to strict criteria set out at the start of the study, and the data from those studies selected for inclusion is analysed and synthesised. Part of this process is an attempt to identify any potential source of bias in existing findings. A systematic review will offer a summary of the available research findings, and offer conclusions on the basis of these, taking into account any flaws or limitations in the original studies.
A systematic review can offer more generalisability and consistency of research findings than the individual studies on which it is based.
Systematic reviews may employ quantitative, qualitative (experiential), or mixed-methods approaches.
- << Previous: Introduction
- Next: Primary research >>
- Last Updated: May 19, 2023 10:51 AM
- URL: https://libguides.sgul.ac.uk/researchdesign
Literature Reviews: Types of Literature
- Library Basics
- 1. Choose Your Topic
- How to Find Books
- Types of Clinical Study Designs
Types of Literature
- 3. Search the Literature
- 4. Read & Analyze the Literature
- 5. Write the Review
- Keeping Track of Information
- Style Guides
- Books, Tutorials & Examples
Different types of publications have different characteristics.
Primary Literature Primary sources means original studies, based on direct observation, use of statistical records, interviews, or experimental methods, of actual practices or the actual impact of practices or policies. They are authored by researchers, contains original research data, and are usually published in a peer-reviewed journal. Primary literature may also include conference papers, pre-prints, or preliminary reports. Also called empirical research .
Secondary Literature Secondary literature consists of interpretations and evaluations that are derived from or refer to the primary source literature. Examples include review articles (such as meta-analysis and systematic reviews) and reference works. Professionals within each discipline take the primary literature and synthesize, generalize, and integrate new research.
Tertiary Literature Tertiary literature consists of a distillation and collection of primary and secondary sources such as textbooks, encyclopedia articles, and guidebooks or handbooks. The purpose of tertiary literature is to provide an overview of key research findings and an introduction to principles and practices within the discipline.
Adapted from the Information Services Department of the Library of the Health Sciences-Chicago , University of Illinois at Chicago.
Types of Scientific Publications
These examples and descriptions of publication types will give you an idea of how to use various works and why you would want to write a particular kind of paper.
- Scholarly article aka empirical article
- Review article
- Conference paper
Scholarly (aka empirical) article -- example
Empirical studies use data derived from observation or experiment. Original research papers (also called primary research articles) that describe empirical studies and their results are published in academic journals. Articles that report empirical research contain different sections which relate to the steps of the scientific method.
Abstract - The abstract provides a very brief summary of the research.
Introduction - The introduction sets the research in a context, which provides a review of related research and develops the hypotheses for the research.
Method - The method section describes how the research was conducted.
Results - The results section describes the outcomes of the study.
Discussion - The discussion section contains the interpretations and implications of the study.
References - A references section lists the articles, books, and other material cited in the report.
Review article -- example
A review article summarizes a particular field of study and places the recent research in context. It provides an overview and is an excellent introduction to a subject area. The references used in a review article are helpful as they lead to more in-depth research.
Many databases have limits or filters to search for review articles. You can also search by keywords like review article, survey, overview, summary, etc.
Conference proceedings, abstracts and reports -- example
Conference proceedings, abstracts and reports are not usually peer-reviewed. A conference article is similar to a scholarly article insofar as it is academic. Conference articles are published much more quickly than scholarly articles. You can find conference papers in many of the same places as scholarly articles.
How Do You Identify Empirical Articles?
To identify an article based on empirical research, look for the following characteristics:
The article is published in a peer-reviewed journal .
The article includes charts, graphs, or statistical analysis .
The article is substantial in size , likely to be more than 5 pages long.
The article contains the following parts (the exact terms may vary): abstract, introduction, method, results, discussion, references .
- << Previous: Types of Clinical Study Designs
- Next: 3. Search the Literature >>
- Last Updated: Apr 13, 2023 9:35 AM
- URL: https://research.library.gsu.edu/litrev
Chapter Six: Reviewing the Secondary Literature / Types of Literature Reviews / Reading Like a Researcher
Reviewing the Secondary Literature
Topics discussed on this page include:
What is the Purpose of the Review?
The annotated bibliography, the literature review, what do we mean by literature, what is the scope of the review.
The literature review provides your reader of an overview of the existing research about your topic or problem. Creating the literature review involves more than gathering citations. It is a qualitative process through which you will discover what is already known about your topic, and identify the key authorities, methods, and theoretical foundations so you can begin to position your contributions within the scholarly conversation.
Further, the literature review sharpens the focus of your research and demonstrates your knowledge and understanding of the scholarly conversation around your topic, which in turn helps establish your credibility as a researcher.
We suggest you begin putting your research together by making an annotated bibliography (or annotated list of sources ), then synthesize your research sources by looking for through lines in them (arguments, narratives, trends, etc.), then determine which type of literature review works best for your project (we discuss these types on the following page). To help you gather annotated materials in one place, we provide a matrix tool that helps you organize and synthesize your research. The annotated bibliography serves numerous purposes:
- It organizes your research findings in one place, and provides a handy reference while you are completing your research project.
- If you will be writing a literature review for your research project, compiling an annotated bibliography is a great first step.
- If you decide to include the annotated bibliography in your research project, it will allow readers to explore these sources on their own.
The annotated bibliography, unlike the literature review, does not need to be essayistic. To create an annotated bibliography, use either the matrix tool or write a separate paragraph for each entry. An annotated bibliography organizes sources alphabetically and explains not only a summary of each source, but also addresses the source’s credibility and explains its relevance to your research project. An example of an annotated bibliography , created by UCF student Dolores Batten, explains how her readings related to her research project (which was to develop methods for improving student writing).
Writing a literary studies research paper involves time and effort, with much of it going towards the development of a literature review . A literature review might fill several pages of your research paper and usually appears soon after an introduction and before you present your analysis. A literature review provides your audience with an overview of the available research about your area(s) of study, including the literary work, your theory, and methodology. The literature review demonstrates how these scholarly discussions have changed over time and it allows you to position your research in relation to research that has come before yours. Your aim is to narrate the discussion up to this point. Depending on the nature of the assignment, you may also include your critical commentary on prior research, noting among this material the weaker and stronger arguments, breakthroughs and dead ends, blind spots and opportunities, the invention of key terms and methods, mistakes as well as misreadings, and so on.
Once you have gathered the research materials you need for your literature review, you have yet another task in front of you: conducting an analysis on said research for your original contribution, which is the part where you discover and bring something new to the conversation. As the saying goes, “we are standing on the shoulders of giants.” Your job is to show a portrait to your audience of these giants and to show how your work relates to it.
Some beginning researchers try to tear down the work of other researchers in an effort to make their own work look good by comparison. It rarely works. First, it tends to make your audience justly skeptical of your claims. Second, it ignores the fact that even the mistakes, blind spots, and failures of other researchers contribute something to our knowledge. Albert Einstein didn’t trash Sir Isaac Newton by saying his theory of space was wrong and terrible and that his own theory was great by comparison. He built upon Newton’s work, showing how it could be improved. If, however, a researcher willfully set out to deceive others, then their work does not deserve such deference.
Before you begin work on your literature review, let’s discuss what we mean by “literature,” understand the purpose and scope of the review, establish criteria for selecting, organizing, and interpreting your findings, and, finally, discuss how to connect your findings to your research question.
When we use the word “literature” in the phrase “literature review,” we are not talking about literary writing such as novels, poems, and plays, but about scholarly research. Our objective is to tell the story of research up to the point when you add your own contribution. You should use this time to think about what types of information and resources you will need to complete your project. In the case of literary studies, we often start with peer-reviewed journal articles and scholarly monographs (books) that can be accessed through the library catalog and subject databases. These are both essential resources, but you may need more.
For Jada’s research project about James Baldwin’s ‘Sonny’s Blues,” we might also think about exploring newspapers and primary source collections related to civil rights, African American studies, and social activism. Other topics might require different types of media, data sets, case studies, etc.
More about searching for these sources will be discussed in the library resources portion. In the meantime, let’s break down the literature review a little further.
Defining the scope of your review will also help you establish criteria to determine the relevance of the sources you are finding. At this stage, you are not reading in-depth; you are taking snapshots of what has been published, identifying major concepts, theories, methodologies, and methods while identifying connections, tensions, and contradictions within what Michael Patton calls the “intellectual heritage” of your topic or problem.
This work involves building on the knowledge of others and understanding what methods, measures, and models we have inherited from previous researchers in our field.
For more about Dr. Patton’s thoughts on the literature review, watch this short video:
Video provided courtesy of the Center for Quality Research (CQR)
Before we take a look at types of reviews, here are some key Dos and Don’ts:
Strategies for Conducting Literary Research Copyright © 2021 by Barry Mauer & John Venecek is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License , except where otherwise noted.
Share This Book
An official website of the United States government
The .gov means it's official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you're on a federal government site.
The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.
- Account settings
- Browse Titles
NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.
Lau F, Kuziemsky C, editors. Handbook of eHealth Evaluation: An Evidence-based Approach [Internet]. Victoria (BC): University of Victoria; 2017 Feb 27.
Handbook of eHealth Evaluation: An Evidence-based Approach [Internet].
Chapter 9 methods for literature reviews.
Guy Paré and Spyros Kitsiou .
Literature reviews play a critical role in scholarship because science remains, first and foremost, a cumulative endeavour ( vom Brocke et al., 2009 ). As in any academic discipline, rigorous knowledge syntheses are becoming indispensable in keeping up with an exponentially growing eHealth literature, assisting practitioners, academics, and graduate students in finding, evaluating, and synthesizing the contents of many empirical and conceptual papers. Among other methods, literature reviews are essential for: (a) identifying what has been written on a subject or topic; (b) determining the extent to which a specific research area reveals any interpretable trends or patterns; (c) aggregating empirical findings related to a narrow research question to support evidence-based practice; (d) generating new frameworks and theories; and (e) identifying topics or questions requiring more investigation ( Paré, Trudel, Jaana, & Kitsiou, 2015 ).
Literature reviews can take two major forms. The most prevalent one is the “literature review” or “background” section within a journal paper or a chapter in a graduate thesis. This section synthesizes the extant literature and usually identifies the gaps in knowledge that the empirical study addresses ( Sylvester, Tate, & Johnstone, 2013 ). It may also provide a theoretical foundation for the proposed study, substantiate the presence of the research problem, justify the research as one that contributes something new to the cumulated knowledge, or validate the methods and approaches for the proposed study ( Hart, 1998 ; Levy & Ellis, 2006 ).
The second form of literature review, which is the focus of this chapter, constitutes an original and valuable work of research in and of itself ( Paré et al., 2015 ). Rather than providing a base for a researcher’s own work, it creates a solid starting point for all members of the community interested in a particular area or topic ( Mulrow, 1987 ). The so-called “review article” is a journal-length paper which has an overarching purpose to synthesize the literature in a field, without collecting or analyzing any primary data ( Green, Johnson, & Adams, 2006 ).
When appropriately conducted, review articles represent powerful information sources for practitioners looking for state-of-the art evidence to guide their decision-making and work practices ( Paré et al., 2015 ). Further, high-quality reviews become frequently cited pieces of work which researchers seek out as a first clear outline of the literature when undertaking empirical studies ( Cooper, 1988 ; Rowe, 2014 ). Scholars who track and gauge the impact of articles have found that review papers are cited and downloaded more often than any other type of published article ( Cronin, Ryan, & Coughlan, 2008 ; Montori, Wilczynski, Morgan, Haynes, & Hedges, 2003 ; Patsopoulos, Analatos, & Ioannidis, 2005 ). The reason for their popularity may be the fact that reading the review enables one to have an overview, if not a detailed knowledge of the area in question, as well as references to the most useful primary sources ( Cronin et al., 2008 ). Although they are not easy to conduct, the commitment to complete a review article provides a tremendous service to one’s academic community ( Paré et al., 2015 ; Petticrew & Roberts, 2006 ). Most, if not all, peer-reviewed journals in the fields of medical informatics publish review articles of some type.
The main objectives of this chapter are fourfold: (a) to provide an overview of the major steps and activities involved in conducting a stand-alone literature review; (b) to describe and contrast the different types of review articles that can contribute to the eHealth knowledge base; (c) to illustrate each review type with one or two examples from the eHealth literature; and (d) to provide a series of recommendations for prospective authors of review articles in this domain.
9.2. Overview of the Literature Review Process and Steps
As explained in Templier and Paré (2015) , there are six generic steps involved in conducting a review article:
- formulating the research question(s) and objective(s),
- searching the extant literature,
- screening for inclusion,
- assessing the quality of primary studies,
- extracting data, and
- analyzing data.
Although these steps are presented here in sequential order, one must keep in mind that the review process can be iterative and that many activities can be initiated during the planning stage and later refined during subsequent phases ( Finfgeld-Connett & Johnson, 2013 ; Kitchenham & Charters, 2007 ).
Formulating the research question(s) and objective(s): As a first step, members of the review team must appropriately justify the need for the review itself ( Petticrew & Roberts, 2006 ), identify the review’s main objective(s) ( Okoli & Schabram, 2010 ), and define the concepts or variables at the heart of their synthesis ( Cooper & Hedges, 2009 ; Webster & Watson, 2002 ). Importantly, they also need to articulate the research question(s) they propose to investigate ( Kitchenham & Charters, 2007 ). In this regard, we concur with Jesson, Matheson, and Lacey (2011) that clearly articulated research questions are key ingredients that guide the entire review methodology; they underscore the type of information that is needed, inform the search for and selection of relevant literature, and guide or orient the subsequent analysis. Searching the extant literature: The next step consists of searching the literature and making decisions about the suitability of material to be considered in the review ( Cooper, 1988 ). There exist three main coverage strategies. First, exhaustive coverage means an effort is made to be as comprehensive as possible in order to ensure that all relevant studies, published and unpublished, are included in the review and, thus, conclusions are based on this all-inclusive knowledge base. The second type of coverage consists of presenting materials that are representative of most other works in a given field or area. Often authors who adopt this strategy will search for relevant articles in a small number of top-tier journals in a field ( Paré et al., 2015 ). In the third strategy, the review team concentrates on prior works that have been central or pivotal to a particular topic. This may include empirical studies or conceptual papers that initiated a line of investigation, changed how problems or questions were framed, introduced new methods or concepts, or engendered important debate ( Cooper, 1988 ). Screening for inclusion: The following step consists of evaluating the applicability of the material identified in the preceding step ( Levy & Ellis, 2006 ; vom Brocke et al., 2009 ). Once a group of potential studies has been identified, members of the review team must screen them to determine their relevance ( Petticrew & Roberts, 2006 ). A set of predetermined rules provides a basis for including or excluding certain studies. This exercise requires a significant investment on the part of researchers, who must ensure enhanced objectivity and avoid biases or mistakes. As discussed later in this chapter, for certain types of reviews there must be at least two independent reviewers involved in the screening process and a procedure to resolve disagreements must also be in place ( Liberati et al., 2009 ; Shea et al., 2009 ). Assessing the quality of primary studies: In addition to screening material for inclusion, members of the review team may need to assess the scientific quality of the selected studies, that is, appraise the rigour of the research design and methods. Such formal assessment, which is usually conducted independently by at least two coders, helps members of the review team refine which studies to include in the final sample, determine whether or not the differences in quality may affect their conclusions, or guide how they analyze the data and interpret the findings ( Petticrew & Roberts, 2006 ). Ascribing quality scores to each primary study or considering through domain-based evaluations which study components have or have not been designed and executed appropriately makes it possible to reflect on the extent to which the selected study addresses possible biases and maximizes validity ( Shea et al., 2009 ). Extracting data: The following step involves gathering or extracting applicable information from each primary study included in the sample and deciding what is relevant to the problem of interest ( Cooper & Hedges, 2009 ). Indeed, the type of data that should be recorded mainly depends on the initial research questions ( Okoli & Schabram, 2010 ). However, important information may also be gathered about how, when, where and by whom the primary study was conducted, the research design and methods, or qualitative/quantitative results ( Cooper & Hedges, 2009 ). Analyzing and synthesizing data : As a final step, members of the review team must collate, summarize, aggregate, organize, and compare the evidence extracted from the included studies. The extracted data must be presented in a meaningful way that suggests a new contribution to the extant literature ( Jesson et al., 2011 ). Webster and Watson (2002) warn researchers that literature reviews should be much more than lists of papers and should provide a coherent lens to make sense of extant knowledge on a given topic. There exist several methods and techniques for synthesizing quantitative (e.g., frequency analysis, meta-analysis) and qualitative (e.g., grounded theory, narrative analysis, meta-ethnography) evidence ( Dixon-Woods, Agarwal, Jones, Young, & Sutton, 2005 ; Thomas & Harden, 2008 ).
9.3. Types of Review Articles and Brief Illustrations
EHealth researchers have at their disposal a number of approaches and methods for making sense out of existing literature, all with the purpose of casting current research findings into historical contexts or explaining contradictions that might exist among a set of primary research studies conducted on a particular topic. Our classification scheme is largely inspired from Paré and colleagues’ (2015) typology. Below we present and illustrate those review types that we feel are central to the growth and development of the eHealth domain.
9.3.1. Narrative Reviews
The narrative review is the “traditional” way of reviewing the extant literature and is skewed towards a qualitative interpretation of prior knowledge ( Sylvester et al., 2013 ). Put simply, a narrative review attempts to summarize or synthesize what has been written on a particular topic but does not seek generalization or cumulative knowledge from what is reviewed ( Davies, 2000 ; Green et al., 2006 ). Instead, the review team often undertakes the task of accumulating and synthesizing the literature to demonstrate the value of a particular point of view ( Baumeister & Leary, 1997 ). As such, reviewers may selectively ignore or limit the attention paid to certain studies in order to make a point. In this rather unsystematic approach, the selection of information from primary articles is subjective, lacks explicit criteria for inclusion and can lead to biased interpretations or inferences ( Green et al., 2006 ). There are several narrative reviews in the particular eHealth domain, as in all fields, which follow such an unstructured approach ( Silva et al., 2015 ; Paul et al., 2015 ).
Despite these criticisms, this type of review can be very useful in gathering together a volume of literature in a specific subject area and synthesizing it. As mentioned above, its primary purpose is to provide the reader with a comprehensive background for understanding current knowledge and highlighting the significance of new research ( Cronin et al., 2008 ). Faculty like to use narrative reviews in the classroom because they are often more up to date than textbooks, provide a single source for students to reference, and expose students to peer-reviewed literature ( Green et al., 2006 ). For researchers, narrative reviews can inspire research ideas by identifying gaps or inconsistencies in a body of knowledge, thus helping researchers to determine research questions or formulate hypotheses. Importantly, narrative reviews can also be used as educational articles to bring practitioners up to date with certain topics of issues ( Green et al., 2006 ).
Recently, there have been several efforts to introduce more rigour in narrative reviews that will elucidate common pitfalls and bring changes into their publication standards. Information systems researchers, among others, have contributed to advancing knowledge on how to structure a “traditional” review. For instance, Levy and Ellis (2006) proposed a generic framework for conducting such reviews. Their model follows the systematic data processing approach comprised of three steps, namely: (a) literature search and screening; (b) data extraction and analysis; and (c) writing the literature review. They provide detailed and very helpful instructions on how to conduct each step of the review process. As another methodological contribution, vom Brocke et al. (2009) offered a series of guidelines for conducting literature reviews, with a particular focus on how to search and extract the relevant body of knowledge. Last, Bandara, Miskon, and Fielt (2011) proposed a structured, predefined and tool-supported method to identify primary studies within a feasible scope, extract relevant content from identified articles, synthesize and analyze the findings, and effectively write and present the results of the literature review. We highly recommend that prospective authors of narrative reviews consult these useful sources before embarking on their work.
Darlow and Wen (2015) provide a good example of a highly structured narrative review in the eHealth field. These authors synthesized published articles that describe the development process of mobile health ( m-health ) interventions for patients’ cancer care self-management. As in most narrative reviews, the scope of the research questions being investigated is broad: (a) how development of these systems are carried out; (b) which methods are used to investigate these systems; and (c) what conclusions can be drawn as a result of the development of these systems. To provide clear answers to these questions, a literature search was conducted on six electronic databases and Google Scholar . The search was performed using several terms and free text words, combining them in an appropriate manner. Four inclusion and three exclusion criteria were utilized during the screening process. Both authors independently reviewed each of the identified articles to determine eligibility and extract study information. A flow diagram shows the number of studies identified, screened, and included or excluded at each stage of study selection. In terms of contributions, this review provides a series of practical recommendations for m-health intervention development.
9.3.2. Descriptive or Mapping Reviews
The primary goal of a descriptive review is to determine the extent to which a body of knowledge in a particular research topic reveals any interpretable pattern or trend with respect to pre-existing propositions, theories, methodologies or findings ( King & He, 2005 ; Paré et al., 2015 ). In contrast with narrative reviews, descriptive reviews follow a systematic and transparent procedure, including searching, screening and classifying studies ( Petersen, Vakkalanka, & Kuzniarz, 2015 ). Indeed, structured search methods are used to form a representative sample of a larger group of published works ( Paré et al., 2015 ). Further, authors of descriptive reviews extract from each study certain characteristics of interest, such as publication year, research methods, data collection techniques, and direction or strength of research outcomes (e.g., positive, negative, or non-significant) in the form of frequency analysis to produce quantitative results ( Sylvester et al., 2013 ). In essence, each study included in a descriptive review is treated as the unit of analysis and the published literature as a whole provides a database from which the authors attempt to identify any interpretable trends or draw overall conclusions about the merits of existing conceptualizations, propositions, methods or findings ( Paré et al., 2015 ). In doing so, a descriptive review may claim that its findings represent the state of the art in a particular domain ( King & He, 2005 ).
In the fields of health sciences and medical informatics, reviews that focus on examining the range, nature and evolution of a topic area are described by Anderson, Allen, Peckham, and Goodwin (2008) as mapping reviews . Like descriptive reviews, the research questions are generic and usually relate to publication patterns and trends. There is no preconceived plan to systematically review all of the literature although this can be done. Instead, researchers often present studies that are representative of most works published in a particular area and they consider a specific time frame to be mapped.
An example of this approach in the eHealth domain is offered by DeShazo, Lavallie, and Wolf (2009). The purpose of this descriptive or mapping review was to characterize publication trends in the medical informatics literature over a 20-year period (1987 to 2006). To achieve this ambitious objective, the authors performed a bibliometric analysis of medical informatics citations indexed in medline using publication trends, journal frequencies, impact factors, Medical Subject Headings (MeSH) term frequencies, and characteristics of citations. Findings revealed that there were over 77,000 medical informatics articles published during the covered period in numerous journals and that the average annual growth rate was 12%. The MeSH term analysis also suggested a strong interdisciplinary trend. Finally, average impact scores increased over time with two notable growth periods. Overall, patterns in research outputs that seem to characterize the historic trends and current components of the field of medical informatics suggest it may be a maturing discipline (DeShazo et al., 2009).
9.3.3. Scoping Reviews
Scoping reviews attempt to provide an initial indication of the potential size and nature of the extant literature on an emergent topic (Arksey & O’Malley, 2005; Daudt, van Mossel, & Scott, 2013 ; Levac, Colquhoun, & O’Brien, 2010). A scoping review may be conducted to examine the extent, range and nature of research activities in a particular area, determine the value of undertaking a full systematic review (discussed next), or identify research gaps in the extant literature ( Paré et al., 2015 ). In line with their main objective, scoping reviews usually conclude with the presentation of a detailed research agenda for future works along with potential implications for both practice and research.
Unlike narrative and descriptive reviews, the whole point of scoping the field is to be as comprehensive as possible, including grey literature (Arksey & O’Malley, 2005). Inclusion and exclusion criteria must be established to help researchers eliminate studies that are not aligned with the research questions. It is also recommended that at least two independent coders review abstracts yielded from the search strategy and then the full articles for study selection ( Daudt et al., 2013 ). The synthesized evidence from content or thematic analysis is relatively easy to present in tabular form (Arksey & O’Malley, 2005; Thomas & Harden, 2008 ).
One of the most highly cited scoping reviews in the eHealth domain was published by Archer, Fevrier-Thomas, Lokker, McKibbon, and Straus (2011) . These authors reviewed the existing literature on personal health record ( phr ) systems including design, functionality, implementation, applications, outcomes, and benefits. Seven databases were searched from 1985 to March 2010. Several search terms relating to phr s were used during this process. Two authors independently screened titles and abstracts to determine inclusion status. A second screen of full-text articles, again by two independent members of the research team, ensured that the studies described phr s. All in all, 130 articles met the criteria and their data were extracted manually into a database. The authors concluded that although there is a large amount of survey, observational, cohort/panel, and anecdotal evidence of phr benefits and satisfaction for patients, more research is needed to evaluate the results of phr implementations. Their in-depth analysis of the literature signalled that there is little solid evidence from randomized controlled trials or other studies through the use of phr s. Hence, they suggested that more research is needed that addresses the current lack of understanding of optimal functionality and usability of these systems, and how they can play a beneficial role in supporting patient self-management ( Archer et al., 2011 ).
9.3.4. Forms of Aggregative Reviews
Healthcare providers, practitioners, and policy-makers are nowadays overwhelmed with large volumes of information, including research-based evidence from numerous clinical trials and evaluation studies, assessing the effectiveness of health information technologies and interventions ( Ammenwerth & de Keizer, 2004 ; Deshazo et al., 2009 ). It is unrealistic to expect that all these disparate actors will have the time, skills, and necessary resources to identify the available evidence in the area of their expertise and consider it when making decisions. Systematic reviews that involve the rigorous application of scientific strategies aimed at limiting subjectivity and bias (i.e., systematic and random errors) can respond to this challenge.
Systematic reviews attempt to aggregate, appraise, and synthesize in a single source all empirical evidence that meet a set of previously specified eligibility criteria in order to answer a clearly formulated and often narrow research question on a particular topic of interest to support evidence-based practice ( Liberati et al., 2009 ). They adhere closely to explicit scientific principles ( Liberati et al., 2009 ) and rigorous methodological guidelines (Higgins & Green, 2008) aimed at reducing random and systematic errors that can lead to deviations from the truth in results or inferences. The use of explicit methods allows systematic reviews to aggregate a large body of research evidence, assess whether effects or relationships are in the same direction and of the same general magnitude, explain possible inconsistencies between study results, and determine the strength of the overall evidence for every outcome of interest based on the quality of included studies and the general consistency among them ( Cook, Mulrow, & Haynes, 1997 ). The main procedures of a systematic review involve:
- Formulating a review question and developing a search strategy based on explicit inclusion criteria for the identification of eligible studies (usually described in the context of a detailed review protocol).
- Searching for eligible studies using multiple databases and information sources, including grey literature sources, without any language restrictions.
- Selecting studies, extracting data, and assessing risk of bias in a duplicate manner using two independent reviewers to avoid random or systematic errors in the process.
- Analyzing data using quantitative or qualitative methods.
- Presenting results in summary of findings tables.
- Interpreting results and drawing conclusions.
Many systematic reviews, but not all, use statistical methods to combine the results of independent studies into a single quantitative estimate or summary effect size. Known as meta-analyses , these reviews use specific data extraction and statistical techniques (e.g., network, frequentist, or Bayesian meta-analyses) to calculate from each study by outcome of interest an effect size along with a confidence interval that reflects the degree of uncertainty behind the point estimate of effect ( Borenstein, Hedges, Higgins, & Rothstein, 2009 ; Deeks, Higgins, & Altman, 2008 ). Subsequently, they use fixed or random-effects analysis models to combine the results of the included studies, assess statistical heterogeneity, and calculate a weighted average of the effect estimates from the different studies, taking into account their sample sizes. The summary effect size is a value that reflects the average magnitude of the intervention effect for a particular outcome of interest or, more generally, the strength of a relationship between two variables across all studies included in the systematic review. By statistically combining data from multiple studies, meta-analyses can create more precise and reliable estimates of intervention effects than those derived from individual studies alone, when these are examined independently as discrete sources of information.
The review by Gurol-Urganci, de Jongh, Vodopivec-Jamsek, Atun, and Car (2013) on the effects of mobile phone messaging reminders for attendance at healthcare appointments is an illustrative example of a high-quality systematic review with meta-analysis. Missed appointments are a major cause of inefficiency in healthcare delivery with substantial monetary costs to health systems. These authors sought to assess whether mobile phone-based appointment reminders delivered through Short Message Service ( sms ) or Multimedia Messaging Service ( mms ) are effective in improving rates of patient attendance and reducing overall costs. To this end, they conducted a comprehensive search on multiple databases using highly sensitive search strategies without language or publication-type restrictions to identify all rct s that are eligible for inclusion. In order to minimize the risk of omitting eligible studies not captured by the original search, they supplemented all electronic searches with manual screening of trial registers and references contained in the included studies. Study selection, data extraction, and risk of bias assessments were performed independently by two coders using standardized methods to ensure consistency and to eliminate potential errors. Findings from eight rct s involving 6,615 participants were pooled into meta-analyses to calculate the magnitude of effects that mobile text message reminders have on the rate of attendance at healthcare appointments compared to no reminders and phone call reminders.
Meta-analyses are regarded as powerful tools for deriving meaningful conclusions. However, there are situations in which it is neither reasonable nor appropriate to pool studies together using meta-analytic methods simply because there is extensive clinical heterogeneity between the included studies or variation in measurement tools, comparisons, or outcomes of interest. In these cases, systematic reviews can use qualitative synthesis methods such as vote counting, content analysis, classification schemes and tabulations, as an alternative approach to narratively synthesize the results of the independent studies included in the review. This form of review is known as qualitative systematic review.
A rigorous example of one such review in the eHealth domain is presented by Mickan, Atherton, Roberts, Heneghan, and Tilson (2014) on the use of handheld computers by healthcare professionals and their impact on access to information and clinical decision-making. In line with the methodological guidelines for systematic reviews, these authors: (a) developed and registered with prospero ( www.crd.york.ac.uk/ prospero / ) an a priori review protocol; (b) conducted comprehensive searches for eligible studies using multiple databases and other supplementary strategies (e.g., forward searches); and (c) subsequently carried out study selection, data extraction, and risk of bias assessments in a duplicate manner to eliminate potential errors in the review process. Heterogeneity between the included studies in terms of reported outcomes and measures precluded the use of meta-analytic methods. To this end, the authors resorted to using narrative analysis and synthesis to describe the effectiveness of handheld computers on accessing information for clinical knowledge, adherence to safety and clinical quality guidelines, and diagnostic decision-making.
In recent years, the number of systematic reviews in the field of health informatics has increased considerably. Systematic reviews with discordant findings can cause great confusion and make it difficult for decision-makers to interpret the review-level evidence ( Moher, 2013 ). Therefore, there is a growing need for appraisal and synthesis of prior systematic reviews to ensure that decision-making is constantly informed by the best available accumulated evidence. Umbrella reviews , also known as overviews of systematic reviews, are tertiary types of evidence synthesis that aim to accomplish this; that is, they aim to compare and contrast findings from multiple systematic reviews and meta-analyses ( Becker & Oxman, 2008 ). Umbrella reviews generally adhere to the same principles and rigorous methodological guidelines used in systematic reviews. However, the unit of analysis in umbrella reviews is the systematic review rather than the primary study ( Becker & Oxman, 2008 ). Unlike systematic reviews that have a narrow focus of inquiry, umbrella reviews focus on broader research topics for which there are several potential interventions ( Smith, Devane, Begley, & Clarke, 2011 ). A recent umbrella review on the effects of home telemonitoring interventions for patients with heart failure critically appraised, compared, and synthesized evidence from 15 systematic reviews to investigate which types of home telemonitoring technologies and forms of interventions are more effective in reducing mortality and hospital admissions ( Kitsiou, Paré, & Jaana, 2015 ).
9.3.5. Realist Reviews
Realist reviews are theory-driven interpretative reviews developed to inform, enhance, or supplement conventional systematic reviews by making sense of heterogeneous evidence about complex interventions applied in diverse contexts in a way that informs policy decision-making ( Greenhalgh, Wong, Westhorp, & Pawson, 2011 ). They originated from criticisms of positivist systematic reviews which centre on their “simplistic” underlying assumptions ( Oates, 2011 ). As explained above, systematic reviews seek to identify causation. Such logic is appropriate for fields like medicine and education where findings of randomized controlled trials can be aggregated to see whether a new treatment or intervention does improve outcomes. However, many argue that it is not possible to establish such direct causal links between interventions and outcomes in fields such as social policy, management, and information systems where for any intervention there is unlikely to be a regular or consistent outcome ( Oates, 2011 ; Pawson, 2006 ; Rousseau, Manning, & Denyer, 2008 ).
To circumvent these limitations, Pawson, Greenhalgh, Harvey, and Walshe (2005) have proposed a new approach for synthesizing knowledge that seeks to unpack the mechanism of how “complex interventions” work in particular contexts. The basic research question — what works? — which is usually associated with systematic reviews changes to: what is it about this intervention that works, for whom, in what circumstances, in what respects and why? Realist reviews have no particular preference for either quantitative or qualitative evidence. As a theory-building approach, a realist review usually starts by articulating likely underlying mechanisms and then scrutinizes available evidence to find out whether and where these mechanisms are applicable ( Shepperd et al., 2009 ). Primary studies found in the extant literature are viewed as case studies which can test and modify the initial theories ( Rousseau et al., 2008 ).
The main objective pursued in the realist review conducted by Otte-Trojel, de Bont, Rundall, and van de Klundert (2014) was to examine how patient portals contribute to health service delivery and patient outcomes. The specific goals were to investigate how outcomes are produced and, most importantly, how variations in outcomes can be explained. The research team started with an exploratory review of background documents and research studies to identify ways in which patient portals may contribute to health service delivery and patient outcomes. The authors identified six main ways which represent “educated guesses” to be tested against the data in the evaluation studies. These studies were identified through a formal and systematic search in four databases between 2003 and 2013. Two members of the research team selected the articles using a pre-established list of inclusion and exclusion criteria and following a two-step procedure. The authors then extracted data from the selected articles and created several tables, one for each outcome category. They organized information to bring forward those mechanisms where patient portals contribute to outcomes and the variation in outcomes across different contexts.
9.3.6. Critical Reviews
Lastly, critical reviews aim to provide a critical evaluation and interpretive analysis of existing literature on a particular topic of interest to reveal strengths, weaknesses, contradictions, controversies, inconsistencies, and/or other important issues with respect to theories, hypotheses, research methods or results ( Baumeister & Leary, 1997 ; Kirkevold, 1997 ). Unlike other review types, critical reviews attempt to take a reflective account of the research that has been done in a particular area of interest, and assess its credibility by using appraisal instruments or critical interpretive methods. In this way, critical reviews attempt to constructively inform other scholars about the weaknesses of prior research and strengthen knowledge development by giving focus and direction to studies for further improvement ( Kirkevold, 1997 ).
Kitsiou, Paré, and Jaana (2013) provide an example of a critical review that assessed the methodological quality of prior systematic reviews of home telemonitoring studies for chronic patients. The authors conducted a comprehensive search on multiple databases to identify eligible reviews and subsequently used a validated instrument to conduct an in-depth quality appraisal. Results indicate that the majority of systematic reviews in this particular area suffer from important methodological flaws and biases that impair their internal validity and limit their usefulness for clinical and decision-making purposes. To this end, they provide a number of recommendations to strengthen knowledge development towards improving the design and execution of future reviews on home telemonitoring.
Table 9.1 outlines the main types of literature reviews that were described in the previous sub-sections and summarizes the main characteristics that distinguish one review type from another. It also includes key references to methodological guidelines and useful sources that can be used by eHealth scholars and researchers for planning and developing reviews.
Typology of Literature Reviews (adapted from Paré et al., 2015).
As shown in Table 9.1 , each review type addresses different kinds of research questions or objectives, which subsequently define and dictate the methods and approaches that need to be used to achieve the overarching goal(s) of the review. For example, in the case of narrative reviews, there is greater flexibility in searching and synthesizing articles ( Green et al., 2006 ). Researchers are often relatively free to use a diversity of approaches to search, identify, and select relevant scientific articles, describe their operational characteristics, present how the individual studies fit together, and formulate conclusions. On the other hand, systematic reviews are characterized by their high level of systematicity, rigour, and use of explicit methods, based on an “a priori” review plan that aims to minimize bias in the analysis and synthesis process (Higgins & Green, 2008). Some reviews are exploratory in nature (e.g., scoping/mapping reviews), whereas others may be conducted to discover patterns (e.g., descriptive reviews) or involve a synthesis approach that may include the critical analysis of prior research ( Paré et al., 2015 ). Hence, in order to select the most appropriate type of review, it is critical to know before embarking on a review project, why the research synthesis is conducted and what type of methods are best aligned with the pursued goals.
9.5. Concluding Remarks
In light of the increased use of evidence-based practice and research generating stronger evidence ( Grady et al., 2011 ; Lyden et al., 2013 ), review articles have become essential tools for summarizing, synthesizing, integrating or critically appraising prior knowledge in the eHealth field. As mentioned earlier, when rigorously conducted review articles represent powerful information sources for eHealth scholars and practitioners looking for state-of-the-art evidence. The typology of literature reviews we used herein will allow eHealth researchers, graduate students and practitioners to gain a better understanding of the similarities and differences between review types.
We must stress that this classification scheme does not privilege any specific type of review as being of higher quality than another ( Paré et al., 2015 ). As explained above, each type of review has its own strengths and limitations. Having said that, we realize that the methodological rigour of any review — be it qualitative, quantitative or mixed — is a critical aspect that should be considered seriously by prospective authors. In the present context, the notion of rigour refers to the reliability and validity of the review process described in section 9.2. For one thing, reliability is related to the reproducibility of the review process and steps, which is facilitated by a comprehensive documentation of the literature search process, extraction, coding and analysis performed in the review. Whether the search is comprehensive or not, whether it involves a methodical approach for data extraction and synthesis or not, it is important that the review documents in an explicit and transparent manner the steps and approach that were used in the process of its development. Next, validity characterizes the degree to which the review process was conducted appropriately. It goes beyond documentation and reflects decisions related to the selection of the sources, the search terms used, the period of time covered, the articles selected in the search, and the application of backward and forward searches ( vom Brocke et al., 2009 ). In short, the rigour of any review article is reflected by the explicitness of its methods (i.e., transparency) and the soundness of the approach used. We refer those interested in the concepts of rigour and quality to the work of Templier and Paré (2015) which offers a detailed set of methodological guidelines for conducting and evaluating various types of review articles.
To conclude, our main objective in this chapter was to demystify the various types of literature reviews that are central to the continuous development of the eHealth field. It is our hope that our descriptive account will serve as a valuable source for those conducting, evaluating or using reviews in this important and growing domain.
- Ammenwerth E., de Keizer N. An inventory of evaluation studies of information technology in health care. Trends in evaluation research, 1982-2002. International Journal of Medical Informatics. 2004; 44 (1):44–56. [ PubMed : 15778794 ]
- Anderson S., Allen P., Peckham S., Goodwin N. Asking the right questions: scoping studies in the commissioning of research on the organisation and delivery of health services. Health Research Policy and Systems. 2008; 6 (7):1–12. [ PMC free article : PMC2500008 ] [ PubMed : 18613961 ] [ CrossRef ]
- Archer N., Fevrier-Thomas U., Lokker C., McKibbon K. A., Straus S.E. Personal health records: a scoping review. Journal of American Medical Informatics Association. 2011; 18 (4):515–522. [ PMC free article : PMC3128401 ] [ PubMed : 21672914 ]
- Arksey H., O’Malley L. Scoping studies: towards a methodological framework. International Journal of Social Research Methodology. 2005; 8 (1):19–32.
- A systematic, tool-supported method for conducting literature reviews in information systems. Paper presented at the Proceedings of the 19th European Conference on Information Systems ( ecis 2011); June 9 to 11; Helsinki, Finland. 2011.
- Baumeister R. F., Leary M.R. Writing narrative literature reviews. Review of General Psychology. 1997; 1 (3):311–320.
- Becker L. A., Oxman A.D. In: Cochrane handbook for systematic reviews of interventions. Higgins J. P. T., Green S., editors. Hoboken, nj : John Wiley & Sons, Ltd; 2008. Overviews of reviews; pp. 607–631.
- Borenstein M., Hedges L., Higgins J., Rothstein H. Introduction to meta-analysis. Hoboken, nj : John Wiley & Sons Inc; 2009.
- Cook D. J., Mulrow C. D., Haynes B. Systematic reviews: Synthesis of best evidence for clinical decisions. Annals of Internal Medicine. 1997; 126 (5):376–380. [ PubMed : 9054282 ]
- Cooper H., Hedges L.V. In: The handbook of research synthesis and meta-analysis. 2nd ed. Cooper H., Hedges L. V., Valentine J. C., editors. New York: Russell Sage Foundation; 2009. Research synthesis as a scientific process; pp. 3–17.
- Cooper H. M. Organizing knowledge syntheses: A taxonomy of literature reviews. Knowledge in Society. 1988; 1 (1):104–126.
- Cronin P., Ryan F., Coughlan M. Undertaking a literature review: a step-by-step approach. British Journal of Nursing. 2008; 17 (1):38–43. [ PubMed : 18399395 ]
- Darlow S., Wen K.Y. Development testing of mobile health interventions for cancer patient self-management: A review. Health Informatics Journal. 2015 (online before print). [ PubMed : 25916831 ] [ CrossRef ]
- Daudt H. M., van Mossel C., Scott S.J. Enhancing the scoping study methodology: a large, inter-professional team’s experience with Arksey and O’Malley’s framework. bmc Medical Research Methodology. 2013; 13 :48. [ PMC free article : PMC3614526 ] [ PubMed : 23522333 ] [ CrossRef ]
- Davies P. The relevance of systematic reviews to educational policy and practice. Oxford Review of Education. 2000; 26 (3-4):365–378.
- Deeks J. J., Higgins J. P. T., Altman D.G. In: Cochrane handbook for systematic reviews of interventions. Higgins J. P. T., Green S., editors. Hoboken, nj : John Wiley & Sons, Ltd; 2008. Analysing data and undertaking meta-analyses; pp. 243–296.
- Deshazo J. P., Lavallie D. L., Wolf F.M. Publication trends in the medical informatics literature: 20 years of “Medical Informatics” in mesh . bmc Medical Informatics and Decision Making. 2009; 9 :7. [ PMC free article : PMC2652453 ] [ PubMed : 19159472 ] [ CrossRef ]
- Dixon-Woods M., Agarwal S., Jones D., Young B., Sutton A. Synthesising qualitative and quantitative evidence: a review of possible methods. Journal of Health Services Research and Policy. 2005; 10 (1):45–53. [ PubMed : 15667704 ]
- Finfgeld-Connett D., Johnson E.D. Literature search strategies for conducting knowledge-building and theory-generating qualitative systematic reviews. Journal of Advanced Nursing. 2013; 69 (1):194–204. [ PMC free article : PMC3424349 ] [ PubMed : 22591030 ]
- Grady B., Myers K. M., Nelson E. L., Belz N., Bennett L., Carnahan L. … Guidelines Working Group. Evidence-based practice for telemental health. Telemedicine Journal and E Health. 2011; 17 (2):131–148. [ PubMed : 21385026 ]
- Green B. N., Johnson C. D., Adams A. Writing narrative literature reviews for peer-reviewed journals: secrets of the trade. Journal of Chiropractic Medicine. 2006; 5 (3):101–117. [ PMC free article : PMC2647067 ] [ PubMed : 19674681 ]
- Greenhalgh T., Wong G., Westhorp G., Pawson R. Protocol–realist and meta-narrative evidence synthesis: evolving standards ( rameses ). bmc Medical Research Methodology. 2011; 11 :115. [ PMC free article : PMC3173389 ] [ PubMed : 21843376 ]
- Gurol-Urganci I., de Jongh T., Vodopivec-Jamsek V., Atun R., Car J. Mobile phone messaging reminders for attendance at healthcare appointments. Cochrane Database System Review. 2013; 12 cd 007458. [ PMC free article : PMC6485985 ] [ PubMed : 24310741 ] [ CrossRef ]
- Hart C. Doing a literature review: Releasing the social science research imagination. London: SAGE Publications; 1998.
- Higgins J. P. T., Green S., editors. Cochrane handbook for systematic reviews of interventions: Cochrane book series. Hoboken, nj : Wiley-Blackwell; 2008.
- Jesson J., Matheson L., Lacey F.M. Doing your literature review: traditional and systematic techniques. Los Angeles & London: SAGE Publications; 2011.
- King W. R., He J. Understanding the role and methods of meta-analysis in IS research. Communications of the Association for Information Systems. 2005; 16 :1.
- Kirkevold M. Integrative nursing research — an important strategy to further the development of nursing science and nursing practice. Journal of Advanced Nursing. 1997; 25 (5):977–984. [ PubMed : 9147203 ]
- Kitchenham B., Charters S. ebse Technical Report Version 2.3. Keele & Durham. uk : Keele University & University of Durham; 2007. Guidelines for performing systematic literature reviews in software engineering.
- Kitsiou S., Paré G., Jaana M. Systematic reviews and meta-analyses of home telemonitoring interventions for patients with chronic diseases: a critical assessment of their methodological quality. Journal of Medical Internet Research. 2013; 15 (7):e150. [ PMC free article : PMC3785977 ] [ PubMed : 23880072 ]
- Kitsiou S., Paré G., Jaana M. Effects of home telemonitoring interventions on patients with chronic heart failure: an overview of systematic reviews. Journal of Medical Internet Research. 2015; 17 (3):e63. [ PMC free article : PMC4376138 ] [ PubMed : 25768664 ]
- Levac D., Colquhoun H., O’Brien K. K. Scoping studies: advancing the methodology. Implementation Science. 2010; 5 (1):69. [ PMC free article : PMC2954944 ] [ PubMed : 20854677 ]
- Levy Y., Ellis T.J. A systems approach to conduct an effective literature review in support of information systems research. Informing Science. 2006; 9 :181–211.
- Liberati A., Altman D. G., Tetzlaff J., Mulrow C., Gøtzsche P. C., Ioannidis J. P. A. et al. Moher D. The prisma statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: Explanation and elaboration. Annals of Internal Medicine. 2009; 151 (4):W-65. [ PubMed : 19622512 ]
- Lyden J. R., Zickmund S. L., Bhargava T. D., Bryce C. L., Conroy M. B., Fischer G. S. et al. McTigue K. M. Implementing health information technology in a patient-centered manner: Patient experiences with an online evidence-based lifestyle intervention. Journal for Healthcare Quality. 2013; 35 (5):47–57. [ PubMed : 24004039 ]
- Mickan S., Atherton H., Roberts N. W., Heneghan C., Tilson J.K. Use of handheld computers in clinical practice: a systematic review. bmc Medical Informatics and Decision Making. 2014; 14 :56. [ PMC free article : PMC4099138 ] [ PubMed : 24998515 ]
- Moher D. The problem of duplicate systematic reviews. British Medical Journal. 2013; 347 (5040) [ PubMed : 23945367 ] [ CrossRef ]
- Montori V. M., Wilczynski N. L., Morgan D., Haynes R. B., Hedges T. Systematic reviews: a cross-sectional study of location and citation counts. bmc Medicine. 2003; 1 :2. [ PMC free article : PMC281591 ] [ PubMed : 14633274 ]
- Mulrow C. D. The medical review article: state of the science. Annals of Internal Medicine. 1987; 106 (3):485–488. [ PubMed : 3813259 ] [ CrossRef ]
- Evidence-based information systems: A decade later. Proceedings of the European Conference on Information Systems ; 2011. Retrieved from http://aisel .aisnet.org/cgi/viewcontent .cgi?article =1221&context =ecis2011 .
- Okoli C., Schabram K. A guide to conducting a systematic literature review of information systems research. ssrn Electronic Journal. 2010
- Otte-Trojel T., de Bont A., Rundall T. G., van de Klundert J. How outcomes are achieved through patient portals: a realist review. Journal of American Medical Informatics Association. 2014; 21 (4):751–757. [ PMC free article : PMC4078283 ] [ PubMed : 24503882 ]
- Paré G., Trudel M.-C., Jaana M., Kitsiou S. Synthesizing information systems knowledge: A typology of literature reviews. Information & Management. 2015; 52 (2):183–199.
- Patsopoulos N. A., Analatos A. A., Ioannidis J.P. A. Relative citation impact of various study designs in the health sciences. Journal of the American Medical Association. 2005; 293 (19):2362–2366. [ PubMed : 15900006 ]
- Paul M. M., Greene C. M., Newton-Dame R., Thorpe L. E., Perlman S. E., McVeigh K. H., Gourevitch M.N. The state of population health surveillance using electronic health records: A narrative review. Population Health Management. 2015; 18 (3):209–216. [ PubMed : 25608033 ]
- Pawson R. Evidence-based policy: a realist perspective. London: SAGE Publications; 2006.
- Pawson R., Greenhalgh T., Harvey G., Walshe K. Realist review—a new method of systematic review designed for complex policy interventions. Journal of Health Services Research & Policy. 2005; 10 (Suppl 1):21–34. [ PubMed : 16053581 ]
- Petersen K., Vakkalanka S., Kuzniarz L. Guidelines for conducting systematic mapping studies in software engineering: An update. Information and Software Technology. 2015; 64 :1–18.
- Petticrew M., Roberts H. Systematic reviews in the social sciences: A practical guide. Malden, ma : Blackwell Publishing Co; 2006.
- Rousseau D. M., Manning J., Denyer D. Evidence in management and organizational science: Assembling the field’s full weight of scientific knowledge through syntheses. The Academy of Management Annals. 2008; 2 (1):475–515.
- Rowe F. What literature review is not: diversity, boundaries and recommendations. European Journal of Information Systems. 2014; 23 (3):241–255.
- Shea B. J., Hamel C., Wells G. A., Bouter L. M., Kristjansson E., Grimshaw J. et al. Boers M. amstar is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. Journal of Clinical Epidemiology. 2009; 62 (10):1013–1020. [ PubMed : 19230606 ]
- Shepperd S., Lewin S., Straus S., Clarke M., Eccles M. P., Fitzpatrick R. et al. Sheikh A. Can we systematically review studies that evaluate complex interventions? PLoS Medicine. 2009; 6 (8):e1000086. [ PMC free article : PMC2717209 ] [ PubMed : 19668360 ]
- Silva B. M., Rodrigues J. J., de la Torre Díez I., López-Coronado M., Saleem K. Mobile-health: A review of current state in 2015. Journal of Biomedical Informatics. 2015; 56 :265–272. [ PubMed : 26071682 ]
- Smith V., Devane D., Begley C., Clarke M. Methodology in conducting a systematic review of systematic reviews of healthcare interventions. bmc Medical Research Methodology. 2011; 11 (1):15. [ PMC free article : PMC3039637 ] [ PubMed : 21291558 ]
- Sylvester A., Tate M., Johnstone D. Beyond synthesis: re-presenting heterogeneous research literature. Behaviour & Information Technology. 2013; 32 (12):1199–1215.
- Templier M., Paré G. A framework for guiding and evaluating literature reviews. Communications of the Association for Information Systems. 2015; 37 (6):112–137.
- Thomas J., Harden A. Methods for the thematic synthesis of qualitative research in systematic reviews. bmc Medical Research Methodology. 2008; 8 (1):45. [ PMC free article : PMC2478656 ] [ PubMed : 18616818 ]
- Reconstructing the giant: on the importance of rigour in documenting the literature search process. Paper presented at the Proceedings of the 17th European Conference on Information Systems ( ecis 2009); Verona, Italy. 2009.
- Webster J., Watson R.T. Analyzing the past to prepare for the future: Writing a literature review. Management Information Systems Quarterly. 2002; 26 (2):11.
- Whitlock E. P., Lin J. S., Chou R., Shekelle P., Robinson K.A. Using existing systematic reviews in complex systematic reviews. Annals of Internal Medicine. 2008; 148 (10):776–782. [ PubMed : 18490690 ]
This publication is licensed under a Creative Commons License, Attribution-Noncommercial 4.0 International License (CC BY-NC 4.0): see https://creativecommons.org/licenses/by-nc/4.0/
- Cite this Page Paré G, Kitsiou S. Chapter 9 Methods for Literature Reviews. In: Lau F, Kuziemsky C, editors. Handbook of eHealth Evaluation: An Evidence-based Approach [Internet]. Victoria (BC): University of Victoria; 2017 Feb 27.
- PDF version of this title (4.5M)
- Disable Glossary Links
In this Page
- Overview of the Literature Review Process and Steps
- Types of Review Articles and Brief Illustrations
- Concluding Remarks
- PMC PubMed Central citations
- PubMed Links to PubMed
- Chapter 9 Methods for Literature Reviews - Handbook of eHealth Evaluation: An Ev... Chapter 9 Methods for Literature Reviews - Handbook of eHealth Evaluation: An Evidence-based Approach
Your browsing activity is empty.
Activity recording is turned off.
Turn recording back on
Connect with NLM
National Library of Medicine 8600 Rockville Pike Bethesda, MD 20894
Web Policies FOIA HHS Vulnerability Disclosure
Help Accessibility Careers
- Teesside University Student & Library Services
- Learning Hub Group
- Primary Research
What is Secondary Research?
Advantages and disadvantages of secondary research, secondary research in literature reviews, secondary research - going beyond literature reviews, main stages of secondary research, useful resources, using material on this page.
- Quantitative Research This link opens in a new window
- Qualitative Research This link opens in a new window
- Being Critical This link opens in a new window
- Subject LibGuides This link opens in a new window
Secondary research uses research and data that has already been carried out. It is sometimes referred to as desk research. It is a good starting point for any type of research as it enables you to analyse what research has already been undertaken and identify any gaps.
You may only need to carry out secondary research for your assessment or you may need to use secondary research as a starting point, before undertaking your own primary research .
Searching for both primary and secondary sources can help to ensure that you are up to date with what research has already been carried out in your area of interest and to identify the key researchers in the field.
"Secondary sources are the books, articles, papers and similar materials written or produced by others that help you to form your background understanding of the subject. You would use these to ﬁnd out about experts’ ﬁndings, analyses or perspectives on the issue and decide whether to draw upon these explicitly in your research." (Cottrell, 2014, p. 123).
Examples of secondary research sources include:.
- journal articles
- official statistics, such as government reports or organisations which have collected and published data
Primary research involves gathering data which has not been collected before. Methods to collect it can include interviews, focus groups, controlled trials and case studies. Secondary research often comments on and analyses this primary research.
Gopalakrishnan and Ganeshkumar (2013, p. 10) explain the difference between primary and secondary research:
"Primary research is collecting data directly from patients or population, while secondary research is the analysis of data already collected through primary research. A review is an article that summarizes a number of primary studies and may draw conclusions on the topic of interest which can be traditional (unsystematic) or systematic".
As secondary data has already been collected by someone else for their research purposes, it may not cover all of the areas of interest for your research topic. This research will need to be analysed alongside other research sources and data in the same subject area in order to confirm, dispute or discuss the findings in a wider context.
"Secondary source data, as the name infers, provides second-hand information. The data come ‘pre-packaged’, their form and content reflecting the fact that they have been produced by someone other than the researcher and will not have been produced specifically for the purpose of the research project. The data, none the less, will have some relevance for the research in terms of the information they contain, and the task for the researcher is to extract that information and re-use it in the context of his/her own research project." (Denscombe, 2021, p. 268)
In the video below Dr. Benedict Wheeler (Senior Research Fellow at the European Center for Environment and Human Health at the University of Exeter Medical School) discusses secondary data analysis. Secondary data was used for his research on how the environment affects health and well-being and utilising this secondary data gave access to a larger data set.
As with all research, an important part of the process is to critically evaluate any sources you use. There are tools to help with this in the Being Critical section of the guide.
Louise Corti, from the UK Data Archive, discusses using secondary data in the video below. T he importance of evaluating secondary research is discussed - this is to ensure the data is appropriate for your research and to investigate how the data was collected.
There are advantages and disadvantages to secondary research:
- Usually low cost
- Easily accessible
- Provides background information to clarify / refine research areas
- Increases breadth of knowledge
- Shows different examples of research methods
- Can highlight gaps in the research and potentially outline areas of difficulty
- Can incorporate a wide range of data
- Allows you to identify opposing views and supporting arguments for your research topic
- Highlights the key researchers and work which is being undertaken within the subject area
- Helps to put your research topic into perspective
- Can be out of date
- Might be unreliable if it is not clear where or how the research has been collected - remember to think critically
- May not be applicable to your specific research question as the aims will have had a different focus
Secondary research for your major project may take the form of a literature review . this is where you will outline the main research which has already been written on your topic. this might include theories and concepts connected with your topic and it should also look to see if there are any gaps in the research., as the criteria and guidance will differ for each school, it is important that you check the guidance which you have been given for your assessment. this may be in blackboard and you can also check with your supervisor..
The videos below include some insights from academics regarding the importance of literature reviews.
Secondary research which goes beyond literature reviews
For some dissertations/major projects there might only be a literature review (discussed above ). For others there could be a literature review followed by primary research and for others the literature review might be followed by further secondary research.
You may be asked to write a literature review which will form a background chapter to give context to your project and provide the necessary history for the research topic. However, you may then also be expected to produce the rest of your project using additional secondary research methods, which will need to produce results and findings which are distinct from the background chapter t o avoid repetition .
Remember, as the criteria and guidance will differ for each School, it is important that you check the guidance which you have been given for your assessment. This may be in Blackboard and you can also check with your supervisor.
Although this type of secondary research will go beyond a literature review, it will still rely on research which has already been undertaken. And, "just as in primary research, secondary research designs can be either quantitative, qualitative, or a mixture of both strategies of inquiry" (Manu and Akotia, 2021, p. 4) .
Your secondary research may use the literature review to focus on a specific theme, which is then discussed further in the main project. Or it may use an alternative approach. Some examples are included below. Remember to speak with your supervisor if you are struggling to define these areas.
Some approaches of how to conduct secondary research include:
- A systematic review is a structured literature review that involves identifying all of the relevant primary research using a rigorous search strategy to answer a focused research question.
- This involves comprehensive searching which is used to identify themes or concepts across a number of relevant studies.
- The review will assess the q uality of the research and provide a summary and synthesis of all relevant available research on the topic.
- The systematic review LibGuide goes into more detail about this process (The guide is aimed a PhD/Researcher students. However, students on other levels of study may find parts of the guide helpful too).
- Scoping reviews aim to identify and assess available research on a specific topic (which can include ongoing research).
- They are "particularly useful when a body of literature has not yet been comprehensively reviewed, or exhibits a complex or heterogeneous nature not amenable to a more precise systematic review of the evidence. While scoping reviews may be conducted to determine the value and probable scope of a full systematic review, they may also be undertaken as exercises in and of themselves to summarize and disseminate research findings, to identify research gaps, and to make recommendations for the future research." (Peters et al., 2015) .
- This is designed to summarise the current knowledge and provide priorities for future research.
- "A state-of-the-art review will often highlight new ideas or gaps in research with no official quality assessment." (Baguss, 2020) .
- "Bibliometric analysis is a popular and rigorous method for exploring and analyzing large volumes of scientific data." (Donthu et al., 2021)
- Quantitative methods and statistics are used to analyse the bibliographic data of published literature. This can be used to measure the impact of authors, publications, or topics within a subject area.
The bibliometric analysis often uses the data from a citation source such as Scopus or Web of Science .
- This is a technique used to combine the statistic results of prior quantitative studies in order to increase precision and validity.
- "It goes beyond the parameters of a literature review, which assesses existing literature, to actually perform calculations based on the results collated, thereby coming up with new results" (Curtis and Curtis, 2011, p. 220)
(Adapted from: Grant and Booth, 2009, cited in Sarhan and Manu, 2021, p. 72 )
- Grounded Theory is used to create explanatory theory from data which has been collected.
- "Grounded theory data analysis strategies can be used with different types of data, including secondary data." ( Whiteside, Mills and McCalman, 2012 )
- This allows you to use a specific theory or theories which can then be applied to your chosen topic/research area.
- You could focus on one case study which is analysed in depth, or you could examine more than one in order to compare and contrast the important aspects of your research question.
- "Good case studies often begin with a predicament that is poorly comprehended and is inadequately explained or traditionally rationalised by numerous conflicting accounts. Therefore, the aim is to comprehend an existent problem and to use the acquired understandings to develop new theoretical outlooks or explanations." ( Papachroni and Lochrie, 2015, p. 81 )
Main stages of secondary research for a dissertation/major project
In general, the main stages for conducting secondary research for your dissertation or major project will include:
Click on the image below to access the reading list which includes resources used in this guide as well as some additional useful resources.
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License .
- << Previous: Primary Research
- Next: Quantitative Research >>
- Last Updated: Aug 11, 2022 3:41 PM
- URL: https://libguides.tees.ac.uk/researchmethods
- University of Arkansas
- Blackboard Learn
- Exchange Mail
- Research Guides
International and Global Studies
- Understanding Sources
- Getting Started
- Searching OneSearch
Popular Vs. Scholarly Sources
The peer review process, understanding primary, secondary, and tertiary literature.
- Searching and Evaluating Resources
- Databases & Ebooks
- Dissertations & Theses
- InterLibrary Loan
- Citing Your Sources
- Writing Tips
Chat with Us
Image created by Laura Cameron, Librarian
Peer review is the process by which articles or other works are critiqued before they are published. Authors send articles to an editor, who decides whether the work should be forwarded to reviewers for the journal. The most stringent form is anonymous or blind review, where neither the author nor the reviewers know whose work is being examined by whom. This helps reduce bias.
Reviewers are usually well-published researchers and experts themselves. The articles are sent back to the editor with remarks and recommendations, usually 1) publish as is (rare), 2) publish if edited or changed in specific ways, or 3) don't publish. Editors will usually go with the recommendation of the majority of the reviewers. If revision is recommended, the reviewers' comments may be returned with the draft. Authors who have been asked for revisions normally make the recommended changes and resubmit the article to the editor.
The process is intended to improve the content of studies published-- more eyes on a project, and one's reputation on the line with peers, tends to improve the quality of what's published. There are cases where it hasn't worked, and critics of the cycle, but it is the best system that has been developed to this point.
What is considered a primary source varies somewhat by discipline. In any case, think of a primary source as first-hand knowledge, eyewitness accounts, or reports about a specific topic.
- In the arts - a piece of art, such as a painting or sculpture, a musical score, a poem, a book or chapter, or an essay--whatever is created by the artist, writer, photographer, etc.
- In the sciences - the first report of research; it may be published as a journal article, or sometimes as a research report or conference presentation
- In some of the social sciences - a primary source may be the first report of a piece of research, especially of empirical studies, or it may be something closer to primary sources in history, since some areas of these fields depend on direct observation, data, personal narratives or commentary, as from interviews or case studies.
- For history - a primary source is a letter, a diary, speech, lecture, piece of legislation, document or manuscript-- an original source which forms the basis, with other sources, of secondary work, such as a study of life in eighteenth century Ireland
A secondary source is based on a primary source or other sources. It includes analysis, criticism, or other intellectual input. Secondary sources can include books, book chapters, articles, especially literature reviews, and some book reviews.
A tertiary source is commonly a resource or tool that helps people find primary or secondary sources. Tertiary sources include most bibliographies, databases and indexes, and library catalogs.
- << Previous: Searching OneSearch
- Next: Keywords >>
- Last Updated: Nov 7, 2023 8:46 AM
- URL: https://uark.libguides.com/INST
- See us on Instagram
- Follow us on Twitter
- Like us on Facebook
- Libraries on Pinterest
- Phone: 479-575-4104
Understanding Nursing Research
- Primary Research
- Qualitative vs. Quantitative Research
- Experimental Design
- Is it a Nursing journal?
- Is it Written by a Nurse?
Secondary Research and Systematic Reviews
- Quality Improvement Plans
Secondary Research is when researchers collect lots of research that has already been published on a certain subject. They conduct searches in databases, go through lots of primary research articles, and analyze the findings in those pieces of primary research. The goal of secondary research is to pull together lots of diverse primary research (like studies and trials), with the end goal of making a generalized statement. Primary research can only make statements about the specific context in which their research was conducted (for example, this specific intervention worked in this hospital with these participants), but secondary research can make broader statements because it compiled lots of primary research together. So rather than saying, "this specific intervention worked at this specific hospital with these specific participants, a piece of secondary research can say, "This intervention works at hospitals that serve this population."
Systematic Reviews are a kind of secondary research. The creators of systematic reviews are very intentional about their inclusion/exclusion criteria, or which articles they'll include in their review and the goal is to make a generalized statement so other researchers can build upon the practices or interventions they recommend. Use the chart below to understand the differences between a systematic review and a literature review.
Check out the video below to watch the Nursing and Health Sciences librarian describe the differences between primary and secondary research.
- "Literature Reviews and Systematic Reviews: What Is the Difference?" This article explains in depth the differences between Literature Reviews and Systematic Reviews. It is from the journal RADIOLOGIC TECHNOLOGY, Nov/Dec 2013, v. 85, #2. It is one to which Bell Library subscribes and meets copyright clearance requirements through our subscription to CCC.
- << Previous: Is it Written by a Nurse?
- Next: Permalinks >>
- Last Updated: Sep 22, 2023 3:09 PM
- URL: https://guides.library.tamucc.edu/nursingresearch
How to Write a Literature Review: Primary and Secondary Sources
- Writing a Literature Review in APA Format
- Chicago/Turabian Citation Style
- Primary and Secondary Sources
- Basic Research Strategies
- Evaluating Sources
- Using the Library's Ebooks
- Using the Library's Catalog
- Copyright Information
- Contact Information & Feedback
Primary versus Secondary Sources
Primary vs. secondary videos.
Primary, Secondary, & Tertiary Sources The content of research papers may come from different types of sources, such as:
- Your own opinion and analysis
- Primary sources
- Secondary sources
- Tertiary sources
It may not be necessary to include each of these types of sources in every paper you write, but your instructor may require you to include them. It is important to understand the characteristics of primary, secondary and tertiary sources–they each serve a different purpose throughout the research process, and can strengthen your assignment, too.
It can be difficult to figure out if a source is considered primary, secondary, or tertiary. We will explain the differences and provide examples of each in this tutorial. If you are still not sure if a source you would like to use is primary, secondary, or tertiary, ask a librarian or teacher.
What is a Primary Source? Primary sources are first-hand, authoritative accounts of an event, topic, or historical time period. They are typically produced at the time of the event by a person who experienced it, but can also be made later on in the form of personal memoirs or oral histories.
Anything that contains original information on a topic is considered a primary source. Usually, primary sources are the object discussed in your paper. For instance, if you are writing an analysis on Upton Sinclair’s The Jungle, the book would be a primary source. But, just because a source is old does not mean it is a primary source.
Some examples of original, first-hand, authoritative accounts include:
- Letters, diaries or journals (Personal thoughts)
- Original photographs
- First-hand newspaper reports
- Speeches, autobiographies
- Creative works like plays, paintings and songs
- Research data and surveys
What is a Secondary Source? Secondary sources interpret or critique primary sources. They often include an analysis of the event that was discussed or featured in the primary source. They are second-hand accounts that interpret or draw conclusions from one or more primary sources.
Some examples of works that interpret or critique primary sources include:
- Textbooks (May also be considered tertiary)
- Essays or reviews
- Articles that analyze or discuss ideas and events
- Criticisms or commentaries
What is a Tertiary Source? Tertiary sources generally provide an overview or summary of a topic, and may contain both primary and secondary sources. The information is displayed as entirely factual, and does not include analysis or critique. Tertiary sources can also be collections of primary and secondary sources, such as databases, bibliographies and directories.
Some examples of sources that provide a summary or collection of a topic include:
- Textbooks (May also be considered secondary)
- Bibliographies or abstracts
- Wikipedia articles
Using Primary, secondary and Tertiary Sources in Research Let’s say you are writing a research paper on the Equal Rights Amendment (ERA) of 1972, but you are unfamiliar with it. A good place to gather a general idea or understanding of the ERA would be a tertiary source, such as Wikipedia or the Encyclopedia Britannica. There, you can read a summary of events on its history, key people involved, and legislation.
To find more in-depth analysis on the Equal Rights Amendment, you consult a secondary source: the nonfiction book Why We Lost the ERA by Jane Mansbridge and a newspaper article from the 1970’s that discuss and review the legislation. These provide a more focused analysis of the Equal Rights Amendment that you can include as sources in your paper (make sure you cite them!). A primary source that could bolster your research would be a government document detailing the ERA legislation that initially passed in Congress, giving a first-hand account of the legislation that went through the House and Senate in 1972.
This video provides a great overview of primary and secondary sources: [ youtube http://www.youtube.com/watch?v= PgfQC4d3pKc &w=420&h=315]
- << Previous: Guide to Writing a Research Paper
- Next: Basic Research Strategies >>
- Last Updated: Sep 25, 2023 2:24 PM
- URL: https://tuskegee.libguides.com/c.php?g=692585
- University of La Verne
- Subject Guides
Literature Review Basics
- Primary & Secondary Sources
- Literature Review Introduction
- Writing Literature Reviews
- Tutorials & Samples
The Literature refers to the collection of scholarly writings on a topic. This includes peer-reviewed articles, books, dissertations and conference papers.
- When reviewing the literature, be sure to include major works as well as studies that respond to major works. You will want to focus on primary sources, though secondary sources can be valuable as well.
The term primary source is used broadly to embody all sources that are original. P rimary sources provide first-hand information that is closest to the object of study. Primary sources vary by discipline.
- In the natural and social sciences, original reports of research found in academic journals detailing the methodology used in the research, in-depth descriptions, and discussions of the findings are considered primary sources of information.
- Other common examples of primary sources include speeches, letters, diaries, autobiographies, interviews, official reports, court records, artifacts, photographs, and drawings.
Galvan, J. L. (2013). Writing literature reviews: A guide for students of the social and behavioral sciences . Glendale, CA: Pyrczak.
A secondary source is a source that provides non-original or secondhand data or information.
- Secondary sources are written about primary sources.
- Research summaries reported in textbooks, magazines, and newspapers are considered secondary sources. They typically provide global descriptions of results with few details on the methodology. Other examples of secondary sources include biographies and critical studies of an author's work.
Secondary Source. (2005). In W. Paul Vogt (Ed.), Dictionary of Statistics & Methodology. (3 rd ed., p. 291). Thousand Oaks, CA: SAGE Publications, Inc.
Weidenborner, S., & Caruso, D. (1997). Writing research papers: A guide to the process . New York: St. Martin's Press.
More Examples of Primary and Secondary Sources
- << Previous: Writing Literature Reviews
- Next: Tutorials & Samples >>
- Last Updated: Jun 28, 2023 9:19 AM
- URL: https://laverne.libguides.com/litreviews
Secondary research – the basics of narrative reviews, systematic reviews, and meta-analysis
Planning to Write
Dr. Shivanee Shah
When it comes to academic publishing, typically, researchers think of publishing original articles , which form the bulk of primary research articles. Research can also be published by summarizing, collating, or analyzing existing research data. Research that falls into this category is somewhat less common than primary research publications and is called secondary research.
In secondary research publications, field observations that have been published by other authors are re-interpreted and re-written as a secondary research article. Let us now look at secondary publications in more detail.
Why secondary research publications are important
Secondary research publications provide a different perspective or an additional analysis of the current literature. This can help identify gaps in the current knowledge in a field and highlight future directions. Secondary research publications include narrative reviews, systematic reviews, or meta-analysis.
Since no new data is required for such publications, often, they may offer early career researchers a chance to begin publishing. In addition, they are also important for busy scientists to remain up-to-date in their fields.
Digging deeper into secondary research publications
So what exactly is a narrative review, a systematic review, or a meta-analysis? And how does one go about writing them? Read on to learn more about these secondary research article types.
1. Narrative reviews
These reviews typically provide an overview of the current knowledge in a given field or topic. They are written with the idea of summarizing what is known and highlighting either new perspectives in the field, or drawing out pending questions that are yet to be unanswered. There is no new analysis in these reviews and depending on the topic, may be quite exhaustive, with a long list of references. Most journals allow for longer review articles – even longer than original primary research articles–often ranging from 5000-10000 words.
Narrative reviews typically include:
- an abstract to summarize the article,
- an introduction to provide a background on the topic and to detail why such a review is required,
- a main text section that may be broken down to smaller sub-parts to discuss different aspects in the field,
- a conclusion section to highlight the gaps and future directions,
- and a detailed reference list.
Since the pre-existing knowledge in the field might be quite extensive, and multiple similar review articles may already have been published, it is very important to consider the scope and novelty of the topic to be covered before starting to write the review.
Also, it is highly recommended that authors contact the editors of suitable target journals prior to starting to write such reviews. This is to ensure that the editor would indeed consider the review article as many journals publish only solicited review articles.
2. Systematic reviews
Systematic reviews are more detailed and rigorous, and review a well-defined research question rather than a field or topic as in narrative reviews. Systematic reviews follow a systematic and reproducible methodology for searching all previous publications that have addressed the same question, and for critically assessing and analyzing the results from these previous publications in a review format.
Systematic reviews can be qualitative or quantitative. Qualitative systematic reviews derive data from descriptive or qualitative results, whereas quantitative systematic reviews include studies with numerical data.
Systematic reviews are structured in the typical IMRAD structure as are primary research articles, in which an abstract is followed by the main test including introduction, methods, results, and discussion and conclusions, followed by a reference list.
Meta-analysis is a type of systematic review, but a systematic review in which statistical analysis is carried out to compare previously published studies and derive new interpretations or new findings. These are immensely helpful to understand the global effects of a particular finding rather than depending on the results of a single, isolated study in a particular region.
Meta-analysis may be considered more reliable than primary research as they include all the previously published data on the same research question and discuss the differences in study outcomes and the reasons behind them. They are becoming very popular especially in cases where a lot of individual and smaller studies have already been performed.
For instance, there may be several different studies on the effect of treatment A for colon cancer. These maybe case studies in random cities where a particular treatment plan was effective or smaller studies claiming the benefits of the same treatment plan. In a meta-analysis, all these published studies are analyzed together to determine the global effectiveness of treatment A.
Like systematic reviews, they also follow the IMRAD structure and are written as regular research articles. Many journals consider meta-analysis articles as original articles due to the statistical analysis performed and the subsequent new data gathered.
Guidelines for authors planning to write secondary research publications
If you are considering working on systematic reviews and meta-analysis, you need to keep a few additional points in mind: Like any primary research, systematic reviews and meta-analysis also need a protocol for the methodology. Such protocols can be published in a similar manner as done in case of primary research articles.
In addition, like primary research, journals request that a checklist be followed while writing the manuscripts. For systematic reviews and meta-analysis, you should follow the PRISMA checklist . Some journals may even request the protocol and completed checklist to be submitted along with the manuscript during submission to the journal.
Further, since the research involved needs to be exhaustive, and there is no point in duplicating similar secondary research, it is recommended that potential systematic studies be registered in a common registry, just like clinical trials are required to be registered. Registering is beneficial in holding stake for the study and preventing another group from working on the same topic. Multiple registries are available for registering your systematic reviews and meta-analysis. For instance, PROSPERO is a well-documented, open registry for all systematic reviews.
As you can see, all the above types of publications can be completed without any infrastructure, experimental setup, or large expenses. Thus, secondary research may also be referred to ‘desk research,’ as it can be completely carried out by sitting at a desk.
A major benefit of performing secondary research is its cost-effectiveness and shorter time frames. Primary research can lead to huge expenses depending on the type of experimentation and surveys done to gather data. In case of secondary research, data has already been collected and stored during primary research, making it more economic.
In addition to reviews, systematic reviews and meta-analysis, smaller articles such as perspectives, opinion articles, or commentaries are also secondary research and may provide an outlet to get your ideas and hypotheses out before you are able to implement the research.
So consider your options to progress your research career as there are several opportunities to publish even while you may be facing dipping funds or may be in a difficult position to carry out primary research!
- A young researcher's guide to perspective, commentary, and opinion articles
- How to write a book review
- 6 Article types that journals publish: A guide for early career researchers
for this article
Published on: Dec 21, 2018
- Academic Writing
- Journal Article Types
You're looking to give wings to your academic career and publication journey. We like that!
Why don't we give you complete access! Create a free account and get unlimited access to all resources & a vibrant researcher community.
One click sign-in with your social accounts
Sign up via email
1536 visitors saw this today and 1210 signed up.
Subscribe to Conducting Research
Confirm that you would also like to sign up for free personalized email coaching for this stage.
A young researcher's guide to a systematic review
The basics of converting your PhD thesis into journal articles
A young researcher's guide to writing a clinical case report
Secondary research – the basics of narrative reviews, systematic reviews, and meta-analysis 6 min read
A young researcher's guide to a systematic review 6 min read
6 Article types that journals publish: A guide for early career researchers 6 min read
How to write the literature review of your research paper 7 min read
Publication and reporting biases and how they impact publication of research 8 min read
- Statement of the problem
- Background of study
- Scope of the study
- Types of qualitative research
- Rationale of the study
- Concept paper
- Literature review
- Introduction in research
- Under "Editor Evaluation"
- Ethics in research
- Review paper
- Responding to reviewer comments
- Predatory publishers
- Scope and delimitations
- Open access
- Plagiarism in research
- Journal selection tips
- Editor assigned
- Types of articles
- "Reject and Resubmit" status
- Decision in process
- Conflict of interest
Have a language expert improve your writing
Run a free plagiarism check in 10 minutes, generate accurate citations for free.
- Knowledge Base
- Systematic Review | Definition, Example, & Guide
Systematic Review | Definition, Example & Guide
Published on June 15, 2022 by Shaun Turney . Revised on June 22, 2023.
A systematic review is a type of review that uses repeatable methods to find, select, and synthesize all available evidence. It answers a clearly formulated research question and explicitly states the methods used to arrive at the answer.
They answered the question “What is the effectiveness of probiotics in reducing eczema symptoms and improving quality of life in patients with eczema?”
In this context, a probiotic is a health product that contains live microorganisms and is taken by mouth. Eczema is a common skin condition that causes red, itchy skin.
Table of contents
What is a systematic review, systematic review vs. meta-analysis, systematic review vs. literature review, systematic review vs. scoping review, when to conduct a systematic review, pros and cons of systematic reviews, step-by-step example of a systematic review, other interesting articles, frequently asked questions about systematic reviews.
A review is an overview of the research that’s already been completed on a topic.
What makes a systematic review different from other types of reviews is that the research methods are designed to reduce bias . The methods are repeatable, and the approach is formal and systematic:
- Formulate a research question
- Develop a protocol
- Search for all relevant studies
- Apply the selection criteria
- Extract the data
- Synthesize the data
- Write and publish a report
Although multiple sets of guidelines exist, the Cochrane Handbook for Systematic Reviews is among the most widely used. It provides detailed guidelines on how to complete each step of the systematic review process.
Systematic reviews are most commonly used in medical and public health research, but they can also be found in other disciplines.
Systematic reviews typically answer their research question by synthesizing all available evidence and evaluating the quality of the evidence. Synthesizing means bringing together different information to tell a single, cohesive story. The synthesis can be narrative ( qualitative ), quantitative , or both.
Receive feedback on language, structure, and formatting
Professional editors proofread and edit your paper by focusing on:
- Academic style
- Vague sentences
- Style consistency
See an example
Systematic reviews often quantitatively synthesize the evidence using a meta-analysis . A meta-analysis is a statistical analysis, not a type of review.
A meta-analysis is a technique to synthesize results from multiple studies. It’s a statistical analysis that combines the results of two or more studies, usually to estimate an effect size .
A literature review is a type of review that uses a less systematic and formal approach than a systematic review. Typically, an expert in a topic will qualitatively summarize and evaluate previous work, without using a formal, explicit method.
Although literature reviews are often less time-consuming and can be insightful or helpful, they have a higher risk of bias and are less transparent than systematic reviews.
Similar to a systematic review, a scoping review is a type of review that tries to minimize bias by using transparent and repeatable methods.
However, a scoping review isn’t a type of systematic review. The most important difference is the goal: rather than answering a specific question, a scoping review explores a topic. The researcher tries to identify the main concepts, theories, and evidence, as well as gaps in the current research.
Sometimes scoping reviews are an exploratory preparation step for a systematic review, and sometimes they are a standalone project.
Prevent plagiarism. Run a free check.
A systematic review is a good choice of review if you want to answer a question about the effectiveness of an intervention , such as a medical treatment.
To conduct a systematic review, you’ll need the following:
- A precise question , usually about the effectiveness of an intervention. The question needs to be about a topic that’s previously been studied by multiple researchers. If there’s no previous research, there’s nothing to review.
- If you’re doing a systematic review on your own (e.g., for a research paper or thesis ), you should take appropriate measures to ensure the validity and reliability of your research.
- Access to databases and journal archives. Often, your educational institution provides you with access.
- Time. A professional systematic review is a time-consuming process: it will take the lead author about six months of full-time work. If you’re a student, you should narrow the scope of your systematic review and stick to a tight schedule.
- Bibliographic, word-processing, spreadsheet, and statistical software . For example, you could use EndNote, Microsoft Word, Excel, and SPSS.
A systematic review has many pros .
- They minimize research bias by considering all available evidence and evaluating each study for bias.
- Their methods are transparent , so they can be scrutinized by others.
- They’re thorough : they summarize all available evidence.
- They can be replicated and updated by others.
Systematic reviews also have a few cons .
- They’re time-consuming .
- They’re narrow in scope : they only answer the precise research question.
The 7 steps for conducting a systematic review are explained with an example.
Step 1: Formulate a research question
Formulating the research question is probably the most important step of a systematic review. A clear research question will:
- Allow you to more effectively communicate your research to other researchers and practitioners
- Guide your decisions as you plan and conduct your systematic review
A good research question for a systematic review has four components, which you can remember with the acronym PICO :
- Population(s) or problem(s)
You can rearrange these four components to write your research question:
- What is the effectiveness of I versus C for O in P ?
Sometimes, you may want to include a fifth component, the type of study design . In this case, the acronym is PICOT .
- Type of study design(s)
- The population of patients with eczema
- The intervention of probiotics
- In comparison to no treatment, placebo , or non-probiotic treatment
- The outcome of changes in participant-, parent-, and doctor-rated symptoms of eczema and quality of life
- Randomized control trials, a type of study design
Their research question was:
- What is the effectiveness of probiotics versus no treatment, a placebo, or a non-probiotic treatment for reducing eczema symptoms and improving quality of life in patients with eczema?
Step 2: Develop a protocol
A protocol is a document that contains your research plan for the systematic review. This is an important step because having a plan allows you to work more efficiently and reduces bias.
Your protocol should include the following components:
- Background information : Provide the context of the research question, including why it’s important.
- Research objective (s) : Rephrase your research question as an objective.
- Selection criteria: State how you’ll decide which studies to include or exclude from your review.
- Search strategy: Discuss your plan for finding studies.
- Analysis: Explain what information you’ll collect from the studies and how you’ll synthesize the data.
If you’re a professional seeking to publish your review, it’s a good idea to bring together an advisory committee . This is a group of about six people who have experience in the topic you’re researching. They can help you make decisions about your protocol.
It’s highly recommended to register your protocol. Registering your protocol means submitting it to a database such as PROSPERO or ClinicalTrials.gov .
Step 3: Search for all relevant studies
Searching for relevant studies is the most time-consuming step of a systematic review.
To reduce bias, it’s important to search for relevant studies very thoroughly. Your strategy will depend on your field and your research question, but sources generally fall into these four categories:
- Databases: Search multiple databases of peer-reviewed literature, such as PubMed or Scopus . Think carefully about how to phrase your search terms and include multiple synonyms of each word. Use Boolean operators if relevant.
- Handsearching: In addition to searching the primary sources using databases, you’ll also need to search manually. One strategy is to scan relevant journals or conference proceedings. Another strategy is to scan the reference lists of relevant studies.
- Gray literature: Gray literature includes documents produced by governments, universities, and other institutions that aren’t published by traditional publishers. Graduate student theses are an important type of gray literature, which you can search using the Networked Digital Library of Theses and Dissertations (NDLTD) . In medicine, clinical trial registries are another important type of gray literature.
- Experts: Contact experts in the field to ask if they have unpublished studies that should be included in your review.
At this stage of your review, you won’t read the articles yet. Simply save any potentially relevant citations using bibliographic software, such as Scribbr’s APA or MLA Generator .
- Databases: EMBASE, PsycINFO, AMED, LILACS, and ISI Web of Science
- Handsearch: Conference proceedings and reference lists of articles
- Gray literature: The Cochrane Library, the metaRegister of Controlled Trials, and the Ongoing Skin Trials Register
- Experts: Authors of unpublished registered trials, pharmaceutical companies, and manufacturers of probiotics
Step 4: Apply the selection criteria
Applying the selection criteria is a three-person job. Two of you will independently read the studies and decide which to include in your review based on the selection criteria you established in your protocol . The third person’s job is to break any ties.
To increase inter-rater reliability , ensure that everyone thoroughly understands the selection criteria before you begin.
If you’re writing a systematic review as a student for an assignment, you might not have a team. In this case, you’ll have to apply the selection criteria on your own; you can mention this as a limitation in your paper’s discussion.
You should apply the selection criteria in two phases:
- Based on the titles and abstracts : Decide whether each article potentially meets the selection criteria based on the information provided in the abstracts.
- Based on the full texts: Download the articles that weren’t excluded during the first phase. If an article isn’t available online or through your library, you may need to contact the authors to ask for a copy. Read the articles and decide which articles meet the selection criteria.
It’s very important to keep a meticulous record of why you included or excluded each article. When the selection process is complete, you can summarize what you did using a PRISMA flow diagram .
Next, Boyle and colleagues found the full texts for each of the remaining studies. Boyle and Tang read through the articles to decide if any more studies needed to be excluded based on the selection criteria.
When Boyle and Tang disagreed about whether a study should be excluded, they discussed it with Varigos until the three researchers came to an agreement.
Step 5: Extract the data
Extracting the data means collecting information from the selected studies in a systematic way. There are two types of information you need to collect from each study:
- Information about the study’s methods and results . The exact information will depend on your research question, but it might include the year, study design , sample size, context, research findings , and conclusions. If any data are missing, you’ll need to contact the study’s authors.
- Your judgment of the quality of the evidence, including risk of bias .
You should collect this information using forms. You can find sample forms in The Registry of Methods and Tools for Evidence-Informed Decision Making and the Grading of Recommendations, Assessment, Development and Evaluations Working Group .
Extracting the data is also a three-person job. Two people should do this step independently, and the third person will resolve any disagreements.
They also collected data about possible sources of bias, such as how the study participants were randomized into the control and treatment groups.
Step 6: Synthesize the data
Synthesizing the data means bringing together the information you collected into a single, cohesive story. There are two main approaches to synthesizing the data:
- Narrative ( qualitative ): Summarize the information in words. You’ll need to discuss the studies and assess their overall quality.
- Quantitative : Use statistical methods to summarize and compare data from different studies. The most common quantitative approach is a meta-analysis , which allows you to combine results from multiple studies into a summary result.
Generally, you should use both approaches together whenever possible. If you don’t have enough data, or the data from different studies aren’t comparable, then you can take just a narrative approach. However, you should justify why a quantitative approach wasn’t possible.
Boyle and colleagues also divided the studies into subgroups, such as studies about babies, children, and adults, and analyzed the effect sizes within each group.
Step 7: Write and publish a report
The purpose of writing a systematic review article is to share the answer to your research question and explain how you arrived at this answer.
Your article should include the following sections:
- Abstract : A summary of the review
- Introduction : Including the rationale and objectives
- Methods : Including the selection criteria, search method, data extraction method, and synthesis method
- Results : Including results of the search and selection process, study characteristics, risk of bias in the studies, and synthesis results
- Discussion : Including interpretation of the results and limitations of the review
- Conclusion : The answer to your research question and implications for practice, policy, or research
To verify that your report includes everything it needs, you can use the PRISMA checklist .
Once your report is written, you can publish it in a systematic review database, such as the Cochrane Database of Systematic Reviews , and/or in a peer-reviewed journal.
If you want to know more about statistics , methodology , or research bias , make sure to check out some of our other articles with explanations and examples.
- Student’s t -distribution
- Normal distribution
- Null and Alternative Hypotheses
- Chi square tests
- Confidence interval
- Quartiles & Quantiles
- Cluster sampling
- Stratified sampling
- Data cleansing
- Reproducibility vs Replicability
- Peer review
- Prospective cohort study
- Implicit bias
- Cognitive bias
- Placebo effect
- Hawthorne effect
- Hindsight bias
- Affect heuristic
- Social desirability bias
A literature review is a survey of scholarly sources (such as books, journal articles, and theses) related to a specific topic or research question .
It is often written as part of a thesis, dissertation , or research paper , in order to situate your work in relation to existing knowledge.
A literature review is a survey of credible sources on a topic, often used in dissertations , theses, and research papers . Literature reviews give an overview of knowledge on a subject, helping you identify relevant theories and methods, as well as gaps in existing research. Literature reviews are set up similarly to other academic texts , with an introduction , a main body, and a conclusion .
An annotated bibliography is a list of source references that has a short description (called an annotation ) for each of the sources. It is often assigned as part of the research process for a paper .
A systematic review is secondary research because it uses existing research. You don’t collect new data yourself.
Cite this Scribbr article
If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.
Turney, S. (2023, June 22). Systematic Review | Definition, Example & Guide. Scribbr. Retrieved November 8, 2023, from https://www.scribbr.com/methodology/systematic-review/
Is this article helpful?
Other students also liked, how to write a literature review | guide, examples, & templates, how to write a research proposal | examples & templates, what is critical thinking | definition & examples, what is your plagiarism score.