[Accessed 6 April 2020].
Kind regards,
Thank u Derek Your lectures are effectively informative and easily understood. They are clear and organized.
I’m so glad I came across this website. Thank you Derek !!!
You’re welcome, Rabia 🙂
Dear Derek,
Thanks for your extremely useful video tutorials. Please can you send me a link to open your Lit Review Excel Templates.
Thanks for your feedback. You can download the template following the instructions in the orange box.
All the best with your studies!
great effort
Thank you for this document. I am in the beginning stages of the dissertation process.
You’re welcome, Jerry 🙂
The video on literature review was very useful. I especially like the cataloging suggestion.
Great to hear that, Rishi. All the best for your research!
I`m busy writing my minor dissertation my master’s in engineering. I’m following your videos on youtube for writing a literature review. I’m looking for the excel sheet to save a list of references.
The literature video was helpful. Thank you. I haven’t received the excel template its been a week now . Please assist me .
Hi Morakane
That’s very unusual. At most, it should take a few minutes. Please re-attempt the download (use an alternative address if need be).
Thank you for the template. it seems useful to organize my literature review.
You’re very welcome.
Thank you for this informative site and all the tips. Very useful for my research.
You’re welcome, Feyi.
Currently writing a dissertation for a masters in social sciences. Found the youtube videos which are of great help.
You’re most welcome 🙂 Good luck with your literature review.
Thank you very much for the support!!
your videos are great helpful.
Glad to hear that! Good luck with your lit review.
You are the BEST
Thanks for the feedback, Annie. I wish you best of luck with your literature review.
where is the download link for the excel template?
Hi Lebogang
The download is below the first image. Good luck with your literature review!
Your videos literally saved me!!!! Due to recent issues, most of my classes were cancelled and i was completely lost. No words can explain how much grateful i am to you!!
Glad to hear we helped you! Good luck with your literature review.
You guys are the kinds of people who should survive covid-19. You are the type of people we really need in this planet. You are a star. You really saved me from many headaches. Thank you very much for the useful videos and the literature organisation spreadsheet.
Thanks for the kind words, Abel. All the best for your literature review!
Thanks so much for your video. I have consistently received comments that my arguments don’t flow well and I could never figure out how to successfully fix this issue. Now I feel I have a plan and someone to help me and provide feedback if I still don’t get it quite well. Looking forward to getting an improved mark on my next Lit review Thank you
Great to hear that, Helen. Good luck with your future literature reviews!
Excellent lecture
Thanks Biren – good luck with your literature review
i have watched your video on three steps to write a literature review and i found it very useful. thank you for sharing. keep it up.
Thanks again
Thank you for providing such excellent information and sources. Your videos helped me so much. I was on the verge of quitting. Thank you again for your videos and recommended tools.
Great to hear that, Tanya. All the best for your literature review, and for your research.
The video was very informative and timely for me. I am about to start, so Gradcoach is a source I will be revisiitng
Thanks, Nina – glad to hear that. Good luck with your literature review 🙂
hey your video is awesome I had to make an assignment on literature Review and it helped me to get an outline on how I should start ! i was fed of reading books and online articles. Your video served as a boon and clarified my thought process – how I should move forward .Thank you so much!
Great to hear that, Kavita. All the best with your literature review!
Hi Derek, I have tried unsuccessfully to download the Excel template but it keeps bringing me back to this comment section. Is it a technical issue? Kindly help.
Sorry about that. Can you please send me a screenshot of what you’re seeing – [email protected] . I will send you the template as well.
Best of luck with your literature review.
This is so very helpful!! I am writing my first lit review within a proposal (rather last minute, yikes) and this is so helpful to stay organized!
Pleasure. Good luck with your lit review 🙂
Hi i like the video,it is very helpful especially now that I am working on my proposal for thesis project….Hope I will be able to use the excel template to organize for my literature review
Great to hear that, Faith. All the best with your literature review!
hey Derek this video is absolutely amazing. One problem though I’m one of the few that are struggling to download excel. I keep clicking on it and nothing happens.
Thanks for letting us know. Please email me a screenshot of your error and I’ll sort you out – [email protected]
Thanks, Derek
Thanks a lot! Very well explained and easy to follow…now I guess I have no excuse to actually do the work 😉
Thanks for your comment! Good luck with your literature review.
Your video is very informative and useful. Thanks a lot. I also want to try the template but I can’t the find the download link…
The download button is below the main image.
Very helpful thank you
Hi! It is a big help for beginners, such as me. Thanks a lot for sharing!
Thank you. All the best with your literature review.
This is brilliant, Pls sir, in writing a review article, how deep can u go. Is it necessary to go down to the inception of your area of research?
How do I know the country/region of research article?
This will usually be covered somewhere in the article itself.
I would like this free resource
You’re welcome to download it. The download button is below the main image. Good luck with your literature review 🙂
I’d love to have this resource pls. Thank you so much
You’re welcome to download it. The download button is below the main image.
Thanks for the you tube videos. they are very informative
Thank you so much for the full tutorial with so much detailed information. I’m a Ph.D. Candidate in China. The whole syllabus of the Ph.D. program sucks. Thanks again for sharing all this helpful information. I hope your team will getting better in the future!
You’re very welcome. Good luck writing your literature review.
It’s such a big help for me. Thank you!
I’ve watched your video on writing a research proposal. I am interested in the lit review excel template.
I have watched your lecture video on writing a research proposal. I am interested in the literature review excel template and the book write smarter not harder.
It is very helpful. Thank you for your experience sharing.
You’re welcome – good luck with your literature review 🙂
Good afternoon, I recall listening/seeing in 1 of your videos *of saving the abstract (PDF) together with the excel database. How do you do this? is it also with this excel sheet
Thank you ..your videos are a confidence booster
*How To Write A Literature Review In 3 Steps (Full Tutorial)
Wonderful work !!! Please share more !!!! I will be very happy.
Thanks so much for your precision in your presentation. I have not yet started practicing but it’s one of the best I have come across. More grease to your elbows.
I love every video on research that you ve made so far. Thanks a lot
انت رائع جدا
You’re welcome. Good luck with your literature review 🙂
Excellent work. Very helpful. I am starting in this beautiful activity of writing papers with my research . I am learning a lot. Thank you very much.
Glad to hear that. Good luck writing up your research papers!
Thank you so much for the free Excel document! It’s such a huge time-saver!
You’re most welcome, Rebecca. Good luck writing your literature review 🙂
I am so grateful that I have found you on YouTube!
In the meantime, is it better to make another excel file for another variable of the same thesis or just put all articles of all variables in 1 excel file?
Thank you very much!
The notes have been very helpful to me thank you very much for sharing
You’re most welcome, Juan 🙂
Just recently seen your youtube video. Its very information. I usually gets running out of words while writing literature review. Example: XX et al investigated, YY et al shown that, ZZ et al demonstrated…….. After 4-5 references, I feels like again am repeating the words investigated, demonstrated… Could you please shoe some references with a set of vocabularies that can be used while writing literature review section.
Thank you in advance
Thank you so much. Amazing tutorial. Am feeling educated now. Lol…
Glad to hear that, Frank. Good luck with your literature review!
Very helpful stuff, thank you so much for the free Excel! I’m going to use it for my DBA and get your YT channel.
Hi, thank you for the great insights! I was unable download the template even though I completed the form. Would you be able to help me?
Derek thanks for sharing your sacrifices. I love the clarity and confidence, it takes experience to do such.
I just downloaded the excel template for LR coupled with the explanation on how to use. I found it useful, thank you!
Do you have any recommendations for adding Key quotes from a reference ?
Great content. Template very useful
Awesome! An answer to my prayer. I found this in time I need it most. Thank you for the spirit of service.
You’re most welcome, Jojo. Good luck with your literature review.
I am really impressed. This discussion helped me a lot to reconsider a lot of issues.
Thanks for the kind words. Good luck with your literature review!
This is amazing! I really like the guidance you are giving here. However, can you throw more light on the ‘category’ columns for me? I’m really nit clear on that. Thanks
Thanks for your comment. Please see my reply to Sasquia’s question re the same thing.
Good luck with your research!
I have been sitting on an enormous amount of articles for months with difficulties in organizing them until i discovered your video on literature review (YouTube). It brought me to this page where you also had a free template for us. Research process is so much bearable now than i expected. Highly recommended for all researchers. Thank you very much.
Great template. Quick question: Are the categories KEYWORDS that I draw from each source? or pre-planned TOPICS that I come up with to organize the source content?
Thanks for your comment. You can use the category columns in whichever way works for you. It would be different for each student depending on the nature of their research and their research objectives.
Hi there, can you suggest how the corresponding literature resources are best saved into a document folder for retrieval later.
I have seen some suggesting using a unique identifier in a master tab in the spreadsheet so as to be able to create a separate tab for quotes or similar thus using the id as the link
But no one has gone on to say if they are also saving the source document in a folder and naming it 57 or author last name, title or other.
I checked out your Literature Kickstarter and the screen shot of the articles didn’t look to correspond with the catalogue. Have been meaning to sort out my reference folders for sometime and am inspired by the use of an excel spreadsheet but not sure what to name files (currently saved in theme folders) Any help would be gratefully received. Thanks
I am happy if I get a catalogue excel template on the research are of zeolite synthesis from local clay for water treatment mechanism. I need help.
I love the template! But I would like to change the name of some of the headings, used in the dropdown, i.e. change “Audio Recording” to “Podcast”. How could I do that?
Very helpful!
Great!!! Very handy.
Thankyou so much. The excel file is really helpful. This really means and is helping a lot for me.
Hello, please, how can i get your excel document to catacogue the ideas for my literature review. Can you also assist on how to build the methodology section of my literature review? Thank you in advance.
I’m a student from Indonesia..This is very useful for me.. Thank you Derek..
What is the better, download all literatures and then log them into the excel sheet or do that for one by one?
Dear Derek.
I was utterly stressed when taking on an MSc Educational Leadership distance learning degree after 30 years of no academic studying. However, I found your literature review tutorial on Youtube and I immediately experienced a sense of calm direction. I am working full time in the Cayman Islands and am native Afrikaans speaking, so it was such a great help with my literature review for my first assignment. However, I have to write an evaluative essay for my second module and can not find any tutorial done by you about this. Do you perhaps have a template I could use? I have also used your services for editing and proofreading and am super grateful for the amazing help I have received! THANK YOU!
Hi Mr Derek,
It really really helps me to summarise my LR in Excel form and start-up writing
Hi Derek I have tried to download the template and it has failed to. I am not receiving the email either, could this be network issues.
Hi Derek I have been able to download the template. thank you for all your support. let me get started
I have downloaded the template. I would like to print out the guide so I can easily follow. Hope that is fine with you.
THANKS A LOTTTTT This template is exactly the one I needed when reading the literature review for my Bachelor’s dissertation
Thank you so much for your support ,I have downloaded your template and it is amazing .
Derek, The products you and the team members have put together continue to provide exemplary help as I finish the journey toward completing my dissertation! I wish I would have known of GradCoach during both of my MBAs. It could have helped alleviate a lot of time and frustration! I look forward to learning and seeing new things as I complete the dissertation.
Thanks for the kind words 🙂
Can data will be entered in excel sheet automatically like in Mendeley or i have to enter manually, pl?
Thank you GRADCOACH, I’m keenly following your tutorials as I’m about to start my literature review. These videos have been very helpful. So for the literature review you recommend only checking abstract, introduction and conclusion of the relevant literature?
Thanks for providing such an amazing resource.
I wish I knew about this when I was doing my masters. I’m doing my PhD now and sitting on Word files of reference lists and quotes I made for my MEd. This catalog will help me to keep everything more organized in one place. I’ve already started making your template my own by adding additional columns that are important for my research topic. One of the best features of your template is the Literature summary page. My question is how do I get the information I put into my new columns to auto-populate with descriptive statistics on the Literature Summary page?
Hi, I still don’t understand what you would put as ‘Category 1’, ‘Category 2’, ‘Category X’. Are they like the sort of big topics covered in the paper?
This is very helpful
Your email address will not be published. Required fields are marked *
Save my name, email, and website in this browser for the next time I comment.
An official website of the United States government
The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.
The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.
Marco pautasso.
1 Centre for Functional and Evolutionary Ecology (CEFE), CNRS, Montpellier, France
2 Centre for Biodiversity Synthesis and Analysis (CESAB), FRB, Aix-en-Provence, France
Literature reviews are in great demand in most scientific fields. Their need stems from the ever-increasing output of scientific publications [1] . For example, compared to 1991, in 2008 three, eight, and forty times more papers were indexed in Web of Science on malaria, obesity, and biodiversity, respectively [2] . Given such mountains of papers, scientists cannot be expected to examine in detail every single new paper relevant to their interests [3] . Thus, it is both advantageous and necessary to rely on regular summaries of the recent literature. Although recognition for scientists mainly comes from primary research, timely literature reviews can lead to new synthetic insights and are often widely read [4] . For such summaries to be useful, however, they need to be compiled in a professional way [5] .
When starting from scratch, reviewing the literature can require a titanic amount of work. That is why researchers who have spent their career working on a certain research issue are in a perfect position to review that literature. Some graduate schools are now offering courses in reviewing the literature, given that most research students start their project by producing an overview of what has already been done on their research issue [6] . However, it is likely that most scientists have not thought in detail about how to approach and carry out a literature review.
Reviewing the literature requires the ability to juggle multiple tasks, from finding and evaluating relevant material to synthesising information from various sources, from critical thinking to paraphrasing, evaluating, and citation skills [7] . In this contribution, I share ten simple rules I learned working on about 25 literature reviews as a PhD and postdoctoral student. Ideas and insights also come from discussions with coauthors and colleagues, as well as feedback from reviewers and editors.
How to choose which topic to review? There are so many issues in contemporary science that you could spend a lifetime of attending conferences and reading the literature just pondering what to review. On the one hand, if you take several years to choose, several other people may have had the same idea in the meantime. On the other hand, only a well-considered topic is likely to lead to a brilliant literature review [8] . The topic must at least be:
Ideas for potential reviews may come from papers providing lists of key research questions to be answered [9] , but also from serendipitous moments during desultory reading and discussions. In addition to choosing your topic, you should also select a target audience. In many cases, the topic (e.g., web services in computational biology) will automatically define an audience (e.g., computational biologists), but that same topic may also be of interest to neighbouring fields (e.g., computer science, biology, etc.).
After having chosen your topic and audience, start by checking the literature and downloading relevant papers. Five pieces of advice here:
The chances are high that someone will already have published a literature review ( Figure 1 ), if not exactly on the issue you are planning to tackle, at least on a related topic. If there are already a few or several reviews of the literature on your issue, my advice is not to give up, but to carry on with your own literature review,
The bottom-right situation (many literature reviews but few research papers) is not just a theoretical situation; it applies, for example, to the study of the impacts of climate change on plant diseases, where there appear to be more literature reviews than research studies [33] .
When searching the literature for pertinent papers and reviews, the usual rules apply:
If you read the papers first, and only afterwards start writing the review, you will need a very good memory to remember who wrote what, and what your impressions and associations were while reading each single paper. My advice is, while reading, to start writing down interesting pieces of information, insights about how to organize the review, and thoughts on what to write. This way, by the time you have read the literature you selected, you will already have a rough draft of the review.
Of course, this draft will still need much rewriting, restructuring, and rethinking to obtain a text with a coherent argument [11] , but you will have avoided the danger posed by staring at a blank document. Be careful when taking notes to use quotation marks if you are provisionally copying verbatim from the literature. It is advisable then to reformulate such quotes with your own words in the final draft. It is important to be careful in noting the references already at this stage, so as to avoid misattributions. Using referencing software from the very beginning of your endeavour will save you time.
After having taken notes while reading the literature, you will have a rough idea of the amount of material available for the review. This is probably a good time to decide whether to go for a mini- or a full review. Some journals are now favouring the publication of rather short reviews focusing on the last few years, with a limit on the number of words and citations. A mini-review is not necessarily a minor review: it may well attract more attention from busy readers, although it will inevitably simplify some issues and leave out some relevant material due to space limitations. A full review will have the advantage of more freedom to cover in detail the complexities of a particular scientific development, but may then be left in the pile of the very important papers “to be read” by readers with little time to spare for major monographs.
There is probably a continuum between mini- and full reviews. The same point applies to the dichotomy of descriptive vs. integrative reviews. While descriptive reviews focus on the methodology, findings, and interpretation of each reviewed study, integrative reviews attempt to find common ideas and concepts from the reviewed material [12] . A similar distinction exists between narrative and systematic reviews: while narrative reviews are qualitative, systematic reviews attempt to test a hypothesis based on the published evidence, which is gathered using a predefined protocol to reduce bias [13] , [14] . When systematic reviews analyse quantitative results in a quantitative way, they become meta-analyses. The choice between different review types will have to be made on a case-by-case basis, depending not just on the nature of the material found and the preferences of the target journal(s), but also on the time available to write the review and the number of coauthors [15] .
Whether your plan is to write a mini- or a full review, it is good advice to keep it focused 16 , 17 . Including material just for the sake of it can easily lead to reviews that are trying to do too many things at once. The need to keep a review focused can be problematic for interdisciplinary reviews, where the aim is to bridge the gap between fields [18] . If you are writing a review on, for example, how epidemiological approaches are used in modelling the spread of ideas, you may be inclined to include material from both parent fields, epidemiology and the study of cultural diffusion. This may be necessary to some extent, but in this case a focused review would only deal in detail with those studies at the interface between epidemiology and the spread of ideas.
While focus is an important feature of a successful review, this requirement has to be balanced with the need to make the review relevant to a broad audience. This square may be circled by discussing the wider implications of the reviewed topic for other disciplines.
Reviewing the literature is not stamp collecting. A good review does not just summarize the literature, but discusses it critically, identifies methodological problems, and points out research gaps [19] . After having read a review of the literature, a reader should have a rough idea of:
It is challenging to achieve a successful review on all these fronts. A solution can be to involve a set of complementary coauthors: some people are excellent at mapping what has been achieved, some others are very good at identifying dark clouds on the horizon, and some have instead a knack at predicting where solutions are going to come from. If your journal club has exactly this sort of team, then you should definitely write a review of the literature! In addition to critical thinking, a literature review needs consistency, for example in the choice of passive vs. active voice and present vs. past tense.
Like a well-baked cake, a good review has a number of telling features: it is worth the reader's time, timely, systematic, well written, focused, and critical. It also needs a good structure. With reviews, the usual subdivision of research papers into introduction, methods, results, and discussion does not work or is rarely used. However, a general introduction of the context and, toward the end, a recapitulation of the main points covered and take-home messages make sense also in the case of reviews. For systematic reviews, there is a trend towards including information about how the literature was searched (database, keywords, time limits) [20] .
How can you organize the flow of the main body of the review so that the reader will be drawn into and guided through it? It is generally helpful to draw a conceptual scheme of the review, e.g., with mind-mapping techniques. Such diagrams can help recognize a logical way to order and link the various sections of a review [21] . This is the case not just at the writing stage, but also for readers if the diagram is included in the review as a figure. A careful selection of diagrams and figures relevant to the reviewed topic can be very helpful to structure the text too [22] .
Reviews of the literature are normally peer-reviewed in the same way as research papers, and rightly so [23] . As a rule, incorporating feedback from reviewers greatly helps improve a review draft. Having read the review with a fresh mind, reviewers may spot inaccuracies, inconsistencies, and ambiguities that had not been noticed by the writers due to rereading the typescript too many times. It is however advisable to reread the draft one more time before submission, as a last-minute correction of typos, leaps, and muddled sentences may enable the reviewers to focus on providing advice on the content rather than the form.
Feedback is vital to writing a good review, and should be sought from a variety of colleagues, so as to obtain a diversity of views on the draft. This may lead in some cases to conflicting views on the merits of the paper, and on how to improve it, but such a situation is better than the absence of feedback. A diversity of feedback perspectives on a literature review can help identify where the consensus view stands in the landscape of the current scientific understanding of an issue [24] .
In many cases, reviewers of the literature will have published studies relevant to the review they are writing. This could create a conflict of interest: how can reviewers report objectively on their own work [25] ? Some scientists may be overly enthusiastic about what they have published, and thus risk giving too much importance to their own findings in the review. However, bias could also occur in the other direction: some scientists may be unduly dismissive of their own achievements, so that they will tend to downplay their contribution (if any) to a field when reviewing it.
In general, a review of the literature should neither be a public relations brochure nor an exercise in competitive self-denial. If a reviewer is up to the job of producing a well-organized and methodical review, which flows well and provides a service to the readership, then it should be possible to be objective in reviewing one's own relevant findings. In reviews written by multiple authors, this may be achieved by assigning the review of the results of a coauthor to different coauthors.
Given the progressive acceleration in the publication of scientific papers, today's reviews of the literature need awareness not just of the overall direction and achievements of a field of inquiry, but also of the latest studies, so as not to become out-of-date before they have been published. Ideally, a literature review should not identify as a major research gap an issue that has just been addressed in a series of papers in press (the same applies, of course, to older, overlooked studies (“sleeping beauties” [26] )). This implies that literature reviewers would do well to keep an eye on electronic lists of papers in press, given that it can take months before these appear in scientific databases. Some reviews declare that they have scanned the literature up to a certain point in time, but given that peer review can be a rather lengthy process, a full search for newly appeared literature at the revision stage may be worthwhile. Assessing the contribution of papers that have just appeared is particularly challenging, because there is little perspective with which to gauge their significance and impact on further research and society.
Inevitably, new papers on the reviewed topic (including independently written literature reviews) will appear from all quarters after the review has been published, so that there may soon be the need for an updated review. But this is the nature of science [27] – [32] . I wish everybody good luck with writing a review of the literature.
Many thanks to M. Barbosa, K. Dehnen-Schmutz, T. Döring, D. Fontaneto, M. Garbelotto, O. Holdenrieder, M. Jeger, D. Lonsdale, A. MacLeod, P. Mills, M. Moslonka-Lefebvre, G. Stancanelli, P. Weisberg, and X. Xu for insights and discussions, and to P. Bourne, T. Matoni, and D. Smith for helpful comments on a previous draft.
This work was funded by the French Foundation for Research on Biodiversity (FRB) through its Centre for Synthesis and Analysis of Biodiversity data (CESAB), as part of the NETSEED research project. The funders had no role in the preparation of the manuscript.
Jump to navigation
Chapter 14: completing ‘summary of findings’ tables and grading the certainty of the evidence.
Holger J Schünemann, Julian PT Higgins, Gunn E Vist, Paul Glasziou, Elie A Akl, Nicole Skoetz, Gordon H Guyatt; on behalf of the Cochrane GRADEing Methods Group (formerly Applicability and Recommendations Methods Group) and the Cochrane Statistical Methods Group
Cite this chapter as: Schünemann HJ, Higgins JPT, Vist GE, Glasziou P, Akl EA, Skoetz N, Guyatt GH. Chapter 14: Completing ‘Summary of findings’ tables and grading the certainty of the evidence. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.4 (updated August 2023). Cochrane, 2023. Available from www.training.cochrane.org/handbook .
14.1.1 introduction to ‘summary of findings’ tables.
‘Summary of findings’ tables present the main findings of a review in a transparent, structured and simple tabular format. In particular, they provide key information concerning the certainty or quality of evidence (i.e. the confidence or certainty in the range of an effect estimate or an association), the magnitude of effect of the interventions examined, and the sum of available data on the main outcomes. Cochrane Reviews should incorporate ‘Summary of findings’ tables during planning and publication, and should have at least one key ‘Summary of findings’ table representing the most important comparisons. Some reviews may include more than one ‘Summary of findings’ table, for example if the review addresses more than one major comparison, or includes substantially different populations that require separate tables (e.g. because the effects differ or it is important to show results separately). In the Cochrane Database of Systematic Reviews (CDSR), all ‘Summary of findings’ tables for a review appear at the beginning, before the Background section.
Planning for the ‘Summary of findings’ table starts early in the systematic review, with the selection of the outcomes to be included in: (i) the review; and (ii) the ‘Summary of findings’ table. This is a crucial step, and one that review authors need to address carefully.
To ensure production of optimally useful information, Cochrane Reviews begin by developing a review question and by listing all main outcomes that are important to patients and other decision makers (see Chapter 2 and Chapter 3 ). The GRADE approach to assessing the certainty of the evidence (see Section 14.2 ) defines and operationalizes a rating process that helps separate outcomes into those that are critical, important or not important for decision making. Consultation and feedback on the review protocol, including from consumers and other decision makers, can enhance this process.
Critical outcomes are likely to include clearly important endpoints; typical examples include mortality and major morbidity (such as strokes and myocardial infarction). However, they may also represent frequent minor and rare major side effects, symptoms, quality of life, burdens associated with treatment, and resource issues (costs). Burdens represent the impact of healthcare workload on patient function and well-being, and include the demands of adhering to an intervention that patients or caregivers (e.g. family) may dislike, such as having to undergo more frequent tests, or the restrictions on lifestyle that certain interventions require (Spencer-Bonilla et al 2017).
Frequently, when formulating questions that include all patient-important outcomes for decision making, review authors will confront reports of studies that have not included all these outcomes. This is particularly true for adverse outcomes. For instance, randomized trials might contribute evidence on intended effects, and on frequent, relatively minor side effects, but not report on rare adverse outcomes such as suicide attempts. Chapter 19 discusses strategies for addressing adverse effects. To obtain data for all important outcomes it may be necessary to examine the results of non-randomized studies (see Chapter 24 ). Cochrane, in collaboration with others, has developed guidance for review authors to support their decision about when to look for and include non-randomized studies (Schünemann et al 2013).
If a review includes only randomized trials, these trials may not address all important outcomes and it may therefore not be possible to address these outcomes within the constraints of the review. Review authors should acknowledge these limitations and make them transparent to readers. Review authors are encouraged to include non-randomized studies to examine rare or long-term adverse effects that may not adequately be studied in randomized trials. This raises the possibility that harm outcomes may come from studies in which participants differ from those in studies used in the analysis of benefit. Review authors will then need to consider how much such differences are likely to impact on the findings, and this will influence the certainty of evidence because of concerns about indirectness related to the population (see Section 14.2.2 ).
Non-randomized studies can provide important information not only when randomized trials do not report on an outcome or randomized trials suffer from indirectness, but also when the evidence from randomized trials is rated as very low and non-randomized studies provide evidence of higher certainty. Further discussion of these issues appears also in Chapter 24 .
Several alternative standard versions of ‘Summary of findings’ tables have been developed to ensure consistency and ease of use across reviews, inclusion of the most important information needed by decision makers, and optimal presentation (see examples at Figures 14.1.a and 14.1.b ). These formats are supported by research that focused on improved understanding of the information they intend to convey (Carrasco-Labra et al 2016, Langendam et al 2016, Santesso et al 2016). They are available through GRADE’s official software package developed to support the GRADE approach: GRADEpro GDT (www.gradepro.org).
Standard Cochrane ‘Summary of findings’ tables include the following elements using one of the accepted formats. Further guidance on each of these is provided in Section 14.1.6 .
Ideally, ‘Summary of findings’ tables are supported by more detailed tables (known as ‘evidence profiles’) to which the review may be linked, which provide more detailed explanations. Evidence profiles include the same important health outcomes, and provide greater detail than ‘Summary of findings’ tables of both of the individual considerations feeding into the grading of certainty and of the results of the studies (Guyatt et al 2011a). They ensure that a structured approach is used to rating the certainty of evidence. Although they are rarely published in Cochrane Reviews, evidence profiles are often used, for example, by guideline developers in considering the certainty of the evidence to support guideline recommendations. Review authors will find it easier to develop the ‘Summary of findings’ table by completing the rating of the certainty of evidence in the evidence profile first in GRADEpro GDT. They can then automatically convert this to one of the ‘Summary of findings’ formats in GRADEpro GDT, including an interactive ‘Summary of findings’ for publication.
As a measure of the magnitude of effect for dichotomous outcomes, the ‘Summary of findings’ table should provide a relative measure of effect (e.g. risk ratio, odds ratio, hazard) and measures of absolute risk. For other types of data, an absolute measure alone (such as a difference in means for continuous data) might be sufficient. It is important that the magnitude of effect is presented in a meaningful way, which may require some transformation of the result of a meta-analysis (see also Chapter 15, Section 15.4 and Section 15.5 ). Reviews with more than one main comparison should include a separate ‘Summary of findings’ table for each comparison.
Figure 14.1.a provides an example of a ‘Summary of findings’ table. Figure 15.1.b provides an alternative format that may further facilitate users’ understanding and interpretation of the review’s findings. Evidence evaluating different formats suggests that the ‘Summary of findings’ table should include a risk difference as a measure of the absolute effect and authors should preferably use a format that includes a risk difference .
A detailed description of the contents of a ‘Summary of findings’ table appears in Section 14.1.6 .
Figure 14.1.a Example of a ‘Summary of findings’ table
Summary of findings (for interactive version click here )
a All the stockings in the nine studies included in this review were below-knee compression stockings. In four studies the compression strength was 20 mmHg to 30 mmHg at the ankle. It was 10 mmHg to 20 mmHg in the other four studies. Stockings come in different sizes. If a stocking is too tight around the knee it can prevent essential venous return causing the blood to pool around the knee. Compression stockings should be fitted properly. A stocking that is too tight could cut into the skin on a long flight and potentially cause ulceration and increased risk of DVT. Some stockings can be slightly thicker than normal leg covering and can be potentially restrictive with tight foot wear. It is a good idea to wear stockings around the house prior to travel to ensure a good, comfortable fit. Participants put their stockings on two to three hours before the flight in most of the studies. The availability and cost of stockings can vary.
b Two studies recruited high risk participants defined as those with previous episodes of DVT, coagulation disorders, severe obesity, limited mobility due to bone or joint problems, neoplastic disease within the previous two years, large varicose veins or, in one of the studies, participants taller than 190 cm and heavier than 90 kg. The incidence for the seven studies that excluded high risk participants was 1.45% and the incidence for the two studies that recruited high-risk participants (with at least one risk factor) was 2.43%. We have used 10 and 30 per 1000 to express different risk strata, respectively.
c The confidence interval crosses no difference and does not rule out a small increase.
d The measurement of oedema was not validated (indirectness of the outcome) or blinded to the intervention (risk of bias).
e If there are very few or no events and the number of participants is large, judgement about the certainty of evidence (particularly judgements about imprecision) may be based on the absolute effect. Here the certainty rating may be considered ‘high’ if the outcome was appropriately assessed and the event, in fact, did not occur in 2821 studied participants.
f None of the other studies reported adverse effects, apart from four cases of superficial vein thrombosis in varicose veins in the knee region that were compressed by the upper edge of the stocking in one study.
Figure 14.1.b Example of alternative ‘Summary of findings’ table
The GRADE Working Group’s software, GRADEpro GDT ( www.gradepro.org ), including GRADE’s interactive handbook, is available to assist review authors in the preparation of ‘Summary of findings’ tables. GRADEpro can use data on the comparator group risk and the effect estimate (entered by the review authors or imported from files generated in RevMan) to produce the relative effects and absolute risks associated with experimental interventions. In addition, it leads the user through the process of a GRADE assessment, and produces a table that can be used as a standalone table in a review (including by direct import into software such as RevMan or integration with RevMan Web), or an interactive ‘Summary of findings’ table (see help resources in GRADEpro).
14.1.5.1 dichotomous outcomes.
‘Summary of findings’ tables should include both absolute and relative measures of effect for dichotomous outcomes. Risk ratios, odds ratios and risk differences are different ways of comparing two groups with dichotomous outcome data (see Chapter 6, Section 6.4.1 ). Furthermore, there are two distinct risk ratios, depending on which event (e.g. ‘yes’ or ‘no’) is the focus of the analysis (see Chapter 6, Section 6.4.1.5 ). In the presence of a non-zero intervention effect, any variation across studies in the comparator group risks (i.e. variation in the risk of the event occurring without the intervention of interest, for example in different populations) makes it impossible for more than one of these measures to be truly the same in every study.
It has long been assumed in epidemiology that relative measures of effect are more consistent than absolute measures of effect from one scenario to another. There is empirical evidence to support this assumption (Engels et al 2000, Deeks and Altman 2001, Furukawa et al 2002). For this reason, meta-analyses should generally use either a risk ratio or an odds ratio as a measure of effect (see Chapter 10, Section 10.4.3 ). Correspondingly, a single estimate of relative effect is likely to be a more appropriate summary than a single estimate of absolute effect. If a relative effect is indeed consistent across studies, then different comparator group risks will have different implications for absolute benefit. For instance, if the risk ratio is consistently 0.75, then the experimental intervention would reduce a comparator group risk of 80% to 60% in the intervention group (an absolute risk reduction of 20 percentage points), but would also reduce a comparator group risk of 20% to 15% in the intervention group (an absolute risk reduction of 5 percentage points).
‘Summary of findings’ tables are built around the assumption of a consistent relative effect. It is therefore important to consider the implications of this effect for different comparator group risks (these can be derived or estimated from a number of sources, see Section 14.1.6.3 ), which may require an assessment of the certainty of evidence for prognostic evidence (Spencer et al 2012, Iorio et al 2015). For any comparator group risk, it is possible to estimate a corresponding intervention group risk (i.e. the absolute risk with the intervention) from the meta-analytic risk ratio or odds ratio. Note that the numbers provided in the ‘Corresponding risk’ column are specific to the ‘risks’ in the adjacent column.
For the meta-analytic risk ratio (RR) and assumed comparator risk (ACR) the corresponding intervention risk is obtained as:
As an example, in Figure 14.1.a , the meta-analytic risk ratio for symptomless deep vein thrombosis (DVT) is RR = 0.10 (95% CI 0.04 to 0.26). Assuming a comparator risk of ACR = 10 per 1000 = 0.01, we obtain:
For the meta-analytic odds ratio (OR) and assumed comparator risk, ACR, the corresponding intervention risk is obtained as:
Upper and lower confidence limits for the corresponding intervention risk are obtained by replacing RR or OR by their upper and lower confidence limits, respectively (e.g. replacing 0.10 with 0.04, then with 0.26, in the example). Such confidence intervals do not incorporate uncertainty in the assumed comparator risks.
When dealing with risk ratios, it is critical that the same definition of ‘event’ is used as was used for the meta-analysis. For example, if the meta-analysis focused on ‘death’ (as opposed to survival) as the event, then corresponding risks in the ‘Summary of findings’ table must also refer to ‘death’.
In (rare) circumstances in which there is clear rationale to assume a consistent risk difference in the meta-analysis, in principle it is possible to present this for relevant ‘assumed risks’ and their corresponding risks, and to present the corresponding (different) relative effects for each assumed risk.
The risk difference expresses the difference between the ACR and the corresponding intervention risk (or the difference between the experimental and the comparator intervention).
For the meta-analytic risk ratio (RR) and assumed comparator risk (ACR) the corresponding risk difference is obtained as (note that risks can also be expressed using percentage or percentage points):
As an example, in Figure 14.1.b the meta-analytic risk ratio is 0.41 (95% CI 0.29 to 0.55) for diarrhoea in children less than 5 years of age. Assuming a comparator group risk of 22.3% we obtain:
For the meta-analytic odds ratio (OR) and assumed comparator risk (ACR) the absolute risk difference is obtained as (percentage points):
Upper and lower confidence limits for the absolute risk difference are obtained by re-running the calculation above while replacing RR or OR by their upper and lower confidence limits, respectively (e.g. replacing 0.41 with 0.28, then with 0.55, in the example). Such confidence intervals do not incorporate uncertainty in the assumed comparator risks.
Time-to-event outcomes measure whether and when a particular event (e.g. death) occurs (van Dalen et al 2007). The impact of the experimental intervention relative to the comparison group on time-to-event outcomes is usually measured using a hazard ratio (HR) (see Chapter 6, Section 6.8.1 ).
A hazard ratio expresses a relative effect estimate. It may be used in various ways to obtain absolute risks and other interpretable quantities for a specific population. Here we describe how to re-express hazard ratios in terms of: (i) absolute risk of event-free survival within a particular period of time; (ii) absolute risk of an event within a particular period of time; and (iii) median time to the event. All methods are built on an assumption of consistent relative effects (i.e. that the hazard ratio does not vary over time).
(i) Absolute risk of event-free survival within a particular period of time Event-free survival (e.g. overall survival) is commonly reported by individual studies. To obtain absolute effects for time-to-event outcomes measured as event-free survival, the summary HR can be used in conjunction with an assumed proportion of patients who are event-free in the comparator group (Tierney et al 2007). This proportion of patients will be specific to a period of time of observation. However, it is not strictly necessary to specify this period of time. For instance, a proportion of 50% of event-free patients might apply to patients with a high event rate observed over 1 year, or to patients with a low event rate observed over 2 years.
As an example, suppose the meta-analytic hazard ratio is 0.42 (95% CI 0.25 to 0.72). Assuming a comparator group risk of event-free survival (e.g. for overall survival people being alive) at 2 years of ACR = 900 per 1000 = 0.9 we obtain:
so that that 956 per 1000 people will be alive with the experimental intervention at 2 years. The derivation of the risk should be explained in a comment or footnote.
(ii) Absolute risk of an event within a particular period of time To obtain this absolute effect, again the summary HR can be used (Tierney et al 2007):
In the example, suppose we assume a comparator group risk of events (e.g. for mortality, people being dead) at 2 years of ACR = 100 per 1000 = 0.1. We obtain:
so that that 44 per 1000 people will be dead with the experimental intervention at 2 years.
(iii) Median time to the event Instead of absolute numbers, the time to the event in the intervention and comparison groups can be expressed as median survival time in months or years. To obtain median survival time the pooled HR can be applied to an assumed median survival time in the comparator group (Tierney et al 2007):
In the example, assuming a comparator group median survival time of 80 months, we obtain:
For all three of these options for re-expressing results of time-to-event analyses, upper and lower confidence limits for the corresponding intervention risk are obtained by replacing HR by its upper and lower confidence limits, respectively (e.g. replacing 0.42 with 0.25, then with 0.72, in the example). Again, as for dichotomous outcomes, such confidence intervals do not incorporate uncertainty in the assumed comparator group risks. This is of special concern for long-term survival with a low or moderate mortality rate and a corresponding high number of censored patients (i.e. a low number of patients under risk and a high censoring rate).
14.1.6.1 table title and header.
The title of each ‘Summary of findings’ table should specify the healthcare question, framed in terms of the population and making it clear exactly what comparison of interventions are made. In Figure 14.1.a , the population is people taking long aeroplane flights, the intervention is compression stockings, and the control is no compression stockings.
The first rows of each ‘Summary of findings’ table should provide the following ‘header’ information:
Patients or population This further clarifies the population (and possibly the subpopulations) of interest and ideally the magnitude of risk of the most crucial adverse outcome at which an intervention is directed. For instance, people on a long-haul flight may be at different risks for DVT; those using selective serotonin reuptake inhibitors (SSRIs) might be at different risk for side effects; while those with atrial fibrillation may be at low (< 1%), moderate (1% to 4%) or high (> 4%) yearly risk of stroke.
Setting This should state any specific characteristics of the settings of the healthcare question that might limit the applicability of the summary of findings to other settings (e.g. primary care in Europe and North America).
Intervention The experimental intervention.
Comparison The comparator intervention (including no specific intervention).
The rows of a ‘Summary of findings’ table should include all desirable and undesirable health outcomes (listed in order of importance) that are essential for decision making, up to a maximum of seven outcomes. If there are more outcomes in the review, review authors will need to omit the less important outcomes from the table, and the decision selecting which outcomes are critical or important to the review should be made during protocol development (see Chapter 3 ). Review authors should provide time frames for the measurement of the outcomes (e.g. 90 days or 12 months) and the type of instrument scores (e.g. ranging from 0 to 100).
Note that review authors should include the pre-specified critical and important outcomes in the table whether data are available or not. However, they should be alert to the possibility that the importance of an outcome (e.g. a serious adverse effect) may only become known after the protocol was written or the analysis was carried out, and should take appropriate actions to include these in the ‘Summary of findings’ table.
The ‘Summary of findings’ table can include effects in subgroups of the population for different comparator risks and effect sizes separately. For instance, in Figure 14.1.b effects are presented for children younger and older than 5 years separately. Review authors may also opt to produce separate ‘Summary of findings’ tables for different populations.
Review authors should include serious adverse events, but it might be possible to combine minor adverse events as a single outcome, and describe this in an explanatory footnote (note that it is not appropriate to add events together unless they are independent, that is, a participant who has experienced one adverse event has an unaffected chance of experiencing the other adverse event).
Outcomes measured at multiple time points represent a particular problem. In general, to keep the table simple, review authors should present multiple time points only for outcomes critical to decision making, where either the result or the decision made are likely to vary over time. The remainder should be presented at a common time point where possible.
Review authors can present continuous outcome measures in the ‘Summary of findings’ table and should endeavour to make these interpretable to the target audience. This requires that the units are clear and readily interpretable, for example, days of pain, or frequency of headache, and the name and scale of any measurement tools used should be stated (e.g. a Visual Analogue Scale, ranging from 0 to 100). However, many measurement instruments are not readily interpretable by non-specialist clinicians or patients, for example, points on a Beck Depression Inventory or quality of life score. For these, a more interpretable presentation might involve converting a continuous to a dichotomous outcome, such as >50% improvement (see Chapter 15, Section 15.5 ).
Review authors should provide up to three typical risks for participants receiving the comparator intervention. For dichotomous outcomes, we recommend that these be presented in the form of the number of people experiencing the event per 100 or 1000 people (natural frequency) depending on the frequency of the outcome. For continuous outcomes, this would be stated as a mean or median value of the outcome measured.
Estimated or assumed comparator intervention risks could be based on assessments of typical risks in different patient groups derived from the review itself, individual representative studies in the review, or risks derived from a systematic review of prognosis studies or other sources of evidence which may in turn require an assessment of the certainty for the prognostic evidence (Spencer et al 2012, Iorio et al 2015). Ideally, risks would reflect groups that clinicians can easily identify on the basis of their presenting features.
An explanatory footnote should specify the source or rationale for each comparator group risk, including the time period to which it corresponds where appropriate. In Figure 14.1.a , clinicians can easily differentiate individuals with risk factors for deep venous thrombosis from those without. If there is known to be little variation in baseline risk then review authors may use the median comparator group risk across studies. If typical risks are not known, an option is to choose the risk from the included studies, providing the second highest for a high and the second lowest for a low risk population.
For dichotomous outcomes, review authors should provide a corresponding absolute risk for each comparator group risk, along with a confidence interval. This absolute risk with the (experimental) intervention will usually be derived from the meta-analysis result presented in the relative effect column (see Section 14.1.6.6 ). Formulae are provided in Section 14.1.5 . Review authors should present the absolute effect in the same format as the risks with comparator intervention (see Section 14.1.6.3 ), for example as the number of people experiencing the event per 1000 people.
For continuous outcomes, a difference in means or standardized difference in means should be presented with its confidence interval. These will typically be obtained directly from a meta-analysis. Explanatory text should be used to clarify the meaning, as in Figures 14.1.a and 14.1.b .
For dichotomous outcomes, the risk difference can be provided using one of the ‘Summary of findings’ table formats as an additional option (see Figure 14.1.b ). This risk difference expresses the difference between the experimental and comparator intervention and will usually be derived from the meta-analysis result presented in the relative effect column (see Section 14.1.6.6 ). Formulae are provided in Section 14.1.5 . Review authors should present the risk difference in the same format as assumed and corresponding risks with comparator intervention (see Section 14.1.6.3 ); for example, as the number of people experiencing the event per 1000 people or as percentage points if the assumed and corresponding risks are expressed in percentage.
For continuous outcomes, if the ‘Summary of findings’ table includes this option, the mean difference can be presented here and the ‘corresponding risk’ column left blank (see Figure 14.1.b ).
The relative effect will typically be a risk ratio or odds ratio (or occasionally a hazard ratio) with its accompanying 95% confidence interval, obtained from a meta-analysis performed on the basis of the same effect measure. Risk ratios and odds ratios are similar when the comparator intervention risks are low and effects are small, but may differ considerably when comparator group risks increase. The meta-analysis may involve an assumption of either fixed or random effects, depending on what the review authors consider appropriate, and implying that the relative effect is either an estimate of the effect of the intervention, or an estimate of the average effect of the intervention across studies, respectively.
This column should include the number of participants assessed in the included studies for each outcome and the corresponding number of studies that contributed these participants.
Review authors should comment on the certainty of the evidence (also known as quality of the body of evidence or confidence in the effect estimates). Review authors should use the specific evidence grading system developed by the GRADE Working Group (Atkins et al 2004, Guyatt et al 2008, Guyatt et al 2011a), which is described in detail in Section 14.2 . The GRADE approach categorizes the certainty in a body of evidence as ‘high’, ‘moderate’, ‘low’ or ‘very low’ by outcome. This is a result of judgement, but the judgement process operates within a transparent structure. As an example, the certainty would be ‘high’ if the summary were of several randomized trials with low risk of bias, but the rating of certainty becomes lower if there are concerns about risk of bias, inconsistency, indirectness, imprecision or publication bias. Judgements other than of ‘high’ certainty should be made transparent using explanatory footnotes or the ‘Comments’ column in the ‘Summary of findings’ table (see Section 14.1.6.10 ).
The aim of the ‘Comments’ field is to help interpret the information or data identified in the row. For example, this may be on the validity of the outcome measure or the presence of variables that are associated with the magnitude of effect. Important caveats about the results should be flagged here. Not all rows will need comments, and it is best to leave a blank if there is nothing warranting a comment.
Detailed explanations should be included as footnotes to support the judgements in the ‘Summary of findings’ table, such as the overall GRADE assessment. The explanations should describe the rationale for important aspects of the content. Table 14.1.a lists guidance for useful explanations. Explanations should be concise, informative, relevant, easy to understand and accurate. If explanations cannot be sufficiently described in footnotes, review authors should provide further details of the issues in the Results and Discussion sections of the review.
Table 14.1.a Guidance for providing useful explanations in ‘Summary of findings’ (SoF) tables. Adapted from Santesso et al (2016)
14.2.1 the grade approach.
The Grades of Recommendation, Assessment, Development and Evaluation Working Group (GRADE Working Group) has developed a system for grading the certainty of evidence (Schünemann et al 2003, Atkins et al 2004, Schünemann et al 2006, Guyatt et al 2008, Guyatt et al 2011a). Over 100 organizations including the World Health Organization (WHO), the American College of Physicians, the American Society of Hematology (ASH), the Canadian Agency for Drugs and Technology in Health (CADTH) and the National Institutes of Health and Clinical Excellence (NICE) in the UK have adopted the GRADE system ( www.gradeworkinggroup.org ).
Cochrane has also formally adopted this approach, and all Cochrane Reviews should use GRADE to evaluate the certainty of evidence for important outcomes (see MECIR Box 14.2.a ).
MECIR Box 14.2.a Relevant expectations for conduct of intervention reviews
For systematic reviews, the GRADE approach defines the certainty of a body of evidence as the extent to which one can be confident that an estimate of effect or association is close to the quantity of specific interest. Assessing the certainty of a body of evidence involves consideration of within- and across-study risk of bias (limitations in study design and execution or methodological quality), inconsistency (or heterogeneity), indirectness of evidence, imprecision of the effect estimates and risk of publication bias (see Section 14.2.2 ), as well as domains that may increase our confidence in the effect estimate (as described in Section 14.2.3 ). The GRADE system entails an assessment of the certainty of a body of evidence for each individual outcome. Judgements about the domains that determine the certainty of evidence should be described in the results or discussion section and as part of the ‘Summary of findings’ table.
The GRADE approach specifies four levels of certainty ( Figure 14.2.a ). For interventions, including diagnostic and other tests that are evaluated as interventions (Schünemann et al 2008b, Schünemann et al 2008a, Balshem et al 2011, Schünemann et al 2012), the starting point for rating the certainty of evidence is categorized into two types:
There are many instances in which review authors rely on information from NRSI, in particular to evaluate potential harms (see Chapter 24 ). In addition, review authors can obtain relevant data from both randomized trials and NRSI, with each type of evidence complementing the other (Schünemann et al 2013).
In GRADE, a body of evidence from randomized trials begins with a high-certainty rating while a body of evidence from NRSI begins with a low-certainty rating. The lower rating with NRSI is the result of the potential bias induced by the lack of randomization (i.e. confounding and selection bias).
However, when using the new Risk Of Bias In Non-randomized Studies of Interventions (ROBINS-I) tool (Sterne et al 2016), an assessment tool that covers the risk of bias due to lack of randomization, all studies may start as high certainty of the evidence (Schünemann et al 2018). The approach of starting all study designs (including NRSI) as high certainty does not conflict with the initial GRADE approach of starting the rating of NRSI as low certainty evidence. This is because a body of evidence from NRSI should generally be downgraded by two levels due to the inherent risk of bias associated with the lack of randomization, namely confounding and selection bias. Not downgrading NRSI from high to low certainty needs transparent and detailed justification for what mitigates concerns about confounding and selection bias (Schünemann et al 2018). Very few examples of where not rating down by two levels is appropriate currently exist.
The highest certainty rating is a body of evidence when there are no concerns in any of the GRADE factors listed in Figure 14.2.a . Review authors often downgrade evidence to moderate, low or even very low certainty evidence, depending on the presence of the five factors in Figure 14.2.a . Usually, certainty rating will fall by one level for each factor, up to a maximum of three levels for all factors. If there are very severe problems for any one domain (e.g. when assessing risk of bias, all studies were unconcealed, unblinded and lost over 50% of their patients to follow-up), evidence may fall by two levels due to that factor alone. It is not possible to rate lower than ‘very low certainty’ evidence.
Review authors will generally grade evidence from sound non-randomized studies as low certainty, even if ROBINS-I is used. If, however, such studies yield large effects and there is no obvious bias explaining those effects, review authors may rate the evidence as moderate or – if the effect is large enough – even as high certainty ( Figure 14.2.a ). The very low certainty level is appropriate for, but is not limited to, studies with critical problems and unsystematic clinical observations (e.g. case series or case reports).
Figure 14.2.a Levels of the certainty of a body of evidence in the GRADE approach. *Upgrading criteria are usually applicable to non-randomized studies only (but exceptions exist).
We now describe in more detail the five reasons (or domains) for downgrading the certainty of a body of evidence for a specific outcome. In each case, if no reason is found for downgrading the evidence, it should be classified as 'no limitation or not serious' (not important enough to warrant downgrading). If a reason is found for downgrading the evidence, it should be classified as 'serious' (downgrading the certainty rating by one level) or 'very serious' (downgrading the certainty grade by two levels). For non-randomized studies assessed with ROBINS-I, rating down by three levels should be classified as 'extremely' serious.
(1) Risk of bias or limitations in the detailed design and implementation
Our confidence in an estimate of effect decreases if studies suffer from major limitations that are likely to result in a biased assessment of the intervention effect. For randomized trials, these methodological limitations include failure to generate a random sequence, lack of allocation sequence concealment, lack of blinding (particularly with subjective outcomes that are highly susceptible to biased assessment), a large loss to follow-up or selective reporting of outcomes. Chapter 8 provides a discussion of study-level assessments of risk of bias in the context of a Cochrane Review, and proposes an approach to assessing the risk of bias for an outcome across studies as ‘Low’ risk of bias, ‘Some concerns’ and ‘High’ risk of bias for randomized trials. Levels of ‘Low’. ‘Moderate’, ‘Serious’ and ‘Critical’ risk of bias arise for non-randomized studies assessed with ROBINS-I ( Chapter 25 ). These assessments should feed directly into this GRADE domain. In particular, ‘Low’ risk of bias would indicate ‘no limitation’; ‘Some concerns’ would indicate either ‘no limitation’ or ‘serious limitation’; and ‘High’ risk of bias would indicate either ‘serious limitation’ or ‘very serious limitation’. ‘Critical’ risk of bias on ROBINS-I would indicate extremely serious limitations in GRADE. Review authors should use their judgement to decide between alternative categories, depending on the likely magnitude of the potential biases.
Every study addressing a particular outcome will differ, to some degree, in the risk of bias. Review authors should make an overall judgement on whether the certainty of evidence for an outcome warrants downgrading on the basis of study limitations. The assessment of study limitations should apply to the studies contributing to the results in the ‘Summary of findings’ table, rather than to all studies that could potentially be included in the analysis. We have argued in Chapter 7, Section 7.6.2 , that the primary analysis should be restricted to studies at low (or low and unclear) risk of bias where possible.
Table 14.2.a presents the judgements that must be made in going from assessments of the risk of bias to judgements about study limitations for each outcome included in a ‘Summary of findings’ table. A rating of high certainty evidence can be achieved only when most evidence comes from studies that met the criteria for low risk of bias. For example, of the 22 studies addressing the impact of beta-blockers on mortality in patients with heart failure, most probably or certainly used concealed allocation of the sequence, all blinded at least some key groups and follow-up of randomized patients was almost complete (Brophy et al 2001). The certainty of evidence might be downgraded by one level when most of the evidence comes from individual studies either with a crucial limitation for one item, or with some limitations for multiple items. An example of very serious limitations, warranting downgrading by two levels, is provided by evidence on surgery versus conservative treatment in the management of patients with lumbar disc prolapse (Gibson and Waddell 2007). We are uncertain of the benefit of surgery in reducing symptoms after one year or longer, because the one study included in the analysis had inadequate concealment of the allocation sequence and the outcome was assessed using a crude rating by the surgeon without blinding.
(2) Unexplained heterogeneity or inconsistency of results
When studies yield widely differing estimates of effect (heterogeneity or variability in results), investigators should look for robust explanations for that heterogeneity. For instance, drugs may have larger relative effects in sicker populations or when given in larger doses. A detailed discussion of heterogeneity and its investigation is provided in Chapter 10, Section 10.10 and Section 10.11 . If an important modifier exists, with good evidence that important outcomes are different in different subgroups (which would ideally be pre-specified), then a separate ‘Summary of findings’ table may be considered for a separate population. For instance, a separate ‘Summary of findings’ table would be used for carotid endarterectomy in symptomatic patients with high grade stenosis (70% to 99%) in which the intervention is, in the hands of the right surgeons, beneficial, and another (if review authors considered it relevant) for asymptomatic patients with low grade stenosis (less than 30%) in which surgery appears harmful (Orrapin and Rerkasem 2017). When heterogeneity exists and affects the interpretation of results, but review authors are unable to identify a plausible explanation with the data available, the certainty of the evidence decreases.
(3) Indirectness of evidence
Two types of indirectness are relevant. First, a review comparing the effectiveness of alternative interventions (say A and B) may find that randomized trials are available, but they have compared A with placebo and B with placebo. Thus, the evidence is restricted to indirect comparisons between A and B. Where indirect comparisons are undertaken within a network meta-analysis context, GRADE for network meta-analysis should be used (see Chapter 11, Section 11.5 ).
Second, a review may find randomized trials that meet eligibility criteria but address a restricted version of the main review question in terms of population, intervention, comparator or outcomes. For example, suppose that in a review addressing an intervention for secondary prevention of coronary heart disease, most identified studies happened to be in people who also had diabetes. Then the evidence may be regarded as indirect in relation to the broader question of interest because the population is primarily related to people with diabetes. The opposite scenario can equally apply: a review addressing the effect of a preventive strategy for coronary heart disease in people with diabetes may consider studies in people without diabetes to provide relevant, albeit indirect, evidence. This would be particularly likely if investigators had conducted few if any randomized trials in the target population (e.g. people with diabetes). Other sources of indirectness may arise from interventions studied (e.g. if in all included studies a technical intervention was implemented by expert, highly trained specialists in specialist centres, then evidence on the effects of the intervention outside these centres may be indirect), comparators used (e.g. if the comparator groups received an intervention that is less effective than standard treatment in most settings) and outcomes assessed (e.g. indirectness due to surrogate outcomes when data on patient-important outcomes are not available, or when investigators seek data on quality of life but only symptoms are reported). Review authors should make judgements transparent when they believe downgrading is justified, based on differences in anticipated effects in the group of primary interest. Review authors may be aided and increase transparency of their judgements about indirectness if they use Table 14.2.b available in the GRADEpro GDT software (Schünemann et al 2013).
(4) Imprecision of results
When studies include few participants or few events, and thus have wide confidence intervals, review authors can lower their rating of the certainty of the evidence. The confidence intervals included in the ‘Summary of findings’ table will provide readers with information that allows them to make, to some extent, their own rating of precision. Review authors can use a calculation of the optimal information size (OIS) or review information size (RIS), similar to sample size calculations, to make judgements about imprecision (Guyatt et al 2011b, Schünemann 2016). The OIS or RIS is calculated on the basis of the number of participants required for an adequately powered individual study. If the 95% confidence interval excludes a risk ratio (RR) of 1.0, and the total number of events or patients exceeds the OIS criterion, precision is adequate. If the 95% CI includes appreciable benefit or harm (an RR of under 0.75 or over 1.25 is often suggested as a very rough guide) downgrading for imprecision may be appropriate even if OIS criteria are met (Guyatt et al 2011b, Schünemann 2016).
(5) High probability of publication bias
The certainty of evidence level may be downgraded if investigators fail to report studies on the basis of results (typically those that show no effect: publication bias) or outcomes (typically those that may be harmful or for which no effect was observed: selective outcome non-reporting bias). Selective reporting of outcomes from among multiple outcomes measured is assessed at the study level as part of the assessment of risk of bias (see Chapter 8, Section 8.7 ), so for the studies contributing to the outcome in the ‘Summary of findings’ table this is addressed by domain 1 above (limitations in the design and implementation). If a large number of studies included in the review do not contribute to an outcome, or if there is evidence of publication bias, the certainty of the evidence may be downgraded. Chapter 13 provides a detailed discussion of reporting biases, including publication bias, and how it may be tackled in a Cochrane Review. A prototypical situation that may elicit suspicion of publication bias is when published evidence includes a number of small studies, all of which are industry-funded (Bhandari et al 2004). For example, 14 studies of flavanoids in patients with haemorrhoids have shown apparent large benefits, but enrolled a total of only 1432 patients (i.e. each study enrolled relatively few patients) (Alonso-Coello et al 2006). The heavy involvement of sponsors in most of these studies raises questions of whether unpublished studies that suggest no benefit exist (publication bias).
A particular body of evidence can suffer from problems associated with more than one of the five factors listed here, and the greater the problems, the lower the certainty of evidence rating that should result. One could imagine a situation in which randomized trials were available, but all or virtually all of these limitations would be present, and in serious form. A very low certainty of evidence rating would result.
Table 14.2.a Further guidelines for domain 1 (of 5) in a GRADE assessment: going from assessments of risk of bias in studies to judgements about study limitations for main outcomes across studies
Table 14.2.b Judgements about indirectness by outcome (available in GRADEpro GDT)
Intervention:
Comparator:
Direct comparison:
Final judgement about indirectness across domains:
Although NRSI and downgraded randomized trials will generally yield a low rating for certainty of evidence, there will be unusual circumstances in which review authors could ‘upgrade’ such evidence to moderate or even high certainty ( Table 14.3.a ).
Review authors should report the grading of the certainty of evidence in the Results section for each outcome for which this has been performed, providing the rationale for downgrading or upgrading the evidence, and referring to the ‘Summary of findings’ table where applicable.
Table 14.3.a provides a framework and examples for how review authors can justify their judgements about the certainty of evidence in each domain. These justifications should also be included in explanatory notes to the ‘Summary of Findings’ table (see Section 14.1.6.10 ).
Chapter 15, Section 15.6 , describes in more detail how the overall GRADE assessment across all domains can be used to draw conclusions about the effects of the intervention, as well as providing implications for future research.
Table 14.3.a Framework for describing the certainty of evidence and justifying downgrading or upgrading
Authors: Holger J Schünemann, Julian PT Higgins, Gunn E Vist, Paul Glasziou, Elie A Akl, Nicole Skoetz, Gordon H Guyatt; on behalf of the Cochrane GRADEing Methods Group (formerly Applicability and Recommendations Methods Group) and the Cochrane Statistical Methods Group
Acknowledgements: Andrew D Oxman contributed to earlier versions. Professor Penny Hawe contributed to the text on adverse effects in earlier versions. Jon Deeks provided helpful contributions on an earlier version of this chapter. For details of previous authors and editors of the Handbook , please refer to the Preface.
Funding: This work was in part supported by funding from the Michael G DeGroote Cochrane Canada Centre and the Ontario Ministry of Health.
Alonso-Coello P, Zhou Q, Martinez-Zapata MJ, Mills E, Heels-Ansdell D, Johanson JF, Guyatt G. Meta-analysis of flavonoids for the treatment of haemorrhoids. British Journal of Surgery 2006; 93 : 909-920.
Atkins D, Best D, Briss PA, Eccles M, Falck-Ytter Y, Flottorp S, Guyatt GH, Harbour RT, Haugh MC, Henry D, Hill S, Jaeschke R, Leng G, Liberati A, Magrini N, Mason J, Middleton P, Mrukowicz J, O'Connell D, Oxman AD, Phillips B, Schünemann HJ, Edejer TT, Varonen H, Vist GE, Williams JW, Jr., Zaza S. Grading quality of evidence and strength of recommendations. BMJ 2004; 328 : 1490.
Balshem H, Helfand M, Schünemann HJ, Oxman AD, Kunz R, Brozek J, Vist GE, Falck-Ytter Y, Meerpohl J, Norris S, Guyatt GH. GRADE guidelines: 3. Rating the quality of evidence. Journal of Clinical Epidemiology 2011; 64 : 401-406.
Bhandari M, Busse JW, Jackowski D, Montori VM, Schünemann H, Sprague S, Mears D, Schemitsch EH, Heels-Ansdell D, Devereaux PJ. Association between industry funding and statistically significant pro-industry findings in medical and surgical randomized trials. Canadian Medical Association Journal 2004; 170 : 477-480.
Brophy JM, Joseph L, Rouleau JL. Beta-blockers in congestive heart failure. A Bayesian meta-analysis. Annals of Internal Medicine 2001; 134 : 550-560.
Carrasco-Labra A, Brignardello-Petersen R, Santesso N, Neumann I, Mustafa RA, Mbuagbaw L, Etxeandia Ikobaltzeta I, De Stio C, McCullagh LJ, Alonso-Coello P, Meerpohl JJ, Vandvik PO, Brozek JL, Akl EA, Bossuyt P, Churchill R, Glenton C, Rosenbaum S, Tugwell P, Welch V, Garner P, Guyatt G, Schünemann HJ. Improving GRADE evidence tables part 1: a randomized trial shows improved understanding of content in summary of findings tables with a new format. Journal of Clinical Epidemiology 2016; 74 : 7-18.
Deeks JJ, Altman DG. Effect measures for meta-analysis of trials with binary outcomes. In: Egger M, Davey Smith G, Altman DG, editors. Systematic Reviews in Health Care: Meta-analysis in Context . 2nd ed. London (UK): BMJ Publication Group; 2001. p. 313-335.
Devereaux PJ, Choi PT, Lacchetti C, Weaver B, Schünemann HJ, Haines T, Lavis JN, Grant BJ, Haslam DR, Bhandari M, Sullivan T, Cook DJ, Walter SD, Meade M, Khan H, Bhatnagar N, Guyatt GH. A systematic review and meta-analysis of studies comparing mortality rates of private for-profit and private not-for-profit hospitals. Canadian Medical Association Journal 2002; 166 : 1399-1406.
Engels EA, Schmid CH, Terrin N, Olkin I, Lau J. Heterogeneity and statistical significance in meta-analysis: an empirical study of 125 meta-analyses. Statistics in Medicine 2000; 19 : 1707-1728.
Furukawa TA, Guyatt GH, Griffith LE. Can we individualize the 'number needed to treat'? An empirical study of summary effect measures in meta-analyses. International Journal of Epidemiology 2002; 31 : 72-76.
Gibson JN, Waddell G. Surgical interventions for lumbar disc prolapse: updated Cochrane Review. Spine 2007; 32 : 1735-1747.
Guyatt G, Oxman A, Vist G, Kunz R, Falck-Ytter Y, Alonso-Coello P, Schünemann H. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ 2008; 336 : 3.
Guyatt G, Oxman AD, Akl EA, Kunz R, Vist G, Brozek J, Norris S, Falck-Ytter Y, Glasziou P, DeBeer H, Jaeschke R, Rind D, Meerpohl J, Dahm P, Schünemann HJ. GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings tables. Journal of Clinical Epidemiology 2011a; 64 : 383-394.
Guyatt GH, Oxman AD, Kunz R, Brozek J, Alonso-Coello P, Rind D, Devereaux PJ, Montori VM, Freyschuss B, Vist G, Jaeschke R, Williams JW, Jr., Murad MH, Sinclair D, Falck-Ytter Y, Meerpohl J, Whittington C, Thorlund K, Andrews J, Schünemann HJ. GRADE guidelines 6. Rating the quality of evidence--imprecision. Journal of Clinical Epidemiology 2011b; 64 : 1283-1293.
Iorio A, Spencer FA, Falavigna M, Alba C, Lang E, Burnand B, McGinn T, Hayden J, Williams K, Shea B, Wolff R, Kujpers T, Perel P, Vandvik PO, Glasziou P, Schünemann H, Guyatt G. Use of GRADE for assessment of evidence about prognosis: rating confidence in estimates of event rates in broad categories of patients. BMJ 2015; 350 : h870.
Langendam M, Carrasco-Labra A, Santesso N, Mustafa RA, Brignardello-Petersen R, Ventresca M, Heus P, Lasserson T, Moustgaard R, Brozek J, Schünemann HJ. Improving GRADE evidence tables part 2: a systematic survey of explanatory notes shows more guidance is needed. Journal of Clinical Epidemiology 2016; 74 : 19-27.
Levine MN, Raskob G, Landefeld S, Kearon C, Schulman S. Hemorrhagic complications of anticoagulant treatment: the Seventh ACCP Conference on Antithrombotic and Thrombolytic Therapy. Chest 2004; 126 : 287S-310S.
Orrapin S, Rerkasem K. Carotid endarterectomy for symptomatic carotid stenosis. Cochrane Database of Systematic Reviews 2017; 6 : CD001081.
Salpeter S, Greyber E, Pasternak G, Salpeter E. Risk of fatal and nonfatal lactic acidosis with metformin use in type 2 diabetes mellitus. Cochrane Database of Systematic Reviews 2007; 4 : CD002967.
Santesso N, Carrasco-Labra A, Langendam M, Brignardello-Petersen R, Mustafa RA, Heus P, Lasserson T, Opiyo N, Kunnamo I, Sinclair D, Garner P, Treweek S, Tovey D, Akl EA, Tugwell P, Brozek JL, Guyatt G, Schünemann HJ. Improving GRADE evidence tables part 3: detailed guidance for explanatory footnotes supports creating and understanding GRADE certainty in the evidence judgments. Journal of Clinical Epidemiology 2016; 74 : 28-39.
Schünemann HJ, Best D, Vist G, Oxman AD, Group GW. Letters, numbers, symbols and words: how to communicate grades of evidence and recommendations. Canadian Medical Association Journal 2003; 169 : 677-680.
Schünemann HJ, Jaeschke R, Cook DJ, Bria WF, El-Solh AA, Ernst A, Fahy BF, Gould MK, Horan KL, Krishnan JA, Manthous CA, Maurer JR, McNicholas WT, Oxman AD, Rubenfeld G, Turino GM, Guyatt G. An official ATS statement: grading the quality of evidence and strength of recommendations in ATS guidelines and recommendations. American Journal of Respiratory and Critical Care Medicine 2006; 174 : 605-614.
Schünemann HJ, Oxman AD, Brozek J, Glasziou P, Jaeschke R, Vist GE, Williams JW, Jr., Kunz R, Craig J, Montori VM, Bossuyt P, Guyatt GH. Grading quality of evidence and strength of recommendations for diagnostic tests and strategies. BMJ 2008a; 336 : 1106-1110.
Schünemann HJ, Oxman AD, Brozek J, Glasziou P, Bossuyt P, Chang S, Muti P, Jaeschke R, Guyatt GH. GRADE: assessing the quality of evidence for diagnostic recommendations. ACP Journal Club 2008b; 149 : 2.
Schünemann HJ, Mustafa R, Brozek J. [Diagnostic accuracy and linked evidence--testing the chain]. Zeitschrift für Evidenz, Fortbildung und Qualität im Gesundheitswesen 2012; 106 : 153-160.
Schünemann HJ, Tugwell P, Reeves BC, Akl EA, Santesso N, Spencer FA, Shea B, Wells G, Helfand M. Non-randomized studies as a source of complementary, sequential or replacement evidence for randomized controlled trials in systematic reviews on the effects of interventions. Research Synthesis Methods 2013; 4 : 49-62.
Schünemann HJ. Interpreting GRADE's levels of certainty or quality of the evidence: GRADE for statisticians, considering review information size or less emphasis on imprecision? Journal of Clinical Epidemiology 2016; 75 : 6-15.
Schünemann HJ, Cuello C, Akl EA, Mustafa RA, Meerpohl JJ, Thayer K, Morgan RL, Gartlehner G, Kunz R, Katikireddi SV, Sterne J, Higgins JPT, Guyatt G, Group GW. GRADE guidelines: 18. How ROBINS-I and other tools to assess risk of bias in nonrandomized studies should be used to rate the certainty of a body of evidence. Journal of Clinical Epidemiology 2018.
Spencer-Bonilla G, Quinones AR, Montori VM, International Minimally Disruptive Medicine W. Assessing the Burden of Treatment. Journal of General Internal Medicine 2017; 32 : 1141-1145.
Spencer FA, Iorio A, You J, Murad MH, Schünemann HJ, Vandvik PO, Crowther MA, Pottie K, Lang ES, Meerpohl JJ, Falck-Ytter Y, Alonso-Coello P, Guyatt GH. Uncertainties in baseline risk estimates and confidence in treatment effects. BMJ 2012; 345 : e7401.
Sterne JAC, Hernán MA, Reeves BC, Savović J, Berkman ND, Viswanathan M, Henry D, Altman DG, Ansari MT, Boutron I, Carpenter JR, Chan AW, Churchill R, Deeks JJ, Hróbjartsson A, Kirkham J, Jüni P, Loke YK, Pigott TD, Ramsay CR, Regidor D, Rothstein HR, Sandhu L, Santaguida PL, Schünemann HJ, Shea B, Shrier I, Tugwell P, Turner L, Valentine JC, Waddington H, Waters E, Wells GA, Whiting PF, Higgins JPT. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ 2016; 355 : i4919.
Thompson DC, Rivara FP, Thompson R. Helmets for preventing head and facial injuries in bicyclists. Cochrane Database of Systematic Reviews 2000; 2 : CD001855.
Tierney JF, Stewart LA, Ghersi D, Burdett S, Sydes MR. Practical methods for incorporating summary time-to-event data into meta-analysis. Trials 2007; 8 .
van Dalen EC, Tierney JF, Kremer LCM. Tips and tricks for understanding and using SR results. No. 7: time‐to‐event data. Evidence-Based Child Health 2007; 2 : 1089-1090.
For permission to re-use material from the Handbook (either academic or commercial), please see here for full details.
Chapter: appendix a: literature review tables, appendix a literature review tables.
E vidence on the effectiveness of continuing education (CE) and CE methods was identified through a literature review. Although nonexhaustive, the review included a comprehensive search of the Research and Development Resource Base (RDRB), a bibliographic database of more than 18,000 articles from fields including CE, knowledge translation, interprofessional literature, and faculty development. Articles in the RDRB are culled from Medline, the Cumulative Index to Nursing and Allied Health Literature (CINAHL), Excerpta Medica Database (EMBASE), Education Resources Information Center (ERIC), Sociological Abstracts, PsychoInfo, Library Information and Science Abstracts (LISA), and business databases, as well as automatic retrieval of articles from journals dedicated to medical education (e.g., Journal of Continuing Education in the Health Professions , Medical Education , Studies in Continuing Education ).
The RDRB was searched using keywords, 1 and the results of the searches were culled by two independent reviewers using an iterative approach. Studies collected were from 1989 to April 2009.
Abstracts of search results were reviewed to eliminate articles that clearly did not pertain to CE methods, cost-effectiveness, or educational theory and to categorize the studies as informative, equivocal, or not informative of CE effectiveness. A wide range of designs were classified as informative, including randomized controlled trials, prospective cohort studies, observational studies, and studies with pre- and post-intervention assessment methodologies. Quantitative and qualitative approaches were included, and inclusion was not limited to studies with positive results. The most common reasons articles were classified as not informative were absence of a trial design, small sample size, and high likelihood of confounding factors in the design that could affect outcomes. The two reviewers independently classified abstracts and full texts of the articles and then compared their classification results. Interreviewer reliability was greater than 80 percent, and discrepancies were resolved by a consensus process. A third reviewer verified the results classified as informative or equivocal in a final round of detailed assessment of the study design, populations, intervention, type of outcome, and conclusions for each article. Systematic reviews and metaanalyses are included in Table A-1 ; studies and articles are included in Table A-2 .
Table A-1 begins on the next page.
TABLE A-1 Summary of Systematic Reviews on Effectiveness of CE Methods
TABLE A-2 Literature Review on the Effectiveness of CE Methods
This page intentionally left blank.
Today in the United States, the professional health workforce is not consistently prepared to provide high quality health care and assure patient safety, even as the nation spends more per capita on health care than any other country. The absence of a comprehensive and well-integrated system of continuing education (CE) in the health professions is an important contributing factor to knowledge and performance deficiencies at the individual and system levels.
To be most effective, health professionals at every stage of their careers must continue learning about advances in research and treatment in their fields (and related fields) in order to obtain and maintain up-to-date knowledge and skills in caring for their patients. Many health professionals regularly undertake a variety of efforts to stay up to date, but on a larger scale, the nation's approach to CE for health professionals fails to support the professions in their efforts to achieve and maintain proficiency.
Redesigning Continuing Education in the Health Professions illustrates a vision for a better system through a comprehensive approach of continuing professional development, and posits a framework upon which to develop a new, more effective system. The book also offers principles to guide the creation of a national continuing education institute.
You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.
Do you want to take a quick tour of the OpenBook's features?
Show this book's table of contents , where you can jump to any chapter by name.
...or use these buttons to go back to the previous chapter or skip to the next one.
Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.
Switch between the Original Pages , where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.
To search the entire text of this book, type in your search term here and press Enter .
Share a link to this book page on your preferred social network or via email.
View our suggested citation for this chapter.
Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.
Do you enjoy reading reports from the Academies online for free ? Sign up for email notifications and we'll let you know about new publications in your areas of interest when they're released.
BMC Health Services Research volume 24 , Article number: 202 ( 2024 ) Cite this article
137 Accesses
6 Altmetric
Metrics details
Healthcare systems around the world are facing large challenges. There are increasing demands and costs while at the same time a diminishing health workforce. Without reform, healthcare systems are unsustainable. Relocating care, for example, from hospitals to sites closer to patients’ homes, is expected to make a key contribution to keeping healthcare sustainable. Given the significant impact of this initiative on citizens, we conducted a scoping review to provide insight into the factors that influence citizens’ attitudes towards relocating care.
A scoping review was conducted. The search was performed in the following databases: Pubmed, Embase, Cinahl, and Scopus. Articles had to include relocating healthcare and citizens’ perspectives on this topic and the articles had to be about a European country with a strong primary care system. After applying the inclusion and exclusion criteria, 70 articles remained.
Factors positively influencing citizens’ attitudes towards relocating care included: convenience, familiarity, accessibility, patients having more control over their disease, and privacy. Factors influencing negative attitudes included: concerns about the quality of care, familiarity, the lack of physical examination, contact with others, convenience, and privacy. Furthermore, in general, most citizens preferred to relocate care in the studies we found, especially from the hospital to care provided at home.
Several factors influencing the attitude of citizens towards relocating care were found. These factors are very important when determining citizens’ preferences for the location of their healthcare. The majority of studies in this review reported that citizens are in favour of relocating care. In general citizens’ perspectives on relocating care are very often missing in articles. It was significant that very few studies on relocation from the hospital to the general practitioner were identified.
Peer Review reports
Demand for healthcare is increasing across the world due to a number of developments including populations ageing, technical advances in medical care, and rising incomes [ 1 , 2 , 3 ]. With an increase in demand, costs will also rise, while at the same time a diminishing health workforce. [ 1 , 2 , 3 , 4 , 5 ]. Consequently, reforms within the healthcare system will be necessary in order to control increasing healthcare costs and staff shortages [ 1 , 2 , 3 ]. It is assumed that reforming healthcare systems with a view to making better use of resources will make a key contribution to keeping healthcare sustainable. Estimates suggest that one fifth of health spending could be channelled towards better use, thus improving healthcare efficiency [ 6 ]. Increased efficiency could be accomplished in several ways. These may include: reducing the number of patients who receive low-value or unnecessary care; providing the same care with fewer resources, for instance by providing care in more cost-effective settings rather than in hospitals; or by reducing administrative processes that add no value [ 6 ]. This article focuses on providing care with fewer resources by relocating it to more cost-effective settings. This, in the first instance, would mean from secondary care to primary care. The thought behind this is that general practitioners (GPs) can generally provide care at less expense than hospitals for certain procedures that do not need hospital staff or environment [ 6 ]. These may include minor interventions, such as the placement of an intra-uterine device (IUD), or follow-up care, such as yearly blood- and ultrasounds, for patients who have been treated for cancer[ 6 , 7 , 8 , 9 ]. Relocating care to control costs could also include relocating care from secondary to homecare, self-care or eHealth [ 10 ]. Delivering care digitally can prevent a patient from having to go to the hospital. For example, an app could be used to monitor a patient receiving oxygen at home. Care commonly provided by the GP could also be relocated, to self-care, eHealth or to other healthcare providers (HCPs), like a physiotherapist or dietitian. This could result in more time for the GP to take on other secondary or primary care tasks.
It is important for relocating care to succeed, to get insights into the perspectives and needs of healthcare providers and citizens. Although involving citizens is a very important aspect of policy-making processes, it is an often overlooked form of evidence according to the World Health Organization (WHO) [ 11 ]. Citizen engagement will strengthen societal trust, will lead to more effective public policies and will lead to an improved quality of care. Furthermore, citizen engagement is essential because healthcare systems are transitioning towards a patient-centered approach, where citizens' perspectives on quality are inherently meaningful and should be a primary focus within healthcare systems [ 12 ].Extensive research has already been undertaken regarding the perspective of healthcare providers [ 9 , 13 , 14 , 15 , 16 ], the quality and outcomes of care [ 17 , 18 , 19 , 20 ] and the cost perspectives [ 10 , 17 , 18 , 20 , 21 ], but not regarding the citizens' perspective on relocating care. To our knowledge, a review about citizens’ perspectives on relocating care does not exist yet. We have, therefore, conducted a scoping review with the goal of describing the findings and range of research concerning citizens’ perspectives on relocating care in more detail. A strong primary care system is required to make relocating care possible [ 6 ]. We, therefore, searched for studies that were undertaken in countries in Europe with a strong primary care [ 22 ]. Table 1 describes the characteristics of countries with strong primary care. The research questions answered in this review are: (1) Which factors influence citizens’ attitudes towards relocating care? (2) What are citizens’ preferences towards the location of care?
The aim of this review is to understand citizens’ attitudes and preferences towards relocating care. As this topic is quite broad and may be studied using many different study designs, and considering that we are not aware of any prior synthesis on this topic, a scoping review rather than a systematic review was conducted. This scoping review was carried out on the basis of the guideline by Arksey and O’Malley [ 23 ]. The review includes the following key phases: 1) identifying the research question; 2) identifying relevant studies; 3) study selection; 4) charting the data, and; 5) collating, summarising, and reporting the results.
An initial broad search of the literature was undertaken by the first author in order to identify relevant articles that could be used for designing a search strategy. During this search, 18 key articles were identified, which included citizens, preference, and relocating care, these three terms formed the basis of our search strategy. A qualified medical information specialist was consulted in order to design and execute a sensitive search strategy. The medical information specialist also advised on which databases were most likely to contain the type of studies we were seeking and thus constituted an initial search strategy. This was refined several times after consultation. The final version was first used on the Pubmed database and then converted for each of these subsequent databases, Embase, Cinahl, and Scopus. The final search strategy, shown in Appendix A , was able to find 16 out of the 18 key articles identified. In total, it identified 19.587 articles. Duplicate references were removed, leaving 11.080 unique references. The most recent search was executed on 5 July 2022.
The selection process was performed by all authors. First, inclusion and exclusion criteria were developed. There were several inclusion criteria for this scoping review. The topic of the articles had to be citizens’ perspectives on relocating care. Only articles related to European countries with strong primary care systems were included, as a strong primary care system is required to make relocating care possible [ 6 ]. These countries were: the Netherlands, the United Kingdom, Belgium, Spain, Portugal, Finland, Estonia, Lithuania, Denmark, and Slovenia [ 22 ]. Only articles written in English, Dutch, or German were included as these were languages sufficiently mastered by the authors. In addition, all study designs were included. An overview of inclusion and exclusion criteria are shown in Table 2 . In order to calibrate the inclusion process, the researchers independently applied the inclusion and exclusion criteria to a selection of three hundred articles. The task was to include, or exclude, articles based on the title alone. The results were discussed by the researchers to see if there was a maximum margin of disagreement up to 10%. This percentage was agreed in advance by the researchers. During this process, the inclusion and exclusion criteria were further refined (See Table 2 ). As disagreement remained, a second round of calibration was performed on 50 articles, including both titles and abstracts. The disagreement rate was now only 4% and therefore all the remaining articles were distributed among the reviewers to be scored, based on the title and abstract. After screening on the title and abstract, 167 references remained and two key articles that were not found with the search were added. These articles were distributed among the researchers once more in order to read the full text. While reading the full texts, another three relevant articles were identified through the references. These were then added too. This resulted in a total of 172 full text articles. Results from included articles were charted in a spreadsheet, which was tested by the researchers before using it. When one of the reviewers had doubts about an article, it was read by a second reviewer and the outcomes were discussed until the two researchers came to an agreement.
A spreadsheet was created to categorise the information that contributed to answering the research questions.
The information extracted from the articles was structured according to the type of relocation, including: relocating from the hospital to the GP, to care at home, to self-care, or to eHealth, and relocating from the GP to self-care, to care at home, or to eHealth. The difference between self-care and care at home is that self-care does not involve a healthcare provider, unlike care at home. Both forms of relocating do not involve eHealth. When the article was about eHealth it was catalogued with the eHealth category. Articles that remained, of which there was only one, were placed within the category ‘other’.
The information extracted included factors that determined citizens’ attitudes towards relocating care. All of these factors were coded by highlighter and categorised. The categories were discussed within the research team. Subsequently, we made a top three of factors for each form of relocation that occurred most often.
Furthermore, we extracted information regarding preferences for healthcare location in the articles. Citizens could have a preference for either keeping care its current location, relocating care, or a combination of both, suggesting that citizens may prefer a hybrid approach where some aspects of healthcare are relocated, while others remain in their current location. Citizens could also express equal preferences for both locations. In addition, we compared the outcomes of the one-armed, the two-armed, and the hypothetical studies, to see if there were major differences, in the preferences for healthcare location, resulting from their methodological approaches. In the one-armed studies, care was relocated for all participants in the study [ 24 ]. In the two-armed studies there was one group of participants where care was relocated, but also one group who received care as usual. The outcomes of the two groups were then compared. Hypothetical studies, presented scenarios without actual choices. They asked citizens how they would feel if care were relocated. Two-armed studies are generally considered of higher quality than one-armed and hypothetical studies, due to the presence of both an experimental group and a control group, which increases their internal validity [ 25 ].
A total of 19,587 references were identified from the databases, of which 8,507 were duplicates, as shown in Fig. 1 . At the end of the selection process, 70 full text articles were included. The characteristics of these studies are shown in Table 3 .
Flowchart of the review process
The majority of studies of citizens’ perspectives on relocating care took place in the UK ( N = 44), followed by the Netherlands ( N = 13), and Denmark ( N = 11). One study is from Spain and one from Estonia. Most studies are one-armed ( N = 42), followed by two-armed ( n = 19), and nine studies were hypothetical. While eight studies are from 2013, most studies were published quite recently in 2019 ( N = 7), 2020 ( N = 6), 2021 ( N = 16), and 2022 ( N = 9). Relocating care from the hospital to eHealth is the form of relocating that is most often examined within the studies identified ( N = 28) [ 26 , 27 , 28 , 29 , 30 , 31 , 32 , 33 , 34 , 35 , 36 , 37 , 38 , 39 , 40 , 41 , 42 , 43 , 44 , 45 , 46 , 47 , 48 , 49 , 50 , 51 , 52 , 53 ]. This is followed by relocating from the hospital to self-care ( N = 15) [ 54 , 55 , 56 , 57 , 58 , 59 , 60 , 61 , 62 , 63 , 64 , 65 , 66 , 67 ] and care at home ( N = 13) [ 30 , 68 , 69 , 70 , 71 , 72 , 73 , 74 , 75 , 76 , 77 , 78 , 79 , 80 ]. Forms of relocating care that are not frequently studied include relocating from the hospital to the GP ( N = 7) [ 16 , 69 , 81 , 82 , 83 , 84 , 85 ] and from the GP to self-care ( N = 4) [ 86 , 87 , 88 , 89 ]. Five more forms of relocating are listed under the heading “other”. These include: relocating from the hospital to a community-based clinic [ 90 ]; from outpatient visits to a one-stop clinic [ 91 ]; nurse home visits that were replaced by eHealth [ 92 ]; hospital care relocated to a mobile chemotherapy unit [ 93 ]; and, care relocated from the GP to eHealth [ 94 ]. Most studies are about the relocation of care for oncology patients ( N = 19), followed by citizens in general ( N = 10), and cardiology patients ( N = 8).
Convenience.
The most frequently cited factors influencing citizens’ attitudes towards relocating care are shown in Table 4 . Convenience was most often reported, from the citizens’ perspective, as an advantage of relocating care. This was true for all forms of relocation [ 27 , 28 , 29 , 30 , 32 , 33 , 34 , 38 , 41 , 42 , 45 , 47 , 49 , 52 , 53 , 54 , 58 , 59 , 60 , 65 , 66 , 67 , 69 , 70 , 73 , 78 , 82 , 84 , 85 , 86 , 88 , 90 , 93 , 94 ]. Citizens think of relocating as convenient because in most cases it saves travel time [ 26 , 29 , 53 ]. It saves costs [ 26 , 69 ]. It avoids stress due to factors such as transport problems, busy traffic, travelling while you are sick, or long sojourns in waiting rooms [ 26 , 53 , 73 , 93 ]. When relocating to self-care it was very often mentioned that it is an advantage to have more flexibility [ 30 , 86 ]. Citizens can do a self-test whenever and wherever they want, without having to consider opening hours, for example [ 59 , 66 , 67 ]. Convenience was also mentioned as a reason for not wanting to relocate care. This factor was especially mentioned when relocating from the hospital or GP to self-care [ 59 , 60 , 86 ]. With regard to home dialysis, some citizens said that they did not have the space at home to do this. It was, therefore, not convenient [ 60 ]. In addition, for citizens living close to the hospital, self-care was sometimes more expensive and did not save time [ 59 , 86 ].
Familiarity was another factor which was reported as important to citizens regarding their attitude towards relocating care [ 29 , 30 , 31 , 32 , 33 , 58 , 61 , 67 , 68 , 69 , 70 , 73 , 74 , 77 , 83 , 84 , 85 , 86 , 90 , 94 ]. Some citizens feel more familiar with their GP than with a hospital specialist and would, therefore, want to relocate care [ 83 , 84 ]. Other citizens experience a sense of familiarity due to the environment in which care is provided. When receiving care at home, citizens feel more familiar, because they are in their own environment with their own support system [ 29 , 30 , 50 , 58 , 70 , 77 ]. In addition, when receiving care at home, the HCP enters the personal space of the patient. This, according to some of the patients, provided a better and more personal connection with the HCP. As shown in Table 4 , familiarity is also named as a reason not to want to relocate. While some citizens said that they had a better relationship with their GP, others said they were more familiar with the specialist so they would rather go there [ 85 ]. Some citizens thought that personal contact was reduced when using eHealth. They felt that it was more distant [ 31 , 33 , 36 , 47 , 51 ]. In addition, during telephone consultations, citizens did not feel a sense of familiarity if they had never seen the HCP before and therefore could not picture the face belonging to the voice. [ 29 ]. With regard to self-care, some citizens did not feel a sense of familiarity because this care is usually performed alone, while they preferred to have the support of a HCP [ 60 , 63 ].
The third most frequently mentioned factor that influenced citizens’ perceptions of relocating care was “accessibility”. Citizens were more willing to relocate care when waiting times became shorter and so the accessibility became better [ 28 , 29 , 30 , 45 , 49 , 54 , 58 , 82 , 83 , 84 , 88 , 90 , 91 , 93 ]. For example when relocating from the hospital to the GP [ 82 , 83 , 84 ]. Regarding self-tests, citizens mentioned that they had very rapid access. They can pick up the test and then apply it directly, without having to make an appointment with a HCP, who is often not immediately available [ 30 , 54 , 55 , 58 ]. In addition, with a self-test you often get the results without delay [ 55 , 59 ]. With regard to eHealth, citizens said that access to the HCP improved because they could contact them easily when they had questions [ 28 , 49 ].
Another advantage of relocating care, mentioned by citizens, is being more in control, especially when relocating care from the hospital to eHealth, self-care, or to care at home [ 30 , 54 , 58 , 60 , 70 , 73 ]. The sense of increased control can stem from two primary factors. Firstly, patients become more actively engaged in their healthcare, leading to a better understanding of their diagnoses and consequently, greater control over their condition [ 38 , 49 , 53 , 59 , 86 ]. Secondly, citizens felt more involved in the process of decision making regarding their healthcare, affording them the ability to influence what happens and when [ 49 , 50 , 59 , 74 ]. This gives them the feeling of having more control over their lives.
The last factor named as an advantage, but also as a disadvantage of relocating care, is ‘privacy’. Citizens who saw it as an advantage mentioned that there is more privacy at home using eHealth or self-care than there is in a hospital [ 53 , 54 , 55 , 58 , 60 , 66 , 69 , 70 , 74 ]. With regard to self-care there are a lot of articles about using self-tests to check for sexually transmitted infections or about administering drugs oneself at home in order to induce an abortion. Citizens indicated that having such tests carried out at a clinic may cause a lot of embarrassment [ 54 ]. You may run into acquaintances for example [ 67 ]. Self-care, on the other hand, is more anonymous and thus offers more privacy [ 55 ]. However, privacy is also named as an disadvantage by citizens. Regarding eHealth, some citizens are concerned about whether the privacy of their data can be guaranteed [ 33 ]. In addition, some citizens said that it was hard to find a private space in their house during the covid-19 crisis [ 30 ]. Furthermore, when care is being given at home, some citizens do not like the fact that other family members may witness them being treated [ 69 ] or that caregivers are having to enter their home, thus violating their privacy [ 70 ].
The most frequently mentioned factor for having a negative attitude towards relocating care is that citizens have concerns about the quality of care when care is being relocated, due to less expertise of the HCP or insufficient quality of the instrument or self-test, which will be involved in the new location [ 28 , 32 , 33 , 34 , 36 , 47 , 51 , 54 , 55 , 59 , 60 , 63 , 65 , 67 , 69 , 70 , 73 , 77 , 82 , 85 , 86 , 87 , 90 , 94 ]. Regarding relocating care to eHealth or self-care a lack of trust in eHealth technology [ 33 , 34 , 36 , 47 ], or a particular self-care device, [ 54 , 55 , 59 , 60 , 63 , 65 , 67 ] was reported very often. Citizens fear technical problems or that important factors might be overlooked. Neither do some citizens feel that they have the right skills for using the new eHealth technology [ 36 ] or performing self-care in the right way [ 54 , 60 , 65 , 67 ]. Regarding care at home, citizens were concerned with the absence of constant surveillance and a diminished contact with the doctor. Moreover, citizens felt that the hospital is better equipped [ 77 ]. With regard to relocating from the hospital to the GP, some citizens thought that the specialist had more expertise which was a reason for them not wanting to relocate [ 82 , 85 ].
Another factor for not wanting to relocate care is where it results in an absence of physical examination. This reason was named many times when relocating care from the hospital to eHealth [ 27 , 29 , 31 , 34 , 47 , 51 , 52 ] and relocating from the GP to self-care [ 86 , 89 ]. With regard to eHealth, some citizens say that they found it difficult because they are not able to demonstrate physical symptoms and they find it hard to describe problems without seeing the HCP [ 31 , 33 ].
The last factor, frequently mentioned as a disadvantage of relocating care, is less contact with their peers. This aspect was most mentioned regarding relocating from the hospital to care at home [ 69 , 70 , 73 ]. Some citizens enjoyed going to the hospital because of the social interaction with other citizens. They were afraid of social isolation [ 60 ].
A total of 49 articles investigated citizens’ preferences regarding the location of healthcare. Their location preferences for each form of relocating care will be discussed below and are shown in Table 5 .
Within the articles about relocating from the hospital to eHealth, 23 articles out of 28 provided the preferences of respondents towards the location of care. In ten articles there was a preference for eHealth [ 28 , 32 , 33 , 34 , 42 , 44 , 45 , 46 , 50 , 53 ] and in six articles a preference for the hospital [ 26 , 31 , 36 , 39 , 43 , 48 ]. In four articles, citizens expressed a wish for a combination of eHealth and face to face contact [ 37 , 47 , 49 , 52 ]. In the remaining articles ( N = 3), the preference was equal for the hospital and for eHealth [ 35 , 41 , 51 ].
Eight out of 15 articles about relocating from the hospital to self-care investigated citizens preferences for the location of care. In five articles citizens showed a preference for self-care [ 56 , 57 , 61 , 64 , 66 ] and in three articles for the hospital [ 55 , 60 , 65 ].
With regard to articles about relocating from the hospital to care at home, ten out of 13 articles investigated a preference for healthcare location. In eight articles, the participants had a preference for care at home [ 68 , 69 , 72 , 74 , 75 , 78 , 79 , 80 ]. In two articles, preferences for care at home and the hospital were equal [ 71 , 76 ]. There were no articles with a preference for the hospital.
Regarding relocating from the hospital to the GP, there were five out of seven articles investigating citizens preferences regarding healthcare location. In two articles, participants preferred the hospital over the GP [ 81 , 85 ]. In one they preferred the GP [ 84 ], and in the other, preferences were equal [ 16 ]. In the fifth study citizens could choose between three locations: the hospital, the GP, or care at Home. Here they preferred care at home followed by care at the general practice [ 69 ].
Two out of four articles about relocating from the GP to self-care investigated a preference for a healthcare location. In one article, citizens preferred self-care [ 86 ], and in the other, they preferred the GP [ 89 ].
Within the category “other”, there were two articles which investigated a preference for a healthcare location. In the article about relocating from the hospital to one-and-a-half line care, citizens preferred one-and-a-half line care [ 91 ]. The last article was about nurse home visits that were relocated to eHealth. Here, citizens preferred eHealth over the nurse visits [ 93 ].
Most articles adopted a one-armed approach. Since two-armed articles are often of higher quality, we compared the results of the one-armed, and the two-armed, articles. In total there were 19 two-armed articles of which 14 investigated a preference for healthcare location. In nine out of 14 articles citizens preferred relocating healthcare and in two articles they did not. In the other articles, preferences were equal. Of the 35 one-armed articles which investigated healthcare preferences in 18 articles, citizens gave a preference for relocating healthcare. Thus, in both cases, there is a preference for relocating care in just over half of the articles. We see here a different outcome than with the hypothetical studies ( N = 10). Here there was no preference for relocating care in five out of seven articles.
This scoping review was conducted in order to provide insight into the factors that influence citizens attitudes towards relocating care. Seventy articles were included and most which were found were about relocating care from the hospital to eHealth. Most of these articles about eHealth were published in 2020 or later ( N = 20). Only eight articles were published in 2019 or earlier. This is likely due to covid-19, which started in 2020 in Europe and required healthcare providers in many places to offer care online.
The first research question concerned which factors influence citizens attitudes towards relocating care. The most frequent reported factor for a positive attitude towards relocating care is “convenience”, according to citizens, followed by “familiarity”. Other factors that were in the top three of reasons for a positive attitude towards relocating care were “accessibility”, “patients have more control”, and “privacy”. The positive drivers for relocating care are almost the same for all forms of relocating. The two most mentioned factors for a negative attitude towards relocating care are, first of all, citizens having concerns about the quality of care and, secondly, citizens feel less familiar when care is being relocated. Other reasons to have a negative attitude towards relocating are “the lack of physical examination”, “contact with others”, “convenience”, and “privacy”.
The second research question concerned citizens’ preferences for healthcare location. In general, as far as the conditions and treatments mentioned in the articles are concerned, most citizens favoured relocating healthcare. Especially with regard to care at home, there were no articles found where citizens had a preference for the hospital instead of care at home. In addition, eHealth and self-care are also carried out from home. Citizens thus prefer receiving care at home.
Not all articles investigated preferences for the location of healthcare, and of those which did, most were one-armed. However, there were no major differences found when comparing the outcomes of the one-armed and two-armed studies. This contrasted with the hypothetical studies, where citizens did not prefer relocating care in the majority of cases. This may be due to the fact that citizens are familiar with the current situation and do not know, or find it difficult to imagine, what a new situation will look like. Citizens may not want to relocate because familiarity is an important aspect of healthcare, as described earlier.
The articles found included a wide variety of conditions and phases of treatment. We would have preferred to distinguish between different conditions and treatment phases, as these aspects may determine the preference for healthcare location. For example, it might be the case that citizens would like to relocate follow-up cancer care to care at home, while keeping the treatment itself in the hospital. However, the large variation in conditions and phases of treatment resulted in a small N per condition or phase of treatment and this hampered further in-depth analysis.
Relocating care often involves not only the location changing, but also other aspects. For instance, the care provider may change too, for example a telephone consultation with a nurse instead of a face to face appointment with the specialist in the hospital [ 32 , 53 ]. And in some cases, the purpose of treatment changed, for example, a telephone consultation that was meant for providing information and supporting patients, while a face to face consultation was more focused on looking for signs of recurrent disease [ 29 ]. All of these factors together determine the preference for healthcare location. So it is not only the location on which citizens base their preference. It is, therefore, important to take all aspects into account, not only the geography when investigating the preferences for healthcare location.
A strength of this scoping review is that it has a broad search strategy developed together with a medical information specialist. This resulted in over 11.000 references that were all assessed. However, the search strategy may not have been broad enough, as some articles were missed, including two of the 18 key articles. This was known beforehand and so we investigated why the two key articles were not found. One key article was not found because we did not use the word “experience” [ 16 ] while the other focused on the terms “breast cancer”, “follow-up care”, and “healthcare models” [ 81 ], which we did not use in our search strategy. The words used in these two articles were not words we saw repeated in other relevant articles. Adding any of the key words yielded about 5,800 additional results in Pubmed alone. Therefore, we chose to add the key articles manually and left these words out of the search string. All statements made in this article are based on the conditions and forms of care that recurred in the studies we found. There may be other forms of care that could be relocated that have not been discussed in this article.
Another limitation of this study is that the articles are not double reviewed because of the large number of references found. However, to calibrate the inclusion process, the researchers applied the inclusion and exclusion criteria to a selection of 350 articles. Also, it was decided to start with reviewing abstracts, instead of titles, which is the normal procedure [ 23 ].
A limitation of a scoping review is that it analyses studies that use a range of data collection and techniques. This makes it more difficult to synthesise the results of the studies [ 23 ]. A strong point of this review is that we made a comparison between one-armed and two-armed articles and that approximately the same results emerged in the articles.
A knowledge gap we identified is that citizens’ perspectives on relocating care received relatively little attention within the current literature. In particular, we found limited literature focusing on citizens’ perspectives regarding the relocation of care from the hospital to the GP. This gap is significant, because this is one of the forms of relocating that governments think of first in order to limit healthcare costs [ 6 , 7 , 8 ]. There are several studies about this subject but they do not involve the citizens’ perspective. Despite the importance of including citizens' perspectives in policy-making processes, it often remains underrepresented in the literature [ 11 ]. The World Health Organization (WHO) emphasizes that citizen engagement can enhance societal trust and lead to more effective public policies.
Another knowledge gap we identified is that insufficient research has been done on different treatment phases and conditions in healthcare with regard to citizens’ perspectives and relocating care. To fill this gap, future research should delve deeper into the relationship between the factors leading to particular attitudes towards relocating care, and preferences for location of care and different conditions and treatment phases, including diagnosis, treatment phase and aftercare.
Our study has also revealed practical implications that can inform healthcare policy and decision-making. Firstly, the factors we have identified can serve as conditions that governments can use to improve acceptance among citizens regarding healthcare location. They can be used as conditions that have to be met, and that can be used to direct citizens to a particular location. Secondly, it's evident from our findings that citizens generally prefer receiving care from home. This preference presents an opportunity for governments to invest in home-based healthcare services, potentially leading to higher citizen satisfaction and more cost-effective healthcare delivery.
Positive factors influencing the attitude of citizens towards relocating care are almost the same for all forms of this development—with convenience as the most important. The most often reported factor for having a negative attitude towards relocating care are concerns about the quality of care. The factors found are very important when determining a citizens’ preference for a particular healthcare location. The majority of studies in this review reported that citizens are in favour of relocating care, especially to care at home. Several knowledge gaps were identified. Strikingly, very few studies on relocation from the hospital to the GP were identified.
Not applicable. The studies we used are accessible to everyone. All studies used are included in the references.
General practitioner
Healthcare provider
Intra-uterine device
Organisation for Economic Co-operation and Development. OECD work on health. Paris: OECD; 2021. p. 1–44.
Google Scholar
Rechel B, Doyle Y, Grundy E, McKee M. How can health systems respond to population ageing? Technical report. Copenhagen: World Health Organization; 2009. Report No.: 10.
Rudnicka E, Napierała P, Podfigurna A, Męczekalski B, Smolarczyk R, Grymowicz M. The World Health Organization (WHO) approach to healthy ageing. Maturitas. 2020;139:6–11.
Article PubMed PubMed Central Google Scholar
Liu JX, Goryakin Y, Maeda A, Bruckner T, Scheffler R. Global health workforce labor market projections for 2030. Hum Resour Health. 2017;15(1):11.
Boniol M, Kunjumen T, Nair TS, Siyam A, Campbell J, Diallo K. The global health workforce stock and distribution in 2020 and 2030: a threat to equity and ‘universal’ health coverage? BMJ Glob Health. 2022;7(6):e009316.
Organisation for Economic Co-operation and Development. Tackling wasteful spending on Health. Paris: OECD; 2017.
Taskforce De Juiste Zorg Op de Juiste Plek. De Juiste zorg op de juiste plek rapport: Wie durft? Den Haag: Ministerie van Volksgezondheid, Welzijn en Sport; 2018.
Sibbald B, McDonald R, Roland M. Shifting care from hospitals to the community: a review of the evidence on quality and efficiency. J Health Serv Res Policy. 2007;12(2):110–7.
Article PubMed Google Scholar
Liemburg GB, Korevaar JC, van Zomeren WT, Berendsen AJ, Brandenbarg D. Follow-up of curatively treated cancer in primary care: a qualitative study of the views of Dutch GPs. Br J Gen Pract. 2022;72(721):e592–600.
Gupta Strategists. No place like home. An analysis of the growing movement away from hospitals towards providing medical care to patients in their own homes. Amsterdam: Gutpa Strategists; 2016.
World Health Organization. Implementing citizen engagement within evidence-informed policy-making: an overview of purpose and methods. Geneva: World Health Organization; 2022.
Sofaer S, Firminger K. Patient perceptions of the quality of health services. Annu Rev Public Health. 2005;26:513–59.
Van Hoof SJM, Kroese MEAL, Spreeuwenberg MD, Elissen AMJ, Meerlo RJ, Hanraets MMH, et al. Substitution of hospital care with primary care: defining the conditions of primary Care Plus. Int J Integr Care. 2016;16(1):12.
Firet L, de Bree C, Verhoeks CM, Teunissen DA, Lagro-Janssen AL. Mixed feelings: general practitioners’ attitudes towards eHealth for stress urinary incontinence-a qualitative study. BMC Fam Pract. 2019;20:1–8.
Article Google Scholar
Noels EC, Wakkee M, van den Bos RR, Bindels PJE, Nijsten T, Lugtenberg M. Substitution of low-risk skin cancer hospital care towards primary care: a qualitative study on views of general practitioners and dermatologists. Plos One. 2019;14(3): e0213595.
Article CAS PubMed PubMed Central Google Scholar
Van Hoof SJ, Spreeuwenberg MD, Kroese ME, Steevens J, Meerlo RJ, Hanraets MM, et al. Substitution of outpatient care with primary care: a feasibility study on the experiences among general practitioners, medical specialists and patients. BMC Fam Pract. 2016;17(1):1–9.
Crawford DC, Li CS, Sprague S, Bhandari M. Clinical and cost implications of inpatient versus outpatient orthopedic surgeries: a systematic review of the published literature. Orthop Rev (Pavia). 2015;7(4):6177.
PubMed Google Scholar
Calkins TE, Mosher ZA, Throckmorton TW, Brolin TJ. Safety and cost effectiveness of outpatient total shoulder Arthroplasty: a systematic review. J Am Acad Orthop Surg. 2022;30(2):e233–41.
Qin C, Dekker RG, Blough JT, Kadakia AR. Safety and outcomes of inpatient compared with outpatient surgical procedures for ankle fractures. J Bone Joint Surg Am. 2016;98(20):1699–705.
van Hoof SJM, Quanjel TCC, Kroese M, Spreeuwenberg MD, Ruwaard D. Substitution of outpatient hospital care with specialist care in the primary care setting: a systematic review on quality of care, health and costs. Plos One. 2019;14(8):e0219957.
Article ADS PubMed PubMed Central Google Scholar
Cryer L, Shannon SB, Van Amsterdam M, Leff B. Costs for “hospital at home” patients were 19 percent lower, with equal or better outcomes compared to similar inpatients. Health Aff (Millwood). 2012;31(6):1237–43.
Kringos D, Boerma W, Bourgueil Y, Cartier T, Dedeu T, Hasvold T, et al. The strength of primary care in Europe: an international comparative study. Br J Gen Pract. 2013;63(616):e742–50.
Arksey H, O’Malley L. Scoping studies: towards a methodological framework. Int J Soc Res Methodol. 2005;8(1):19–32.
Evans SR. Clinical trial structures. J Exp Stroke Transl Med. 2010;3(1):8–18.
Malay S, Chung KC. The choice of controls for providing validity and evidence in clinical research. Plast Reconstr Surg. 2012;130(4):959–65.
Abdelmotagly Y, Noureldin M, Paramore L, Kummar R, Nedas T, Hindley R, et al. The impact of remote urology outpatient clinics during the COVID-19 pandemic. J Endoluminal Endourol. 2021;4(3):e17–25.
Barsom EZ, Jansen M, Tanis PJ, van de Ven AWH, van Blussé M, Buskens CJ, et al. Video consultation during follow up care: effect on quality of care and patient- and provider attitude in patients with colorectal cancer. Surg Endosc. 2021;35(3):1278–87.
Bager P, Hentze R, Nairn C. Outpatients with inflammatory bowel disease (IBD) strongly prefer annual telephone calls from an IBD nurse instead of outpatient visits. Gastroenterol Nurs. 2013;36(2):92–6.
Beaver K, Williamson S, Chalmers K. Telephone follow-up after treatment for breast cancer: views and experiences of patients and specialist breast care nurses. J Clin Nurs. 2010;19(19–20):2916–24.
Boydell N, Reynolds-Wright JJ, Cameron ST, Harden J. Women’s experiences of a telemedicine abortion service (up to 12 weeks) implemented during the coronavirus (COVID‐19) pandemic: a qualitative evaluation. BJOG: Int J Obstet Gynecol. 2021;128(11):1752–61.
Article CAS Google Scholar
Brewer A, Coleman V. Adaptation of a stoma care pathway and use of telephone clinics during the pandemic: patient experience survey. Br J Nurs. 2022;31(1):8–14.
Casey R, Powell L, Braithwaite M, Booth C, Sizer B, Corr J. Nurse-led phone call follow-up clinics are effective for patients with prostate cancer. J Patient Exp. 2017;4(3):114–20.
Damery S, Jones J, O’Connell Francischetto E, Jolly K, Lilford R, Ferguson J. Remote consultations versus standard face-to-face appointments for liver transplant patients in routine hospital care: feasibility randomized controlled trial of myVideoClinic. J Med Internet Res. 2021;23(9):e19232.
Duncan H, Russell RK. Role for structured telephone clinics in paediatric gastroenterology: reflections, lessons and patient feedback. BMJ Open Gastroenterol. 2019;6(1):e000245.
Hansen JB, Sørensen JF, Glassou EN, Homilius M, Hansen TB. Reducing patient–staff contact in fast-track total hip arthroplasty has no effect on patient-reported outcomes, but decreases satisfaction amongst patients with self-perceived complications: analysis of 211 patients. Acta Orthop. 2022;93:264.
Heeno E, Biesenbach I, Englund C, Lund M, Toft A, Lund L. Patient perspective on telemedicine replacing physical consultations in urology during the COVID-19 lockdown in Denmark. Scandinavian J Urol. 2021;55(3):177–83.
Jones MT, Arif R, Rai A. Patient experiences with telemedicine in a national health service rheumatology outpatient department during coronavirus disease-19. J Patient Exp. 2021;8:23743735211034972.
PubMed PubMed Central Google Scholar
Khan Z, Kershaw V, Madhuvrata P, Radley S, Connor M. Patient experience of telephone consultations in gynaecology: a service evaluation. BJOG: Int J Obstet Gynecol. 2021;128(12):1958–65.
Kimman ML, Dellaert BG, Boersma LJ, Lambin P, Dirksen CD. Follow-up after treatment for breast cancer: one strategy fits all? An investigation of patient preferences using a discrete choice experiment. Acta Oncol. 2010;49(3):328–37.
Kjeldsted E, Lindblad KV, Bødtcher H, Sørensen DM, Rosted E, Christensen HG, et al. A population-based survey of patients’ experiences with teleconsultations in cancer care in Denmark during the COVID-19 pandemic. Acta Oncol. 2021;60(10):1352–60.
Article CAS PubMed Google Scholar
Knudsen LR, de Thurah A, Lomborg K. Experiences with telehealth followup in patients with rheumatoid arthritis: a qualitative interview study. Arthritis Care Res. 2018;70(9):1366–72.
Lee J, Hynes C, Humphries G, Thumbikat P. Pilot study to explore the use of video consultation for outpatient follow up of spinal cord injury (SCI) patients. Clin Rehabil. 2017;31(12):1690.
Lim K, Neal-Smith G, Mitchell C, Xerri J, Chuanromanee P. Perceptions of the use of artificial intelligence in the diagnosis of skin cancer: an outpatient survey. Clin Exp Dermatol. 2022;47(3):542–6.
Lo WB, Herbert K, Rodrigues D. Clinical effectiveness of and family experience with telephone consultation in a regional pediatric neurosurgery center in the United Kingdom. J Neurosurg Pediatr. 2021;28(4):483–9.
Patel S, Douglas-Moore J. A reflection on an adapted approach from face‐to‐face to telephone consultations in our Urology Outpatient Department during the COVID‐19 pandemic–a pathway for change to future practice? BJU Int. 2020;126(3):339–41.
Rovira A, Brar S, Munroe-Gray T, Ofo E, Rodriguez C, Kim D. Telephone consultation for two-week-wait ENT and head and neck cancer referrals: initial evaluation including patient satisfaction. J Laryngology Otology. 2022;136(7):615–21.
Singh N, Datta M. Single-centre telephone survey on patients’ perspectives regarding remote paediatric outpatient consultations in a district general hospital. BMJ Paediatrics Open. 2020;4(1):e000885.
Stavrou M, Lioutas E, Lioutas J, Davenport RJ. Experiences of remote consulting for patients and neurologists during the COVID-19 pandemic in Scotland. BMJ Neurol Open. 2021;3(2):e000173.
Trace S, Collinson A, Searle A, Lithander F. Using videoconsultations to deliver dietary advice to children with chronic kidney disease: a qualitative study of parent and child perspectives. J Hum Nutr Dietetics. 2020;33(6):881–9.
Tyler JM, Pratt AC, Wooster J, Vasilakis C, Wood RM. The impact of increased outpatient telehealth during COVID-19: retrospective analysis of patient survey and routine activity data from a major healthcare system in England. Int J Health Plann Manag. 2021;36(4):1338–45.
Van Erkel FM, Pet MJ, Bossink EH, Van de Graaf CF, Hodes MT, Van Ogtrop SN, et al. Experiences of patients and health care professionals on the quality of telephone follow-up care during the COVID-19 pandemic: a large qualitative study in a multidisciplinary academic setting. BMJ Open. 2022;12(3):e058361.
Watters C, Miller B, Kelly M, Burnay V, Karagama Y, Chevretton E. Virtual voice clinics in the COVID-19 era: have they been helpful? Eur Arch Otorhinolaryngol. 2021;278:4113–8.
Williamson S, Chalmers K, Beaver K. Patient experiences of nurse-led telephone follow-up following treatment for colorectal cancer. Eur J Oncol Nurs. 2015;19(3):237–43.
Aicken CR, Fuller SS, Sutcliffe LJ, Estcourt CS, Gkatzidou V, Oakeshott P, et al. Young people’s perceptions of smartphone-enabled self-testing and online care for sexually transmitted infections: qualitative interview study. BMC Public Health. 2016;16(1):1–11.
Baraitser P, Brown KC, Gleisner Z, Pearce V, Kumar U, Brady M. ‘Do it yourself’sexual health care: the user experience. Sex Health. 2011;8(1):23–9.
Boons CC, Timmers L, Janssen JJ, Swart EL, Hugtenburg JG, Hendrikse NH. Feasibility of and patients’ perspective on nilotinib dried blood spot self-sampling. Eur J Clin Pharmacol. 2019;75:825–9.
Bundgaard JS, Raaschou-Pedersen DT, Todsen T, Ringgaard A, Torp-Pedersen C, Von Buchwald C, et al. Danish citizens’ preferences for at-home oropharyngeal/nasal SARS-CoV-2 specimen collection. Int J Infect Dis. 2021;109:195–8.
Cameron S, Glasier A, Dewart H, Johnstone A. Women’s experiences of the final stage of early medical abortion at home: results of a pilot survey. BMJ Sex Reprod Health. 2010;36(4):213–6.
Grogan A, Coughlan M, Prizeman G, O’Connell N, O’Mahony N, Quinn K, et al. The patients’ perspective of international normalized ratio self-testing, remote communication of test results and confidence to move to self-management. J Clin Nurs. 2017;26(23–24):4379–89.
Haroon S, Griva K, Davenport A. Factors affecting uptake of home hemodialysis among self-care dialysis unit patients. Hemodial Int. 2020;24(4):460–9.
Hope J. A patient perspective on the barriers to home dialysis. J Ren care. 2013;39(S1):3–8.
Article MathSciNet PubMed Google Scholar
Hoyos J, Maté T, Guerras J-M, Donat M, Agustí C, Kuske M, et al. Preference towards HIV Self-Testing above other testing options in a sample of men who have sex with men from five European countries. Int J Environ Res Public Health. 2021;18(9):4804.
Den Oudendammer WM, Broerse JE. Towards a decision aid for self-tests: users’ experiences in the Netherlands. Health Expect. 2019;22(5):983–92.
Tompson AC, Ward AM, McManus RJ, Perera R, Thompson MJ, Heneghan CJ, et al. Acceptability and psychological impact of out-of-office monitoring to diagnose hypertension: an evaluation of survey data from primary care patients. Br J Gen Pract. 2019;69(683):e389–97.
Tonna A, Anthony G, Tonna I, Paudyal V, Forbes-McKay K, Laing R, et al. Home self-administration of intravenous antibiotics as part of an outpatient parenteral antibiotic therapy service: a qualitative study of the perspectives of patients who do not self-administer. BMJ Open. 2019;9(1):e027475.
Veerus P, Hallik R, Jänes J, Jõers K, Paapsi K, Laidra K, et al. Human papillomavirus self-sampling for long-term non-attenders in cervical cancer screening: a randomised feasibility study in Estonia. J Med Screen. 2022;29(1):53–60.
Witzel T, Bourne A, Burns F, Rodger A, McCabe L, Gabriel M, et al. HIV self-testing intervention experiences and kit usability: results from a qualitative study among men who have sex with men in the SELPHI (Self‐Testing Public Health Intervention) randomized controlled trial in England and Wales. HIV Med. 2020;21(3):189–97.
Bendien S, van Leeuwen M, Lau H, Ten Brinke A, Visser L, de Koning E, et al. Home-based intravenous treatment with reslizumab for severe asthma in the Netherlands–An evaluation. Respir Med. 2022;194:106776.
Corrie P, Moody A, Armstrong G, Nolasco S, Lao-Sirieix S, Bavister L, et al. Is community treatment best? A randomised trial comparing delivery of cancer treatment in the hospital, home and GP surgery. Br J Cancer. 2013;109(6):1549–55.
Dismore LL, Echevarria C, Van Wersch A, Gibson J, Bourke S. What are the positive drivers and potential barriers to implementation of hospital at home selected by low-risk DECAF score in the UK: a qualitative study embedded within a randomised controlled trial. BMJ Open. 2019;9(4):e026609.
Goossens LM, Utens CM, Smeenk FW, Donkers B, van Schayck OC, Rutten-van Mölken MP. Should I stay or should I go home? A latent class analysis of a discrete choice experiment on hospital-at-home. Value Health. 2014;17(5):588–96.
Hansson H, Kjaergaard H, Johansen C, Hallström I, Christensen J, Madsen M, et al. Hospital-based home care for children with cancer: feasibility and psychosocial impact on children and their families. Pediatr Blood Cancer. 2013;60(5):865–72.
Hansson H, Kjaergaard H, Schmiegelow K, Hallström I. Hospital-based home care for children with cancer: a qualitative exploration of family members’ experiences in Denmark. Eur J Cancer Care. 2012;21(1):59–66.
Jepsen LØ, Høybye MT, Hansen DG, Marcher CW, Friis LS. Outpatient management of intensively treated acute leukemia patients—the patients’ perspective. Support Care Cancer. 2016;24:2111–8.
Lohr PA, Wade J, Riley L, Fitzgibbon A, Furedi A. Women’s opinions on the home management of early medical abortion in the UK. BMJ Sex Reprod Health. 2010;36(1):21–5.
Rosted E, Aabom B, Hølge-Hazelton B, Raunkiær M. Comparing two models of outpatient specialised palliative care. BMC Palliat Care. 2021;20(1):1–13.
Schiff R, Oyston M, Quinn M, Walters S, McEnhill P, Collins M. Hospital at home: another piece of the armoury against COVID-19. Future Healthc J. 2022;9(1):90–5.
Uitdehaag MJ, Van Putten PG, Van Eijck CH, Verschuur EM, Van der Gaast A, Pek CJ, et al. Nurse-led follow-up at home vs. conventional medical outpatient clinic follow-up in patients with incurable upper gastrointestinal cancer: a randomized study. J Pain Symptom Manag. 2014;47(3):518–30.
Utens CM, Goossens LM, Van Schayck OC, Rutten-van Mölken MP, Van Litsenburg W, Janssen A, et al. Patient preference and satisfaction in hospital-at-home and usual hospital care for COPD exacerbations: results of a randomised controlled trial. Int J Nurs Stud. 2013;50(11):1537–49.
Van Ramshorst J, Duffels M, De Boer S, Bos-Schaap A, Drexhage O, Walburg S, et al. Connected care for endocarditis and heart failure patients: a hospital-at-home programme. Neth Heart J. 2022;30(6):319–27.
Baena-Cañada JM, Ramirez-Daffos P, Cortes-Carmona C, Rosado-Varela P, Nieto-Vera J, Benitez-Rodriguez E. Follow-up of long-term survivors of breast cancer in primary care versus specialist attention. Fam Pract. 2013;30(5):525–32.
Van Bodegom-Vos L, De Jong JD, Spreeuwenberg P, Curfs EC, Groenewegen PP. Are patients’ preferences for shifting services from medical specialists to general practitioners related to the type of medical intervention? Qual Prim Care. 2013;21(2):81–95.
Pollard L, Rogers S, Shribman J, Sprigings D, Sinfield P. A study of role expansion: a new GP role in cardiology care. BMC Health Serv Res. 2014;14(1):1–10.
Milosevic S, Joseph-Williams N, Pell B, Cain E, Hackett R, Murdoch F, et al. Managing lower urinary tract symptoms in primary care: qualitative study of GPs’ and patients’ experiences. Br J Gen Pract. 2021;71(710):e685–92.
Wildeboer JA, Van de Ven ART, De Boer D. Substitution of care for chronic heart failure from the hospital to the general practice: patients’ perspectives. BMC Fam Pract. 2018;19(1):8.
Cottrell E, McMillan K, Chambers R. A cross-sectional survey and service evaluation of simple telehealth in primary care: what do patients think? BMJ Open. 2012;2(6):e001392.
Scott A, Jones C. An exploration of the attitudes and perceptions of the UK public towards self-care for minor ailments. Br J Nurs. 2020;29(1):44–9.
McAteer A, Yi D, Watson V, Norwood P, Ryan M, Hannaford PC, et al. Exploring preferences for symptom management in primary care: a discrete choice experiment using a questionnaire survey. Br J Gen Pract. 2015;65(636):e478–88.
Fletcher B, Hinton L, McManus R, Rivero-Arias O. Patient preferences for management of high blood pressure in the UK: a discrete choice experiment. Br J Gen Pract. 2019;69(686):e629–37.
Heath G, Greenfield S, Redwood S. The meaning of ‘place’in families’ lived experiences of paediatric outpatient care in different settings: a descriptive phenomenological study. Health Place. 2015;31:46–53.
King KE. Patient satisfaction in a one-stop Haematuria clinic and urology outpatients: a comparison of clinics. Int J Urol Nurs. 2016;10(3):127–36.
Fitzsimmons DA, Thompson J, Bentley CL, Mountain GA. Comparison of patient perceptions of Telehealth-supported and specialist nursing interventions for early stage COPD: a qualitative study. BMC Health Serv Res. 2016;16:1–12.
Mitchell T. Patients’ experiences of receiving chemotherapy in outpatient clinic and/or onboard a unique nurse-led mobile chemotherapy unit: a qualitative study. Eur J Cancer Care. 2013;22(4):430–9.
Cook EJ, Randhawa G, Large S, Guppy A, Chater AM, Ali N. Barriers and facilitators to using NHS Direct: a qualitative study ofusers’ andnon-users’. BMC Health Serv Res. 2014;14(1):1–12.
Download references
We would like to thank Linda Schoonmade, medical information specialist, for her contribution to this research in helping develop the search strategy.
The authors received no specific funding for this work.
Authors and affiliations.
Nivel, Netherlands Institute for Health Services Research, Utrecht, the Netherlands
L. J. Damen, L. H. D. Van Tuyl, J. C. Korevaar, B. J. Knottnerus & J. D. De Jong
CAPHRI, Maastricht University, PO Box 616, 6200 MD Maastricht, Maastricht, the Netherlands
J. D. De Jong
The Hague University of Applied Sciences, The Hague, the Netherlands
J. C. Korevaar
You can also search for this author in PubMed Google Scholar
The selection process of articles was performed by all authors. L.D. wrote the main manuscript text. All authors reviewed and edited the manuscript. L.T., J.J., B.K. and J.K. supervised.
Correspondence to L. J. Damen .
Ethics approval and consent to participate.
Not applicable.
Competing interests.
The authors declare no competing interests.
Publisher’s note.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Additional file 1: appendix a..
Search string Pubmed.
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.
Reprints and permissions
Cite this article.
Damen, L.J., Van Tuyl, L.H.D., Korevaar, J.C. et al. Citizens’ perspectives on relocating care: a scoping review. BMC Health Serv Res 24 , 202 (2024). https://doi.org/10.1186/s12913-024-10671-3
Download citation
Received : 16 June 2023
Accepted : 01 February 2024
Published : 14 February 2024
DOI : https://doi.org/10.1186/s12913-024-10671-3
Anyone you share the following link with will be able to read this content:
Sorry, a shareable link is not currently available for this article.
Provided by the Springer Nature SharedIt content-sharing initiative
ISSN: 1472-6963
Check out the full question paper for the cbse class 10 hindi board exam 2024 here..
Listen to Story
The Central Board of Secondary Education (CBSE) has successfully conducted the CBSE Class 10 Hindi board exam today, February 21, 2024. As per the initial reactions, students found the paper was moderate in terms of difficulty level. The paper was held for three hours, and students were given another 15 minutes to go through it.
"The paper was slightly lengthy. The questions weren't anything that hadn't been practised in class before. The grammar section was moderate and the topics that had come for the writing section weren't anything too different from those in the sample papers" said Saanvi Singh, a Class 10 student from Seth Anandram Jaipuria School, Lucknow.
IMAGES
VIDEO
COMMENTS
Step 1 - Search for relevant literature Step 2 - Evaluate and select sources Step 3 - Identify themes, debates, and gaps Step 4 - Outline your literature review's structure Step 5 - Write your literature review Free lecture slides Other interesting articles Frequently asked questions Introduction Quick Run-through Step 1 & 2 Step 3 Step 4 Step 5
Literature Review Matrix. As you read and evaluate your literature there are several different ways to organize your research. Courtesy of Dr. Gary Burkholder in the School of Psychology, these sample matrices are one option to help organize your articles. These documents allow you to compile details about your sources, such as the foundational ...
Figure 1 provides an example of a literature summary table from a scoping review. 3 Download figure Open in new tab Download powerpoint Figure 1 Tabular literature summaries from a scoping review. Source: Rasheed et al. 3 The provision of information about conceptual and theoretical frameworks and methods is useful for several reasons.
Here's an example of the layout visualised in a literature review introduction: Your introduction should also outline your topic (including any tricky terminology or jargon) and provide an explanation of the scope of your literature review - in other words, what you will and won't be covering (the delimitations ).
This is an example of a synthesis table or synthesis matrix, in which you organize and analyze your research by listing each source and indicating whether a given finding or result occurred in a particular study or article ( each row lists an individual source, and each finding has its own column, in which X = yes, blank = no).
Definition Examples A synthesis matrix helps you record the main points of each source and document how sources relate to each other. After summarizing and evaluating your sources, arrange them in a matrix or use a citation manager to help you see how they relate to each other and apply to each of your themes or variables.
Examples of Literature Tables Camak, D.J. (2015), Addressing the burden of stroke caregivers: a literature review. J Clin Nurs, 24: 2376-2382. doi: 10.1111/jocn.12884 Balcombe, L., Miller, C., & McGuiness, W. (2017). Approaches to the application and removal of compression therapy: A literature review.
This template is structure is based on the tried and trusted best-practice format for formal academic research projects such as dissertations and theses. The literature review template includes the following sections: Before you start - essential groundwork to ensure you're ready The introduction section The core/body section
A literature review is a survey of scholarly sources on a specific topic. It provides an overview of current knowledge, allowing you to identify relevant theories, methods, and gaps in the existing research. There are five key steps to writing a literature review: Search for relevant literature. Evaluate sources. Identify themes, debates and gaps.
Steps to Completing a Literature Review. Conduct searches for relevant information. Critically review your sources. Determine the most important and relevant information from each source, theories, findings, etc. Create a synthesis matrix to find connections between resources, and ensure your sources relate to your main ideas. Use the synthesis ...
Cite Using citation machines responsibly Powered by A literature review can be a part of a research paper or scholarly article, usually falling after the introduction and before the research methods sections.
Literature Review Sample 2 Literature Review Sample 3 Have an exemplary literature review? Have you written a stellar literature review you care to share for teaching purposes? Are you an instructor who has received an exemplary literature review and have permission from the student to post? Please contact Britt McGowan at [email protected] for ...
Common Assignments: Literature Reviews. Include an introduction and conclusion. Even if the literature review will be part of a longer document, introductory and concluding paragraphs can act as bookends to your material. Provide background information for your reader, such as including references to the pioneers in the field in the beginning ...
50 Smart Literature Review Templates (APA) A literary review template is a type of written work that discusses published information about a specific subject matter. The length of the review doesn't matter. It can be as simple as a summary of sources or can be as long as several pages.
- The presence of negative consequences cited in the literature indicates a need to further evaluate the risks associated with eHealth Harrington, L., D. Kennerly, and C. Johnson. 2011. Safety issues related to the electronic medi-cal record (EMR): Synthesis of the literature from the last decade, 2000-2009. Journal of Health Care Management
This is called a review matrix. When you create a review matrix, the first few columns should include (1) the authors, title, journal, (2) publication year, and (3) purpose of the paper. The remaining columns should identify important aspects of each study such as methodology and findings. Click on the image below to view a sample review matrix.
Descriptive: Used for reviews. They summarize the subject of the review and the approach the reviewer has taken in his or her coverage of the subject. This type of abstract does not report original findings. A Good Literature Review is.
Literature Review Examples For the following types of literature review, I present an explanation and overview of the type, followed by links to some real-life literature reviews on the topics. 1. Narrative Review Examples
Download The Literature Catalogue Template (Free) Need a template for the actual literature review chapter? You can get that here. How to use the Excel template effectively. A quick overview. The first tab (labelled "Literature") is where you'll record specific details of all the reading you'll do.
Literature reviews are in great demand in most scientific fields. Their need stems from the ever-increasing output of scientific publications .For example, compared to 1991, in 2008 three, eight, and forty times more papers were indexed in Web of Science on malaria, obesity, and biodiversity, respectively .Given such mountains of papers, scientists cannot be expected to examine in detail every ...
Table 14.3.a provides a framework and examples for how review authors can justify their judgements about the certainty of evidence in each domain. These justifications should also be included in explanatory notes to the 'Summary of Findings' table (see Section 14.1.6.10 ).
Literature Review Examples #01 #02 #03 #04 #05 #06 #07 #08 #09 #10 #11 #12 #13 #14 #15 #16 #17 #18 Purpose of Literature Review Literature reviews have different purposes in scholarly articles, research papers, and books, depending on the discipline at hand.
Evidence on the effectiveness of continuing education (CE) and CE methods was identified through a literature review.Although nonexhaustive, the review included a comprehensive search of the Research and Development Resource Base (RDRB), a bibliographic database of more than 18,000 articles from fields including CE, knowledge translation, interprofessional literature, and faculty development.
Data extraction. A spreadsheet was created to categorise the information that contributed to answering the research questions. The information extracted from the articles was structured according to the type of relocation, including: relocating from the hospital to the GP, to care at home, to self-care, or to eHealth, and relocating from the GP to self-care, to care at home, or to eHealth. The ...
The Central Board of Secondary Education (CBSE) has successfully conducted the CBSE Class 10 Hindi board exam today, February 21, 2024. As per the initial reactions, students found the paper was moderate in terms of difficulty level. The paper was held for three hours, and students were given another 15 minutes to go through it.