• En español – ExME
  • Em português – EME

How to read a funnel plot

Posted on 27th June 2023 by Jehath Syed

cochrane systematic review funnel plot

What is a funnel plot?

A funnel plot is a scatter plot that compares the precision (how close the estimated intervention effect size is to the true effect size) and results of individual studies. It is commonly used in meta-analyses to visually detect publication bias.

The term ‘funnel plot’ refers to the fact that the precision of the estimated intervention effect increases with the size of the study. Small study effect estimates will typically scatter more widely at the bottom of the graph, with the spread narrowing among larger studies as they are more precise and closer to the true effect.

How do you read a funnel plot?

The plot should ideally look like a pyramid or a symmetrical inverted funnel, as seen in Image A.

""

  • Each included study is represented as a dot.
  • The y-axis represents a measure of study precision, with standard error being commonly used. Larger studies with greater precision are displayed at the top and studies with lower precision at the bottom. Other measures such as the reciprocal of the standard error, the reciprocal of the sample size, or variance of the estimated effect can also be used as the y-axis.
  • The x-axis displays the study estimated effect size for an outcome. The scale for the x-axis can include risk ratios or odds ratios (which should be plotted on a logarithmic scale), or continuous measures such as mean difference or standardised mean difference.
  • In the absence of both bias and heterogeneity, 95% of studies would be expected to lie within the diagonal dotted ‘95% Confidence Interval’ lines, as shown in Figure A.

As a rule of thumb, tests for funnel plot asymmetry should only be used when at least 10 studies are included in the meta-analysis, because the power of the tests is low when there are fewer studies.

What are you looking for?

Image A is shown again below alongside Image B, which depicts an asymmetrical funnel due to presence of bias (the points are now predominantly towards the left). In this case, the meta-analysis summary estimate will tend to overestimate the intervention effect. The greater the asymmetry, the greater the likelihood that the amount of bias in the meta-analysis will be significant.

""

Possible reasons for asymmetry in a funnel plot are:

  • Non-reporting bias: Some studies, or specific results, are less likely to be published if they are not statistically significant, or if the effect size is very small or non-existent.
  • Poor methodological quality leading to exaggerated effects: Studies with inferior methods may show larger effect estimates of an intervention than would have been observed in a well-designed study.
  • True heterogeneity : Sometimes a significant benefit of an intervention can only be observed in patients who are at high risk for the outcome targeted by the intervention. These high-risk patients are more likely to be included in small, early trials, leading to asymmetry in the funnel plot.
  • Artefactual: Certain effect estimates, such as odds ratios or standardised mean differences, are inherently correlated with their standard errors. This correlation can create a false asymmetry in a funnel plot even when there is no bias.
  • Chance: With a small number of studies and their heterogeneity (variation), the analysis of the relationships between studies in a meta-analysis is more prone to false positives. This can affect the symmetry of the funnel plot.

References (pdf)

You may also be interested in the following resources related to this topic:

  • Medical Statistics X: Funnel Plots (Rob Radcliffe, School of Surgery) – YouTube video (9 minutes)

  • Catalogue of Bias from the Centre of Evidence Based Medicine, Oxford (website)
  • Funnel plots in meta-analysis (pdf) (paper)
  • A beginner’s guide to interpreting odds ratios, confidence intervals and p-values (blog)

' src=

Jehath Syed

Leave a reply cancel reply.

Your email address will not be published. Required fields are marked *

Save my name, email, and website in this browser for the next time I comment.

Subscribe to our newsletter

You will receive our monthly newsletter and free access to Trip Premium.

Related Articles

""

Systematic reviews vs meta-analysis: what’s the difference?

What are the differences between a systematic review and a meta-analysis? Here are some tips to help you understand these two different yet related types of study.

""

Heterogeneity in meta-analysis

When you bring studies together in a meta-analysis, one of the things you need to consider is the variability in your studies – this is called heterogeneity. This blog presents the three types of heterogeneity, considers the different types of outcome data, and delves a little more into dealing with the variations.

""

Internal and external validity: what are they and how do they differ?

Is this study valid? Can I trust this study’s methods and design? Can I apply the results of this study to other contexts? Learn more about internal and external validity in research to help you answer these questions when you next look at a paper.

10.4.2   Different reasons for funnel plot asymmetry

Although funnel plot asymmetry has long been equated with publication bias (Light 1984, Begg 1988) , the funnel plot should be seen as a generic means of displaying small-study effects – a tendency for the intervention effects estimated in smaller studies to differ from those estimated in larger studies (Sterne 2000) . Small-study effects may be due to reasons other than publication bias (Egger 1997a, Sterne 2000) . Some of these are shown in Table 10.4.a .

Differences in methodological quality are an important potential source of funnel plot asymmetry. Smaller studies tend to be conducted and analysed with less methodological rigour than larger studies (Egger 2003) . Trials of lower quality also tend to show larger intervention effects (Schulz 1995) . Therefore trials that would have been ‘negative’, if conducted and analysed properly, may become ‘positive’ ( Figure 10.4.a , Panel C).

True heterogeneity in intervention effects may also lead to funnel plot asymmetry. For example, substantial benefit may be seen only in patients at high risk for the outcome which is affected by the intervention and these high risk patients are usually more likely to be included in early, small studies (Davey Smith 1994, Glasziou 1995) . In addition, small trials are generally conducted before larger trials are established and in the intervening years standard treatment may have improved (resulting in smaller intervention effects in the larger trials). Furthermore, some interventions may have been implemented less thoroughly in larger trials and may, therefore, have resulted in smaller estimates of the intervention effect (Stuck 1998) . Finally, it is of course possible that an asymmetrical funnel plot arises merely by the play of chance. Terrin et al. have suggested that the funnel plot is inappropriate for heterogeneous meta-analyses, drawing attention to the premise that the studies come from a single underlying population given by the originators of the funnel plot (Light 1984, Terrin 2003).

A proposed enhancement (Peters 2008) to the funnel plot is to include contour lines corresponding to perceived ‘milestones’ of statistical significance (P = 0.01, 0.05, 0.1 etc). This allows the statistical significance of study estimates, and areas in which studies are perceived to be missing, to be considered. Such ‘contour-enhanced’ funnel plots may help review authors to differentiate asymmetry due to publication bias from that due to other factors. For example if studies appear to be missing in areas of statistical non-significance (see Figure 10.4.b , Panel A for an example) then this adds credence to the possibility that the asymmetry is due to publication bias. Conversely, if the supposed missing studies are in areas of higher statistical significance (see Figure 10.4.b , Panel B for an example), this would suggest the cause of the asymmetry may be more likely to be due to factors other than publication bias (see Table 10.4.a ). If there are no statistically significant studies then publication bias may not be a plausible explanation for funnel plot asymmetry (Ioannidis 2007b).

In interpreting funnel plots, systematic review authors thus need to distinguish the different possible reasons for funnel plot asymmetry listed in Table 10.4.a . Knowledge of the particular intervention, and the circumstances in which it was implemented in different studies, can help identify true heterogeneity as a cause of funnel plot asymmetry. There remains a concern that visual interpretation of funnel plots is inherently subjective. Therefore, we now discuss statistical tests for funnel plot asymmetry, and the extent to which they may assist in the objective interpretation of funnel plots. When review authors are concerned that small study effects are influencing the results of a meta-analysis, they may want to conduct sensitivity analyses in order to explore the robustness of the meta-analysis’ conclusions to different assumptions about the causes of funnel plot asymmetry: these are discussed in Section 10.4.4 .

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • View all journals
  • Explore content
  • About the journal
  • Publish with us
  • Sign up for alerts
  • Published: 23 December 2011

The role of the funnel plot in detecting publication and related biases in meta-analysis

  • Joseph LY Liu 1  

Evidence-Based Dentistry volume  12 ,  pages 121–122 ( 2011 ) Cite this article

3648 Accesses

31 Citations

4 Altmetric

Metrics details

  • Statistical methods

You have full access to this article via your institution.

What is publication bias and why is it important?

The late and distinguished Alvan Feinstein expressed concern about the proliferation of positive associations with the ‘menace[s] of everyday life’ in risk factor epidemiology. 1 Publication bias is a long-standing problem in clinical research, in which treatments that demonstrate a significant health benefit are more likely to be published than those that do not. 2 , 3 The problem was highlighted in the 1950s when a survey of empirical studies published in leading psychology journals discovered an astonishing 97% reported statistically significant results. 4 Meta-analyses which include only published studies are therefore likely to be biased.

The impact of publication bias on the integrity of medical research and the findings of meta-analyses have been widely discussed and evaluated. 2 , 3 , 5 , 6 , 7 , 8 In an early example (1986), Simes compared the benefit of treating advanced ovarian cancer using an alkylating agent with combination chemotherapy. He conducted a meta-analysis of published trials only and a meta-analysis of all registered trials. 2 The pooled median survival ratio for the published trials was 1.16 (95% CI 1.06 to 1.27), which suggested improved survival using combination therapy, while the pooled ratio for the registered trials was 1.06 (95% CI 0.97-1.15), a smaller and less optimistic improvement that was not statistically significant. The difference between the two pooled estimates was attributed to publication bias. 2

Publication bias in medical research continues to be a problem today, as shown by a recent meta-analysis which compared the effectiveness of reboxetine with selective serotonin reuptake inhibitors (SSRI) or placebo for treating major depression. 8 The investigators found that 74% of data were not published and ‘published data overestimated the benefit of reboxetine versus placebo by up to 115% and reboxetine versus SSRIs by up to 23%, and also underestimated harm’. 8 Previous findings of reboxetine's effectiveness were reversed.

In dentistry, Scholey and Harrison demonstrated that more than 50% of studies presented at international conferences in dental health were unpublished five years after the conference, suggesting that a meta-analysis in a dental health-related area could be biased if a search for unpublished studies is not included. 9 A review of five orthodontic journals shows that 88% of studies demonstrated statistically significant results. 10 A similar study of journals in maxillofacial surgery came up with a figure of 77%. 11 Like other health care domains, publication bias is clearly a problem in dental research. 9 , 10 , 11 , 12

The funnel plot as a tool to detect publication bias

The funnel plot is a commonly used graphical device to detect publication bias in systematic reviews. Originally advocated by Light and Pillemer 13 , it is a plot of a study-specific effect estimate against an estimate of the study's precision. Precision may be assessed in various ways, through a function of the standard error for an effect measure or simply the sample size in each study of a systematic review. 13 , 14 , 15 , 16 , 17 Effect estimates include relative risk, risk ratio, odds ratio, absolute risk and logarithmic transformations of these measures.

In the absence of publication bias, the points will be symmetrically distributed around the true effect in the shape of an inverted funnel ( Figure 1 ). 13 , 14 , 15 , 16 , 17 As the sample size of studies in an unbiased meta-analysis increases, the effect estimates become more precise. The scatter of effect estimates around the true effect would therefore be expected to become wider at the bottom end of the plot, where the smaller studies are located. The scatter becomes progressively narrower as the studies increase in size going up the plot. In principle, the pooled effect estimate should reflect the true effect. In the presence of publication bias the shape of the funnel plot will be asymmetric, with negative effect estimates from smaller studies missing from the plot and the pooled effect estimate diverging from the true effect. 18 ( Figure 2 ).

figure 1

Symmetrical funnel plot

figure 2

Asymmetric funnel plot

However, the empirical evidence strongly suggests that visual inspection of funnel plots alone is not a reliable way to assess the shape of the funnel plot. 19

Statistical tests to assess shape of funnel plot

Various statistical tests have thus been developed to more objectively evaluate the shape of the funnel plot and test for the presence of publication bias. Regression models are the most commonly used, 20 and test for an association between the study's effect estimate and its precision, as measured for example by the inverse of the standard error. 21 Publication bias may be present if the fitted regression model suggests that the less precise or smaller studies have bigger effect estimates than the more precise or larger studies. 21 A weakness of these tests is that they have low statistical power and sometimes fail to detect publication bias when it actually exists. The Cochrane Handbook for Systematic Reviews of Interventions, available online at http://www.cochrane-handbook.org/ , provides a list of different regression models. 20

The results of these regression models can vary, depending on the choice of effect measure and the choice of precision measure used to construct the funnel plot. 15 This issue has been discussed in various contexts for many years 15 , 16 , 19 , 22 , 23 , 24 , 25 , 26 and is beyond the scope of the present Toolbox article. The reader is advised to refer to the Cochrane Collaboration's latest recommendations, which undergo regular review and periodic changes. 20

Publication bias as part of a more general phenomenon

It should be noted that publication bias is part of a more general type of bias:

Reporting biases arise when the dissemination of research findings is influenced by the nature and direction of results. Statistically significant, ‘positive’ results that indicate that an intervention works are more likely to be published, more likely to be published rapidly, more likely to be published in English, more likely to be published more than once, more likely to be published in high impact journals and, related to the last point, more likely to be cited by others. 20

The Cochrane Handbook provides a detailed discussion and typology of reporting biases. 20 Reporting bias (including publication bias), true heterogeneity and chance, could all account for an asymmetric funnel plot. 15 , 20 , 27 True heterogeneity refers to genuine variation in effect size according to the size of the study. This could happen if the characteristics of patients in small trials are different from those in large trials, eg small trials may enrol higher risk patients than large trials and effect size may depend on the underlying patient risk. 27 , 28 , 29 Finally, play of chance could also result in an asymmetric plot.

Implications for dental health research

Like other health care domains, publication bias is a serious problem in dentistry. 9 , 10 , 11 , 12 Dental health researchers might quite understandably think of the funnel plot and associated statistical tests as tools to detect only publication bias. A funnel plot can be asymmetric for a number of reasons and statistical tests to assess reporting bias have their limitations. In the absence of universal access to trial data in an ideal world, these tests serve as an aid to interpretation. 30 , 31 Dental health researchers and practitioners need to assess funnel plots with the above caveats in mind when reading systematic reviews and to seek advice from experienced systematic reviewers in using funnel plots and associated tests when conducting their own reviews.

Feinstein AR . Scientific standards in epidemiologic studies of the menace of daily life. Science 1988; 242 : 1257–1263.

Article   Google Scholar  

Simes RJ . Publication bias: the case for an international registry of clinical trials. J Clin Oncol 1986; 4 : 1529–1541.

Easterbrook PJ, Berlin JA, Gopalan R, Matthews DR . Publication bias in clinical research. Lancet 1991; 337(8746) : 867–872.

Sterling TD . Publication decisions and their possible effects on inferences drawn from tests of significance – or vice versa. J Am Stat Assoc 1959; 54 : 30–34.

Google Scholar  

Kennedy D . The old file-drawer problem. Science 2004; 305 : 451.

Liu JL, Altman DG . Conduct and reporting of clinical research. Science 2005; 308 : 201–202.

Rosenthal R . The file drawer problem and tolerance for null results. Psychol Bull 1979; 86 : 638–641.

Eyding D, Lelgemann M, Grouven U, et al . Reboxetine for acute treatment of major depression: systematic review and meta-analysis of published and unpublished placebo and selective serotonin reuptake inhibitor controlled trials. BMJ 2010; 341 : c4737.

Scholey JM, Harrison JE . Delay and failure to publish dental research. Evid Based Dent 2005; 6 : 58–61.

Koletsi D, Karagianni A, Pandis N, Makou M, Polychronopoulou A, Eliades T . Are studies reporting significant results more likely to be published? Am J Orthod Dentofacial Orthop 2009; 136 : 632.

Pitak-Arnnop P, Sader R, Rapidis AD, et al . Publication bias in oral and maxillofacial surgery journals: an observation on published controlled trials. J Craniomaxillofac Surg 2010; 38 : 4–10.

Scholey JM, Harrison JE . Publication bias: raising awareness of a potential problem in dental research. Br Dent J 2003; 194 : 235–237.

Light R, Pillemer DB . Summing up: the science of reviewing research . Cambridge: Harvard University Press, 1984.

Sutton AJ, Abrams KR, Jones DR, Sheldon TA, Song F . Methods for meta-analysis in medical research . Chichester: John Wiley & Sons, 2000.

Tang JL, Liu JL . Misleading funnel plot for detection of bias in meta-analysis. J Clin Epidemiol 2000; 53 : 477–484.

Lau J, Ioannidis JP, Terrin N, Schmid CH, Olkin I . The case of the misleading funnel plot. BMJ 2006; 333 : 597–600.

Petitti DB . Meta-analysis, decision analysis, and cost effectiveness analysis: methods for quantitative synthesis in medicine . 2nd ed. New York: Oxford University Press, 2000.

Peters JL, Sutton AJ, Jones DR, Abrams KR, Rushton L . Comparison of two methods to detect publication bias in meta-analysis. JAMA 2006; 295 : 676–680.

Terrin N, Schmid CH, Lau J . In an empirical evaluation of the funnel plot, researchers could not visually identify publication bias. J Clin Epidemiol 2005; 58 : 894–901.

Higgins JPT, Green S . Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [Updated March 2011] ( http://www.cochrane-handbook.org/ ). 2011.

Moreno SG, Sutton AJ, Ades AE, et al . Assessment of regression-based methods to adjust for publication bias through a comprehensive simulation study. BMC Med Res Methodol 2009; 9 : 2 doi: 10.1186/1471-2288-9-2.

Article   PubMed   PubMed Central   Google Scholar  

Pfeiffer T, Bertram L, Ioannidis JP . Quantifying selective reporting and the Proteus phenomenon for multiple datasets with similar bias. PLoS ONE 2011; 6 : e18362.

Sterne JA, Egger M . Funnel plots for detecting bias in meta-analysis: guidelines on choice of axis. J Clin Epidemiol 2001; 54 : 1046–1055.

Gigerenzer G, Wegwarth O, Feufel M . Misleading communication of risk. BMJ 2010; 341 : 791–792.

Rucker G, Schwarzer G, Carpenter JR, Binder H, Schumacher M . Treatment-effect estimates adjusted for small-study effects via a limit meta-analysis. Biostatistics 2011; 12 : 122–142.

Ioannidis JP . Interpretation of tests of heterogeneity and bias in meta-analysis. J Eval Clin Pract 2008; 14 : 951–957.

Egger M, Smith GD, Altman DG, editors. Systematic reviews in health care: meta-analysis in context . London: BMJ Publishing Group, 2001.

Book   Google Scholar  

Altman DG . Practical Statistics for Medical Research . London: Chapman and Hall, 1991.

Smith GD, Song F, Sheldon TA . Cholesterol lowering and mortality: the importance of considering initial level of risk. BMJ 1993; 306 : 1367–1373.

Steinbrook R, Kassirer JP . Free the data. BMJ 2010; 341 : 811–812.

Freemantle N . Commentary: Journals must facilitate the dissemination and scrutiny of clinical research. BMJ 2010; 341 : 812–813.

Download references

Author information

Authors and affiliations.

The University of Dundee, Dental Health Services Research Unit, Scottish Dental Clinical Effectiveness Programme, NHS Education for Scotland, Frankland Building, Small's Wynd Dundee, DD1 4HN, United Kingdom

Joseph LY Liu

You can also search for this author in PubMed   Google Scholar

Rights and permissions

Reprints and permissions

About this article

Cite this article.

Liu, J. The role of the funnel plot in detecting publication and related biases in meta-analysis. Evid Based Dent 12 , 121–122 (2011). https://doi.org/10.1038/sj.ebd.6400831

Download citation

Published : 23 December 2011

Issue Date : December 2011

DOI : https://doi.org/10.1038/sj.ebd.6400831

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

This article is cited by

Menstrual hygiene practice and associated factors among adolescent girls in sub-saharan africa: a systematic review and meta-analysis.

  • Etsay Woldu Anbesu
  • Dejen Kahsay Asgedom

BMC Public Health (2023)

Male involvement in family planning use and its determinants in Ethiopia: a systematic review and meta-analysis protocol

  • Setognal Birara Aychiluhm
  • Zinabu Hadush Kahsay

Systematic Reviews (2022)

Determinants of birth asphyxia among preterm newborns in Ethiopia: a systematic review and meta-analysis of observational studies protocol

  • Abay Woday Tadesse
  • Muluken Dessalegn Muluneh
  • Gebeyaw Biset Wagaw

Depressive symptoms and their determinants in patients who are on antiretroviral therapy in the case of a low-income country, Ethiopia: a systematic review and meta-analysis

  • Mogesie Necho
  • Asmare Belete
  • Mekonnen Tsehay

International Journal of Mental Health Systems (2021)

The prevalence and risk factors of work-related musculoskeletal disorders among adults in Ethiopia: a study protocol for extending a systematic review with meta-analysis of observational studies

  • Tsiwaye Gebreyesus
  • Kalkidan Nigussie
  • Balamurugan Janakiraman

Systematic Reviews (2020)

Quick links

  • Explore articles by subject
  • Guide to authors
  • Editorial policies

cochrane systematic review funnel plot

Jump to navigation

Home

Cochrane Training

Identifying publication bias in meta-analyses of continuous outcomes.

cochrane systematic review funnel plot

These videos, originally part of the Cochrane Learning Live webinar series, are highly relevant to reviewers, editors and statisticians with interests in dealing with bias in meta-analysis. Although there is some statistical content, concepts are explained visually where possible keeping much of the material accessible.

The webinar was delivered in July 2020 and below you will find the videos from the webinar, together with accompanying slides to download [PDF].

Part 1: Introduction to publication bias Part 2: Example: postoperative pain Part 3: Simulation study, limitations and conclusions Part 4: Questions and answers

Presenter bios.

Suzanne Freeman is an NIHR Research Fellow and member of the NIHR Complex Reviews Support Unit. Her research interests include network meta-analysis, individual participant data meta-analysis, synthesis of time-to-event and continuous outcomes and diagnostic test accuracy meta-analysis.

Alex Sutton has had a long-standing interest in methodology for evidence synthesis and has worked on developing methods for publication bias, diagnostic test accuracy meta-analysis and network meta-analysis. He is currently particularly interested in methods for visual communication of information and its application to synthesis methods.

Part 1: Introduction to publication bias

Part 2: Example: postoperative pain

Part 3: Simulation study, limitations and conclusions

Part 4: Questions and answers

Additional materials

Download the slides from the webinar [PDF]

  • Open access
  • Published: 16 April 2007

Assessment of funnel plot asymmetry and publication bias in reproductive health meta-analyses: an analytic survey

  • João P Souza 1 , 2 ,
  • Cynthia Pileggi 2 &
  • José G Cecatti 1  

Reproductive Health volume  4 , Article number:  3 ( 2007 ) Cite this article

14k Accesses

63 Citations

Metrics details

Despite efforts to assure high methodological standards, systematic reviews may be affected by publication bias. The objective of this study was to evaluate the occurrence of publication bias in a collection of high quality systematic reviews on reproductive health.

Systematic reviews included in the Reproductive Health Library (RHL), issue No 9, were assessed. Funnel plot was used to assess meta-analyses containing 10 or more trials reporting a binary outcome. A funnel plot, the estimated number of missing studies and the adjusted combined effect size were obtained using the "trim and fill method". Meta-analyses results that were not considered to be robust due to a possible publication bias were submitted to a more detailed assessment.

A total of 21 systematic reviews were assessed. The number of trials comprising each one ranged from 10 to 83 (median = 13), totaling 379 trials, whose results have been summarized. None of the reviews had reported any evaluation of publication bias or funnel plot asymmetry. Some degree of asymmetry in funnel plots was observed in 18 of the 21 meta-analyses evaluated (85.7%), with the estimated number of missing studies ranging from 1 to 18 (median = 3). Only for three meta-analyses, the conclusion could not be considered robust due to a possible publication bias.

Asymmetry is a frequent finding in funnel plots of meta-analyses in reproductive health, but according to the present evaluation, less than 15% of meta-analyses report conclusions that would not be considered robust. Publication bias and other sources of asymmetry in funnel plots should be systematically addressed by reproductive health meta-analysts. Next amendments in Cochrane systematic reviews should include this type of evaluation. Further studies regarding the evolution of effect size and publication bias over time in systematic reviews in reproductive health are needed.

Peer Review reports

Implementing best practices is a major goal in health services [ 1 ]. However, the identification of such practices depends on the evaluation and synthesis of a large amount of scientific information. This may be achieved by carrying out systematic reviews and meta-analyses, which have come to represent important sources of evidence-based knowledge for clinicians, policy makers and researchers [ 2 ].

A systematic review is an observational study of the scientific literature based on individual studies. It may contain meta-analyses, which are statistical procedures developed for summarizing effects across individual studies. The ideal meta-analysis should combine data appropriately to produce a more complete and meaningful estimate of the overall effect [ 3 ]. Nevertheless, despite efforts to assure high methodological standards, systematic reviews may be affected by publication bias, one of the major drawbacks of such studies and a threat to their validity. Publication bias occurs whenever the results of a set of published studies differ from the results of all the research performed on a specific topic [ 3 ]. A publication-biased meta-analysis may present an ineffective or unsafe intervention as being effective or safe, or not recommend an effective or safe intervention because the results of some studies already performed are not included. Furthermore, publication bias may be partially responsible for occasional discrepancies between the conclusions of previous meta-analyses and subsequent large multicenter trials [ 4 ].

The assessment of publication bias is a relatively new recommendation but reported in several relevant meta-analyses reporting guidelines [ 5 – 8 ]. However, it has been observed that only few meta-analyses have actually evaluated publication bias (3.2% – 6.5%) [ 9 ]. It is also unclear whether reproductive health meta-analyses are affected by publication bias, since we have been unable to identify any previous reports assessing publication bias and its effects on meta-analyses in reproductive health. The objective of this survey was, therefore, to evaluate the occurrence of publication bias in a collection of high quality systematic reviews on reproductive health.

This is an analytic survey carried out to evaluate the impact of publication bias on the results of meta-analyses of reproductive health interventions. Systematic reviews included in the World Health Organization (WHO) Reproductive Health Library (RHL), issue No 9, were assessed [ 10 ]. The RHL is a WHO instrument for documenting and disseminating best practices in the field. It reproduces the most relevant Cochrane systematic reviews related to reproductive health, adding some practical aspects and pertinent comments for developing country settings, as well as implications for research [ 10 ].

There are several methods of assessing the occurrence of publication bias. A common approach is based on scatter plots of the treatment effect estimated by individual studies versus a measure of study size or precision (the "funnel plot"). In this graphical representation, larger and more precise studies are plotted at the top, near the combined effect size, while smaller and less precise studies will show a wider distribution below. If there is no publication bias, the studies would be expected to be symmetrically distributed on both sides of the combined effect size line. In case of publication bias, the funnel plot may be asymmetrical, since the absence of studies would distort the distribution on the scatter plot [ 3 ].

The "trim and fill" method examines the existence of asymmetry in the funnel plot and is recommended as a tool for the assessment of the robustness of the results of meta-analyses (sensitivity analysis). The method consists of a rank-based data augmentation procedure that statistically estimates the number and location of missing studies. The main application of this method is to adjust for the possible effects of missing studies [ 11 ]. If the conclusion of the meta-analysis remains unchanged following adjustment for the publication bias, the results can be considered reasonably robust, excluding publication bias.

In the present study, funnel plot asymmetry was used to assess meta-analyses containing 10 or more trials reporting a binary outcome. In each review, the meta-analysis with the greatest number of trials was selected for evaluation. Therefore those meta-analyses may not necessarily represent the primary outcomes for the review. If two or more meta-analyses had a similar number of trials, the one listed first in the review was selected [ 12 ]. To achieve consistency across meta-analyses, endpoints were re-coded if necessary so that an effect size below 1 always indicated a beneficial effect of the intervention.

The following data were extracted from each meta-analysis: study name, subgroups within the study, data on effect size, year of publication of the most recent trial included and assessment of publication bias. After extracting the data and compiling a database, statistical analysis was performed using the Comprehensive Meta-Analysis ® software program (version 2.2.034, USA, 2006). The database was checked twice for the presence of inconsistencies. For each meta-analysis, a funnel plot, the estimated number of missing studies and the adjusted combined effect size were obtained using the "trim and fill method". This procedure was applied to both sides of each funnel plot. The model of effect (fixed-effects or random-effects) used was the same as that applied in the primary meta-analysis. Results of "trim and fill method" were validated by using Stata (Stata Corporation, College Station, TX, USA)

In Cochrane systematic reviews, the Mantel-Haenszel method is usually applied to estimate relative risk by using the Review Manager (RevMan) software program [ 8 ]. In order to comply with the requirements of the Comprehensive Meta-Analyses ® software program used in the present study, values obtained for the adjustment for publication bias were converted through the inverse variance method. In case of possible publication bias, the meta-analyses were submitted to a more detailed assessment.

The 9 th issue of the WHO Reproductive Health Library included 105 Cochrane Systematic Reviews. A total of 21 systematic reviews contained meta-analyses with 10 or more trials reporting a binary outcome. In this set of meta-analyses, the number of trials comprising each one ranged from 10 to 83 (median = 13), totaling 379 trials, whose results have been summarized. None of the reviews had reported any evaluation of publication bias or funnel plot asymmetry. During data extraction, the endpoints of one meta-analysis were re-coded to guarantee consistency across meta-analyses [ 13 ].

The main characteristics of the selected meta-analyses and the adjustments performed are shown in Table 1 . According to the "trim and fill method", some degree of asymmetry in funnel plots was observed in 18 of the 21 meta-analyses evaluated (85.7%). In meta-analyses in which asymmetric funnel plot was found, the estimated number of missing studies ranged from 1 to 18 (median = 3). All summary plots of meta-analyses, together with their respective "trimmed and filled" funnel plots, are shown in Appendix 1 [See Additional file 1 ].

In 18 of the 21 meta-analyses evaluated, the assessment procedure and the adjustments for publication bias had no effect on the conclusions (Table 1 ); however, in the remaining three (14.3%, 3:21), the conclusion cannot be considered robust due to a possible publication bias [ 14 – 16 ]. These results were obtained applying the same model of effect used in the primary meta-analysis (fixed-effects or random-effects), but were confirmed using the alternative method (data not shown).

Figures 1 and 2 show filled funnel plots with summary effect estimates before and after adjustment for the publication bias. Both meta-analyses included a similar number of trials and both presented asymmetric funnel plots of identified studies (open circles). After adjustment for the publication bias, the estimated number of missing studies entered into the funnel plot (filled circles) was moderate, 7 and 5 respectively. The summary of estimates obtained before (open diamond) and after the adjustment (filled diamond) indicates that, if really such a number of missing studies exists, the impact on the conclusion may not be negligible. Figure 3 also presents a filled funnel plot and, although the estimates suggest that only one study may be missing, the practical impact on the conclusion should be considered. Table 2 summarizes the practical impact of the adjustment for publication bias on conclusions in these three meta-analyses.

figure 1

A filled funnel plot of the antibiotics prophylaxis regimens for cesarean section data, with filled circles denoting the imputed missing studies. The bottom diamonds show summary effect estimates before (open) and after (filled) publication bias adjustment.

figure 2

A filled funnel plot of the continuous support for women during childbirth data, with filled circles denoting the imputed missing studies. The bottom diamonds show summary effect estimates before (open) and after (filled) publication bias adjustment.

figure 3

A filled funnel plot of the interventions for emergency contraception data, with filled circles denoting the imputed missing studies. The bottom diamonds show summary effect estimates before (open) and after (filled) publication bias adjustment.

The main results of this analytic survey suggest that some degree of asymmetry in funnel plots is a common finding in reproductive health meta-analyses. In about 14% of the selected meta-analyses, this type of asymmetry qualitatively impacted conclusions. However, the present study also has some limitations that have to be taken into consideration for the subject to be evaluated within context. If Cochrane meta-analyses differ in their methods from other meta-analyses or if meta-analyses with fewer than ten trials differ from those with more than ten, a selection bias may exist. Despite this possible selection bias, this source of meta-analyses was chosen because of the consistency in the methodology, associated with a widely recognized standard of quality. The number of trials was established as an inclusion criterion since this factor results in the best performance of the "trim and fill" method. Moreover, although other methods are available for the assessment of asymmetry in funnel plots, no consensus has been reached with respect to the superiority of any single method. Therefore, any method used for detecting asymmetry in funnel plots should be considered indirect and exploratory. In this study, we used the "trim and fill" method as an instrument for sensitivity analysis. Our principal concern was not the exact number of missing studies; we were, in fact, interested in how the effect size estimates would be qualitatively changed by the presence of an underlying publication bias.

Asymmetrical in funnel plots are linked to publication bias although there are other sources of asymmetry that have to be considered, including other dissemination biases, differences in the quality of smaller studies, the existence of true heterogeneity, and chance. Asymmetry in funnel plots may be an indicator that a more detailed investigation should be carried out on the presence of heterogeneity, such as sensitivity analysis.

Nevertheless, none of the meta-analyses evaluated in this study reported the use of sensitivity analysis. In the latest version of the software program generally used by Cochrane reviewers (RevMan, version 4.2.8, The Nordic Cochrane Centre, Rigshospitalet 2003), no formal test for the assessment of funnel plots is available. This software program permits the visual subjective interpretation of funnel plots, but such an approach has been shown to include a significant inter-observer variability [ 17 ]. These limitations may have restricted the use of this method in this selected sample of meta-analyses.

On the other hand, there is some concern regarding the evolution of effect size over time and the impact of including "old" trials in meta-analyses. In the past, it was possible that the determinants of data suppression and the intensity of publication bias were different when compared to those in current use. We observed that in one-third of meta-analyses, the most recent trials had been published prior to 1997 (8:21) and in more than half (11:21), the most recent trials had been published prior to 2000. In fact, it is unclear whether the date of publication would have any impact on meta-analyses in reproductive health, but caution should be taken when summarizing effects across older trials.

Rather than reviewing the conclusions of meta-analyses, the aim of this study was to provide evidence of publication bias and its consequences in selected reproductive health meta-analyses. Consequently, three examples of possible publication bias were identified. Following adjustment, a reduction in the discrepancy between the conclusions of larger trials and the conclusions of meta-analyses was seen (data not shown). In all three meta-analyses whose conclusions were not considered robust, their suggested post-adjustment conclusion was in agreement with those of the respective larger trials included. In the systematic review assessing interventions for emergency contraception [ 16 ], the meta-analysts adopted a different form of sensitivity analysis, by reanalyzing the data, and including only the trials that had adequate allocation concealment. Their findings were similar to the adjustment for funnel plot asymmetry or publication bias.

Implementing Best Practices in Reproductive Health. [ http://www.ibpinitiative.org/ ]

Villar J, Gülmezoglu AM, Khanna J, Carroli G, Hofmeyr GJ, Schulz K, Lumbiganon P: Evidence-based reproductive health in developing countries: The WHO Reproductive Health Library. 1999, Geneva, The World Health Organization, 2

Google Scholar  

Rothstein HR, Sutton AJ, Borenstein M: Publication Bias in Meta-analysis: prevention, assessment and adjustment. 2005, Chichester, UK:John Wiley & Sons, Ltd

Chapter   Google Scholar  

LeLorier J, Grégoire G, Benhaddad A, Lapierre J, Dederian F: Discrepancies between meta-analyses and subsequent large randomized, controlled trials. N Engl J Med. 1997, 337: 536-42. 10.1056/NEJM199708213370806.

Article   CAS   PubMed   Google Scholar  

Oxman AD, Cook DJ, Guyatt GH: User's guide to the medical literature. JAMA. 1994, 272: 1367-71. 10.1001/jama.272.17.1367.

Maher D, Cook DJ, Eastwood S: Improving the quality of reports of meta-analysis of randomized controlled trials: the QUOROM statement. Lancet. 1999, 354: 1896-900. 10.1016/S0140-6736(99)04149-5.

Article   Google Scholar  

Stroup DF, Berlin JA, Morton SC: Meta-analysis of observational studies in epidemiology: a proposal for reporting. JAMA. 2000, 283: 2008-12. 10.1001/jama.283.15.2008.

Higgins JPT, Green S, editors: Cochrane Handbook for Systematic Reviews of Interventions 4.2.5 [updated May 2005]. The Cochrane Library. 2005, Chichester, UK: John Wiley & Sons, Ltd, 3

Tallon D, Schneider M, Egger M: Quality of systematic reviews published in high impact general and specialist journals. Proceedings of 2nd symposium on systematic reviews: beyond the basics. 1999, Oxford

WHO: The Reproductive Health Library. 2006, Oxford: Update Software Ltd, [ http://www.rhlibrary.com ]

Duval S: The trim and fill Method. Publication Bias in Meta-analysis: prevention, assessment and adjustment. Edited by: Rothstein HR, Sutton AJ, Borenstein M. 2005, Chichester, UK: John Wiley & Sons, Ltd

Sutton AJ, Duval SJ, Tweedie RL, Abrams KR, Jones DR: Empirical assessment of effect of publication bias on meta-analysis. BMJ. 2000, 320: 1574-77. 10.1136/bmj.320.7249.1574.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Martin-Hirsch P, Jarvis G, Kitchener H, Lilford R: Collection devices for obtaining cervical cytology samples (Cochrane Review). The Reproductive Health Library. 2006, Oxford: Update Software Ltd, (Reprinted from The Cochrane Library, Issue 1, 2006. Chichester, UK: John Wiley & Sons, Ltd.), [ http://www.rhlibrary.com ]9

Hopkins L, Smaill F: Antibiotic prophylaxis regimens and drugs for cesarean section (Cochrane Review). The Reproductive Health Library. 2006, Oxford: Update Software Ltd, (Reprinted from The Cochrane Library, Issue 1, 2006. Chichester, UK: John Wiley & Sons, Ltd.), [ http://www.rhlibrary.com ]9

Hodnett ED, Gates S, Hofmeyr GJ, Sakala C: Continuous support for women during childbirth (Cochrane Review). The Reproductive Health Library. 2006, Oxford: Update Software Ltd, (Reprinted from The Cochrane Library, Issue 1, 2006. Chichester, UK: John Wiley & Sons, Ltd.), [ http://www.rhlibrary.com ]9

Cheng L, Gülmezoglu AM, Van Oel CJ, Piaggio G, Ezcurra E, Van Look PFA: Interventions for emergency contraception (Cochrane Review). The Reproductive Health Library. 2006, Oxford: Update Software Ltd, (Reprinted from The Cochrane Library, Issue 1, 2006. Chichester, UK: John Wiley & Sons, Ltd.), [ http://www.rhlibrary.com ]9

Villar J, Piaggio G, Carroli G, Donner A: Factors affecting the comparability of meta-analyses and largest trials results in perinatology. J Clin Epidemiol. 1997, 50: 997-1002. 10.1016/S0895-4356(97)00148-0.

Hofmeyr GJ: Amnioinfusion for meconium-stained liquor in labour (Cochrane Review). The Reproductive Health Library. 2006, Oxford: Update Software Ltd, (Reprinted from The Cochrane Library, Issue 1, 2006. Chichester, UK: John Wiley & Sons, Ltd.), [ http://www.rhlibrary.com ]9

Smaill F, Hofmeyr GJ: Antibiotic prophylaxis for cesarean section (Cochrane Review). The Reproductive Health Library. 2006, Oxford: Update Software Ltd, (Reprinted from The Cochrane Library, Issue 1, 2006. Chichester, UK: John Wiley & Sons, Ltd.), [ http://www.rhlibrary.com ]9

Smaill F: Antibiotics for asymptomatic bacteriuria in pregnancy (Cochrane Review). The Reproductive Health Library. 2006, Oxford: Update Software Ltd, (Reprinted from The Cochrane Library, Issue 1, 2006. Chichester, UK: John Wiley & Sons, Ltd.), [ http://www.rhlibrary.com ]9

Kenyon S, Boulvain M, Neilson J: Antibiotics for preterm rupture of membranes (Cochrane Review). The Reproductive Health Library. 2006, Oxford: Update Software Ltd, (Reprinted from The Cochrane Library, Issue 1, 2006. Chichester, UK: John Wiley & Sons, Ltd.), [ http://www.rhlibrary.com ]9

McDonald H, Brocklehurst P, Parsons J: Antibiotics for treating bacterial vaginosis in pregnancy (Cochrane Review). The Reproductive Health Library. 2006, Oxford: Update Software Ltd, (Reprinted from The Cochrane Library, Issue 1, 2006. Chichester, UK: John Wiley & Sons, Ltd.), [ http://www.rhlibrary.com ]9

Abalos E, Duley L, Steyn DW, Henderson-Smart DJ: Antihypertensive drug therapy for mild to moderate hypertension during pregnancy (Cochrane Review). The Reproductive Health Library. 2006, Oxford: Update Software Ltd, (Reprinted from The Cochrane Library, Issue 1, 2006. Chichester, UK: John Wiley & Sons, Ltd.), [ http://www.rhlibrary.com ]9

Knight M, Duley L, Henderson-Smart DJ, King JF: Antiplatelet agents for preventing and treating pre-eclampsia (Cochrane Review). The Reproductive Health Library. 2006, Oxford: Update Software Ltd, (Reprinted from The Cochrane Library, Issue 1, 2006. Chichester, UK: John Wiley & Sons, Ltd.), [ http://www.rhlibrary.com ]9

King JF, Flenady VJ, Papatsonis DNM, Dekker GA, Carbonne B: Calcium channel blockers for inhibiting preterm labour (Cochrane Review). The Reproductive Health Library. 2006, Oxford: Update Software Ltd, (Reprinted from The Cochrane Library, Issue 1, 2006. Chichester, UK: John Wiley & Sons, Ltd.), [ http://www.rhlibrary.com ]9

Atallah AN, Hofmeyr GJ, Duley L: Calcium supplementation during pregnancy for preventing hypertensive disorders and related problems (Cochrane Review). The Reproductive Health Library. 2006, Oxford: Update Software Ltd, (Reprinted from The Cochrane Library, Issue 1, 2006. Chichester, UK: John Wiley & Sons, Ltd.), [ http://www.rhlibrary.com ]9

Thacker SB, Stroup D, Chang M: Continuous electronic heart rate monitoring for fetal assessment during labor (Cochrane Review). The Reproductive Health Library. 2006, Oxford: Update Software Ltd, (Reprinted from The Cochrane Library, Issue 1, 2006. Chichester, UK: John Wiley & Sons, Ltd.), [ http://www.rhlibrary.com ]9

Crowley P: Interventions for preventing or improving the outcome of delivery at or beyond term (Cochrane Review). The Reproductive Health Library. 2006, Oxford: Update Software Ltd, (Reprinted from The Cochrane Library, Issue 1, 2006. Chichester, UK: John Wiley & Sons, Ltd.), [ http://www.rhlibrary.com ]9

Oates-Whitehead RM, Haas DM, Carrier JAK: Progestogen for preventing miscarriage (Cochrane Review). The Reproductive Health Library. 2006, Oxford: Update Software Ltd, (Reprinted from The Cochrane Library, Issue 1, 2006. Chichester, UK: John Wiley & Sons, Ltd.), [ http://www.rhlibrary.com ]9

Hodnett ED, Fredericks S: Support during pregnancy for women at increased risk of low birthweight babies (Cochrane Review). The Reproductive Health Library. 2006, Oxford: Update Software Ltd, (Reprinted from The Cochrane Library, Issue 1, 2006. Chichester, UK: John Wiley & Sons, Ltd.), [ http://www.rhlibrary.com ]9

Carroli G, Bergel E: Umbilical vein injection for management of retained placenta (Cochrane Review). The Reproductive Health Library. 2006, Oxford: Update Software Ltd, (Reprinted from The Cochrane Library, Issue 1, 2006. Chichester, UK: John Wiley & Sons, Ltd.), [ http://www.rhlibrary.com ]9

Johanson RB, Menon V: Vacuum extraction versus forceps for assisted vaginal delivery (Cochrane Review). The Reproductive Health Library. 2006, Oxford: Update Software Ltd, (Reprinted from The Cochrane Library, Issue 1, 2006. Chichester, UK: John Wiley & Sons, Ltd.), [ http://www.rhlibrary.com ]9

Hofmeyr GJ, Gülmezoglu AM: Vaginal misoprostol for cervical ripening and induction of labour (Cochrane Review). The Reproductive Health Library. 2006, Oxford: Update Software Ltd, (Reprinted from The Cochrane Library, Issue 1, 2006. Chichester, UK: John Wiley & Sons, Ltd.), [ http://www.rhlibrary.com ]9

Download references

Acknowledgements

The authors are pleased to thank Steven Tarlow for helping in data analysis.

Author information

Authors and affiliations.

Department of Obstetrics and Gynecology, School of Medical Sciences, University of Campinas, Brazil

João P Souza & José G Cecatti

Clinical Epidemiology Collaborative Group, Women's Integrated Health Care Center, University of Campinas, Brazil

João P Souza & Cynthia Pileggi

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to João P Souza .

Additional information

Competing interests.

The author(s) declare that they have no competing interests.

Authors' contributions

JPS participated in all the steps of the project, including the project development, data extraction, data analysis and writing the final report. CP participated in the project development and data extraction. JGC participated in the project development, data analysis and writing the final report. All authors provided suggestions for the manuscript, read it carefully, agreed on its content and approved the final version.

Electronic supplementary material

Additional file 1: appendix 1. filled funnel plots of 21 reproductive health meta-analyses. (doc 768 kb), authors’ original submitted files for images.

Below are the links to the authors’ original submitted files for images.

Authors’ original file for figure 1

Authors’ original file for figure 2, authors’ original file for figure 3, rights and permissions.

This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License ( http://creativecommons.org/licenses/by/2.0 ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Reprints and permissions

About this article

Cite this article.

Souza, J.P., Pileggi, C. & Cecatti, J.G. Assessment of funnel plot asymmetry and publication bias in reproductive health meta-analyses: an analytic survey. Reprod Health 4 , 3 (2007). https://doi.org/10.1186/1742-4755-4-3

Download citation

Received : 22 February 2007

Accepted : 16 April 2007

Published : 16 April 2007

DOI : https://doi.org/10.1186/1742-4755-4-3

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

  • Publication Bias

Reproductive Health

  • Emergency Contraception
  • Cochrane Systematic Review
  • Analytic Survey

ISSN: 1742-4755

cochrane systematic review funnel plot

  • - Google Chrome

Intended for healthcare professionals

  • Access provided by Google Indexer
  • My email alerts
  • BMA member login
  • Username * Password * Forgot your log in details? Need to activate BMA Member Log In Log in via OpenAthens Log in via your institution

Home

Search form

  • Advanced search
  • Search responses
  • Search blogs
  • Bias in meta-analysis...

Bias in meta-analysis detected by a simple, graphical test

  • Related content
  • Peer review
  • Matthias Egger , reader in social medicine and epidemiology ( m.egger{at}bristol.ac.uk ) a ,
  • George Davey Smith , professor of clinical epidemiology a ,
  • Martin Schneider , research associate b ,
  • Christoph Minder , head, medical statistics unit b
  • a Department of Social Medicine, University of Bristol, Bristol BS8 2PR
  • b Department of Social and Preventive Medicine, University of Berne, CH-3012 Berne, Switzerland
  • Correspondence to: Dr Egger
  • Accepted 26 August 1997

Objective: Funnel plots (plots of effect estimates against sample size) may be useful to detect bias in meta-analyses that were later contradicted by large trials. We examined whether a simple test of asymmetry of funnel plots predicts discordance of results when meta-analyses are compared to large trials, and we assessed the prevalence of bias in published meta-analyses.

Design: Medline search to identify pairs consisting of a meta-analysis and a single large trial (concordance of results was assumed if effects were in the same direction and the meta-analytic estimate was within 30% of the trial); analysis of funnel plots from 37 meta-analyses identified from a hand search of four leading general medicine journals 1993-6 and 38 meta-analyses from the second 1996 issue of the Cochrane Database of Systematic Reviews .

Main outcome measure: Degree of funnel plot asymmetry as measured by the intercept from regression of standard normal deviates against precision.

Results: In the eight pairs of meta-analysis and large trial that were identified (five from cardiovascular medicine, one from diabetic medicine, one from geriatric medicine, one from perinatal medicine) there were four concordant and four discordant pairs. In all cases discordance was due to meta-analyses showing larger effects. Funnel plot asymmetry was present in three out of four discordant pairs but in none of concordant pairs. In 14 (38%) journal meta-analyses and 5 (13%) Cochrane reviews, funnel plot asymmetry indicated that there was bias.

Conclusions: A simple analysis of funnel plots provides a useful test for the likely presence of bias in meta-analyses, but as the capacity to detect bias will be limited when meta-analyses are based on a limited number of small trials the results from such analyses should be treated with considerable caution.

Key messages

Systematic reviews of randomised trials are the best strategy for appraising evidence; however, the findings of some meta-analyses were later contradicted by large trials

Funnel plots, plots of the trials' effect estimates against sample size, are skewed and asymmetrical in the presence of publication bias and other biases

Funnel plot asymmetry, measured by regression analysis, predicts discordance of results when meta-analyses are compared with single large trials

Funnel plot asymmetry was found in 38% of meta-analyses published in leading general medicine journals and in 13% of reviews from the Cochrane Database of Systematic Reviews

Critical examination of systematic reviews for publication and related biases should be considered a routine procedure

Introduction

Systematic reviews of the best available evidence regarding the benefits and risks of medical interventions can inform decision making in clinical practice and public health. 1 2 Such reviews are, whenever possible, based on meta-analysis: “a statistical analysis which combines or integrates the results of several independent clinical trials considered by the analyst to be ‘combinable.’” 3 However, the findings of some meta-analyses have later been contradicted by large randomised controlled trials. 4 Such discrepancies have brought discredit on a technique that has been controversial since the outset. 5 The appearance of misleading meta-analysis is not surprising considering the existence of publication bias and the many other biases that may be introduced in the process of locating, selecting, and combining studies. 6 7 8 9

Funnel plots, plots of the trials' effect estimates against sample size, may be useful to assess the validity of meta-analyses. 4 10 The funnel plot is based on the fact that precision in estimating the underlying treatment effect will increase as the sample size of component studies increases. Results from small studies will scatter widely at the bottom of the graph, with the spread narrowing among larger studies. In the absence of bias the plot will resemble a symmetrical inverted funnel. Conversely, if there is bias, funnel plots will often be skewed and asymmetrical.

The value of the funnel plot has not been systematically examined, and symmetry (or asymmetry) has generally been defined informally, through visual examination. Unsurprisingly, funnel plots have been interpreted differently by different observers. 11 We measured funnel plot asymmetry numerically and examined the extent to which such asymmetry predicts discordance of results when meta-analyses are compared to single large trials of the same issue. We used the same method to assess the prevalence of funnel plot asymmetry, and thus of possible bias, among meta-analyses published in leading general medicine journals and meta-analyses disseminated electronically by the Cochrane Collaboration.

Measures of funnel plot asymmetry

We used a linear regression approach to measure funnel plot asymmetry on the natural logarithm scale of the odds ratio. This corresponds to a regression analysis of Galbraith's radial plot, 12 although in the present context the regression is not constrained to run through the origin. The standard normal deviate (SND), defined as the odds ratio divided by its standard error, is regressed against the estimate's precision, the latter being defined as the inverse of the standard error (regression equation: SND= a + b xprecision). As precision depends largely on sample size, small trials will be close to zero on the × axis. Small trials may produce an odds ratio that differs from unity, but because the standard error will be large, the resulting standard normal deviate will again be close to zero. Small trials will thus be close to zero on both axes—that is, close to the origin. Conversely, large studies will produce precise estimates and, if the treatment is effective, also produce large standard normal deviates. The points from a homogeneous set of trials, not distorted by selection bias, will thus scatter about a line that runs through the origin at standard normal deviate zero ( a =0), with the slope b indicating the size and direction of effect. 12 This situation corresponds to a symmetrical funnel plot.

If there is asymmetry, with smaller studies showing effects that differ systematically from larger studies, the regression line will not run through the origin. The intercept a provides a measure of asymmetry—the larger its deviation from zero the more pronounced the asymmetry. If the smaller studies show big protective effects, they will force the regression line below the origin on the logarithmic scale. Negative values will therefore indicate that smaller studies show more pronounced beneficial effects than larger studies. In some situations (for example, if there are several small trials but only one larger study) power is gained by weighting the analysis by the inverse of the variance of the effect estimate. We performed both weighted and unweighted analyses and used the output from the analysis yielding the intercept with the larger deviation from zero.

In contrast to the overall test of heterogeneity, the test for funnel plot asymmetry assesses a specific type of heterogeneity and provides a more powerful test in this situation. However, any analysis of heterogeneity depends on the number of trials included in a meta-analysis, which is generally small, and this limits the statistical power of the test. We therefore based evidence of asymmetry on P<0.1, and we present intercepts with 90% confidence intervals. The same significance level has been used in previous analyses of heterogeneity in meta-analysis. 13 14

Identification of meta-analyses and matching large randomised trials

A Medline search (Knight Ridder Information Services, Berne, Switzerland) covering the period January 1985 to April 1996 was performed in April 1996 to identify published meta-analyses. For this purpose the word “meta-analysis” was entered in a free text search. The articles identified included all those indexed with the Medical Subject Heading (MeSH) keyword “meta-analysis,” which was introduced in 1989, and articles without the keyword which carried the word meta-analysis in their title or abstract. Results were tabulated by source of publication, and the items published in journals which yielded 30 or more hits were examined further. Meta-analyses of controlled trials combining at least five trials with binary endpoints were identified.

Large scale randomised controlled trials of the same interventions which had been published after the meta-analyses were identified by a Medline search using appropriate keywords. Large trials had to provide an effect estimate with a precision of at least 5. For example, a trial among patients with heart failure in which mortality in the control group at three months is 5% 15 and in which mortality is reduced to 3% among treated patients will need to randomise 2800 patients to measure this effect with a precision of 5 and about 12 000 patients for a precision of 10. Also, the effect estimate from the large trials had to be of equal or greater precision than the meta-analysis. We scrutinised potential matching pairs of meta-analyses and large trials with regard to study participants, interventions, end points and lengths of follow up. In some cases a further Medline search was performed to identify a meta-analysis published in any journal indexed in Medline which would be more suitable for comparison with the large trial.

Some meta-analyses were published several years before the corresponding large trial. In these cases we examined whether the shape of the funnel plot changed when the meta-analysis was updated with trials published in the intervening period.

Concordance and discordance of results

Comparison of results from meta-analyses and large trials required expressing results on a common scale. Odds ratios were used for this purpose. The meta-analysis and the large trial were considered concordant when effects were in the same direction and the estimates from the meta-analysis were within 30% of the estimate of the single trial. A difference of 30% was proposed by Villar et al to denote high similarity between the results from meta-analyses and large trials. 11

SAS version 6.11 software package (Statistical Analysis System, Cary, NC) was used for statistical analysis.

Frequency of asymmetry in funnel plots

We performed a hand search of four leading general medicine journals, Annals of Internal Medicine , BMJ , JAMA , and Lancet , from 1993 to 1996 and examined the second 1996 issue of the Cochrane Database of Systematic Reviews 16 to identify meta-analyses of controlled trials. Analyses that were based on at least five trials with categorical end points were examined further. For each intervention and comparison, the outcome measure which was reported in the largest number of trials was selected. To obtain consistency across reviews, end points were recorded if necessary so that the direction of effect for the expected beneficial outcome was in the same direction. For example, in a review of trials of nicotine patches in smoking cessation, continued smoking rather than quitting was considered to be the outcome, so that an odds ratio above unity indicates an adverse effect.

We identified 38 Cochrane reviews and 37 journal meta-analyses. All references of meta-analyses and trials included are available from the authors on request.

Eight pairs consisting of a meta-analysis and a large trial were identified (table 1 ). 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 Five were from cardiovascular medicine, one from diabetic medicine, one from geriatric medicine, and one from perinatal medicine. Effect estimates from meta-analyses had an average precision of 7.9 compared with 14.4 for large trials. There were four concordant pairs 15 17 18 19 20 21 22 26 and four discordant pairs 14 23 24 25 27 28 29 30 (fig 1 ). In all cases discordance was a consequence of the meta-analyses showing more beneficial effects than the large trials. Three out of four discordant meta-analyses showed significant (P<0.1) funnel plot asymmetry; funnel plots from concordant pairs showed no significant asymmetry ( 2 , table 2 ).

Characteristics of nine pairs of meta-analyses and corresponding large trials

  • View inline

Results from four concordant and four discordant pairs of meta-analysis and large scale randomised controlled trial

  • Download figure
  • Open in new tab
  • Download powerpoint

Funnel plots and single large trials. Points indicate odds ratios from trials included in meta-analysis; squares with horizontal lines show odds ratio from large trial with 95% confidence interval. See table 1 for abbreviations of trial names

Analysis of funnel plot asymmetry

Additional trials were identified for three meta-analyses published several years earlier than the large trial. 26 27 29 These were extracted from more recent meta-analyses. 4 31 32 When the meta-analysis of trials of intravenous magnesium in myocardial infarction was updated with five additional trials the intercept indicated even greater asymmetry (−1.36 (90% confidence interval −2.06 to −0.66), P=0.005). When 13 additional trials were added to the analysis of trials of angiotensin converting enzyme inhibitors in heart failure the plot remained symmetrical (intercept 0.07 (−0.53 to 0.67), P=0.85). When the analysis of aspirin for the prevention of pre-eclampsia was updated with nine additional trials, the funnel plot became asymmetrical (intercept −1.49 (−2.20 to −0.79), P=0.003) (fig 3 ).

Funnel plot of trials of low dose aspirin in the prevention of pre-eclampsia. Trials included in Imperiale and Stollenwerk's 1991 meta-analysis (closed circles), 29 trials published in subsequent years (1990 to 1993, open circles) and the large 1994 CLASP (collaborative low-dose aspirin study in pregnancy) trial (square with horizontal line indicating 95% confidence interval) 30

Figure 4 shows the distribution of regression intercepts from 38 Cochrane reviews and 37 journal meta-analyses. In the absence of bias, random fluctuations should produce a symmetrical distribution of intercepts around a central value of zero, with an equal number of positive and negative values. This is not what was observed. Distributions were shifted towards negative values, with a mean of −0.24 (−0.65 to 0.17) for Cochrane reviews and −1.00 (−1.50 to −0.49) for journal meta-analyses There were 24 negative and 14 positive intercepts among Cochrane reviews (P=0.10 by sign test) and 26 negative and 11 positive intercepts among journal meta-analyses (P=0.007 by sign test). In five (13%) Cochrane reviews and 14 (38%) journal meta-analyses there was evidence of significant (P<0.1) asymmetry.

Distribution of intercepts from regression analysis of funnel plot asymmetry for 38 meta-analyses from the Cochrane Database of Systematic Reviews , 1996 (upper panel) and 37 meta-analyses published in Annals of Internal Medicine , BMJ , JAMA , and Lancet 1993 through 1996 (lower panel)

The selective publication of positive findings from randomised controlled trials is an important concern in meta-analytic reviews of the literature. 9 If the literature is more likely to contain trials showing beneficial effects of treatments, and if equally valid trials showing no effect remain unpublished, how can systematic reviews of this literature serve as an objective guide to decision making in clinical practice and health policy? The potentially serious consequences of such publication bias have been realised for some time, and there have been repeated calls for worldwide registration of clinical trials at inception. 1 4 33 34 35 Although registration of trials and creation of a database holding the results of both published and unpublished trials would solve the problem, it is unlikely that this will be widely instituted in the foreseeable future.

Critical examination for the presence of publication and related biases must therefore become an essential part of meta-analytic studies and systematic reviews. The findings presented here indicate that a simple graphical and statistical method is useful for this purpose. When testing this method on pairs consisting of meta-analyses and single large trials of the same intervention, we found asymmetry in funnel plots in three out of four pairs with discordant results. The fourth was based on only six trials, and asymmetry emerged when it was updated with further studies.

Sources of funnel plot asymmetry

Publication bias has long been associated with funnel plot asymmetry. 10 Among published studies, however, the probability of identifying relevant trials for meta-analysis is also influenced by their results. English language bias—the preferential publication of “negative” findings in journals published in languages other than English—makes the location and inclusion of such studies less likely. 8 As a consequence of citation bias, “negative” studies are quoted less frequently and are therefore more likely to be missed in the search for relevant trials. 7 36 Results of “positive” trials are sometimes reported more than once, increasing the probability that they will be located for meta-analysis (multiple publication bias). 37 These biases are likely to affect smaller studies to a greater degree than large trials.

Another source of asymmetry arises from differences in methodological quality. Smaller studies are, on average, conducted and analysed with less methodological rigour than larger studies. Trials of lower quality also tend to show the larger effects. 38 39 40 The degree of symmetry found in a funnel plot may depend on the statistic used to measure effect. Odds ratios overestimate the relative reduction, or increase, in risk if the event rate is high. 41 This can lead to funnel plot asymmetry if the smaller trials were consistently conducted in patients at higher risk. Similarly, if events accrue at a constant rate, relative risks will move towards unity with increasing length of follow up. In large trials, follow up is often longer than in small studies. Finally, an asymmetrical funnel plot may arise by chance.

The trials displayed in a funnel plot may not estimate the same underlying effect of the intervention, and such heterogeneity between results may lead to asymmetry in funnel plots. For example, if a combined outcome is considered then substantial benefit may be seen only in patients at high risk for the component of the combined outcome that is affected by the intervention. 42 A cholesterol lowering drug that reduces mortality from coronary heart disease will have a greater effect on all cause mortality in high risk patients with established cardiovascular disease than in asymptomatic patients with isolated hypercholesterolaemia. This is because a consistent relative reduction in mortality from coronary heart disease will translate into a greater relative reduction in all cause mortality in high risk patients, in whom a greater proportion of all deaths will be from coronary heart disease. This will produce asymmetry in funnel plots if the smaller trials were performed in high risk patients.

Small trials are generally conducted before larger trials are established. In the intervening years, control treatments may have improved or changed in a way that could reduce the efficacy of the experimental treatment. Such a mechanism has been proposed as an explanation for the discrepant results obtained in clinical trials of the effect of magnesium infusion in myocardial infarction, 43 although this interpretation is not supported by the data from clinical trials. 44 Finally, some interventions may have been implemented less thoroughly in larger trials, thus explaining the more positive results in smaller trials. This could have occurred in one of the interventions considered in our comparison of meta-analysis and single large trials, inpatient geriatric consultation. 14

Very different mechanisms can thus lead to asymmetry in funnel plots, as summarised in the box . It is important to note, however, that this will always be associated with a biased overall estimate of effect when studies are combined in a meta-analysis. The more pronounced the asymmetry, the more likely it is that the amount of bias will be substantial. The exception to this rule arises when asymmetry is produced by chance alone.

Sources of asymmetry in funnel plots

Publication bias

Location biases:

English language bias

Citation bias

Multiple publication bias

True heterogeneity

Size of effect differs according to study size:

Intensity of intervention

Differences in underlying risk

Data irregularities

Poor methodological design of small studies

Inadequate analysis

Artefactual

Choice of effect measure

How frequent is bias in meta-analysis?

Several studies have recently tried to evaluate the validity of meta-analysis. Villar et al analysed 38 meta-analyses from the pregnancy and childbirth module of the 1993 Cochrane database by comparing the results from the largest trial with the remaining smaller studies. 45 On the basis of the direction of estimates of treatment effects, they concluded that 80% of meta-analyses were in total or partial agreement with the results from the larger “gold standard” trial. In a similar study, Cappelleri et al analysed 79 meta-analyses and concluded that there was agreement between smaller trials and large trials in over 80%. 13 In both these analyses, however, the precision of the large trials was low in a sizeable proportion of comparisons. The larger trials in fact often provided an estimate of lower precision than the meta-analysis of the smaller studies. In this situation, concordance between the two could simply be due to the fact that estimates with large, overlapping confidence intervals are unlikely to be classified as discordant. 46

We thought that stringent criteria were necessary for identifying single large trials that could sensibly be used to assess the results from meta-analyses of smaller trials. As a result, the large trials used in our analysis on average provided an estimate of considerably greater precision that the corresponding meta-analyses. Despite an extensive literature search, we identified only eight such pairs. The matched pair approach may therefore not be suitable assessing the frequency of misleading meta-analysis. However, our results indicate that an asymmetrical funnel plot makes bias likely. The prevalence of funnel plot asymmetry may thus provide a useful proxy measure to examine the prevalence of biased analyses in the literature. Our findings indicate that bias may be present in a small proportion of meta-analyses published in the Cochrane Database of Systematic Reviews . Bias may be considerably more prevalent, however, among meta-analyses published in leading general medicine journals. Whether such bias is likely to affect the conclusions of a systematic review or meta-analysis must be carefully assessed for each case.

Begg and Mazumbar proposed a rank correlation test to measure asymmetry in funnel plots. 47 The method is based on the degree of association between the size of effect estimates and their variances. If publication bias is present, the smaller studies will show the larger effects. A positive correlation between effect size and variance emerges in this situation because the variance of the estimates from smaller studies will also be large. When we applied their test to the eight meta-analyses, it indicated significant (P<0.1) asymmetry for only one meta-analysis (inpatient geriatric consultation 14 ). This indicates that the linear regression approach may be more powerful than the rank correlation test.

Conclusions

In the absence of large, conclusive trials for most medical interventions, systematic reviews based on randomised controlled trials are clearly the best strategy for appraising the evidence. Selection bias and other biases pose a serious threat to the validity of this approach, however, and care must be taken to avoid meta-analysis becoming discredited. The technique discussed here should contribute to this goal, providing a reproducible measure for the likely presence, or apparent absence, of such biases. It is easily calculated and provides summary statistics that can be reported when space limitations do not permit the display of funnel plots. Though more methodological research is required, the critical examination for the presence of publication and related biases should be considered a routine procedure. The capacity to unearth such bias will, however, be limited when meta-analyses are based exclusively on small trials. There is no statistical solution in this situation, and the results from such analyses should therefore be treated with caution.

Acknowledgments

We are grateful to Andreas Stuck and Gilbert Ramirez for kindly providing additional data.

Funding: Swiss National Science Foundation (grants 3200-045597 and 3233-038803).

Conflict of interest: None.

  • Chalmers I ,
  • Dickersin K ,
  • Chalmers TC
  • Davey Smith G
  • Easterbrook PJ ,
  • Berlin JA ,
  • Gopalan R ,
  • Matthews DR
  • G⊘tzsche PC
  • Zellweger-Zähner T ,
  • Schneider M ,
  • Lengeler C ,
  • Pillemer DB
  • Piaggio G ,
  • Carroli G ,
  • Galbraith R
  • Cappelleri JC ,
  • Ioannidis JPA ,
  • Schmid CH ,
  • de Ferranti SD ,
  • Chalmers TC ,
  • Wieland GD ,
  • Rubenstein LZ
  • SOLVD investigators.
  • 16. ↵ The Cochrane Database of Systematic Reviews . Oxford : Cochrane Collaboration , 1996 .
  • Collins R ,
  • Furberg C ,
  • Stampfer MJ ,
  • Goldhaber SZ ,
  • Gruppo Italiano per lo Studio della Streptochinasi nell'Infarto Miocardico (GISSI).
  • ISIS-1 Collaborative Group.
  • Diabetes Control and Complications Trial ResearchGroup.
  • Reuben DB ,
  • Wolde-Tsadik G ,
  • Ershoff DH ,
  • Fishman LK ,
  • Ambrosini VL ,
  • MacMahon S ,
  • Gruppo Italiano per lo Studio della Streptochinasinell'Infarto Miocardico (GISSI).
  • Mulrow CD ,
  • Mulrow JP ,
  • Aguilar C ,
  • ISIS-4 Collaborative Group.
  • Imperiale TF ,
  • Stollenwerk Petrullis A
  • CLASP Collaborative Group.
  • for the Collaborative Group on ACEInhibitor Trials
  • Keirse MJNC ,
  • Renfrew MJ ,
  • Savulescu J ,
  • Schulz KF ,
  • Davey Smith G ,
  • Phillips AN
  • Baxter GF ,
  • Sumeray MS ,
  • Flournoy N ,

cochrane systematic review funnel plot

  • Advanced search

CMAJ

Advanced Search

The appropriateness of asymmetry tests for publication bias in meta-analyses: a large survey

  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • Figures & Tables
  • © 2007 Canadian Medical Association or its licensors

Background: Statistical tests for funnel-plot asymmetry are common in meta-analyses. Inappropriate application can generate misleading inferences about publication bias. We aimed to measure, in a survey of meta-analyses, how frequently the application of these tests would be not meaningful or inappropriate.

Methods: We evaluated all meta-analyses of binary outcomes with é 3 studies in the Cochrane Database of Systematic Reviews (2003, issue 2). A separate, restricted analysis was confined to the largest meta-analysis in each of the review articles. In each meta-analysis, we assessed whether criteria to apply asymmetry tests were met: no significant heterogeneity, I2 < 50%, é 10 studies (with statistically significant results in at least 1) and ratio of the maximal to minimal variance across studies > 4. We performed a correlation and 2 regression asymmetry tests and evaluated their concordance. Finally, we sampled 60 meta-analyses from print journals in 2005 that cited use of the standard regression test.

Results: A total of 366 of 6873 (5%) and 98 of 846 meta-analyses (12%) in the wider and restricted Cochrane data set, respectively, would have qualified for use of asymmetry tests. Asymmetry test results were significant in 7%–18% of the meta-analyses. Concordance between the 3 tests was modest (estimated k 0.33–0.66). Of the 60 journal meta-analyses, 7 (12%) would qualify for asymmetry tests; all 11 claims for identification of publication bias were made in the face of large and significant heterogeneity.

Interpretation: Statistical conditions for employing asymmetry tests for publication bias are absent from most meta-analyses; yet, in medical journals these tests are performed often and interpreted erroneously.

Publication bias, the selective publication of studies based on whether results are “positive” or not, is a major threat to the validity of clinical research. 1 – 4 This bias can distort the totality of the available evidence on a research question, which leads to misleading inferences in reviews and meta-analyses. Without up-front study registration, however, this bias is difficult to identify after the fact. 5 Many tests have therefore been proposed to help identify publication bias. 6

The most common approaches try to investigate the presence of asymmetry in (inverted) funnel plots. 7 – 10 A funnel plot shows the relation between study effect size and its precision. The premise is that small studies are more likely to remain unpublished if their results are nonsignificant or unfavourable, whereas larger studies get published regardless. This leads to funnel-plot asymmetry. Although visual inspection of funnel plots is unreliable, 11 , 12 statistical tests can be used to quantify the asymmetry. 7 – 10 These tests have become popular: one relevant article 8 has been cited more than 1000 times.

The limitations of these tests have been documented for some time. Begg and Mazumdar 7 mentioned in 1994 that the false-positive rates of their popular rank-correlation test were too low. In 2000, Sterne and colleagues 13 showed in a simulation study that the regression method described by Egger and associates 8 was more powerful than the rank correlation test, although the power of either method was low for meta-analyses of 10 or fewer trials. False-positive results were found to be a major concern in the presence of heterogeneity. 13 , 14 To reduce the problem, a modified regression test was developed, 10 and several other tests proposed. 6 , 15 Because they differ in their assumptions and statistical properties, discordant results can be expected with different tests.

There are situations when the use of these tests is clearly inappropriate, and others where their use is futile or meaningless. Application of these tests with few studies is not wrong, but has low statistical power. Application in the presence of heterogeneity is more clearly inappropriate, and may lead to false-positive claims for publication bias. 14 , 16 , 17 When all available studies are equally large (i.e., have similar precision), the tests are not meaningful. Finally, it makes no sense to evaluate whether studies with significant results are preferentially published when none with significant results have been published.

Despite these limitations, these tests figure prominently in the medical literature. It would be useful to estimate how often these tests are appropriately or meaningfully applied. We therefore appraised almost 7000 meta-analyses in the Cochrane Database of Systematic Reviews to discover the extent to which tests of funnel-plot asymmetry would be inappropriate or nonconcordant. We also examined the appropriateness of the application of asymmetry testing in meta-analyses recently published in print journals.

We used issue 2, 2003, of the Cochrane Database of Systematic Reviews ( n = 1669 reviews). We imported into Stata software all meta-analyses that had binary outcomes and numerical 2 × 2 table information available ( n = 12 709). 18 We did not consider studies where no patients in either arm of the study had an event, or all patients in both arms had an event; this eliminated 906 meta-analyses. Zero counts in one arm only were handled in the calculations via the addition of 0.5 to all data cells, which allowed an odds ratio to be calculated without distorting the data appreciably. Meta-analysis data sets were further scrutinized for similarity. When numbers of studies, patients and events were all the same and summary results were identical (to 7 digits of accuracy), the meta-analyses were considered to contain duplicate data sets and only one of them was retained: similarity checks eliminated 761 duplicate meta-analyses. We also excluded meta-analyses where only 2 studies were available ( n = 4169), which makes correlation and regression diagnostics impossible to calculate. Thus, our analysis of the wider Cochrane data set included data from 6873 meta-analyses.

The data sets of these meta-analyses are not necessarily independent. Within the same systematic review, different outcomes, contrasts and analyses may be correlated. To minimize correlation, we created a separate, more restricted data set for which we selected one meta-analysis, the one with the largest number of studies, per systematic review. When the largest number of studies was equal in 2 or more of the meta-analyses, we chose the one with the largest number of subjects; if that number was also equal, we chose the one with the largest number of events. The problem of inappropriateness of the asymmetry tests due to limited number of studies was thereby minimized in this analysis of the restricted Cochrane data set of data from 846 meta-analyses.

For each eligible meta-analysis, we evaluated 4 aspects that bear on whether applying an asymmetry test may be meaningful or appropriate. Statistical significance was tested with the χ 2 -based Q statistic and considered significant for p < 0.10 (2-tailed); 19 the extent of between-study heterogeneity was measured with the I 2 statistic and considered large for values of 50% or more. 20 The number of included studies was noted; 10 or more was considered sufficient. To see if the difference in precision of the largest and the smallest study was sufficiently large (ratio of extreme values of variances > 4), we noted the ratio of the maximal versus minimal variance (the square of the standard error of estimates) across the included studies. Finally, we recorded whether at least one study had found formally statistically significant results ( p < 0.05).

Some debate about the extent to which criteria need be fulfilled for asymmetry tests to be meaningful or appropriate is unavoidable. The thresholds listed above are not very demanding, based on the properties of the tests. Results of analyses with alternative, even more lenient criteria are illustrated in Venn diagrams of the 4 overlapping criteria.

The odds ratio was used as the metric of choice for all the meta-analyses. We documented the degree of overlap of the criteria described above and the number of meta-analyses that would qualify, based not only upon each criterion but also on combinations thereof.

We evaluated each meta-analysis by means of 3 asymmetry tests: the 2 most popular tests in the literature (the Begg–Mazumdar τ rank-correlation coefficient, 7 and the standard regression test of the standardized effect size [i.e., the natural logarithm of the odds ratio divided by its standard error] against its precision [the inverse of the standard error] 8 ) and a new variant, a modified version of the regression test, which has a lower false-positive rate. 10 For all tests, statistical significance was claimed for p < 0.10 (2-tailed). 7 , 8 , 10 We estimated inferences on the basis of these 3 tests in the entire data sets and in the subsets of meta-analyses fulfilling the appropriateness criteria already described. Pairwise concordance between the 3 tests was assessed with the κ statistic. 21

The Cochrane Handbook for Systematic Reviews of Interventions 16 has taken a critical stance to the use of these tests. RevMan, the Cochrane Library meta-analysis software, does not include any options for running them, and their use in the Cochrane Library is limited. 22 We therefore used a sample of meta-analyses in printed journals to examine whether these tests are used inappropriately in practice. We examined papers published in 2005 that cited the most common reference for the standard regression test, 8 the asymmetry test most commonly used in the current literature. We screened citations in sequential order (as indexed in the Science Citation Index) until we identified 60 meta-analyses in which asymmetry testing had been employed. The 60 meta-analyses examined were within 24 published articles. Although we focused on the standard regression test, 8 we also recorded results from the other 2 tests whenever such data were reported. We examined whether these 60 meta-analyses fulfilled the criteria that we set, what they found, and how they interpreted the application of the test.

  • View inline
  • Download powerpoint

In terms of fulfillment of criteria, the most common feasibility problem we encountered in both of our Cochrane data-set analyses was too low a number of studies, with three-quarters or more of the meta-analyses examining fewer than 10 studies ( Table 1 ). Lack of significant studies was also a major issue: of the wider and restricted data sets, about half and a third of the meta-analyses, respectively, included no studies with statistically significant results; a fifth/ a quarter had significant or large between-study heterogeneity; and nearly a quarter/ a fifth had a ratio of extreme values of variances of 4 or greater. Only 366 (5%) of the meta-analyses in the wider Cochrane data set and 98 (12%) of those in the restricted Cochrane data set fulfilled all 4 of the original criteria ( Fig. 1 , left).

  • Download figure
  • Open in new tab

Fig. 1: Venn diagrams showing the overlap of the subsets of meta-analyses according to our chosen criteria (diagrams to the left: ≥ 1 study with statistically significant results; ≥ 10 studies in the meta-analysis; I 2 < 50% with nonsignificant Q; ratio of extreme study variances > 4). For comparison, results when a set of very lenient criteria (right: ≥ 1 significant study; ≥ 5 studies; I 2 < 50% regardless of Q; extreme study variances > 2) is used are also depicted. Each set of criteria is likewise shown for our wider data set of meta-analyses (upper diagrams: n = 6873) and for the restricted data set of 1 meta-analysis per systematic review (lower diagrams: n = 846). Shading indicates categories in which substantially more studies met criteria.

Results of the 3 tests showed statistically significant asymmetry in few meta-analyses ( Table 2 ); overall, in the 2 data sets, rates of significant signals (i.e., statistically significant results) varied between 7% and 18%. They tended to be smallest for the correlation test and highest for the unmodified standard regression test, but did not much differ between the 2 data sets. When the data sets were split according to whether meta-analyses met the criteria for applying asymmetry tests or not, significant signals were more prevalent in the meta-analyses that fulfilled the criteria than in those that did not. Nevertheless, even in the former group, the rates of signals varied from 14% to 24%.

The 3 asymmetry tests had modest concordance across the entire data sets ( Table 2 , Fig. 2 ); results were largely similar across the wider and restricted Cochrane data sets. Overall, 3% and 4% of the meta-analyses, respectively, gave a significant signal with all 3 tests. In 19% and 22% of the meta-analyses, a result from at least 1 of the 3 tests was significant. Estimated κ values fell generally below 0.5 (range 0.33–0.45) for the concordance of the correlation test with either of the regression diagnostics, and were somewhat higher (0.64–0.66) for concordance between the unmodified and modified regression diagnostics. When analyses were limited to meta-analyses that fulfilled the criteria for asymmetry tests, concordance slightly improved between the correlation and the regression diagnostics (estimated κ 0.39–0.60) and worsened slightly between the unmodified and modified regression diagnostics (estimated κ 0.57–0.59).

Fig. 2: Venn diagrams disclosing modest concordance in the application of the 3 funnel-plot asymmetry tests to statistically significant results in the wider data set of 6873 meta-analyses (left) and in the restricted data set of 846 meta-analyses (right). Data inside the circles refer only to meta-analyses with significant results with the corresponding test ( p < 0.10).

Of the 60 meta-analyses that stated their use of the regression test within the 24 print articles, use of the test was meaningful or appropriate in 7 of the meta-analyses (12%, 95% confidence interval 5%–23%). Of the 24 articles, 6 had at least one meta-analysis where use of the test was appropriate. Twenty-six meta-analyses had significant heterogeneity (all with I 2 > 50%), and another 4 had I 2 > 50% without statistically significant heterogeneity. Twenty-six meta-analyses were of fewer than 10 studies. Eighteen meta-analyses included no significant studies; 3 had ratios of extreme variances ≤ 4. Four of the 24 articles also reported rank correlation test results (with similar inferences). Another cited the regression test when what had actually been performed were rank correlation tests. One other article apparently used a regression test based on sample size, a different test than the one that was cited.

All 24 articles claimed that the tests were done to estimate publication bias, with a single exception: an article that clarified that the authors tested for “small-study bias, of which publication bias is one potential cause.” Eleven meta-analyses (18%) claimed that there was evidence for publication bias, whereas the other 49 stated that they found no such evidence. All meta-analyses that claimed to have detected publication bias were found to have between-study heterogeneity that was large and statistically significant .

  • Interpretation

In most meta-analyses, the application of funnel-plot asymmetry tests to detect publication bias is inappropriate or not meaningful. We found a major problem to be lack of a sufficient number of studies; lack of studies with significant results and the presence of heterogeneity were also common issues. In a smaller proportion of meta-analyses, differences in the magnitude of the smallest versus the largest studies were negligible.

When each of 3 asymmetry (“publication bias”) tests were applied, we found a minority of the examined meta-analyses to have a positive signal. About a fifth of the meta-analyses gave a signal with any of the 3 tests; 3%–4% gave consistent signals for asymmetry with all diagnostics. In the absence of a criterion standard about the presence of publication bias, it is impossible to decide whether these figures were low because the tests we examined were underpowered or because publication bias is uncommon. Moreover, concordance among the 3 tests was modest. Automatic and undocumented use of these tests may lead to unreliable inferences.

A survey of 60 recently published meta-analyses from 24 published reports that had cited use of the standard regression test 8 revealed that most had used the test inappropriately. With one exception, all these articles misleadingly equated the results of these tests with the presence or absence of publication bias, ignoring numerous other causes that may underlie differences between small and larger studies. 8 Moreover, all signals for publication bias occurred in meta-analyses with large, significant between-study heterogeneity. It is also disquieting that 82% of the meta-analyses were assumed to have no publication bias simply because of a “negative” asymmetry test result.

When these diagnostics give significant signals, this does not necessarily mean that publication bias is present. This applies even when the meta-analyses fulfill all of the 4 eligibility criteria that we considered. In the absence of a prospective registry of studies, publication bias cannot be proven or excluded, because a criterion standard is lacking.

The 4 criteria we used are merely technical and conceptual prerequisites. Even if statistical prerequisites are met, the conceptual assumptions may sometimes not hold. Very large sample size, 11 increased attention to the research question and heightened interest in contradicting previous publications with extreme opposite results may contribute as much or more than statistical significance to dictating publication in selected cases or in entire scientific fields. 23

We used the Cochrane Database of Systematic Reviews because it is by far the largest compilation of meta-analyses. The composition of this database may differ from that of the totality of meta-analyses published. 22 , 24 , 25 Despite some uneven emphasis on specific diseases in the evolving Cochrane Database of Systematic Reviews, 26 this database is likely to be less selective compared with the meta-analyses that appear in the medical journal literature. Meta-analyses published in printed medical journals are larger but also more likely to have large heterogeneity, because they also include a greater share of nonrandomized studies. In the journal literature, the percentage of meta-analyses where asymmetry tests are applied inappropriately is therefore also very high.

There can be some subjectivity about thresholds for a definition of when a statistical test is meaningful or appropriate. Our criteria tended toward the lenient; use of even more lenient criteria would increase the proportion of appropriateness, but not to very high percentages ( Fig. 1 ).

Publication bias is compounded by additional biases that pertain to selective outcome reporting 27 , 28 and “significance-chasing” 29 in the data published. It would be misleading to claim that all these problems can be addressed with asymmetry tests. Occasionally, in a meta-analysis of many studies, the retrieval of unpublished data may “correct” a funnel-plot asymmetry. 30 However, we should caution that, when unpublished data exist, only a portion might possibly be retrievable; so, it is unknown what would happen if data from all studies could be retrieved. Whenever both unpublished and published information is available, the results of these 2 types of evidence should be compared. Nevertheless, as has been stressed repeatedly, prospective registration of clinical studies and of their analyses and outcomes 5 , 31 may be the only means to properly address publication bias.

In conclusion, meta-analysts should refrain from inappropriate or unmeaningful application of funnel-plot asymmetry tests. Readers should not be misled that publication bias has been documented or excluded according to inappropriate use or interpretation of funnel plots.

This article has been peer reviewed.

Contributors: John Ioannidis originated the study concept and wrote the protocol and manuscript, with input and critical revisions by Thomas Trikalinos. John Ioannidis evaluated the meta-analyses published in printed journals; Thomas Trikalinos performed all the statistical analyses. Both authors interpreted the data from their analyses, and approved the final version of the article for publication.

Competing interests: None declared.

  • 1. ↵ Dickersin K, Min YI. Publication bias: the problem that won't go away. Ann N Y Acad Sci 1993 ; 703 : 135 -46. OpenUrl CrossRef PubMed
  • 2. Easterbrook PJ, Berlin JA, Gopalan R, et al. Publication bias in clinical research. Lancet 1991 ; 337 : 867 -72. OpenUrl CrossRef PubMed
  • 3. Ioannidis JP. Effect of the statistical significance of results on the time to completion and publication of randomized efficacy trials. JAMA 1998 ; 279 : 281 -6. OpenUrl CrossRef PubMed
  • 4. ↵ Stern JM, Simes RJ. Publication bias: evidence of delayed publication in a cohort study of clinical research projects. BMJ 1997 ; 315 : 640 -5. OpenUrl Abstract / FREE Full Text
  • 5. ↵ DeAngelis CD, Drazen JM, Frizelle FA, et al; International Committee of Medical Journal Editors. Clinical trial registration: a statement from the International Committee of Medical Journal Editors. JAMA 2004 ; 292 : 1363 -4. OpenUrl CrossRef PubMed
  • 6. ↵ Rothstein HR, Sutton AJ, Borenstein M, editors. Publication bias in meta-analysis. Prevention, assessment and adjustments. Sussex: John Wiley and Sons; 2005.
  • 7. ↵ Begg CB, Mazumdar M. Operating characteristics of a rank correlation test for publication bias. Biometrics 1994 ; 50 : 1088 -101. OpenUrl CrossRef PubMed
  • 8. ↵ Egger M, Davey Smith G, Schneider M, et al. Bias in meta-analysis detected by a simple, graphical test. BMJ 1997 ; 315 : 629 -34. OpenUrl Abstract / FREE Full Text
  • 9. Sterne JA, Egger M. Funnel plots for detecting bias in meta-analysis: guidelines on choice of axis. J Clin Epidemiol 2001 ; 54 : 1046 -55. OpenUrl CrossRef PubMed
  • 10. ↵ Harbord RM, Egger M, Sterne JA. A modified test for small-study effects in meta-analyses of controlled trials with binary endpoints. Stat Med 2006 ; 25 : 3443 -57. OpenUrl CrossRef PubMed
  • 11. ↵ Tang JL, Liu JL. Misleading funnel plot for detection of bias in meta-analysis. J Clin Epidemiol 2000 ; 53 : 477 -84. OpenUrl CrossRef PubMed
  • 12. ↵ Terrin N, Schmid CH, Lau J. In an empirical evaluation of the funnel plot, researchers could not visually identify publication bias. J Clin Epidemiol 2005 ; 58 : 894 -901. OpenUrl CrossRef PubMed
  • 13. ↵ Sterne JA, Gavaghan D, Egger M. Publication and related bias in meta-analysis: power of statistical tests and prevalence in the literature. J Clin Epidemiol 2000 ; 53 : 1119 -29. OpenUrl CrossRef PubMed
  • 14. ↵ Terrin N, Schmid CH, Lau J, et al. Adjusting for publication bias in the presence of heterogeneity. Stat Med 2003 ; 22 : 2113 -26. OpenUrl CrossRef PubMed
  • 15. ↵ Macaskill P, Walter SD, Irwig L. A comparison of methods to detect publication bias in meta-analysis. Stat Med 2001 ; 20 : 641 -54. OpenUrl CrossRef PubMed
  • 16. ↵ The Cochrane Collaboration. Cochrane handbook for systematic reviews of interventions. Available: www.cochrane.org/resources/handbook (accessed 2007 Feb 8).
  • 17. ↵ Ioannidis JP. Differentiating biases from genuine heterogeneity: distinguishing artifactual from substantive effects. In: Rothstein HR, Sutton AJ, Borenstein M, editors. Publication bias in meta-analysis: prevention, assessment and adjustments. Sussex: John Wiley and Sons; 2005. p 287-302.
  • 18. ↵ Ioannidis JP, Trikalinos TA, Zintzaras E. Extreme between-study homogeneity in meta-analyses could offer useful insights. J Clin Epidemiol 2006 ; 59 : 1023 -32. OpenUrl CrossRef PubMed
  • 19. ↵ Lau J, Ioannidis JP, Schmid CH. Quantitative synthesis in systematic reviews. Ann Intern Med 1997 ; 127 : 820 -6. OpenUrl CrossRef PubMed
  • 20. ↵ Higgins JP, Thompson SG. Quantifying heterogeneity in a meta-analysis. Stat Med 2002 ; 21 : 1539 -58. OpenUrl CrossRef PubMed
  • 21. ↵ Cohen J. A coefficient of agreement for nominal scales. Educ Psychol Meas 1960 ; 20 : 37 -46. OpenUrl PubMed
  • 22. ↵ Palma S, Delgado-Rodriguez M. Assessment of publication bias in meta-analyses of cardiovascular diseases. J Epidemiol Community Health 2005 ; 59 : 864 -9. OpenUrl Abstract / FREE Full Text
  • 23. ↵ Ioannidis JP, Trikalinos TA. Early extreme contradictory estimates may appear in published research: the Proteus phenomenon in molecular genetics research and randomized trials. J Clin Epidemiol 2005 ; 58 : 543 -9. OpenUrl CrossRef PubMed
  • 24. ↵ Jadad AR, Cook DJ, Jones A, et al. Methodology and reports of systematic reviews and meta-analyses: a comparison of Cochrane reviews with articles published in paper-based journals. JAMA 1998 ; 280 : 278 -80. OpenUrl CrossRef PubMed
  • 25. ↵ Shea B, Moher D, Graham I, et al. A comparison of the quality of Cochrane reviews and systematic reviews published in paper-based journals. Eval Health Prof 2002 ; 25 : 116 -29. OpenUrl Abstract / FREE Full Text
  • 26. ↵ Mallett S, Clarke M. How many Cochrane reviews are needed to cover existing evidence on the effects of health care interventions? ACP J Club 2003;139:A11.
  • 27. ↵ Chan AW, Altman DG. Identifying outcome reporting bias in randomised trials on PubMed: review of publications and survey of authors. BMJ 2005 ; 330 : 753 . OpenUrl Abstract / FREE Full Text
  • 28. ↵ Chan AW, Krleza-Jeric K, Schmid I, et al. Outcome reporting bias in randomized trials funded by the Canadian Institutes of Health Research. CMAJ 2004 ; 171 : 735 -40. OpenUrl Abstract / FREE Full Text
  • 29. ↵ Ioannidis JP. Why most published research findings are false. PLoS Med 2005;2:e124.
  • 30. ↵ Kyzas PA, Loizou KT, Ioannidis JP. Selective reporting biases in cancer prognostic factor studies. J Natl Cancer Inst 2005 ; 97 : 1043 -55. OpenUrl Abstract / FREE Full Text
  • 31. ↵ Sim I, Detmer DE. Beyond trial registration: a global trial bank for clinical trial reporting. PLoS Med 2005;2:e365.

In this issue

  • Table of Contents
  • Index by author
  • Canadian Adverse Reaction Newsletter (p 1229 - 1236)

Article tools

Thank you for your interest in spreading the word on CMAJ.

NOTE: We only request your email address so that the person you are recommending the page to knows that you wanted them to see it, and that it is not junk mail. We do not capture any email address.

Citation Manager Formats

  • EndNote (tagged)
  • EndNote 8 (xml)
  • RefWorks Tagged
  • Ref Manager

Digg logo

  • Tweet Widget
  • Facebook Like

Jump to section

Related articles.

  • Highlights of this issue
  • Dans ce numéro
  • Google Scholar

Cited By...

  • Efficacy of pharmacological and non-pharmacological therapies on pain intensity and disability for plantar fasciitis: a systematic review and meta-analysis
  • Social media use and health risk behaviours in young people: systematic review and meta-analysis
  • Aspetar clinical practice guideline on rehabilitation after anterior cruciate ligament reconstruction
  • Paediatric traumatic brain injury as a risk factor for psychosis and psychotic symptoms: a systematic review and meta-analysis
  • Preoperative oral fluoroquinolone antibiotics in elective colorectal surgery to prevent surgical site infections: a systematic review and meta-analysis
  • Functional MRS studies of GABA and Glutamate/Glx - a systematic review and meta-analysis
  • Non-pharmacological interventions for the prevention of type 2 diabetes in low-income and middle-income countries: a systematic review of randomised controlled trials
  • Assessing treatment effects and publication bias across different specialties in medicine: a meta-epidemiological study
  • Effectiveness of early versus delayed rehabilitation following rotator cuff repair: systematic review and meta-analyses
  • Effectiveness of exercises by telerehabilitation on pain, physical function and quality of life in people with physical disabilities: a systematic review of randomised controlled trials with GRADE recommendations
  • Mental Health of Clinical Staff Working in High-Risk Epidemic and Pandemic Health Emergencies: A Rapid Review of the Evidence and Meta-Analysis
  • Influence of Chinese eye exercises on myopia control in an East Asian population: a meta-analysis
  • Helicobacter pylori infection and micronutrient deficiency in pregnant women: a systematic review and meta-analysis
  • Association between the ACE I/D gene polymorphism and progressive renal failure in autosomal dominant polycystic kidney disease: A meta-analysis
  • The Prognostic Role of Late Gadolinium Enhancement in Aortic Stenosis: A Systematic Review and Meta-Analysis
  • How strong is the evidence that conservative treatment reduces pain and improves function in individuals with patellar tendinopathy? A systematic review of randomised controlled trials including GRADE recommendations
  • Estimation of fibrosis progression rates for chronic hepatitis C: a systematic review and meta-analysis update
  • Risk of myocardial infarction among people living with HIV: an updated systematic review and meta-analysis
  • Effect of PPARGC1A Gly428Ser (rs8192678) polymorphism on athletic sports performance: A meta-analysis
  • Catalogue of bias: publication bias
  • Marital status and risk of cardiovascular diseases: a systematic review and meta-analysis
  • Pre-symptomatic targeted treatment of patent ductus arteriosus in preterm newborns: A systematic review and meta-analysis
  • Effectiveness of evidence-based treatments of fetal alcohol spectrum disorders in children and adolescents: a systematic review protocol
  • Long-Term Risk of Myocardial Infarction Compared to Recurrent Stroke After Transient Ischemic Attack and Ischemic Stroke: Systematic Review and Meta-Analysis
  • Preterm Delivery and Future Risk of Maternal Cardiovascular Disease: A Systematic Review and Meta-Analysis
  • Effectiveness of early compared with conservative rehabilitation for patients having rotator cuff repair surgery: an overview of systematic reviews
  • Efficacy and adverse effects of collagenase use in the treatment of Dupuytrens disease: a meta-analysis
  • Systematic review and meta-analysis of factors that help or hinder treatment decision-making capacity in psychosis
  • Comparison of efficacy of pharmacological treatments for chronic idiopathic constipation: a systematic review and network meta-analysis
  • Prognostic impact of vascular invasion in differentiated thyroid carcinoma: a systematic review and meta-analysis
  • Risk factors for mortality in adult patients with sickle cell disease: a meta-analysis of studies in North America and Europe
  • Non-pharmacological treatment of depression: a systematic review and evidence map
  • Lactobacillus probiotics in the prevention of diarrhea associated with Clostridium difficile: a systematic review and Bayesian hierarchical meta-analysis
  • Shared treatment decision-making and empowerment-related outcomes in psychosis: systematic review and meta-analysis
  • Transcatheter Aortic Valve Implantation With or Without Preimplantation Balloon Aortic Valvuloplasty: A Systematic Review and Meta-Analysis
  • Review and meta-analysis of ab-interno trabeculectomy outcomes
  • Radiofrequency Ablation Versus Antiarrhythmic Drug Therapy for Atrial Fibrillation: Meta-Analysis of Quality of Life, Morbidity, and Mortality
  • Long-term Glycemic Variability and Risk of Adverse Outcomes: A Systematic Review and Meta-analysis
  • Bone mineral density and fracture risk with long-term use of inhaled corticosteroids in patients with asthma: systematic review and meta-analysis
  • Cardiac resynchronisation therapy is not associated with a reduction in mortality or heart failure hospitalisation in patients with non-left bundle branch block QRS morphology: meta-analysis of randomised controlled trials
  • Habitual chocolate consumption and risk of cardiovascular disease among healthy men and women
  • Mindfulness-based stress reduction (MBSR) as sole intervention for non-somatisation chronic non-cancer pain (CNCP): protocol for a systematic review and meta-analysis of randomised controlled trials
  • Access and Non-Access Site Bleeding After Percutaneous Coronary Intervention and Risk of Subsequent Mortality and Major Adverse Cardiovascular Events: Systematic Review and Meta-Analysis
  • Efficacy of Antiplatelet Therapy in Secondary Prevention Following Lacunar Stroke: Pooled Analysis of Randomized Trials
  • Blood Transfusion After Percutaneous Coronary Intervention and Risk of Subsequent Adverse Outcomes: A Systematic Review and Meta-Analysis
  • Efficacy of psychosocial interventions for psychological and pregnancy outcomes in infertile women and men: a systematic review and meta-analysis
  • Major bleeding after percutaneous coronary intervention and risk of subsequent mortality: a systematic review and meta-analysis
  • Renal denervation and blood pressure reduction in resistant hypertension: a systematic review and meta-analysis
  • Publication bias in the social sciences: Unlocking the file drawer
  • Increased fruit and vegetable intake has no discernible effect on weight loss: a systematic review and meta-analysis
  • Bone Mineral Density and Incidence of Stroke: European Prospective Investigation Into Cancer-Norfolk Population-Based Study, Systematic Review, and Meta-Analysis
  • Adiponectin and Risk of Stroke: Prospective Study and Meta-analysis
  • Effect of systemic steroids on post-tonsillectomy bleeding and reinterventions: systematic review and meta-analysis of randomised controlled trials
  • Reply to Francis: Cumulative power calculations are faulty when based on observed power and a small sample of studies
  • Effects of chocolate, cocoa, and flavan-3-ols on cardiovascular health: a systematic review and meta-analysis of randomized trials
  • Author's reply:
  • Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials
  • Carbohydrate Ingestion during Endurance Exercise Improves Performance in Adults
  • Atrial fibrillation and incidence of dementia: A systematic review and meta-analysis
  • Effectiveness of disease-management programs for improving diabetes care: a meta-analysis
  • Obstetric outcomes after treatment of periodontal disease during pregnancy: systematic review and meta-analysis
  • Influence of Psychological Stress on Upper Respiratory Infection--A Meta-Analysis of Prospective Studies
  • The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration
  • Novel methods to deal with publication biases: secondary analysis of antidepressant trials in the FDA trial registry database and related journal publications
  • Is disclosure of potential conflicts of interest in medicine and public health sufficient to increase transparency and decrease corruption?: How could disclosure of interests work better in medicine, epidemiology and public health?
  • A case of apples and pears?
  • Effects of prenatal multimicronutrient supplementation on pregnancy outcomes: a meta-analysis
  • Intensive insulin therapy and mortality among critically ill patients: a meta-analysis including NICE-SUGAR study data
  • Aldehyde Dehydrogenase 2 and Head and Neck Cancer: A Meta-analysis Implementing a Mendelian Randomization Approach
  • Use of Doppler ultrasonography to predict pre-eclampsia
  • Fluoroquinolones compared with {beta}-lactam antibiotics for the treatment of acute bacterial sinusitis: a meta-analysis of randomized controlled trials
  • Static magnets for reducing pain: systematic review and meta-analysis of randomized trials
  • Erythropoietin-receptor agonists in critically ill patients: a meta-analysis of randomized controlled trials

More in this TOC Section

  • Nunavimmi puvakkut kaggutimik aanniaqarniq: Qanuilirqitaa? Lung cancer in Nunavik: How are we doing? A retrospective matched cohort study
  • Differential effect of ticagrelor versus clopidogrel by homocysteine levels on risk of recurrent stroke: a post hoc analysis of the CHANCE-2 trial
  • One-time screening for abdominal aortic aneurysm in Ontario, Canada: a model-based cost-utility analysis

Similar Articles

Collections.

  • Research methods & statistics

Get the latest content from Canada's leading medical journal, free to all physicians. Click to sign up for CMAJ or JAMC email alerts.

U.S. flag

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

  • Publications
  • Account settings
  • Advanced Search
  • Journal List
  • Reprod Health

Logo of reprohlth

Assessment of funnel plot asymmetry and publication bias in reproductive health meta-analyses: an analytic survey

João p souza.

1 Department of Obstetrics and Gynecology, School of Medical Sciences, University of Campinas, Brazil

2 Clinical Epidemiology Collaborative Group, Women's Integrated Health Care Center, University of Campinas, Brazil

Cynthia Pileggi

José g cecatti.

This is an Open Access article distributed under the terms of the Creative Commons Attribution License ( http://creativecommons.org/licenses/by/2.0 ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Associated Data

Despite efforts to assure high methodological standards, systematic reviews may be affected by publication bias. The objective of this study was to evaluate the occurrence of publication bias in a collection of high quality systematic reviews on reproductive health.

Systematic reviews included in the Reproductive Health Library (RHL), issue No 9, were assessed. Funnel plot was used to assess meta-analyses containing 10 or more trials reporting a binary outcome. A funnel plot, the estimated number of missing studies and the adjusted combined effect size were obtained using the "trim and fill method". Meta-analyses results that were not considered to be robust due to a possible publication bias were submitted to a more detailed assessment.

A total of 21 systematic reviews were assessed. The number of trials comprising each one ranged from 10 to 83 (median = 13), totaling 379 trials, whose results have been summarized. None of the reviews had reported any evaluation of publication bias or funnel plot asymmetry. Some degree of asymmetry in funnel plots was observed in 18 of the 21 meta-analyses evaluated (85.7%), with the estimated number of missing studies ranging from 1 to 18 (median = 3). Only for three meta-analyses, the conclusion could not be considered robust due to a possible publication bias.

Asymmetry is a frequent finding in funnel plots of meta-analyses in reproductive health, but according to the present evaluation, less than 15% of meta-analyses report conclusions that would not be considered robust. Publication bias and other sources of asymmetry in funnel plots should be systematically addressed by reproductive health meta-analysts. Next amendments in Cochrane systematic reviews should include this type of evaluation. Further studies regarding the evolution of effect size and publication bias over time in systematic reviews in reproductive health are needed.

Implementing best practices is a major goal in health services [ 1 ]. However, the identification of such practices depends on the evaluation and synthesis of a large amount of scientific information. This may be achieved by carrying out systematic reviews and meta-analyses, which have come to represent important sources of evidence-based knowledge for clinicians, policy makers and researchers [ 2 ].

A systematic review is an observational study of the scientific literature based on individual studies. It may contain meta-analyses, which are statistical procedures developed for summarizing effects across individual studies. The ideal meta-analysis should combine data appropriately to produce a more complete and meaningful estimate of the overall effect [ 3 ]. Nevertheless, despite efforts to assure high methodological standards, systematic reviews may be affected by publication bias, one of the major drawbacks of such studies and a threat to their validity. Publication bias occurs whenever the results of a set of published studies differ from the results of all the research performed on a specific topic [ 3 ]. A publication-biased meta-analysis may present an ineffective or unsafe intervention as being effective or safe, or not recommend an effective or safe intervention because the results of some studies already performed are not included. Furthermore, publication bias may be partially responsible for occasional discrepancies between the conclusions of previous meta-analyses and subsequent large multicenter trials [ 4 ].

The assessment of publication bias is a relatively new recommendation but reported in several relevant meta-analyses reporting guidelines [ 5 - 8 ]. However, it has been observed that only few meta-analyses have actually evaluated publication bias (3.2% – 6.5%) [ 9 ]. It is also unclear whether reproductive health meta-analyses are affected by publication bias, since we have been unable to identify any previous reports assessing publication bias and its effects on meta-analyses in reproductive health. The objective of this survey was, therefore, to evaluate the occurrence of publication bias in a collection of high quality systematic reviews on reproductive health.

This is an analytic survey carried out to evaluate the impact of publication bias on the results of meta-analyses of reproductive health interventions. Systematic reviews included in the World Health Organization (WHO) Reproductive Health Library (RHL), issue No 9, were assessed [ 10 ]. The RHL is a WHO instrument for documenting and disseminating best practices in the field. It reproduces the most relevant Cochrane systematic reviews related to reproductive health, adding some practical aspects and pertinent comments for developing country settings, as well as implications for research [ 10 ].

There are several methods of assessing the occurrence of publication bias. A common approach is based on scatter plots of the treatment effect estimated by individual studies versus a measure of study size or precision (the "funnel plot"). In this graphical representation, larger and more precise studies are plotted at the top, near the combined effect size, while smaller and less precise studies will show a wider distribution below. If there is no publication bias, the studies would be expected to be symmetrically distributed on both sides of the combined effect size line. In case of publication bias, the funnel plot may be asymmetrical, since the absence of studies would distort the distribution on the scatter plot [ 3 ].

The "trim and fill" method examines the existence of asymmetry in the funnel plot and is recommended as a tool for the assessment of the robustness of the results of meta-analyses (sensitivity analysis). The method consists of a rank-based data augmentation procedure that statistically estimates the number and location of missing studies. The main application of this method is to adjust for the possible effects of missing studies [ 11 ]. If the conclusion of the meta-analysis remains unchanged following adjustment for the publication bias, the results can be considered reasonably robust, excluding publication bias.

In the present study, funnel plot asymmetry was used to assess meta-analyses containing 10 or more trials reporting a binary outcome. In each review, the meta-analysis with the greatest number of trials was selected for evaluation. Therefore those meta-analyses may not necessarily represent the primary outcomes for the review. If two or more meta-analyses had a similar number of trials, the one listed first in the review was selected [ 12 ]. To achieve consistency across meta-analyses, endpoints were re-coded if necessary so that an effect size below 1 always indicated a beneficial effect of the intervention.

The following data were extracted from each meta-analysis: study name, subgroups within the study, data on effect size, year of publication of the most recent trial included and assessment of publication bias. After extracting the data and compiling a database, statistical analysis was performed using the Comprehensive Meta-Analysis ® software program (version 2.2.034, USA, 2006). The database was checked twice for the presence of inconsistencies. For each meta-analysis, a funnel plot, the estimated number of missing studies and the adjusted combined effect size were obtained using the "trim and fill method". This procedure was applied to both sides of each funnel plot. The model of effect (fixed-effects or random-effects) used was the same as that applied in the primary meta-analysis. Results of "trim and fill method" were validated by using Stata (Stata Corporation, College Station, TX, USA)

In Cochrane systematic reviews, the Mantel-Haenszel method is usually applied to estimate relative risk by using the Review Manager (RevMan) software program [ 8 ]. In order to comply with the requirements of the Comprehensive Meta-Analyses ® software program used in the present study, values obtained for the adjustment for publication bias were converted through the inverse variance method. In case of possible publication bias, the meta-analyses were submitted to a more detailed assessment.

The 9 th issue of the WHO Reproductive Health Library included 105 Cochrane Systematic Reviews. A total of 21 systematic reviews contained meta-analyses with 10 or more trials reporting a binary outcome. In this set of meta-analyses, the number of trials comprising each one ranged from 10 to 83 (median = 13), totaling 379 trials, whose results have been summarized. None of the reviews had reported any evaluation of publication bias or funnel plot asymmetry. During data extraction, the endpoints of one meta-analysis were re-coded to guarantee consistency across meta-analyses [ 13 ].

The main characteristics of the selected meta-analyses and the adjustments performed are shown in Table ​ Table1. 1 . According to the "trim and fill method", some degree of asymmetry in funnel plots was observed in 18 of the 21 meta-analyses evaluated (85.7%). In meta-analyses in which asymmetric funnel plot was found, the estimated number of missing studies ranged from 1 to 18 (median = 3). All summary plots of meta-analyses, together with their respective "trimmed and filled" funnel plots, are shown in Appendix 1 [See Additional file 1 ].

Principal characteristics of selected meta-analyses and the adjustment performed according to the "trim and fill method"

* Conclusion is possibly affected by publication bias

In 18 of the 21 meta-analyses evaluated, the assessment procedure and the adjustments for publication bias had no effect on the conclusions (Table ​ (Table1); 1 ); however, in the remaining three (14.3%, 3:21), the conclusion cannot be considered robust due to a possible publication bias [ 14 - 16 ]. These results were obtained applying the same model of effect used in the primary meta-analysis (fixed-effects or random-effects), but were confirmed using the alternative method (data not shown).

Figures ​ Figures1 1 and ​ and2 2 show filled funnel plots with summary effect estimates before and after adjustment for the publication bias. Both meta-analyses included a similar number of trials and both presented asymmetric funnel plots of identified studies (open circles). After adjustment for the publication bias, the estimated number of missing studies entered into the funnel plot (filled circles) was moderate, 7 and 5 respectively. The summary of estimates obtained before (open diamond) and after the adjustment (filled diamond) indicates that, if really such a number of missing studies exists, the impact on the conclusion may not be negligible. Figure ​ Figure3 3 also presents a filled funnel plot and, although the estimates suggest that only one study may be missing, the practical impact on the conclusion should be considered. Table ​ Table2 2 summarizes the practical impact of the adjustment for publication bias on conclusions in these three meta-analyses.

An external file that holds a picture, illustration, etc.
Object name is 1742-4755-4-3-1.jpg

A filled funnel plot of the antibiotics prophylaxis regimens for cesarean section data, with filled circles denoting the imputed missing studies. The bottom diamonds show summary effect estimates before (open) and after (filled) publication bias adjustment.

An external file that holds a picture, illustration, etc.
Object name is 1742-4755-4-3-2.jpg

A filled funnel plot of the continuous support for women during childbirth data, with filled circles denoting the imputed missing studies. The bottom diamonds show summary effect estimates before (open) and after (filled) publication bias adjustment.

An external file that holds a picture, illustration, etc.
Object name is 1742-4755-4-3-3.jpg

A filled funnel plot of the interventions for emergency contraception data, with filled circles denoting the imputed missing studies. The bottom diamonds show summary effect estimates before (open) and after (filled) publication bias adjustment.

The practical impact on conclusions of three meta-analyses submitted to publication bias adjustment.

The main results of this analytic survey suggest that some degree of asymmetry in funnel plots is a common finding in reproductive health meta-analyses. In about 14% of the selected meta-analyses, this type of asymmetry qualitatively impacted conclusions. However, the present study also has some limitations that have to be taken into consideration for the subject to be evaluated within context. If Cochrane meta-analyses differ in their methods from other meta-analyses or if meta-analyses with fewer than ten trials differ from those with more than ten, a selection bias may exist. Despite this possible selection bias, this source of meta-analyses was chosen because of the consistency in the methodology, associated with a widely recognized standard of quality. The number of trials was established as an inclusion criterion since this factor results in the best performance of the "trim and fill" method. Moreover, although other methods are available for the assessment of asymmetry in funnel plots, no consensus has been reached with respect to the superiority of any single method. Therefore, any method used for detecting asymmetry in funnel plots should be considered indirect and exploratory. In this study, we used the "trim and fill" method as an instrument for sensitivity analysis. Our principal concern was not the exact number of missing studies; we were, in fact, interested in how the effect size estimates would be qualitatively changed by the presence of an underlying publication bias.

Asymmetrical in funnel plots are linked to publication bias although there are other sources of asymmetry that have to be considered, including other dissemination biases, differences in the quality of smaller studies, the existence of true heterogeneity, and chance. Asymmetry in funnel plots may be an indicator that a more detailed investigation should be carried out on the presence of heterogeneity, such as sensitivity analysis.

Nevertheless, none of the meta-analyses evaluated in this study reported the use of sensitivity analysis. In the latest version of the software program generally used by Cochrane reviewers (RevMan, version 4.2.8, The Nordic Cochrane Centre, Rigshospitalet 2003), no formal test for the assessment of funnel plots is available. This software program permits the visual subjective interpretation of funnel plots, but such an approach has been shown to include a significant inter-observer variability [ 17 ]. These limitations may have restricted the use of this method in this selected sample of meta-analyses.

On the other hand, there is some concern regarding the evolution of effect size over time and the impact of including "old" trials in meta-analyses. In the past, it was possible that the determinants of data suppression and the intensity of publication bias were different when compared to those in current use. We observed that in one-third of meta-analyses, the most recent trials had been published prior to 1997 (8:21) and in more than half (11:21), the most recent trials had been published prior to 2000. In fact, it is unclear whether the date of publication would have any impact on meta-analyses in reproductive health, but caution should be taken when summarizing effects across older trials.

Rather than reviewing the conclusions of meta-analyses, the aim of this study was to provide evidence of publication bias and its consequences in selected reproductive health meta-analyses. Consequently, three examples of possible publication bias were identified. Following adjustment, a reduction in the discrepancy between the conclusions of larger trials and the conclusions of meta-analyses was seen (data not shown). In all three meta-analyses whose conclusions were not considered robust, their suggested post-adjustment conclusion was in agreement with those of the respective larger trials included. In the systematic review assessing interventions for emergency contraception [ 16 ], the meta-analysts adopted a different form of sensitivity analysis, by reanalyzing the data, and including only the trials that had adequate allocation concealment. Their findings were similar to the adjustment for funnel plot asymmetry or publication bias.

Competing interests

The author(s) declare that they have no competing interests.

Authors' contributions

JPS participated in all the steps of the project, including the project development, data extraction, data analysis and writing the final report. CP participated in the project development and data extraction. JGC participated in the project development, data analysis and writing the final report. All authors provided suggestions for the manuscript, read it carefully, agreed on its content and approved the final version.

Supplementary Material

Appendix 1. Filled funnel plots of 21 reproductive health meta-analyses.

Acknowledgements

The authors are pleased to thank Steven Tarlow for helping in data analysis.

  • Implementing Best Practices in Reproductive Health http://www.ibpinitiative.org/
  • Villar J, Gülmezoglu AM, Khanna J, Carroli G, Hofmeyr GJ, Schulz K, Lumbiganon P. Evidence-based reproductive health in developing countries: The WHO Reproductive Health Library. Geneva, The World Health Organization; 1999. [ Google Scholar ]
  • Rothstein HR, Sutton AJ, Borenstein M. Publication Bias in Meta-analysis: prevention, assessment and adjustment. Chichester, UK:John Wiley & Sons, Ltd; 2005. [ Google Scholar ]
  • LeLorier J, Grégoire G, Benhaddad A, Lapierre J, Dederian F. Discrepancies between meta-analyses and subsequent large randomized, controlled trials. N Engl J Med. 1997; 337 :536–42. doi: 10.1056/NEJM199708213370806. [ PubMed ] [ CrossRef ] [ Google Scholar ]
  • Oxman AD, Cook DJ, Guyatt GH. User's guide to the medical literature. JAMA. 1994; 272 :1367–71. doi: 10.1001/jama.272.17.1367. [ PubMed ] [ CrossRef ] [ Google Scholar ]
  • Maher D, Cook DJ, Eastwood S. Improving the quality of reports of meta-analysis of randomized controlled trials: the QUOROM statement. Lancet. 1999; 354 :1896–900. doi: 10.1016/S0140-6736(99)04149-5. [ PubMed ] [ CrossRef ] [ Google Scholar ]
  • Stroup DF, Berlin JA, Morton SC. Meta-analysis of observational studies in epidemiology: a proposal for reporting. JAMA. 2000; 283 :2008–12. doi: 10.1001/jama.283.15.2008. [ PubMed ] [ CrossRef ] [ Google Scholar ]
  • Higgins JPT, Green S, editors . The Cochrane Library. Chichester, UK: John Wiley & Sons, Ltd; 2005. Cochrane Handbook for Systematic Reviews of Interventions 4.2.5 [updated May 2005] [ Google Scholar ]
  • Tallon D, Schneider M, Egger M. Proceedings of 2nd symposium on systematic reviews: beyond the basics. Oxford; 1999. Quality of systematic reviews published in high impact general and specialist journals. [ Google Scholar ]
  • WHO . The Reproductive Health Library. Oxford: Update Software Ltd; 2006. http://www.rhlibrary.com [ Google Scholar ]
  • Duval S. In: The trim and fill Method. Publication Bias in Meta-analysis: prevention, assessment and adjustment. Rothstein HR, Sutton AJ, Borenstein M, editor. Chichester, UK: John Wiley & Sons, Ltd; 2005. [ Google Scholar ]
  • Sutton AJ, Duval SJ, Tweedie RL, Abrams KR, Jones DR. Empirical assessment of effect of publication bias on meta-analysis. BMJ. 2000; 320 :1574–77. doi: 10.1136/bmj.320.7249.1574. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
  • Martin-Hirsch P, Jarvis G, Kitchener H, Lilford R. The Reproductive Health Library. Oxford: Update Software Ltd; 2006. Collection devices for obtaining cervical cytology samples (Cochrane Review) http://www.rhlibrary.com (Reprinted from The Cochrane Library, Issue 1, 2006. Chichester, UK: John Wiley & Sons, Ltd.) [ Google Scholar ]
  • Hopkins L, Smaill F. The Reproductive Health Library. Oxford: Update Software Ltd; 2006. Antibiotic prophylaxis regimens and drugs for cesarean section (Cochrane Review) http://www.rhlibrary.com (Reprinted from The Cochrane Library, Issue 1, 2006. Chichester, UK: John Wiley & Sons, Ltd.) [ Google Scholar ]
  • Hodnett ED, Gates S, Hofmeyr GJ, Sakala C. The Reproductive Health Library. Oxford: Update Software Ltd; 2006. Continuous support for women during childbirth (Cochrane Review) http://www.rhlibrary.com (Reprinted from The Cochrane Library, Issue 1, 2006. Chichester, UK: John Wiley & Sons, Ltd.) [ Google Scholar ]
  • Cheng L, Gülmezoglu AM, Van Oel CJ, Piaggio G, Ezcurra E, Van Look PFA. The Reproductive Health Library. Oxford: Update Software Ltd; 2006. Interventions for emergency contraception (Cochrane Review) http://www.rhlibrary.com (Reprinted from The Cochrane Library, Issue 1, 2006. Chichester, UK: John Wiley & Sons, Ltd.) [ Google Scholar ]
  • Villar J, Piaggio G, Carroli G, Donner A. Factors affecting the comparability of meta-analyses and largest trials results in perinatology. J Clin Epidemiol. 1997; 50 :997–1002. doi: 10.1016/S0895-4356(97)00148-0. [ PubMed ] [ CrossRef ] [ Google Scholar ]
  • Hofmeyr GJ. The Reproductive Health Library. Oxford: Update Software Ltd; 2006. Amnioinfusion for meconium-stained liquor in labour (Cochrane Review) http://www.rhlibrary.com (Reprinted from The Cochrane Library, Issue 1, 2006. Chichester, UK: John Wiley & Sons, Ltd.) [ Google Scholar ]
  • Smaill F, Hofmeyr GJ. The Reproductive Health Library. Oxford: Update Software Ltd; 2006. Antibiotic prophylaxis for cesarean section (Cochrane Review) http://www.rhlibrary.com (Reprinted from The Cochrane Library, Issue 1, 2006. Chichester, UK: John Wiley & Sons, Ltd.) [ Google Scholar ]
  • Smaill F. The Reproductive Health Library. Oxford: Update Software Ltd; 2006. Antibiotics for asymptomatic bacteriuria in pregnancy (Cochrane Review) http://www.rhlibrary.com (Reprinted from The Cochrane Library, Issue 1, 2006. Chichester, UK: John Wiley & Sons, Ltd.) [ Google Scholar ]
  • Kenyon S, Boulvain M, Neilson J. The Reproductive Health Library. Oxford: Update Software Ltd; 2006. Antibiotics for preterm rupture of membranes (Cochrane Review) http://www.rhlibrary.com (Reprinted from The Cochrane Library, Issue 1, 2006. Chichester, UK: John Wiley & Sons, Ltd.) [ Google Scholar ]
  • McDonald H, Brocklehurst P, Parsons J. The Reproductive Health Library. Oxford: Update Software Ltd; 2006. Antibiotics for treating bacterial vaginosis in pregnancy (Cochrane Review) http://www.rhlibrary.com (Reprinted from The Cochrane Library, Issue 1, 2006. Chichester, UK: John Wiley & Sons, Ltd.) [ Google Scholar ]
  • Abalos E, Duley L, Steyn DW, Henderson-Smart DJ. The Reproductive Health Library. Oxford: Update Software Ltd; 2006. Antihypertensive drug therapy for mild to moderate hypertension during pregnancy (Cochrane Review) http://www.rhlibrary.com (Reprinted from The Cochrane Library, Issue 1, 2006. Chichester, UK: John Wiley & Sons, Ltd.) [ Google Scholar ]
  • Knight M, Duley L, Henderson-Smart DJ, King JF. The Reproductive Health Library. Oxford: Update Software Ltd; 2006. Antiplatelet agents for preventing and treating pre-eclampsia (Cochrane Review) http://www.rhlibrary.com (Reprinted from The Cochrane Library, Issue 1, 2006. Chichester, UK: John Wiley & Sons, Ltd.) [ Google Scholar ]
  • King JF, Flenady VJ, Papatsonis DNM, Dekker GA, Carbonne B. The Reproductive Health Library. Oxford: Update Software Ltd; 2006. Calcium channel blockers for inhibiting preterm labour (Cochrane Review) http://www.rhlibrary.com (Reprinted from The Cochrane Library, Issue 1, 2006. Chichester, UK: John Wiley & Sons, Ltd.) [ Google Scholar ]
  • Atallah AN, Hofmeyr GJ, Duley L. The Reproductive Health Library. Oxford: Update Software Ltd; 2006. Calcium supplementation during pregnancy for preventing hypertensive disorders and related problems (Cochrane Review) http://www.rhlibrary.com (Reprinted from The Cochrane Library, Issue 1, 2006. Chichester, UK: John Wiley & Sons, Ltd.) [ PubMed ] [ Google Scholar ]
  • Thacker SB, Stroup D, Chang M. The Reproductive Health Library. Oxford: Update Software Ltd; 2006. Continuous electronic heart rate monitoring for fetal assessment during labor (Cochrane Review) http://www.rhlibrary.com (Reprinted from The Cochrane Library, Issue 1, 2006. Chichester, UK: John Wiley & Sons, Ltd.) [ Google Scholar ]
  • Crowley P. The Reproductive Health Library. Oxford: Update Software Ltd; 2006. Interventions for preventing or improving the outcome of delivery at or beyond term (Cochrane Review) http://www.rhlibrary.com (Reprinted from The Cochrane Library, Issue 1, 2006. Chichester, UK: John Wiley & Sons, Ltd.) [ Google Scholar ]
  • Oates-Whitehead RM, Haas DM, Carrier JAK. The Reproductive Health Library. Oxford: Update Software Ltd; 2006. Progestogen for preventing miscarriage (Cochrane Review) http://www.rhlibrary.com (Reprinted from The Cochrane Library, Issue 1, 2006. Chichester, UK: John Wiley & Sons, Ltd.) [ Google Scholar ]
  • Hodnett ED, Fredericks S. The Reproductive Health Library. Oxford: Update Software Ltd; 2006. Support during pregnancy for women at increased risk of low birthweight babies (Cochrane Review) http://www.rhlibrary.com (Reprinted from The Cochrane Library, Issue 1, 2006. Chichester, UK: John Wiley & Sons, Ltd.) [ Google Scholar ]
  • Carroli G, Bergel E. The Reproductive Health Library. Oxford: Update Software Ltd; 2006. Umbilical vein injection for management of retained placenta (Cochrane Review) http://www.rhlibrary.com (Reprinted from The Cochrane Library, Issue 1, 2006. Chichester, UK: John Wiley & Sons, Ltd.) [ Google Scholar ]
  • Johanson RB, Menon V. The Reproductive Health Library. Oxford: Update Software Ltd; 2006. Vacuum extraction versus forceps for assisted vaginal delivery (Cochrane Review) http://www.rhlibrary.com (Reprinted from The Cochrane Library, Issue 1, 2006. Chichester, UK: John Wiley & Sons, Ltd.) [ Google Scholar ]
  • Hofmeyr GJ, Gülmezoglu AM. The Reproductive Health Library. Oxford: Update Software Ltd; 2006. Vaginal misoprostol for cervical ripening and induction of labour (Cochrane Review) http://www.rhlibrary.com (Reprinted from The Cochrane Library, Issue 1, 2006. Chichester, UK: John Wiley & Sons, Ltd.) [ Google Scholar ]

Bias in meta-analysis detected by a simple, graphical test

Affiliation.

  • 1 Department of Social Medicine, University of Bristol. [email protected]
  • PMID: 9310563
  • PMCID: PMC2127453
  • DOI: 10.1136/bmj.315.7109.629

Objective: Funnel plots (plots of effect estimates against sample size) may be useful to detect bias in meta-analyses that were later contradicted by large trials. We examined whether a simple test of asymmetry of funnel plots predicts discordance of results when meta-analyses are compared to large trials, and we assessed the prevalence of bias in published meta-analyses.

Design: Medline search to identify pairs consisting of a meta-analysis and a single large trial (concordance of results was assumed if effects were in the same direction and the meta-analytic estimate was within 30% of the trial); analysis of funnel plots from 37 meta-analyses identified from a hand search of four leading general medicine journals 1993-6 and 38 meta-analyses from the second 1996 issue of the Cochrane Database of Systematic Reviews.

Main outcome measure: Degree of funnel plot asymmetry as measured by the intercept from regression of standard normal deviates against precision.

Results: In the eight pairs of meta-analysis and large trial that were identified (five from cardiovascular medicine, one from diabetic medicine, one from geriatric medicine, one from perinatal medicine) there were four concordant and four discordant pairs. In all cases discordance was due to meta-analyses showing larger effects. Funnel plot asymmetry was present in three out of four discordant pairs but in none of concordant pairs. In 14 (38%) journal meta-analyses and 5 (13%) Cochrane reviews, funnel plot asymmetry indicated that there was bias.

Conclusions: A simple analysis of funnel plots provides a useful test for the likely presence of bias in meta-analyses, but as the capacity to detect bias will be limited when meta-analyses are based on a limited number of small trials the results from such analyses should be treated with considerable caution.

Publication types

  • Research Support, U.S. Gov't, Non-P.H.S.
  • Meta-Analysis as Topic*
  • Randomized Controlled Trials as Topic
  • Regression Analysis
  • Statistics as Topic
  • Treatment Outcome

Association of nonpharmacological interventions for cognitive function in older adults with mild cognitive impairment: a systematic review and network meta-analysis

  • Published: 06 January 2023
  • Volume 35 , pages 463–478, ( 2023 )

Cite this article

  • Xueyan Liu   ORCID: orcid.org/0000-0001-6228-2822 1 ,
  • Guangpeng Wang   ORCID: orcid.org/0000-0003-1442-0789 2 &
  • Yingjuan Cao   ORCID: orcid.org/0000-0002-3063-304X 3  

1757 Accesses

5 Citations

2 Altmetric

Explore all metrics

Understanding the effectiveness of nonpharmacological interventions to improve cognitive function in older adults with MCI and identifying the best intervention may help inform ideas for future RCT studies and clinical decision-making.

The main focus of this study was to assess the comparative effectiveness of nonpharmacological interventions on cognitive function in older adults with MCI and to rank the interventions.

RCT studies until September 2022 were searched from six databases, including PubMed, the Cochrane Library, Embase, Web of Science, PsycINFO and CINAHL. The risk of bias in eligible trials was evaluated using the Cochrane Risk of Bias tool. Both pairwise and network meta-analyses were used, and pooled effect sizes were reported using SMD and the corresponding 95% confidence intervals.

A total of 28 RCT studies were included in this study, pooling 18 categories of nonpharmacological interventions. MBE (mind–body exercise) (SMD (standard mean difference): 0.24, 95% CI: 0.08–0.41, P  = 0.004), DTE (dual-task exercise) (SMD: 0.61, 95% CI: 0.09–1.13, P  = 0.02), PE (physical exercise) (SMD: 0.58, 95% CI: 0.04–1.12, P  = 0.03) may be effective in improving cognitive function in older adults with MCI. Acupressure + CT (cognitive training) was the top-ranked intervention among all interventions. No greater benefits of MA (mindful awareness) on cognitive function were found.

Conclusions

Overall, nonpharmacological interventions significantly improved cognitive function in older adults with MCI. Acupressure + CT(cognitive training) was the most effective intervention for managing cognitive impairment. Future studies with high quality and large sample size RCT studies are needed to confirm our results.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price includes VAT (Russian Federation)

Instant access to the full article PDF.

Rent this article via DeepDyve

Institutional subscriptions

cochrane systematic review funnel plot

Salinas-Rodríguez A, Palazuelos-González R, Rivera-Almaraz A et al (2021) Longitudinal association of sarcopenia and mild cognitive impairment among older Mexican adults. J Cachexia Sarcopenia Muscle 12:1848–1859. https://doi.org/10.1002/jcsm.12787

Article   PubMed   PubMed Central   Google Scholar  

Anderson ND (2019) State of the science on mild cognitive impairment (MCI). CNS Spectr 24:78–87. https://doi.org/10.1017/s1092852918001347

Article   PubMed   Google Scholar  

Dunne RA, Aarsland D, O’Brien JT et al (2021) Mild cognitive impairment: the Manchester consensus. Age Ageing 50:72–80. https://doi.org/10.1093/ageing/afaa228

McWhirter L, Ritchie C, Stone J et al (2020) Functional cognitive disorders: a systematic review. Lancet Psychiatry 7:191–207. https://doi.org/10.1016/s2215-0366(19)30405-5

Gómez-Ramírez J, Ávila-Villanueva M, Fernández-Blázquez M (2020) Selecting the most important self-assessed features for predicting conversion to mild cognitive impairment with random forest and permutation-based methods. Sci Rep 10:20630. https://doi.org/10.1038/s41598-020-77296-4

Article   CAS   PubMed   PubMed Central   Google Scholar  

Burks HB, des Bordes JKA, Chadha R et al (2021) Quality of life assessment in older adults with dementia: a systematic review. Dement Geriatr Cogn Disord 50:103–110. https://doi.org/10.1159/000515317

WHO Guidelines Approved by the Guidelines Review Committee (2019) Risk reduction of cognitive decline and dementia: WHO guidelines. World Health Organization, Geneva

Google Scholar  

Marotta N, Calafiore D, Curci C, et al (2022) Integrating virtual reality and exergaming in cognitive rehabilitation of patients with Parkinson disease: a systematic review of randomized controlled trials. Eur J Phys Rehabil Med. https://doi.org/10.23736/s1973-9087.22.07643-2

Gates NJ, Vernooij RW, Di Nisio M et al (2019) Computerised cognitive training for preventing dementia in people with mild cognitive impairment. Cochrane Database Syst Rev 3:Cd012279. https://doi.org/10.1002/14651858.CD012279.pub2

Law CK, Lam FM, Chung RC et al (2020) Physical exercise attenuates cognitive decline and reduces behavioural problems in people with mild cognitive impairment and dementia: a systematic review. J Physiother 66:9–18. https://doi.org/10.1016/j.jphys.2019.11.014

Biazus-Sehn LF, Schuch FB, Firth J et al (2020) Effects of physical exercise on cognitive function of older adults with mild cognitive impairment: a systematic review and meta-analysis. Arch Gerontol Geriatr 89:104048. https://doi.org/10.1016/j.archger.2020.104048

da Costa BR, Juni P (2014) Systematic reviews and meta-analyses of randomized trials: principles and pitfalls. Eur Heart J 35:3336–3345. https://doi.org/10.1093/eurheartj/ehu424

Dias S, Caldwell DM (2019) Network meta-analysis explained. Arch Dis Child Fetal Neonatal Ed 104:F8–F12. https://doi.org/10.1136/archdischild-2018-315224

Gong LL, Tao FY (2021) The effect of biopsychosocial holistic care models on the cognitive function and quality of life of elderly patients with mild cognitive impairment: a randomized trial. Ann Palliat Med 10:5600–5609. https://doi.org/10.21037/apm-21-966

Lam LC, Chan WC, Leung T et al (2015) Would older adults with mild cognitive impairment adhere to and benefit from a structured lifestyle activity intervention to enhance cognition?: a cluster randomized controlled trial. PLoS ONE 10:e0118173. https://doi.org/10.1371/journal.pone.0118173

Lam LCW, Chau RCM, Wong BML et al (2012) A 1-year randomized controlled trial comparing mind body exercise (Tai Chi) with stretching and toning exercise on cognitive function in older chinese adults at risk of cognitive decline. J Am Med Directors Assoc 13:568.e15-568.e20. https://doi.org/10.1016/j.jamda.2012.03.008

Article   Google Scholar  

Li L, Liu M, Zeng H et al (2021) Multi-component exercise training improves the physical and cognitive function of the elderly with mild cognitive impairment: a six-month randomized controlled trial. Ann Palliat Med 10:8919–8929

Liu CL, Cheng FY, Wei MJ et al (2022) Effects of exergaming-based tai chi on cognitive function and dual-task gait performance in older adults with mild cognitive impairment: a randomized control trial. Front Aging Neurosci 14:761053. https://doi.org/10.3389/fnagi.2022.761053

Sun J, Zeng H, Pan L et al (2021) Acupressure and cognitive training can improve cognitive functions of older adults with mild cognitive impairment: a randomized controlled trial. Front Psychol 12:726083. https://doi.org/10.3389/fpsyg.2021.726083

Sun Y, Li Y, Wang J et al (2021) Effectiveness of smartphone-based mindfulness training on maternal perinatal depression: randomized controlled trial. J Med Internet Res 23:e23410. https://doi.org/10.2196/23410

Wang L, Wu B, Tao H et al (2020) Effects and mediating mechanisms of a structured limbs-exercise program on general cognitive function in older adults with mild cognitive impairment: a randomized controlled trial. Int J Nurs Stud 110:103706. https://doi.org/10.1016/j.ijnurstu.2020.103706

Xu Z, Sun W, Zhang D et al (2021) Comparative effectiveness of non-pharmacological interventions for depressive symptoms in mild cognitive impairment: systematic review with network meta-analysis. Aging Ment Health. https://doi.org/10.1080/13607863.2021.1998356

Xue B, Xiao A, Luo X et al (2021) The effect of a game training intervention on cognitive functioning and depression symptoms in the elderly with mild cognitive impairment: A randomized controlled trial. Int J Methods Psychiatr Res 30:e1887. https://doi.org/10.1002/mpr.1887

Yang QH, Lyu X, Lin QR et al (2022) Effects of a multicomponent intervention to slow mild cognitive impairment progression: a randomized controlled trial. Int J Nurs Stud 125:104110. https://doi.org/10.1016/j.ijnurstu.2021.104110

Young KW, Ng P, Kwok T et al (2017) The effects of holistic health group interventions on improving the cognitive ability of persons with mild cognitive impairment: a randomized controlled trial. Clin Interv Aging 12:1543–1552. https://doi.org/10.2147/cia.S142109

Yu L, Liu F, Nie P et al (2021) Systematic review and meta-analysis of randomized controlled trials assessing the impact of Baduanjin exercise on cognition and memory in patients with mild cognitive impairment. Clin Rehabil 35:492–505. https://doi.org/10.1177/0269215520969661

Bae S, Lee S, Lee S et al (2019) The effect of a multicomponent intervention to promote community activity on cognitive function in older adults with mild cognitive impairment: a randomized controlled trial. Complement Ther Med 42:164–169. https://doi.org/10.1016/j.ctim.2018.11.011

Doi T, Verghese J, Makizako H et al (2017) Effects of cognitive leisure activity on cognition in mild cognitive impairment: results of a randomized controlled trial. J Am Med Dir Assoc 18:686–691. https://doi.org/10.1016/j.jamda.2017.02.013

Suzuki T, Shimada H, Makizako H et al (2013) A randomized controlled trial of multicomponent exercise in older adults with mild cognitive impairment. PLoS ONE 8:e61483. https://doi.org/10.1371/journal.pone.0061483

Suzuki T, Shimada H, Makizako H et al (2012) Effects of multicomponent exercise on cognitive function in older adults with amnestic mild cognitive impairment: a randomized controlled trial. BMC Neurol 12:128. https://doi.org/10.1186/1471-2377-12-128

Choi W, Lee S (2019) Virtual Kayak paddling exercise improves postural balance, muscle performance, and cognitive function in older adults with mild cognitive impairment: a randomized controlled trial. J Aging Phys Act. https://doi.org/10.1123/jap2018-0020

Hong SG, Kim JH, Jun TW (2018) Effects of 12-week resistance exercise on electroencephalogram patterns and cognitive function in the elderly with mild cognitive impairment: a randomized controlled trial. Clin J Sport Med 28:500–508. https://doi.org/10.1097/jsm.0000000000000476

Park H, Park JH, Na HR et al (2019) Combined intervention of physical activity, aerobic exercise, and cognitive exercise intervention to prevent cognitive decline for patients with mild cognitive impairment: a randomized controlled clinical study. J Clin Med 8:940. https://doi.org/10.3390/jcm8070940

Carcelén-Fraile MDC, Llera-DelaTorre AM, Aibar-Almazán A et al (2022) Cognitive stimulation as alternative treatment to improve psychological disorders in patients with mild cognitive impairment. J Clin Med 11:3947. https://doi.org/10.3390/jcm11143947

Gómez-Soria I, Brandín-de la Cruz N, Cuenca Zaldívar JN et al (2022) Effectiveness of personalized cognitive stimulation in older adults with mild possible cognitive impairment: a 12-month follow-up cognitive stimulation in mild cognitive impairment. Clin Gerontol 45:878–890. https://doi.org/10.1080/07317115.2021.1937764

de Oliveira Silva F, Ferreira JV, Plácido J et al (2019) Three months of multimodal training contributes to mobility and executive function in elderly individuals with mild cognitive impairment, but not in those with Alzheimer’s disease: a randomized controlled trial. Maturitas 126:28–33. https://doi.org/10.1016/j.maturitas.2019.04.217

Langoni CDS, Resende TL, Barcellos AB et al (2019) Effect of exercise on cognition, conditioning, muscle endurance, and balance in older adults with mild cognitive impairment: a randomized controlled trial. J Geriatr Phys Ther 42:E15–E22. https://doi.org/10.1519/jpt.0000000000000191

Li F, Harmer P, Fitzgerald K et al (2022) A cognitively enhanced online Tai Ji Quan training intervention for community-dwelling older adults with mild cognitive impairment: a feasibility trial. BMC Geriatr 22:76. https://doi.org/10.1186/s12877-021-02747-0

Masika GM, Yu DSF, Li PWC (2021) Can visual art therapy be implemented with illiterate older adults with mild cognitive impairment? A pilot mixed-method randomized controlled trial. J Geriatr Psychiatry Neurol 34:76–86. https://doi.org/10.1177/0891988720901789

Ghafoor U, Lee JH, Hong KS et al (2019) Effects of acupuncture therapy on MCI patients using functional near-infrared spectroscopy. Front Aging Neurosci 11:237. https://doi.org/10.3389/fnagi.2019.00237

Bademli K, Lok N, Canbaz M et al (2019) Effects of Physical Activity Program on cognitive function and sleep quality in elderly with mild cognitive impairment: a randomized controlled trial. Perspect Psychiatr Care 55:401–408. https://doi.org/10.1111/ppc.12324

Xu Z, Zhang D, Lee ATC et al (2020) A pilot feasibility randomized controlled trial on combining mind-body physical exercise, cognitive training, and nurse-led risk factor modification to reduce cognitive decline among older adults with mild cognitive impairment in primary care. PeerJ 8:e9845. https://doi.org/10.7717/peerj.9845

Doi T, Verghese J, Makizako H et al (2017) Effects of cognitive leisure activity on cognition in mild cognitive impairment: results of a randomized controlled trial. J Am Med Directors Assoc 18:686–691. https://doi.org/10.1016/j.jamda.2017.02.013

Lam LC, Chau RC, Wong BM et al (2011) Interim follow-up of a randomized controlled trial comparing Chinese style mind body (Tai Chi) and stretching exercises on cognitive function in subjects at risk of progressive cognitive decline. Int J Geriatr Psychiatry 26:733–740. https://doi.org/10.1002/gps.2602

Lam LC, Chau RC, Wong BM et al (2012) A 1-year randomized controlled trial comparing mind body exercise (Tai Chi) with stretching and toning exercise on cognitive function in older Chinese adults at risk of cognitive decline. J Am Med Dir Assoc 13:568.e515–520. https://doi.org/10.1016/j.jamda.2012.03.008

Seitidis G, Nikolakopoulos S, Hennessy EA et al (2022) Network meta-analysis techniques for synthesizing prevention science evidence. Prev Sci 23:415–424. https://doi.org/10.1007/s11121-021-01289-6

Article   CAS   PubMed   Google Scholar  

Zeng H, Liu M, Wang P et al (2016) The effects of acupressure training on sleep quality and cognitive function of older adults: a 1-year randomized controlled trial. Res Nurs Health 39:328–336. https://doi.org/10.1002/nur.21738

Chen IH, Yeh TP, Yeh YC et al (2019) Effects of acupressure on sleep quality and psychological distress in nursing home residents: a randomized controlled trial. J Am Med Dir Assoc 20:822–829. https://doi.org/10.1016/j.jamda.2019.01.003

Tseng YT, Chen IH, Lee PH et al (2021) Effects of auricular acupressure on depression and anxiety in older adult residents of long-term care institutions: a randomized clinical trial. Geriatr Nurs 42:205–212. https://doi.org/10.1016/j.gerinurse.2020.08.003

Zokaei N, MacKellar C, Čepukaitytė G et al (2017) Cognitive training in the elderly: bottlenecks and new avenues. J Cogn Neurosci 29:1473–1482. https://doi.org/10.1162/jocn_a_01080

McFadden KL, Healy KM, Dettmann ML et al (2011) Acupressure as a non-pharmacological intervention for traumatic brain injury (TBI). J Neurotrauma 28:21–34. https://doi.org/10.1089/neu.2010.1515

Deng X, Teng J, Nong X et al (2021) Characteristics of TCM constitution and related biomarkers for mild cognitive impairment. Neuropsychiatr Dis Treat 17:1115–1124. https://doi.org/10.2147/ndt.S290692

Smith CA, Collins CT, Levett KM et al (2020) Acupuncture or acupressure for pain management during labour. Cochrane Database Syst Rev 2:Cd009232. https://doi.org/10.1002/14651858.CD009232.pub2

McEwen BS, Sapolsky RM (1995) Stress and cognitive function. Curr Opin Neurobiol 5:205–216. https://doi.org/10.1016/0959-4388(95)80028-x

Law R, Clow A (2020) Stress, the cortisol awakening response and cognitive function. Int Rev Neurobiol 150:187–217. https://doi.org/10.1016/bs.irn.2020.01.001

Gates NJ, Rutjes AW, Di Nisio M et al (2020) Computerised cognitive training for 12 or more weeks for maintaining cognitive function in cognitively healthy people in late life. Cochrane Database Syst Rev 2:Cd012277. https://doi.org/10.1002/14651858.CD012277.pub3

Bahar-Fuchs A, Martyr A, Goh AM et al (2019) Cognitive training for people with mild to moderate dementia. Cochrane Database Syst Rev 3:cd013069. https://doi.org/10.1002/14651858.CD013069.pub2

Li BY, He NY, Qiao Y et al (2019) Computerized cognitive training for Chinese mild cognitive impairment patients: a neuropsychological and fMRI study. Neuroimage Clin 22:101691. https://doi.org/10.1016/j.nicl.2019.101691

Loetscher T (2021) Cognitive training interventions for dementia and mild cognitive impairment in Parkinson’s disease—a cochrane review summary with commentary. NeuroRehabilitation 48:385–387. https://doi.org/10.3233/nre-218001

Sala G, Gobet F (2019) Cognitive training does not enhance general cognition. Trends Cogn Sci 23:9–20. https://doi.org/10.1016/j.tics.2018.10.004

Kang JM, Kim N, Lee SY et al (2021) Effect of cognitive training in fully immersive virtual reality on visuospatial function and frontal-occipital functional connectivity in predementia: randomized controlled trial. J Med Internet Res 23:e24526. https://doi.org/10.2196/24526

Ge S, Zhu Z, Wu B et al (2018) Technology-based cognitive training and rehabilitation interventions for individuals with mild cognitive impairment: a systematic review. BMC Geriatr 18:213. https://doi.org/10.1186/s12877-018-0893-1

Xia R, Wan M, Lin H et al (2022) Effects of mind-body exercise Baduanjin on cognition in community-dwelling older people with mild cognitive impairment: a randomized controlled trial. Neuropsychol Rehabil. https://doi.org/10.1080/09602011.2022.2099909

Hölzel BK, Lazar SW, Gard T et al (2011) how does mindfulness meditation work? Proposing mechanisms of action from a conceptual and neural perspective. Perspect Psychol Sci 6:537–559. https://doi.org/10.1177/1745691611419671

Khine WWT, Voong ML, Ng TKS et al (2020) Mental awareness improved mild cognitive impairment and modulated gut microbiome. Aging (Albany NY) 12:24371–24393. https://doi.org/10.18632/aging.202277

Berk L, van Boxtel M, van Os J (2017) Can mindfulness-based interventions influence cognitive functioning in older adults? A review and considerations for future research. Aging Ment Health 21:1113–1120. https://doi.org/10.1080/13607863.2016.1247423

Whitfield T, Barnhofer T, Acabchuk R et al (2022) The effect of mindfulness-based programs on cognitive function in adults: a systematic review and meta-analysis. Neuropsychol Rev 32:677–702. https://doi.org/10.1007/s11065-021-09519-y

Download references

Author information

Authors and affiliations.

School of Nursing and Rehabilitation, Cheeloo College of Medicine, Shandong University, 44 Wenhuaxi Road, Lixia District, Jinan, Shandong Province, China

Xiangya School of Nursing, Central South University, 172 Tongzipo Road, Yuelu District, Changsha, Hunan Province, China

Guangpeng Wang

Department of Nursing, Qilu Hospital, Shandong University, 107 Wenhuaxi Road, Lixia District, Jinan, Shandong Province, China

Yingjuan Cao

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Yingjuan Cao .

Ethics declarations

Conflict of interest.

No conflict of interest to declare.

Ethical approval

This article does not contain any studies with human participants or animals performed by the author.

Informed Consent

For this type of study, formal consent is not required.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (DOCX 1700 KB)

Rights and permissions.

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Liu, X., Wang, G. & Cao, Y. Association of nonpharmacological interventions for cognitive function in older adults with mild cognitive impairment: a systematic review and network meta-analysis. Aging Clin Exp Res 35 , 463–478 (2023). https://doi.org/10.1007/s40520-022-02333-3

Download citation

Received : 09 November 2022

Accepted : 19 December 2022

Published : 06 January 2023

Issue Date : March 2023

DOI : https://doi.org/10.1007/s40520-022-02333-3

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

  • Mild cognitive impairment
  • Nonpharmacological interventions
  • Systematic review
  • Network meta-analysis
  • Find a journal
  • Publish with us
  • Track your research
  • Search Menu
  • Supplements
  • Cohort Profiles
  • Education Corner
  • Author Guidelines
  • Submission Site
  • Open Access
  • About the International Journal of Epidemiology
  • About the International Epidemiological Association
  • Editorial Team
  • Editorial Board
  • Advertising and Corporate Services
  • Journals Career Network
  • Self-Archiving Policy
  • Dispatch Dates
  • Contact the IEA
  • Journals on Oxford Academic
  • Books on Oxford Academic

Article Contents

Introduction, acknowledgements.

  • < Previous

Forest plots in reports of systematic reviews: a cross-sectional study reviewing current practice

  • Article contents
  • Figures & tables
  • Supplementary Data

David L Schriger, Douglas G Altman, Julia A Vetter, Thomas Heafner, David Moher, Forest plots in reports of systematic reviews: a cross-sectional study reviewing current practice, International Journal of Epidemiology , Volume 39, Issue 2, April 2010, Pages 421–429, https://doi.org/10.1093/ije/dyp370

  • Permissions Icon Permissions

Background Forest plots are graphical displays of findings of systematic reviews and meta-analyses. Little is known about the style and content of these plots and whether published plots maximize the graphic’s potential for information exchange.

Methods We examine the number, style and content of forest plots presented in a previously studied cross-sectional sample of 300 systematic reviews. We studied all forest plots in non-Cochrane reviews and a sample of forest plots in Cochrane reviews.

Results The database contained 129 Cochrane reviews and 171 non-Cochrane reviews. All the Cochrane reviews had forest plots (2197 in total), and a random sample of 500 of these plots were included. In total, 28 of the non-Cochrane reviews had forest plots (139 in total), all of which were included. Plots in Cochrane reviews were standardized but often contained little data (80% had three or fewer studies; 10% had no studies) and always presented studies in alphabetical order. Non-Cochrane plots depicted a larger number of studies (60% had four or more studies) and 59% ordered studies by a potentially meaningful characteristic, but important information was often missing. Of the 28 reviews that had a forest plots with at least 10 studies, 3 (11%) had funnel plots.

Conclusions Forest plots in Cochrane reviews were highly standardized but some of the standards do not optimize information exchange, and many of the plots had too little data to be useful. Forest plots in non-Cochrane reviews often omitted key elements but had more data and were often more thoughtfully constructed.

Systematic reviews are an important means of summarizing the methods and results of individual studies and increasingly are being used as a starting point in the development of clinical practice guidelines 1 and have been advocated as the starting and ending point of all randomized trials. 2 Forest plots—the graphical display of individual study results and, usually, the weighted average of studies included in a systematic review—are one way of summarizing the review’s results for a specific outcome. 3–6 Plots of this kind first appeared in the 1970s and were refined over the next two decades; they were first called ‘forest plots’ in the mid-1990s. 4 Since that time the elements contained in a forest plot and the layout of such plots have become somewhat standardized, largely due to the introduction of software that helps authors construct these plots ( Figure 1 ). 7 , 8

A standardized format for forest plots no doubt helps readers because repeated exposure to a familiar format decreases the time and effort required to become oriented to the graphic and likely facilities their interpretation. Nevertheless, many of the de facto standards for the construction of forest plots were not based on theory or empirical evidence regarding optimal information exchange and, for a number of issues, theory would suggest that current practice is suboptimal.

The plot is drawn in STATA 11 (Stata Corp., College Station, TX, USA) from data presented in Ezekowitz et al. 22 Note that studies have been sorted first by whether they addressed primary or secondary prevention and second by year of publication. This organization allows readers to easily determine whether they believe that these variables affect the outcome.

Recently, Moher et al. characterized many of the qualities of the text and tables in a group of systematic reviews. 9 We now examine the forest plots contained in this set of systematic reviews with the goal of describing current practice regarding their construction and display. By defining current practice we hope to identify ways that forest plots in future systematic reviews can be improved.

We used the database assembled by Moher et al. , which consists 300 English language reports of systematic reviews (of various study designs) that were indexed on Medline during November 2004. 9 These papers were found by reviewing 1046 potential citations and keeping those that were systematic reviews, which offered explicit methods for article identification and eligibility. The 300 papers were found in 132 journals, and mainly reviewed therapeutic (71%), epidemiologic (13%) and diagnostic (8%) questions. A total of 54% of the systematic reviews did some type of mathematical pooling of individual study results.

We reviewed each paper for the presence of any graphical attempt to simultaneously portray the results of the individual studies included in the systematic review. We also noted whether each paper contained funnel plots—a graphical display used to assess asymmetry of results, a possible explanation for publication bias. 10 We did not count isolated data tables; we required that there be some form of graphic presentation of the individual study data.

We counted the number of plots in each paper and noted whether each paper was a Cochrane review. 11 All plots from non-Cochrane reviews were included. We sampled the plots in Cochrane reviews because the number of forest plots they contained substantially outnumbered the forest plots found in non-Cochrane reviews. We used the random number function in STATA 9.0 to select two plots from every Cochrane review. We completed the sampling by randomly selecting from the pool of remaining Cochrane review plots sufficient plots to bring the total sample of Cochrane review plots to 500.

Plots were independently rated by two members of the research team (J.V. and T.H.) who had been trained by the principal investigator (D.S.) and had proved their accuracy on a set of training plots. Data forms were reviewed for inconsistencies between the raters, and the principal investigator adjudicated discrepancies. Forest plots were assessed for: whether the individual studies were separated into sub-panels and on what basis; how the studies in each plot (or each sub-panel if there were sub-panels) were ordered—alphabetically, by effect size, by weight, by year of publication, by study characteristic (e.g. dose used); and what measure of effect was used, what scale was used and what data elements were presented in the graphic.

Of the 300 systematic reviews in the data-set, 129 (43%) were Cochrane reviews ( Table 1 ). Although all the reviews had forest plot frameworks, only 115 (89%) had data in the frameworks (in 14 Cochrane reviews all forest plot frameworks were empty as no eligible studies were included). There were 2197 individual forest plots (although ∼10% had no data, see below), a mean of 17 plots per Cochrane review [median 9, interquartile range (IQR) 4–24]. The maximum number of forest plots in a Cochrane review was 125.

Number of forest and funnel plots in 300 systematic reviews

a All 129 Cochrane papers had forest plot frameworks but 14 had no studies in the framework.

Of the 171 non-Cochrane reviews, 28 (16%) had at least one forest plot. Non-Cochrane reviews with a forest plot had a mean of five plots (median 2, IQR 1–4). The three studies with the largest number of plots had 44, 20 and 9. A total of 3% (4/129) of Cochrane and 3% (5/171) of non-Cochrane reviews had funnel plots. Of the 28 reviews that had a forest plot that contained at least 10 studies, 3 (11%) had funnel plots.

The number of individual studies represented within the forest plots is presented in Table 2 . In general, there were fewer studies in the plots presented in Cochrane reviews than in non-Cochrane reviews. This was true for plots that contained a single panel of studies (median one vs seven studies) and plots that divided the studies into sub-panels (median one vs two studies per sub-panel). One paper 12 contributed 44 (32%) of 139 non-Cochrane plots. These plots were atypical of the other non-Cochrane plots—88 of the 92 panels and sub-panels had one study, 3 had two studies and 1 had three studies. Excluding this paper, single panel non-Cochrane plots had a median of eight studies and multi-paneled plots had a median five studies per panel. Although 36% of non-Cochrane plots that used sub-panels (71% if we exclude the one aberrant study) had two sub-panels that contained at least four studies, only 16% of Cochrane plots did so. When present, sub-panels were organized by an explanatory variable in 78% of Cochrane and 70% of non-Cochrane studies with the remaining plots organized by outcome variables. (Table S1, Supplementary data are available at IJE online).

Number of studies in forest plots

a This column reports non-Cochrane results with one large, atypical paper 12 removed. See text.

b We randomly sampled 500 plots from the 2137 Cochrane plots.

All Cochrane review plots displayed individual study results in alphabetical order, either by first author last name or study acronym ( Table 3 ). In contrast, 46% of non-Cochrane review plots displayed study results by year of publication and a smaller number sorted the study results by effect size, and sample size. Ratio measures were the predominant reported outcome and most were scaled between either 0.01 and 100 or 0.1 and 10. Cochrane review plots were always scaled symmetrically whereas non-Cochrane review plots used a variety of scales. A logarithmic scale was used for all plots of ratio measures. In all Cochrane plots the symbol used to indicate the estimated effect size (e.g. mean, relative risk) for each study was sized to reflect that study’s weight. A total of 17 of 28 (61%) non-Cochrane papers (49% of all plots) had sized symbols.

Characteristics of forest plots

a When the one atypical article 12 with 31 plots is removed, ratio measures account for 56% and difference measures for 40% of the remaining 108 studies.

The typical Cochrane review plot includes: (i) a title that states the research question, the comparison being made and the outcome measure; (ii) a description of each study including author last name, publication year, the n / N (binary outcome) or N (continuous outcome) for each group, the point estimate and 95% CI both numerically and as a graphic, and the weight that the study was given if meta-analysis was performed; (iii) for each meta-analysis—a summary diamond, a pooled estimate of the outcome and its CI, the total N for binary measures, a test of heterogeneity and a test for overall effect; and (iv) a scale for the forest plot with labels indicating which direction favours one group or the other ( Table 4 and Figure 1 ). With the exception of one plot that did not indicate which direction favoured the treatment group, all Cochrane review plots contained all these elements except those listed in (iii) above when meta-analysis was not performed. The non-Cochrane review plots were less standardized in this regard, with roughly half of the plots missing many of the elements outlined above ( Table 4 ). In particular, the majority did not include the summary results of each of the studies depicted in the plot.

Items reported in forest plots

SD, Standard deviation.

The majority of Cochrane review plots presented summary diamonds—graphical representations of the summation of the findings of the individual studies derived from meta-analytic techniques (Table S1, Supplementary data are available at IJE online). Summary diamonds were presented in 70% of plots that had only one study and were used more judiciously in non-Cochrane review plots where they were seldom provided unless there were several studies to be combined. Of the 28 non-Cochrane papers with forest plots, 12 (43%) did not state the statistical method by which the summary diamond was created. Of the other 16 articles, 8 (50%) stated the method used random effects, 5 (31%) used fixed effects, 1 (6%) had some plots that used fixed and others random effects and 1 (6%) reported two summary diamonds, one for each method. In the 115 Cochrane reviews 64% used fixed effects, 17% random effects, 15% had some plots for which each method was used and 4% used neither. The ratio of fixed effects to random effects meta-analytic techniques was roughly 4:1 in Cochrane review plots and 3:4 in non-Cochrane review plots.

Plots in all Cochrane and 8 (28%) of the 28 non-Cochrane reviews that had plots appear to have been generated in Review Manager (RevMan), available at the Cochrane website ( http://www.cc-ims.net/RevMan ). Only 8 of the 28 non-Cochrane reviews stated the software used to create the plots (6 RevMan, 2 Stats Direct), although we can assume, based on forest plot style and the software used for the statistical analysis, that some authors used Stata and StatXact. In 18 of 28 papers it was not clear to us the software that was used to make the plot, and in 11 of 28 papers the authors neither stated the software used to make the plot nor the software used to perform the analyses.

Forest plots are a concise graphical way of summarizing the quantitative findings of a systematic review. Such plots are informative whether they contain a summary diamond from a meta-analysis of the included study results or just present the results of individual studies. Our cross-sectional study reveals several important findings. First, authors of Cochrane reviews generally follow a recipe whereby forest plots are created based on the existence of a question rather than the availability of data. As a result, all Cochrane reviews had forest plots, but 10% contained no data and >65% contained just one or two studies. While these sparsely populated plots certainly emphasize that “more research is needed,” plots with 0 or 1 studies serve no other purpose and the message that data are sparse could be made more efficiently. Of note, the 2008 version of the ‘ Cochrane Handbook for Systematic Reviews of Interventions ’ states: ‘Forest plots should not be generated that contain no studies, and are discouraged when only a single study is found for a particular outcome’. 13 The paucity of information contained in forest plots with sparse data is exacerbated when these plots present summary diamonds that ‘summarize’ one study. The information in these diamonds is redundant and may falsely inflate readers’ assessments of the amount of available information.

The majority (84%) of non-Cochrane reviews did not contain even one forest plot. Although it is possible that the authors of some reviews had ample data from many studies but naively did not know to include such a plot, there was little evidence of this. More commonly, when data were sparse and the plot would have contained less than three studies, authors of non-Cochrane reviews wisely decided to omit the plot. As a result, when non-Cochrane reviews did have plots these plots tended to be richer; they contained sufficient papers to make the plot interesting and helpful.

Only 3% of both Cochrane and non-Cochrane reviews presented funnel plots. Some authors believe that when there are a sufficient number of studies (e.g. 10 or more), funnel plots can be very useful for detecting asymmetry, perhaps suggesting publication bias, 10 , 14 whereas others argue that funnel plots are not particularly helpful. 15 , 16 Regardless, our study suggests that they are reported infrequently.

Scientists use graphics to communicate because graphics can have far greater data density and multidimensionality than text. 17 Although the horizontal dimension of each forest plot represents the magnitude and precision of each study’s result, our data demonstrate that authors’ use of forest plots does not exploit the vertical dimension. All Cochrane review plots and 44% of non-Cochrane review plots presented studies in either alphabetical order or some other order that had little potential for illuminating the meaning of the data. This is unfortunate as one of the main benefits of a systematic review is the opportunity to explore why study results differ from one another. 18 For example, when ordered by year of publication, forest plots can reveal trends related to changing technologies (early studies of computerized tomography will have lower sensitivity than more recent ones because scan resolution has improved). They can also be useful when cumulative meta-analysis is performed or to show how beliefs are modified by the addition of new data to an existing meta-analysis (i.e. updating systematic reviews). 2 , 19 Ordering by effect size can aid in the detection of heterogeneity, and ordering on sample size or some analogue (e.g. study weight) can aid in the detection of publication bias (in a manner similar to funnel plots). When ordered by a characteristic of the studies (e.g. dosage used, or severity of illness of the subjects, risk of bias), plots may reveal patterns that would otherwise go unobserved ( Figure 2 ). It is therefore unfortunate that alphabetical order, which wastes the vertical dimension, predominates. We found no directions regarding this concern in the 2008 Cochrane Handbook . 13

Effect of ordering on appearance of forest plot. ( A ) Ordered by author last name. ( B ) Ordered by dose. These data, adapted from Annane et al. 23 , are shown ordered by author last name and the dose employed by each study. Within each dosage level, studies are ordered by effect size. From the bottom panel we easily see that there is a suggestion that dose is an effect modifier. This is not apparent in the upper panel.

All Cochrane review forest plots were created using RevMan software, which ensured that they appeared in a standard format. The benefit of this strategy is homogeneous-appearing forest plots that contain all the desired elements ( Table 4 ). A downside of this approach is that these elements appear even when they are wholly irrelevant. In contrast, authors of non-Cochrane reviews employed a variety of methods to create their plots. As a result, their plots are not standardized and many omit important information ( Table 4 ). However, these plots seldom display nonsense graphics (e.g. meta-analyses of single studies).

Our findings provide a baseline from which authors, peer reviewers and editors can contemplate how to further improve the information content and organization of forest plots. The most obvious first steps are listed in Figure 3 . We emphasize that authors should: (i) only use forest plots when there are sufficient studies to make them of value; (ii) ensure that plots contain all the important elements; and (iii) exploit the plot’s vertical dimension by ordering studies in a way that might illustrate important differences among them, such as by year of publication, effect size or important study characteristic.

Suggestions for making high-quality forest plots. a Median (IQR) may be more appropriate when data are skewed though there is no widely used mechanism for pooling data in this form.

We also encourage authors to consult item 21 of the PRISMA Statement 20 [‘For all outcomes considered (benefits or harms) present, for each study: (i) simple summary data for each intervention group, (ii) effect estimates and confidence intervals, ideally with a forest plot’], and the accompanying PRISMA explanation document. 21 That paper provides examples of ‘good reporting’ including use of tables and graphics to present the results of systematic reviews, along with an explanation and evidence, when available, for reporting this information.

This work was supported in part by an unrestricted grant from the Korein Foundation to D.L.S. Cancer Research UK to D.G.A. and University of Ottawa Research Chair to D.M.

The authors thank Jennifer Tetzlaff for her help with file management, and Jessica Brown, Louis Muscarella and Gina Pang for their help in conducting this research.

Plots in Cochrane reviews were standardized but often contained little data (80% had 3 or fewer studies; 10% had no studies) and always presented studies in alphabetical order.

Non-Cochrane plots depicted a larger number of studies (60% had 4 or more studies) and 59% ordered studies by a potentially meaningful characteristic, but important information was often missing.

We emphasize that authors should: 1) only use forest plots when there are sufficient studies to make them of value 2) ensure that plots contain all of the important elements, and 3) exploit the plot's vertical dimension by ordering studies in a way that might illustrate important differences among them, such as by year of publication, effect size, or important study characteristic.

Google Scholar

Google Preview

  • graphical displays
  • cochrane collaboration

Supplementary data

Email alerts, citing articles via, looking for your next opportunity.

  • About International Journal of Epidemiology
  • Recommend to your Library

Affiliations

  • Online ISSN 1464-3685
  • Copyright © 2024 International Epidemiological Association
  • About Oxford Academic
  • Publish journals with us
  • University press partners
  • What we publish
  • New features  
  • Open access
  • Institutional account management
  • Rights and permissions
  • Get help with access
  • Accessibility
  • Advertising
  • Media enquiries
  • Oxford University Press
  • Oxford Languages
  • University of Oxford

Oxford University Press is a department of the University of Oxford. It furthers the University's objective of excellence in research, scholarship, and education by publishing worldwide

  • Copyright © 2024 Oxford University Press
  • Cookie settings
  • Cookie policy
  • Privacy policy
  • Legal notice

This Feature Is Available To Subscribers Only

Sign In or Create an Account

This PDF is available to Subscribers Only

For full access to this pdf, sign in to an existing account, or purchase an annual subscription.

  • Open access
  • Published: 19 February 2024

Effects of game-based physical education program on enjoyment in children and adolescents: a systematic review and meta-analysis

  • Weilong Mo 1 , 2 ,
  • Jamalsafri Bin Saibon 1 ,
  • Yaming LI 1 ,
  • Jiequan Li 3 &
  • Yanwu He 4  

BMC Public Health volume  24 , Article number:  517 ( 2024 ) Cite this article

231 Accesses

Metrics details

The objective of this study was to conduct a systematic review to summarize and assess the advancements lately made on the enjoyable impacts of game-based physical education interventions on children and adolescents. Additionally, it attempted to identify the effects and variables influencing the enjoyable outcomes of children and adolescents’ engagement in physical education games, through meta-analysis.

This study involves a comprehensive search of different databases like Web of Science, PubMed, Embase, EBSCOhost, Cochrane, and Scopus. Specific criteria are established for the selection process to make sure the relevant literature included. The quality assessment of the included researches is conducted based on the guidelines outlined in the Cochrane 5.1 handbook. Review Manager 5.3 software is employed to synthesis the effect sizes. Additionally, bias is assessed using funnel plots, and to identify potential sources of heterogeneity, subgroup analyses are performed.

A total of 1907 academic papers, out of which 2 articles were identified via other data sources. The present study examined the impact of a pedagogical intervention involving physical education games on the enjoyment experienced by children and adolescents. The results indicated a significant positive effect (MD = 0.53, 95%CI:[0.27,0.79], P  < 0.05) of this intervention on enjoyment. Subgroup analyses further revealed that both boys (MD = 0.31, 95%CI:[0.13,0.50], P  < 0.05) and girls (MD = 0.28, 95%CI:[0.05,0.51], P  < 0.05) experienced increased pleasure compared to traditional physical education. Additionally, children under 12 years of age (MD = 0.41, 95%CI:[0.17,0.64], P  < 0.05) benefited from sessions lasting at least 30 minutes or more per session (MD = 0.40, 95%CI:[0.19,0.60], P  < 0.05), occurring 1 to 3 times per week (MD = 0.28, 95%CI:[0.16,0.40], P  < 0.05), and lasting for more than 3 weeks (MD = 0.81, 95%CI:[0.29,1.34], P  < 0.05). These findings suggest that the implementation of physical education games can be an effective approach to teaching this subject.

Conclusions

1) Interventions using physical games have been shown to yield beneficial outcomes in terms of enhancing the enjoyment experienced by children and adolescents. 2) The effectiveness of treatments aimed at promoting enjoyment among children and adolescents is influenced by several aspects, including gender, age, duration and frequency of physical activity, as well as the specific cycle of activity used.

Peer Review reports

Introduction

Enjoyment is a subjective experience with pleasant emotions, such as pleasure, like, and fun [ 1 ]. Children and adolescents are naturally motivated by enjoyable experiences during the learning process, which enhances their academic achievement, involvement and effort in learning [ 2 , 3 , 4 ], this, in turn, leads to more effective and long-lasting learning [ 5 , 6 ]. In contrast, falling enjoyment can diminish their interest and engagement. Therefore, the cultivation of enjoyable feelings in children and adolescents has a crucial role in enhancing educational achievements.

Studies proved that physical education has a beneficial influence on the psychological and physical health of children and adolescents [ 7 , 8 ], as well as on the prevention of disease problems [ 9 ]. The influence of physical education on the enjoyment of children and adolescents, particularly in relation to emotions, has clear benefits [ 10 ]. Physical education activities for children and adolescents not only have the power to enhance individual happiness, but also foster a positive team atmosphere and promote collaboration and socialization [ 11 ]. Thus, it is essential to explore the positive impact of these activities on enjoyment and mental health.

Compared with traditional physical education, physical education games have several advantages. The implementation of physical play interventions has the potential to facilitate the acquisition of knowledge and skills among children and adolescents [ 12 , 13 ], enabling them to get enjoyment from the process of learning. A combination of entertainment components into traditional physical education (PE) is effective in motivating non-athletic students to actively engage in PE lessons, which cannot be achieved through organized sports [ 14 ]. Liao et al. [ 15 ] further explain that games not only enhance students’ satisfaction with PE lessons, but also facilitate skill development, create a relaxed play environment, foster interpersonal interactions, and offer opportunities for cooperation and socialization.

Hence, the implementation of physical games teaching offers a new and innovative approach within the context of traditional physical education classes [ 16 ]. This approach to learning is not only pleasurable for students, but also meets their requirement for social and physical engagement in the educational process and, most notably, contributes to a key part in sustaining the involvement of children and adolescents in physical education and sports [ 17 ].

Through a comprehensive analysis of 16 academic studies, it has been noticed that further research is needed regarding the efficacy of applying physical education games to enhance enjoyment in children and adolescents. Since the majority of studies show a beneficial effect on enjoyment [ 1 , 18 , 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 ]. However, five studies remain uncertain about the impact of these games [ 14 , 27 , 28 , 29 , 30 ], and there is even a case where teaching with games seems to reduce students’ enjoyment [ 27 ]. Currently, there is not enough comprehensive analysis on how physical game teaching impacts the enjoyable feelings of children and adolescents, and also on the potential factors that may influence those effects (such as gender, age, and duration of the interventions).

The aim of this study was to explore the following topics with a meta-analysis: 1) whether teaching games in physical education has a beneficial influence on enjoyment experienced by children and adolescents, and 2) how other essential elements mitigate the influence of games teaching in physical education on enjoyment experienced by children and adolescents. Due to inconsistent findings from earlier research, there is no consensus on the link of physical games for enjoyment in children and adolescents; yet there is optimism that teaching physical play may have a favorable impact on enjoyment.

Search strategy and standards for selection

This research is conducted under the guidance of Cochrane Handbook for the Systematic Review of Interventions [ 31 ] and the PRISMA Statement Specification for Systematic Review and Meta-analysis [ 32 ].

This research explores six databases, namely Web of Science, PubMed, Embase, EBSCOhost, Scopus and Cochrane. PubMed is primarily adopted to identify medical terms (Mesh). The search period spans from the initiation of database collection until July 25, 2023. Both subject terms and the free word approach are included in the search. The following table consists of many columns. This study firstly focuses on the research object and then emphasizes the intervention strategy. In the third line, the result index is built by connecting the search words using the logical operator “or” inside each group of search terms. Additionally, the search phrases are linked by “or” between each set of search terms. Table  1 displays the whole search words used in the six databases.

Criteria for inclusion and exclusion

Research that meet the following requirements are selected for systematic evaluation: (1) the intervention modality is physical game teaching; (2) the subjects are children and adolescents (3–18 years old); (3) the outcome indicator is the inclusion of enjoyment related MeSH and Entry Terms; (4) the use of a control group; and (5) the articles are written in English.

Researches reach the following standards are omitted from systematic evaluation: (1) the intervention method is not physical game teaching; (2) the experimental subjects are infants, adults, animals, and specific populations (psychiatric patients), etc.; (3) the outcome indicator is the absence of pleasure-related subject terms and free words; (4) no control group is included; (5) the articles are written in other non-English languages; (6) review article; and (7) conference articles.

Screening process

Upload the relevant literature to Endnote (version X9) for organization. Following this, duplicate results are screened by two authors (MWL and LYM) independently. The screening process include reviewing titles, review articles, conference papers, and animal experiments. Read the abstracts to exclude articles that fail to meet such criteria as study subjects or interventions. Finally, read the full text of selected articles to exclude those that are inaccessible, non-English and does not provide end point indicators. The process involves an initial screening of eligible articles, a discussion of any discrepancy and reaching a consensus with the third author (LJQ). Ultimately, 16 articles are selected for the systematic analysis. Detailed information about these steps are presented in the PRISMA flowchart (refer to Fig.  1 ).

figure 1

Flowchart for inclusion and exclusion of studies

Extraction of data and quality evaluation

Three authors (MWL, LYM, and LJQ) extract data from eligible papers in an impartial manner. Any divergence is resolved through discussion until an agreement is reached. The extracted information is then placed in the publications respectively [ 1 ]. the extracted information primarily includes the name of the first author and the publication year [ 2 ]. the subjects’ features encompass the total sample size, age, and gender [ 3 ]. detailed data, like duration, frequency, and cycle, about the teaching process of physical education and sport games are included [ 4 ]. the intervention tool employed in the study is a questionnaire or scale designed to measure the degree of pleasure, satisfaction and motivation before and after physical education [ 5 ]. the intervention items utilized in the study are game items specifically employed for physical education [ 6 ]. the outcome indicators encompass various factors, including the level of pleasure, satisfaction, and motivation before and after physical education and sport activities. Pleasure may be defined as the state of experiencing gratification, enjoyment, satisfaction, delight, or fun [ 7 ]. the writers emphasize the importance of significant results.

The Cochrane 5.1 handbook is applied to assess the quality of bias. The evaluation includes one aspect, namely the random allocation procedure and the concealing of allocation schemes. In this research, the evaluation criteria are: 3) participant blinding and outcome assessment, 4) complete outcome data, 5) selective report findings, and 6) bias from other sources. Each criterion is assessed as having either a low risk (an indication for meeting the criterion), a high risk (an indication for not satisfying the criterion), or a medium risk (if not mentioned), with a note explaining the reason for this assessment. Two researchers assesses the article quality independently. Any divergence in their evaluations are solved by discussing with another author. Figure  2 shows the evaluation findings and the detailed information. A sensitivity analysis is conducted whereby each article is systematically removed one at a time. The analysis reveals that the findings are mostly unchanged, which suggests that the results are robust and reliable.

figure 2

Cochrane risk of bias evaluation

Statistical analysis

Review Manager 5.3, a statistical software, is applied to merge effect sizes and assess for bias. In this analysis, the indicator continuous variable is incorporated, making the results presented as mean ± standard deviation (Mean ± SD). The I 2 and Q tests are adopted to evaluate heterogeneity between studies. The fixed-effects model is applied as the I 2  < 50% or p  > 0.1, indicating the studies lack statistical heterogeneity. In contrast, a random-effects model is adopted to evaluate publication bias using funnel plots and to examine the reliability of the findings.

Search results

A comprehensive search for 1907 articles in total is conducted. The databases for search are Web of Science (1473 articles), PubMed (33 articles), Embase (75 articles), EBSCO host (154 articles), Cochrane (95 articles), and Scopus (75 articles). All identified articles are uploaded to Endnote (version X9), a reference management software. By examining the article titles, a total of 207 duplicate items are eliminated from further analysis. The articles are uploaded to Endnote (version X9), and after examining the titles, a total of 207 duplicates are removed. The dataset comprises 321 conference papers, 44 review articles, 31 articles with inconsistent subjects, 1 article applying inconsistent measurement tools, 94 articles employing inconsistent interventions, 66 articles featuring different endpoints as determined after reading the full text. 14 articles presenting results deviate from the mean ± standard deviation format. A total of 27 publications without a control group are identified, and 8 articles written in non-English languages are excluded. Additionally, the full texts of 26 articles are incomplete or unavailable. Hence, a final set of 16 eligible articles is included in the meta-analysis.

Basic features of the included articles

The analysis encompasses 16 articles, which collectively examines 17 studies. The total sample size has 2181 participants, with 1139 individuals assigned to the experimental group and 1042 individuals assigned to the control group. There were 1096 male participants and 1048 female ones. The age of the samples ranges from 4.9 to 15.62 years. The intervention duration covers from 15 to 90 minutes, with frequencies from 1 to 9 times each week. The intervention cycles span from 2 to 14 weeks. The interventions are concentrated on sports and games programs. The assessed outcome indicators include enjoyment and satisfaction, intervention instruments and major findings. See Table 2 for detailed information.

Quality assessment

This research examines the literature about the random assignment process and specifically focuses on six studies that meet the inclusion criteria [ 20 , 22 , 23 , 26 , 28 , 33 ]. The remaining research do not provide details about the randomization process. None of the 17 studies mention whether the allocation is concealed or not in the allocation scheme concealment. In terms of blinding, researchers on the subjects and inform them about the tests. Thus, the subjects are not blinded. Consequently, all 17 studies are deemed to have a high risk. The evaluation of the findings is featured with uncertainty. Two studies had a high incidence of staff turnover as for the completeness of the outcome data [ 20 , 27 ]. None of the 15 studies shows any subject or data loss, and all of them are considered to have low risk. The included studies show no further selective reporting or biases, and all of them are considered to have low risk of bias.

Tests for bias

This research includes outcome indicators for analysis, and the funnel plot demonstrates a distribution that is symmetrical, indicating the absence of publication bias, as seen in the Fig.  3 .

figure 3

Bias funnel plot

Efficacy tests

The relationship between teaching games in physical education and enjoyment of children and adolescents.

Heterogeneity tests were performed on the articles that were included in the analysis. Out of the total, 17 research (comprising 16 papers) indicated an altered association between the enjoyment experienced by children and adolescents in the context of teaching physical education games [ 1 , 14 , 18 , 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 , 27 , 28 , 29 , 30 , 33 ]. The researchers apply a random effects model to collect the findings about the articles’ outcome indicators. This study includes 17 studies and 2181 participants in total, with 1042 in the control group and 1139 in the experimental group. The present research provides evidence supporting the favorable impact of a physical education intervention using games on the positive emotions of children and adolescents in the experimental group (MD = 0.53, 95% CI: [0.27, 0.79], P  < 0.05), as depicted in Fig.  4 .

figure 4

Forest plot depicting the relation between physical game teaching and enjoyment in children and adolescents

Subgroup analyses

The combined impact size data for physical play teaching interventions on children and adolescents show a significant degree of variation. It is achieved by analyzing subgroups based on gender, age, duration, frequency, and cycling as potential influencing factors.

The results of subgroup analyses examining the influence of gender, age, duration, frequency, and cycling on the effects of games in physical education indicate that such games can enhance enjoyment of boys (MD = 0.31, 95% CI:[0.13,0.50], P  < 0.05) and positively affect girls (MD = 0.28, 95%CI:[0.05,0.51], P  < 0.05). Furthermore, it is observed that children aged 12 experienced an increasing enjoyment (MD = 0.41, 95% CI:[0.17,0.64], P  < 0.05), whereas adolescents aged 12 and above do not show a similar increase ( P  > 0.05). The duration of physical education sessions ranging from 30 to 60 minutes (MD = 0.40,95%CI:[0.19,0.60], P  < 0.05) can provide a favorable impact on enjoyment experienced by children and adolescents. Moreover, extending the duration of physical education beyond 60 minutes (MD = 0.55,95%CI:[0.10,1.00], P  < 0.05) may also improve their enjoyment. However, noticeably, durations shorter than 30 minutes do not show the same good effect ( P  > 0.05). It is more feasible to provide physical game teaching within a frequency range of 1 to 3 sessions per week (MD = 0.28,95%CI:[0.16,0.40], P  < 0.05) to elicit enjoyment among children and adolescents. Conversely, it is unsuitable to give physical game instructions, exceeding the threshold of 3 sessions per week ( P  > 0.05). The optimal duration for physical game teaching to elicit enjoyable outcomes in children and adolescents is between 3 to 6 weeks (MD = 0.81, 95%CI:[0.29,1.34], P  < 0.05), but durations beyond 6 weeks are also considered acceptable (MD = 0.29, 95%CI:[0.10,0.48], P  < 0.05). In contrast, it is not a proper option to be engaged in physical games for less than 3 weeks ( P  > 0.05). Hence, such factors as gender, age, duration, frequency, and cycle contribute significantly to the observed variation in satisfaction, as seen in Figs.  5 , 6 , 7 , 8 , 9 .

figure 5

Forest plot depicting gender subgroup relationship between physical game teaching and enjoyment in children and adolescents

figure 6

Forest plot depicting age subgroup relationship between physical game teaching and enjoyment in children and adolescents

figure 7

Forest plot depicting duration subgroup relationship between physical game teaching and enjoyment in children and adolescents

figure 8

Forest plot depicting frequency subgroup relationship between physical game teaching and enjoyment in children and adolescents

figure 9

Forest plot depicting cycle subgroup relation between physical game teaching and enjoyment in children and adolescents

Main research analyses

The results of this study, including the analysis of 17 studies, show that the adoption of physical education game-based intervention has a beneficial effect on the enjoyment levels of children and adolescents. Such corresponds to the idea offered by Tornero and Capella, 2017, which claims that playing games adjusts to the emotional state of children and adolescents [ 34 ]. This advantageous feeling state can further improve their engagement in school sports activities [ 35 , 36 ]. Physiological studies prove that engaging in physical activity or exercise causes the release of endorphins from the pituitary gland and subthalamus. Endorphins are hormones that induce feelings of calmness and pleasure, enhancing mood and creating an enjoyable experience for children and adolescents during physical education programs, including games [ 37 , 38 , 39 ]. Furthermore, when it comes to content, the teaching of physical education games appears to enhance the enjoyment experienced by children and adolescents to a greater extent than the classes of traditional physical education. In their study, Batez et al. (2021) discovered that adolescents in the experimental group who participated in the Teaching Games for Understanding (mini-volleyball) way experienced a greater sense of satisfaction compared to the control group during the post-test phase [ 23 ]. Lopez-Lemus et al. (2023) noticed that analyzing the pre-test and post-test results of both the experimental and control groups revealed that 67 students who were part of the Sport Education (SE)/Teaching for understanding (TGfU) experimental group, specifically focusing on handball, revealed enhancements in-game performance, enjoyment, perceptual skills, and intentions [ 21 ]. Similarly, researches on dance movement games and basketball games show superior levels of enjoyment compared to traditional teaching methods [ 19 , 40 ]. Hence, this research posits that including games into physical education courses may effectively enhance the enjoyment of children and adolescents, making it a recommended approach compared to programmes that do not use games.

Gender analysis

This study claims that practicing physical sports might affect enjoyment among individuals of different genders, with boys expressing a greater chance of experiencing enjoyment compared to girls. Research has shown that as they get older boys and girls display unique preferences. In a cultural analysis conducted by Joseph et al., 25 African American women were surveyed regarding their engagement in physical activities. The majority of these women reported positive and enjoyable experiences in childhood, but their feelings were not apparent during their youth [ 41 ]. Additionally, female adolescents had a lower frequency of pleasurable meets in physical exercise compared to male adolescents, and they also displayed negative emotions towards engaging in physical activity [ 42 ]. Even so, variations in the level of enjoyment based on gender are likely to be impacted by different types of sports games. Girls have a preference for cooperative activities, particularly dancing games [ 43 ], whereas boys seem to choose competitive fitness games [ 44 ]. In all, both males and females can experience enjoyment in physical education games, still, variations in the level of enjoyment may arise due to factors such as age and the specific type of game. It seems that gender alone is not the sole determinant of enjoyment, and further study is required to identify other contributing factors.

Analysis of age

According to this study, teaching physical education games has a major effect on the enjoyable feelings of children below the age of 12. And yet, it does not have a substantial influence on teenagers aged 12 and above. In the opinion of Velez & Garcia, children between the ages of 9 and 12 have better levels of individual feelings of happiness compared to teenagers aged 13 to 17 [ 45 ]. Play is an essential element in the development of children’s motor skills and is intrinsically linked to enjoyment, which serves as a motivation for children to engage in physical exercise [ 46 ]. According to Bremer et al., a study demonstrated that children between the ages of 6 and 13 with autism who enjoyed their physical education sessions were more likely to engage in other physical activities [ 47 ]. Academic competition at school is an important factor that hinders the development of enjoyable feelings in teens, this is mostly caused by the negative effects of stress-induced depression and anxiety [ 48 ]. Mangerud found that engaging in physical exercise has an impact on the positive emotions of adolescents with anxiety disorders, including their enjoyment of sports circumstances [ 49 ]. As a result, teaching youngsters under the age of 12 physical sports proves to be a more successful method for obtaining enjoyment compared to teenagers aged 12 and above.

Analysis of duration

The present research offers that applying a physical game lesson beyond a duration of 30 minutes has a favorable impact on the enjoyment of children and adolescents, but less than 30 minutes appears to have little to no effect. This corresponds to the findings of the Gil-Madrona, when children participated in 45 minutes of popular cooperative and cooperative-oppositional games [ 50 ]. Physical exercise in children and teenagers increases the release of neurotransmitters including dopamine and (−)-norepinephrine. These substances help to decrease depression and anxiety, leading to increased feelings of euphoria, achievement, and overall well-being, which improve over time [ 51 , 52 , 53 ]. Previous research has shown that children tend to get pleasure from short periods of intense physical activity followed by times of relaxation (similar to outside play), whereas adults may have a preference for lengthier activities [ 54 ]. However, Tobin et al. carried out experiments with participants of varying durations and determined that a 12-minute length of time was considered insufficient for players to become fully engaged in the game, thus serving merely as a warm-up period [ 55 ]. In addition, another study corroborated these findings by establishing that children exhibited diminished motivation and failed to experience enhanced enjoyment when engaging in sports games for a brief duration of 20 minutes [ 56 ]. In the end, engaging in sports games for no less than 30 minutes can lead to improved outcomes and heightened enjoyment for children and adolescents.

Analysis of frequency

The analyses suggested that a teaching intervention based on physical education and games held 1 to 3 times per week is suitable for children and adolescents to experience enjoyment. However, doing more than 3 sessions per week seems unsuitable. Studies indicate a correlation between the frequency of participating in physical activity and experiencing positive emotions [ 57 ]. Furthermore, sustaining a proper frequency of physical activity could promote the feeling of Feelings of happiness. For instance, Batez and Gil-Arias both applied the teaching games for understanding (TGfU) approach in a physical education program, results indicated that students’ level of enjoyment somewhat improved when the games were taught twice weekly [ 23 , 58 ]. However, excessive participation in game activities without sufficient time for rest and recovery can lead to the build-up of lactic acid in the muscles, resulting in increased physical fatigue and negatively impacting the individual’s mood, finally diminishing the enjoyment of the gaming experience [ 59 , 60 , 61 , 62 ]. Temporary breaks can effectively facilitate physical recovery during physical education games, it not only promotes bodily rejuvenation but also enhances the enjoyment of children and adolescents [ 63 ]. Therefore, it is advisable to offer 1–3 lessons per week to optimise the teaching of physical education games.

Analysis of cycles

This study stated that physical education game teaching interventions lasting between 3 to 6 weeks and more than 6 weeks are ideal for improving the enjoyable outcomes of children and adolescents. Conversely, interventions lasting less than 3 weeks are not advisable. This conclusion is supported by the findings of previous studies. Some curriculum interventions like Zetou et al. designed a 4-week ‘Play and Stay’ tennis teaching programme, Jones et al.’s 6-week Teaching Games for Understanding and Fernandez-Rio et al.’s 9-week Gamification [ 22 , 24 , 26 ]. Findings show an increase in students’ enjoyment, linked to the regular meeting of their intrinsic drive in the physical education classroom. Several studies indicated that short physical procedures lasting only 1 week do not effectively assess the intrinsic motivation of participants [ 64 , 65 ], thus posing difficulties with stimulating the generation of enjoyable sensations in persons. Moreover, extended periods of physical play may result in decreased intrinsic motivation or boredom in children and adolescents [ 66 , 67 ]. Fu et al. and Zeng et al. propose that while physical play at first brings joy, enjoyment diminishes over time [ 33 , 68 ]. In conclusion, children and adolescents should engage in playful activities for a minimum of 3 weeks, while also ensuring that the play program offers a variety of activities and rich content to enhance their enjoyment.

This study applies a meta-analysis to examine the significance of teaching games in physical education regarding emotional delight experienced by children and adolescents. Gender, age, duration, frequency and cycles may be the reasons for variances impacting the research outcomes. This research finds that male participants are more likely to show enjoyable behavior compared with their female counterparts as for games teaching in physical courses. However, it should be noted that gender disparities may be influenced by variables like age and the specific kind of sports taught in class. Besides,Children engage in a minimum of 30 minutes every session, attending 1 to 3 sessions per week, so guaranteeing that the physical education and games curriculum is delivered for a span exceeding 3 weeks. This approach aims to foster positive affective experiences among children, thereby facilitating the attainment of optimum outcomes.

Limitations and future research

Apart from the meaningful findings, this research also has some drawbacks. Firstly, it adopts a meta-analytical approach to examine the influence of games teaching in physical education on enjoyment in children and adolescent. It primarily focuses on the outcomes of curriculum and teaching implementation. Consequently, the results may not be applicable to other contexts, such as after-school physical game activities, community physical game activities, and family physical game activities. Furthermore, the 17 studies analyzed in this research have insufficient data on duration, frequency, and period. This insufficient information may influence the statistical accuracy of conducting effect size tests. Additionally, the 17 studies fail to offer any data about the intensity of the activities employed in games teaching in physical education, such as heart rate, oxygen uptake, and respiratory rate. Consequently, future studies can address this gap in knowledge. Additionally, the current research does not ascertain the ideal upper threshold for the duration of engagement in the activity. This aspect warrants further exploration in a later literature review. Additionally, it is crucial to evaluate other variables which may influence the research outcomes, such as the specific nature of the sports game being analyzed. For further study, it would be fruitful to classify different sorts of sports games to improve the whole quality of the research.

Availability of data and materials

All data generated or analysed during this study are included in this published article.

Gao Z, Zhang T, Stodden D. Children’s physical activity levels and psychological correlates in interactive dance versus aerobic dance. J Sport Health Sci. 2013;2(3):146–51.

Article   Google Scholar  

Sharif AF, Kasemy ZA, Alquraishi NA, Alshaikh KN, Alfaraj AH, Alibrahim EA, et al. Inserting an erroneous element in the answer key of grouped team readiness assurance test in team-based learning: a randomized controlled trial in clinical toxicology. Adv Med Educ Pract. 2022;13:865–82.

Izadpanah S. The mediating role of academic passion in determining the relationship between academic self-regulation and goal orientation with academic burnout among English foreign language (EFL) learners. Front Psychol. 2022;13:1–12.

Google Scholar  

Zhang C, Mao L, Li N, Gu X. Chinese EFL Students’ social-emotional competence, grit, and academic engagement. Front Psychol. 2022;13:914759.

Trajkovik V, Malinovski T, Vasileva-Stojanovska T, Vasileva M. Traditional games in elementary school: relationships of student’s personality traits, motivation and experience with learning outcomes. PLoS One. 2018;13(8):e0202172.

Yu J, Huang C, Han Z, He T, Li M. Investigating the influence of interaction on learning persistence in online settings: moderation or mediation of academic emotions? Int J Environ Res Public Health. 2020;17(7):2320.

Yang J, Chen M. Construction of sports and health data resources and transformation of teachers’ orientation based on web database. J Healthc Eng. 2022;2022.

Triaca LM, Frio GS, França MTA. A gender analysis of the impact of physical education on the mental health of brazilian schoolchildren. SSM-Popul Health. 2019;8:100419.

Benka Wallén M, Franzén E, Nero H, Hagströmer M. Levels and patterns of physical activity and sedentary behavior in elderly people with mild to moderate Parkinson disease. Phys Ther. 2015;95(8):1135–41.

Fin G, Moreno-Murcia JA, León J, Baretta E, Nodari Júnior RJ. Teachers’ interpersonal style in physical education: exploring patterns of students’ self-determined motivation and enjoyment of physical activity in a longitudinal study. Front Psychol. 2019;9:2721.

Lim Y, Han A, Lee M, Kim M. The effects of an online-offline hybrid exercise program on the lives of children with cerebral palsy using wheelchairs during the COVID-19 pandemic in Korea. Int J Environ Res Public Health. 2022;19(12).

Simón-Piqueras JÁ, Cano-Noguera F, Zamorano-García D, Gil-Madrona P. Efecto de un programa de iniciación a la natación basado en juegos motores acuáticos sobre la autoestima corporal y socio afectiva en niñas y niños de 5 y 6 años de edad. Rev Electrón Interuniv Form Profr. 2023;26(3):75–88.

Simon-Piqueras JA, Prieto-Ayuso A, Gomez-Moreno E, Martinez-Lopez M, Gil-Madrona P. Evaluation of a program of aquatic motor games in the improvement of motor competence in children from 4 to 5 years old. Children (Basel, Switzerland). 2022;9(8).

Azlan AB, Ismail N, Fauzi NFM, Abd TR. Playing traditional games vs. free-play during physical education lesson to improve physical activity: a comparison study. Pedagogy Phys Cult Sports. 2021;25(3):178–87.

Liao CC, Hsu CH, Kuo KP, Luo YJ, Kao CC. Ability of the sport education model to promote healthy lifestyles in university students: a randomized controlled trial. Int J Environ Res Public Health. 2023;20(3).

Vaghetti CAO, Monteiro-Junior RS, Finco MD, Reategui E, da Costa S, Botelho S. Exergames experience in physical education: a review. Phys Cult Sport Stud Res. 2018;78(1):23–32.

Baysden E, Mendoza N, Callender C, Deng Z, Thompson D. Teen reactions to a self-representational avatar: a qualitative exploration. J Sport Health Sci. 2022;11(2):157–63.

Segura-Robles A, Fuentes-Cabrera A, Parra-Gonzalez ME, Lopez-Belmonte J. Effects on personal factors through flipped learning and gamification as combined methodologies in secondary education. Front Psychol. 2020;11.

Gil-Arias A, Harvey S, García-Herreros F, González-Víllora S, Práxedes A, Moreno A. Effect of a hybrid teaching games for understanding/sport education unit on elementary students’ self-determined motivation in physical education. Eur Phys Educ Rev. 2020;27(2):366–83.

Kolovelonis A, Goudas M. The effects of cognitively challenging physical activity games versus health-related fitness activities on Students’ executive functions and situational interest in physical education: a group-randomized controlled trial. Eur J Investig Health Psychol Educ. 2023;13(5):796–809.

Lopez-Lemus I, Del Villar F, Rodriguez-Gutierrez A, Gonzalez-Silva J, Moreno A. Could the hybridization of the SE/TGfU pedagogical models be an alternative for learning sports and promoting health? School context study. Children-Basel. 2023;10(5).

Fernandez-Rio J, Zumajo-Flores M, Flores-Aguilar G. Motivation, basic psychological needs and intention to be physically active after a gamified intervention Programme. Eur Phys Educ Rev. 2022;28(2):432–45.

Batez M, Petrusic T, Bogataj S, Trajkovic N. Effects of teaching program based on teaching games for understanding model on volleyball skills and enjoyment in secondary school students. Sustainability. 2021;13(2).

Jones RJA, Marshall S, Peters DM. Can we play a game now? The intrinsic benefits of TGfU. Eur J Phys Health Educ. 2010;4(2):57–63.

Navarro-Paton R, Basanta-Camino S, Gomez CA. Cooperative games: incidence in motivation, basic psychological needs and enjoyment in primary school. Sportis-Sci Tech J School Sport Phys Educ Psychomotricity. 2017;3(3):589–604.

Zetou E, Koronas V, Athanailidis I, Koussis P. Learning tennis skill through game play and stay in elementary pupils. J Hum Sport Exerc. 2012;7(2):560–72.

Quintas A, Bustamante JC. Effects of gamified didactic with exergames on the psychological variables associated with promoting physical exercise: results of a natural experiment run in primary schools. Phys Educ Sport Pedagog. 2023;28(5):467–81.

Engels ES, Freund PA. Effects of cooperative games on enjoyment in physical education-how to increase positive experiences in students? PLoS One. 2020;15(12):1–14.

Garcia-Castejon G, Camerino O, Castaner M, Manzano-Sanchez D, Jimenez-Parra JF, Valero-Valenzuela A. Implementation of a hybrid educational program between the model of personal and social responsibility (TPSR) and the teaching games for understanding (TGfU) in physical education and its effects on health: an approach based on mixed methods. Children (Basel, Switzerland). 2021;8(7):1–22.

Paton RN, Ferreiro MC, Nemina RE. Effect of a didactic unit based on competitive games in the motivation, basic psychological needs and enjoyment in primary education students. Sportis-Sci Tech J School Sport Phys Educ Psychomotricity. 2018;4(1):111–25.

Higgins JPT, Thomas J, Chandler J, Cumpston M, li TJ, Page MJ, Welch VA. Cochrane Handbook for Systematic Reviews of Interventions. 2nd ed. Cochrane Collaboration; 2019. p. 13–655.

Moher D, Shamseer L, Clarke M, Ghersi D, Liberati A, Petticrew M, et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Syst Rev. 2015;4(1):1.

Fu Y, Burns RD, Constantino N, Zhang P. Differences in step counts, motor competence, and enjoyment between an exergaming group and a non-exergaming group. Games Health J. 2018;7(5):335–40.

Tornero M, Capella C. Change during psychotherapy through sand play tray in children that have been sexually abused. Front Psychol. 2017;8:617.

Bates LC, Zieff G, Stanford K, Moore JB, Kerr ZY, Hanson ED, et al. COVID-19 impact on behaviors across the 24-hour day in children and adolescents: physical activity, sedentary behavior, and sleep. Children (Basel, Switzerland). 2020;7(9):138.

Gray S, Lorenzino L, Garcia BE. Not academic enough? Enjoyment of physical education and the arts and school engagement in early and middle adolescence. Retos. 2018;35:301–9.

Diggins AD, Hearn LE, Lechner SC, Annane D, Antoni MH, Whitehead NE. Physical activity in black breast cancer survivors: implications for quality of life and mood at baseline and 6-month follow-up. Psycho-Oncology. 2017;26(6):822–8.

Gitimoghaddam M, Vanderloo LM, Hung R, Ryce A, McKellin W, Miller A, et al. Impacts of participation in community-based physical activity programs on cognitive functions of children and youth with neurodevelopmental disabilities: a scoping review. Brain Sci. 2021;11(2):195.

Azevedo Da Silva M, Singh-Manoux A, Brunner EJ, Kaffashian S, Shipley MJ, Kivimaki M, et al. Bidirectional association between physical activity and symptoms of anxiety and depression: the Whitehall II study. Eur J Epidemiol. 2012;27(7):537–46.

Staiano AE, Marker AM, Beyl RA, Hsia DS, Katzmarzyk PT, Newton RL. A randomized controlled trial of dance exergaming for exercise training in overweight and obese adolescent girls. Pediatr Obes. 2017;12(2):120–8.

Article   CAS   Google Scholar  

Joseph RP, Ainsworth BE, Mathis L, Hooker SP, Keller C. Utility of social cognitive theory in intervention Design for Promoting Physical Activity among African-American women: a qualitative study. Am J Health Behav. 2017;41(5):518–33.

Bigman G, Rajesh V, Koehly LM, Strong LL, Oluyomi AO, Strom SS, et al. Family cohesion and moderate-to-vigorous physical activity among Mexican origin adolescents: a longitudinal perspective. J Phys Act Health. 2015;12(7):1023–30.

Gao Z, Zhang P, Podlog LW. Examining elementary school children’s level of enjoyment of traditional tag games vs. interactive dance games. Psychol Health Med. 2014;19(5):605–13.

Buys R, Claes J, Walsh D, Cornelis N, Moran K, Budts W, et al. Cardiac patients show high interest in technology enabled cardiovascular rehabilitation. BMC Med Inform Decis Mak. 2016;16:95.

Velez CM, Garcia Garcia HI. Medicion de la calidad de vida en ninos. Iatreia. 2012;25(3):240–9.

Pesce C, Masci I, Marchetti R, Vazou S, Saakslahti A, Tomporowski PD. Deliberate play and preparation jointly benefit motor and cognitive development: mediated and moderated effects. Front Psychol. 2016;7:349.

Bremer E, Martin Ginis KA, Bassett-Gunter RL, Arbour-Nicitopoulos KP. Factors associated with participation in physical activity among Canadian school-aged children with autism Spectrum disorder: an application of the international classification of functioning, disability and health. Int J Environ Res Public Health. 2020;17(16):5925.

Lupien SJ, McEwen BS, Gunnar MR, Heim C. Effects of stress throughout the lifespan on the brain, behaviour and cognition. Nat Rev Neurosci. 2009;10(6):434–45.

Mangerud WL, Bjerkeset O, Lydersen S, Indredavik MS. Physical activity in adolescents with psychiatric disorders and in the general population. Child Adolesc Psychiatry Ment Health. 2014;8(1):2.

Gil-Madrona P, Pascual-Francés L, Jordá-Espi A, Mujica-Johnson F, Fernández-Revelles AB. Affectivity and motor interaction in popular motor games at school. Apunts Educ Fís Deportes. 2020;139:42–8.

Liu H, Liang J, Wang K, Zhang T, Liu S, Luo J. Mood status response to physical activity and its influence on performance: are Chronotype and exercise timing affect? Int J Environ Res Public Health. 2023;20(4):2822.

Motaghinejad O, Motaghinejad M, Motevalian M, Rahimi-Sharbaf F, Beiranvand T. The effect of maternal forced exercise on offspring pain perception, motor activity and anxiety disorder: the role of 5-HT2 and D2 receptors and CREB gene expression. J Exerc Rehabil. 2017;13(5):514–25.

Lott SA, Burghardt PR, Burghardt KJ, Bly MJ, Grove TB, Ellingrod VL. The influence of metabolic syndrome, physical activity and genotype on catechol-O-methyl transferase promoter-region methylation in schizophrenia. Pharmacogenomics J. 2013;13(3):264–71.

Kashikar-Zuck S, Myer G, Ting TV. Can behavioral treatments be enhanced by integrative neuromuscular training in the treatment of juvenile fibromyalgia? Pain Manag. 2012;2(1):9–12.

Tobin S, Bisson N, Grondin S. An ecological approach to prospective and retrospective timing of long durations: a study involving gamers. PLoS One. 2010;5(2):e9271.

Article   ADS   Google Scholar  

Taylor RW, Farmer VL, Cameron SL, Meredith-Jones K, Williams SM, Mann JI. School playgrounds and physical activity policies as predictors of school and home time activity. Int J Behav Nutr Phys Act. 2011;8:38.

Li L, Ouyang F, He J, Qiu D, Luo D, Xiao S. Associations of socioeconomic status and healthy lifestyle with incidence of dyslipidemia: a prospective Chinese governmental employee cohort study. Front Public Health. 2022;10:878126.

Gil-Arias A, Diloy-Pena S, Sevil-Serrano J, Garcia-Gonzalez L, Abos A. A hybrid TGfU/SE volleyball teaching unit for enhancing motivation in physical education: a mixed-method approach. Int J Environ Res Public Health. 2021;18(1).

Zhang ZT, Du XM, Ma XJ, Zong Y, Chen JK, Yu CL, et al. Activation of the NLRP3 inflammasome in lipopolysaccharide-induced mouse fatigue and its relevance to chronic fatigue syndrome. J Neuroinflammation. 2016;13(1):71.

Yang M, Tao L, Zhao CC, Wang ZL, Yu ZJ, Zhou W, et al. Antifatigue effect of Panax Notoginseng leaves fermented with microorganisms: in-vitro and in-vivo evaluation. Front Nutr. 2022;9:824525.

Ingrassia PL, Mormando G, Giudici E, Strada F, Carfagna F, Lamberti F, et al. Augmented reality learning environment for basic life support and defibrillation training: usability study. J Med Internet Res. 2020;22(5):e14910.

Taylor AK, Loades M, Brigden AL, Collin SM, Crawley E. It’s personal to me’: a qualitative study of depression in young people with CFS/ME. Clin Child Psychol Psychiatry. 2017;22(2):326–40.

Mascarenhas LPG, de Lima VA, Rebesco DB, Franca SN, Cordeiro GR, Mota J, et al. Acute changes in glucose induced by continuous or intermittent exercise in children and adolescents with type 1 diabetes. Arch Endocrinol Metab. 2022;66(2):176–81.

Liao Y, Basen-Engquist KM, Urbauer DL, Bevers TB, Hawk E, Schembre SMJCE, Biomarkers, et al. Using continuous glucose monitoring to motivate physical activity in overweight and obese adults: a pilot study. Cancer Epidemiol Biomarkers Prev. 2020;29(4):761–8.

Mullan B, Olivier C, Thogersen-Ntoumani C. Mind the gap: habit and self-determined motivation predict health behaviours in middle-aged and older adults. Br J Health Psychol. 2021;26(4):1095–113.

Chae H, Park J. The effects of Routinization on radical and incremental creativity: the mediating role of mental workloads. Int J Environ Res Public Health. 2023;20(4):3160.

Simons M, de Vet E, Chinapaw MJ, de Boer M, Seidell JC, Brug J. Personal, social, and game-related correlates of active and non-active gaming among dutch gaming adolescents: survey-based multivariable, multilevel logistic regression analyses. JMIR Serious Games. 2014;2(1):e4.

Zeng N, Gao X, Liu Y, Lee JE, Gao Z. Reliability of using motion sensors to measure Children’s physical activity levels in exergaming. J Clin Med. 2018;7(5):100.

Download references

Acknowledgements

We would like to express our deep appreciation to Professor Jamalsafri Bin Saibon from the School of Educational Studies, Universiti Sains Malaysia (USM), for providing advices and support for this project.

This study was not funded by any organization or institution.

Author information

Authors and affiliations.

Malaysia Postgraduate Department, School of Educational Studies, Universiti Sains Malaysia, 11800, Penang, Malaysia

Weilong Mo, Jamalsafri Bin Saibon & Yaming LI

Zhaoqing College, Zhaoqing, 526061, China

Dinghu First Experimental School, Zhaoqing, 526070, China

Huaqiao Junior High School, Zhaoqing, 526108, China

You can also search for this author in PubMed   Google Scholar

Contributions

MWL performed the experiment, LYM and LJQ performed the data analysis, HYW performs a final check of the data, MWL performed the formal analysis and wrote the manuscript, JBS helped perform the analysis with constructive discussions, All authors were involved in discussing the results.

Corresponding author

Correspondence to Jamalsafri Bin Saibon .

Ethics declarations

Ethics approval and consent to participate.

Not applicable.

Consent for publication

Competing interests.

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Mo, W., Saibon, J.B., LI, Y. et al. Effects of game-based physical education program on enjoyment in children and adolescents: a systematic review and meta-analysis. BMC Public Health 24 , 517 (2024). https://doi.org/10.1186/s12889-024-18043-6

Download citation

Received : 07 November 2023

Accepted : 08 February 2024

Published : 19 February 2024

DOI : https://doi.org/10.1186/s12889-024-18043-6

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

  • Physical education
  • Children and adolescents
  • Meta-analysis

BMC Public Health

ISSN: 1471-2458

cochrane systematic review funnel plot

SYSTEMATIC REVIEW article

Efficacy of non-pharmacological interventions for sleep quality in parkinson’s disease: a systematic review and network meta-analysis.

Rongzhu Tang

  • Department of Neurology, The Second Affiliated Hospital of Chongqing Medical University, Chongqing, China

Background: Sleep disorders are one of the most common non-motor symptoms in PD. It can cause a notable decrease in quality of life and functioning in PD patients, as well as place a huge burden on both patients and caregivers. Currently, there are numerous non-pharmacological interventions available to improve sleep quality in PD, with disagreement as to which intervention is most effective. This network meta-analysis was performed to compare and rank non-pharmacological interventions to explore their efficacy in improving sleep quality in PD and to select the best interventions, with a view to providing references and bases for the development of clinical treatments and care programs.

Methods: The PubMed, Embase, Cochrane Central Register of Controlled Trials (CENTRAL), Web of Science, China National Knowledge Infrastructure (CNKI), and Wanfang databases were searched from inception to December 6, 2023. Two authors independently screened all studies, extracted the data, and evaluated risk of bias of included studies. STATA software version 17.0 was used to conduct the network meta-analysis.

Results: Our network meta-analysis included 29 studies involving 1,477 participants and 16 non-pharmacological interventions. Although most nonpharmacological interventions showed non-significant effects, the surface under the cumulative ranking curve (SUCRA) values indicated that the best non-pharmacological intervention for sleep disorders was massage therapy (97.3%), followed by music therapy (94.2%), and Treadmill training (85.7%).

Conclusion: Massage therapy can be considered as an effective therapy for improving sleep quality in patients with PD. Due to limited quantity and quality of the included studies, more high quality studies are required to verify the conclusions of this network meta-analysis.

Systematic review registration: identifier CRD42023429339, PROSPERO ( york.ac.uk ).

1 Introduction

Parkinson’s disease (PD), the second most common neurodegenerative disease, is a chronic senile disease. Sleep disorder is the most common non-motor symptom in PD, with an incidence of about 47.66% to 89.10% ( Liu et al., 2018 ) and increasing year by year with the course of disease. More and more evidence shows that PD sleep disorders can lead to decreased quality of life ( Zuzuárregui and During, 2020 ), impaired psychosocial and cognitive function ( Riemann, 2019 ), fatigue ( Cao et al., 2020 ), depression ( Demet et al., 1999 ) or substance abuse ( Hasler et al., 2012 ) and may increase the risk of cardiovascular and metabolic diseases ( He et al., 2017 ). In addition to health risks, sleep disorders can also bring significant socio-economic burdens ( Frandsen et al., 2020 ). Studies have shown that there are large individual differences in the manifestation of sleep disorders as assessed by questionnaires, requiring individualized treatment ( Stefani and Högl, 2019 ). The treatment of sleep disorders includes pharmacological therapy and non-pharmacological therapy. Although many medications have been shown to have a certain therapeutic effect on sleep disorders in PD, they also have potential side effects and the overall therapeutic effect is still unsatisfactory. For example, long-term use of sedative-hypnotic drugs may lead to dependence and tolerance, and increase the risk of falls, cognitive impairment and daytime sleepiness ( Zhang H. et al., 2014 ). Therefore, alternative non-pharmacological interventions are needed to improve sleep quality in patients with PD.

Considering the potential side effects and economic costs of pharmacological therapy, while non-pharmacological interventions has the advantages of low incidence of adverse events and strong sustainability. A wide range of non-pharmacological interventions have been used to improve sleep in patients with PD. These can be broadly categorized as follows: environmental interventions (e.g., bright light therapy), psychological interventions (e.g., cognitive behavioral therapy, mindfulness), physical activity interventions (e.g., exercise, tai chi, qigong, and yoga), physical therapy (e.g., rTMS, tDCS), and complementary and alternative therapies (e.g., music therapy, massage therapy, and acupuncture) ( Paus et al., 2007 ; Skogar et al., 2013 ; Cheung et al., 2018 ; Wu et al., 2020 ; Zhu et al., 2020 ; Buchwitz et al., 2021 ; Cristini et al., 2021 ; Luo et al., 2021 ; Shan et al., 2022 ; Wang, 2022 ; Hsu et al., 2023 ). However, most previous studies have focused on comparing the effectiveness of single non-pharmacological intervention with usual care, sham control or waiting list in improving sleep quality in PD. There is still a lack of direct comparative studies between different non-pharmacological interventions, leading to differences in the best effectiveness of non-pharmacological interventions. For PD patients or decision-makers, they are still not known which non-pharmacological intervention is the best treatment for sleep disorders. Network meta-analysis (NMA) has been proposed to be the highest level of evidence in the treatment guideline ( Salanti et al., 2014 ). Different from a conventional pairwise analysis, NMA analyzes simultaneously both the direct and the indirect evidence from different studies, estimation of the relative effectiveness among all interventions, and rank ordering of the interventions ( Caldwell et al., 2005 ; Bafeta et al., 2014 ). The method is helpful to summarize evidence across many interventions and make optimal clinical decision ( Cipriani et al., 2013 ). Therefore, this study used NMA to explore the effect of non-pharmacological interventions on improving sleep quality in PD, in order to provide a scientific basis for clinical medical staff to choose the optimal solution to promote sleep quality in PD patients.

2.1 Protocol and registration

The protocol of this NMA has been registered in PROSPERO (registration number CRD42023429339). In addition, This systematic review was performed according to the Cochrane Handbook for the Systematic Review of Interventions ( Cumpston et al., 2021 ) and according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement ( Hutton et al., 2015 ). The details of the PRISMA Checklist are provided in Supplementary Table S1 .

2.2 Search strategy

Two of the authors (RT and SG) independently searched for randomized controlled trials (RCTs) from inception to December 6, 2023 in the following databases: PubMed, Embase, the Cochrane Central Register of Controlled Trials (CENTRAL), Web of Science, China National Knowledge Infrastructure (CNKI), and Wanfang databases. A combination of Medical Subject Headings (MeSH terms or Emtree terms) and free words related to PD, non-pharmacological interventions, sleep disorders, and RCTs was used, including: (1) Parkinson disease, Parkinson’s disease, Parkinson*, paralysis agitans, PD; (2) non-pharmacolog*, intervention, treatment, training, rehabilitation, exercise, therapy, bright light therapy, BLT, repetitive transcranial magnetic stimulation, rTMS, deep brain stimulation, DBS, cognitive behavioral therapy, CBT, mindfulness meditation, Baduanjin, qigong, continuous positive airway pressure, CPAP, Tai Chi, acupuncture, massage therapy, muscle relaxation, aerobic exercise, resistance training, yoga, dance, music therapy, ultrasound therapy, low-level laser therapy, etc.; (3) dyssomnias, sleep disorders, sleep, sleepiness, sleep quality, insomnia; and (4) randomized controlled trial, randomized controlled trials as topic, controlled clinical trial, randomized, placebo. Medical Subject Headings (MeSH) and free words were linked by “OR” in each group and searched by “AND” to link each group. In addition, the reference lists of the included literature and related articles were also manually searched to identify eligible studies. The search strategies for all databases are listed in Supplementary Table S2 .

2.3 Eligibility criteria

The PICOS (population, intervention, comparison, outcomes, study design) framework ( Hutton et al., 2015 ) was used to operationalize the eligibility criteria of the studies to be included in the review.

A study was included if (1) population: adults (>18 years) diagnosed with PD. All participants in the intervention and control groups who were stably taking antiparkinsonian medications were also eligible, (2) intervention: participants in the experimental groups received non-pharmacological interventions with no limits in frequency, duration, style, form, or setting, (3) comparison: participants in the control groups received sham control, waiting list, or conventional treatment including usual care, supportive instruction (e.g., health education, sleep hygiene advice), and physiotherapy, or other non-pharmacological interventions that differed from the experimental group. However, original trials comparing only different approaches of the same intervention were excluded, (4) outcomes: efficacy outcomes were pre-post changes in the Pittsburgh Sleep Quality Index (PSQI), the Parkinson’s Disease Sleep Scale (PDSS), the Epworth Sleepiness Scale (ESS), the Insomnia Severity Index (ISI), the Parkinson’s disease sleep scale version 2 (PDSS-2), the Scale for Parkinson’s Disease-Sleep (SCOPA-S), and the Mini-sleep Questionnaire (MSQ). Safety outcomes were indicative of adverse events (AEs) after non-pharmacological interventions, (5) study design: only RCTs were included without any regional or publication restrictions.

Studies were excluded if they (1) cannot obtain full text or extract data; (2) were in the trial protocol registration stage and has not yet officially carried out clinical trials; (3) were conference abstracts, masteral dissertation and reviews; (4) were repeatedly published or multiple investigations were based on the same population data, the latest research or articles with comprehensive information would be included.

2.4 Data extraction and quality assessment

Two authors (RT and SG) independently extracted data including the first author, country, year, sample size, baseline characteristics of participants (age, gender) duration of disease, Hoehn–Yahr stage, intervention details (type, frequency, intensity and duration), comparison, and outcomes based on a predesigned form within Microsoft Excel. Two researchers (RT and SG) independently evaluated the quality according to the bias risk assessment scale of randomized controlled trials recommended by Cochrane Handbook 5.1.0 ( Higgins and Green, 2011 ). The scale consists of seven domains: random sequence generation, allocation concealment, whether blind method was used for researchers and subjects, whether blind method was used for outcome evaluation, integrity of outcome data, selective reporting of results and other risk of bias. Each item was assessed as being of “low risk,” “high risk,” or “unclear risk” of bias. The results of the data extraction and quality assessment were cross-checked, and the divergences were resolved through discussion with a third author (CL).

2.5 Statistical analysis

2.5.1 traditional meta-analysis.

Standard mean difference (SMD) with the corresponding 95% confidence interval (CI) was used to express the pooled estimates because all outcomes were continuous variables in this study but were measured using various tools. Due to a wide range of characteristics of the studies included, all analyses were performed using the random effects model. Statistical heterogeneity between the studies was assessed using the I 2 statistic, with I 2 values of 25, 50, and 75% indicating low, moderate, and high heterogeneity, respectively ( Higgins et al., 2003 ). Moreover, subgroup analysis were used to explore the source of heterogeneity. In addition, Egger’s test were used to evaluate publication bias quantitatively ( Egger et al., 1997 ).

2.5.2 Network meta-analysis

The quality evaluation was performed using the bias risk quality evaluation tool in Review Manager 5.3. For all eligible trials, we selected the difference before and after the intervention for comparison, If the difference is not reported in the original literature, the difference is calculated according to the formula in the guide ( Shi et al., 2020 ). Continuous variables were analyzed using SMD with 95% CI, and the significance was set at α  = 0.05.

First, a network map of direct comparisons between different interventions was drawn by using Stata software (version 17.0). Each node in the map represents an intervention, and the size of the node indicates the sample size receiving the intervention. The presence of a line between two nodes indicates that they have a direct comparison relationship, and a thicker line indicates a higher number of comparisons. Subsequently, we examined the global consistency and used the node-split model to determine the local consistency. p  > 0.05 indicated no significant inconsistency between direct and indirect comparisons, and in these cases, the consistency model was adopted; otherwise, the inconsistency model was used.

Additionally, the league table was used to analyze the results of the comparisons among the different interventions based on a NMA. the effects of various interventions were quantitatively analyzed by using the surface under the cumulative ranking (SUCRA) to rank the effects of different interventions. SUCRA values range from 0% to 100%, and if the SUCRA value for an intervention is closer to 100%, it indicates that the intervention is more effective. However, this conclusion should be interpreted with caution if there is not a clinically meaningful difference between the two interventions. Finally, network funnel plots were drawn and visually checked by using the symmetry criterion to determine if there was a possibility of bias leading to NMA publication.

3.1 Study selection

Our study initially retrieved 6,987 articles, of which 2,141 were removed due to duplication, and 4,846 articles remained. Among them, 4,687 articles were excluded because their abstracts and titles did not meet the selection criteria; thus, 159 articles remained. After reading the full text, 130 articles were excluded. Finally, 29 studies were included in this study. The literature screening process was shown in Figure 1 .

www.frontiersin.org

Figure 1 . The study search, selection, eligibility and inclusion process.

3.2 Study characteristics

Table 1 shows the characteristics of 29 eligible RCTs published from inception to 2023 and involving 1,477 participants. In this NMA, The following 16 kinds of non-pharmacological interventions were used: (1) CBT combined with BLT, (2) electroacupuncture, (3) BLT, (4) stretch-balance training, (5) aerobic exercise, (6) rTMS, (7) multimodal exercise, (8) computerized cognitive behavioral therapy, (9) tDCS, (10) baduanjin combined with qigong, (11) mindfulness, (12) qigong, (13) Tai Chi, (14) yoga, (15) music therapy, (16) massage therapy. The comparison mainly consisted of placebo, waitlist, conventional treatment. Among all eligible studies, one study was a three-arm studies ( Altmann et al., 2016 ), and the remaining 28 studies were two-arm studies ( Paus et al., 2007 ; Modugno et al., 2010 ; Nascimento et al., 2013 ; Rios Romenets et al., 2013 ; Skogar et al., 2013 ; Zhang W. et al., 2014 ; Wang et al., 2015 ; Xiao and Zhuang, 2015 ; Patel et al., 2017 ; Videnovic et al., 2017 ; Cheung et al., 2018 ; Rutten et al., 2019 ; Elyazed et al., 2020 ; Kraepelien et al., 2020 ; Moon et al., 2020 ; Wu et al., 2020 ; Xu et al., 2020 ; Zhu et al., 2020 ; Zhuang et al., 2020 ; Buchwitz et al., 2021 ; Li et al., 2021 ; Wu et al., 2021 ; Nazarova et al., 2022 ; Shan et al., 2022 ; Wang, 2022 ; Yu and Chen, 2022 ; Zhang et al., 2022 ; Li et al., 2023 ).

www.frontiersin.org

Table 1 . Characteristics of included trails network meta-analysis.

3.3 Quality assessment

Regarding the risk of bias tool, among the 29 studies included, 20 studies ( Modugno et al., 2010 ; Rios Romenets et al., 2013 ; Skogar et al., 2013 ; Zhang W. et al., 2014 ; Wang et al., 2015 ; Cheung et al., 2018 ; Rutten et al., 2019 ; Elyazed et al., 2020 ; Moon et al., 2020 ; Wu et al., 2020 ; Xu et al., 2020 ; Zhu et al., 2020 ; Buchwitz et al., 2021 ; Li et al., 2021 ; Wu et al., 2021 ; Shan et al., 2022 ; Wang, 2022 ; Yu and Chen, 2022 ; Zhang et al., 2022 ; Li et al., 2023 ) mentioned the use of random number table method or software-generated random numbers for random grouping; two studies ( Patel et al., 2017 ; Kraepelien et al., 2020 ) mentioned the use of envelope method for random grouping; and one study ( Zhuang et al., 2020 ) mentioned the use of coin toss method for random grouping, all were rated as “low risk.” Six studies ( Paus et al., 2007 ; Nascimento et al., 2013 ; Xiao and Zhuang, 2015 ; Altmann et al., 2016 ; Videnovic et al., 2017 ; Nazarova et al., 2022 ) only mentioned the word “random,” did not describe the randomization method, and were rated as having an “unclear risk” of bias in this field. About allocation concealment, Six studies ( Patel et al., 2017 ; Cheung et al., 2018 ; Kraepelien et al., 2020 ; Moon et al., 2020 ; Xu et al., 2020 ; Shan et al., 2022 ) used sealed opaque envelopes for allocation concealment. Rutten et al. (2019) used password-protected secure drives for allocation concealment, and Wu et al. (2021) used password-protected links for allocation concealment. The remaining studies did not describe allocation concealment. Due to the characteristics of some non-drug interventions, it is difficult for researchers to implement blinding in the intervention process. Only eight studies ( Videnovic et al., 2017 ; Cheung et al., 2018 ; Rutten et al., 2019 ; Moon et al., 2020 ; Wu et al., 2020 ; Xu et al., 2020 ; Buchwitz et al., 2021 ; Wu et al., 2021 ) mentioned blinding for participants and researchers, and 10 studies ( Paus et al., 2007 ; Xiao and Zhuang, 2015 ; Cheung et al., 2018 ; Rutten et al., 2019 ; Moon et al., 2020 ; Wu et al., 2020 ; Xu et al., 2020 ; Zhu et al., 2020 ; Buchwitz et al., 2021 ; Wu et al., 2021 ) mentioned blinding for outcome measurers. Among the 29 studies, 20 studies had no dropout cases ( Paus et al., 2007 ; Nascimento et al., 2013 ; Rios Romenets et al., 2013 ; Skogar et al., 2013 ; Zhang W. et al., 2014 ; Wang et al., 2015 ; Altmann et al., 2016 ; Videnovic et al., 2017 ; Cheung et al., 2018 ; Elyazed et al., 2020 ; Moon et al., 2020 ; Wu et al., 2020 ; Zhuang et al., 2020 ; Li et al., 2021 ; Wu et al., 2021 ; Shan et al., 2022 ; Wang, 2022 ; Yu and Chen, 2022 ; Zhang et al., 2022 ; Li et al., 2023 ), and the remaining studies had sample dropout. Among them, seven studies applied appropriate statistical analysis methods (i.e., intentional treatment analysis) ( Modugno et al., 2010 ; Xiao and Zhuang, 2015 ; Rutten et al., 2019 ; Kraepelien et al., 2020 ; Xu et al., 2020 ; Zhu et al., 2020 ; Buchwitz et al., 2021 ), and all explained the groups from which the dropout subjects came and the specific reasons for the dropout, all were rated as “low risk.” Patel et al. (2017) described the reasons for the dropout and applied the intention-to-treat analysis, but there was a high dropout rate, so the risk of bias in this field was assessed as “unclear risk.” Nazarova et al. (2022) did not mention the causes and treatment of dropout, and was rated as having a “high risk” of bias in this field. For other biases, all studies described statistical homogeneity between groups at baseline and were rated as having a “low risk” of bias in this area. Twenty-three studies ( Modugno et al., 2010 ; Nascimento et al., 2013 ; Rios Romenets et al., 2013 ; Wang et al., 2015 ; Xiao and Zhuang, 2015 ; Altmann et al., 2016 ; Patel et al., 2017 ; Videnovic et al., 2017 ; Cheung et al., 2018 ; Rutten et al., 2019 ; Kraepelien et al., 2020 ; Moon et al., 2020 ; Wu et al., 2020 ; Xu et al., 2020 ; Zhu et al., 2020 ; Zhuang et al., 2020 ; Buchwitz et al., 2021 ; Wu et al., 2021 ; Shan et al., 2022 ; Wang, 2022 ; Yu and Chen, 2022 ; Zhang et al., 2022 ; Li et al., 2023 ) described the approval of the Institutional Review Board (IRB). Twenty-seven studies ( Modugno et al., 2010 ; Nascimento et al., 2013 ; Rios Romenets et al., 2013 ; Skogar et al., 2013 ; Zhang W. et al., 2014 ; Wang et al., 2015 ; Xiao and Zhuang, 2015 ; Altmann et al., 2016 ; Videnovic et al., 2017 ; Cheung et al., 2018 ; Rutten et al., 2019 ; Elyazed et al., 2020 ; Kraepelien et al., 2020 ; Moon et al., 2020 ; Wu et al., 2020 ; Xu et al., 2020 ; Zhu et al., 2020 ; Zhuang et al., 2020 ; Buchwitz et al., 2021 ; Li et al., 2021 ; Wu et al., 2021 ; Nazarova et al., 2022 ; Shan et al., 2022 ; Wang, 2022 ; Yu and Chen, 2022 ; Zhang et al., 2022 ; Li et al., 2023 ) described that they had obtained the informed consent of the participants before the experiment. The methodological quality assessments of the eligible RCTs are shown in Figure 2 , ranging from low to high risk.

www.frontiersin.org

Figure 2 . Quality assessment of the eligible studies.

3.4 Traditional meta-analysis

3.4.1 overall effect test.

The meta-analysis of the efficacy of non-pharmacological interventions to improve sleep quality in PD is shown in the forest plot in Supplementary Figure S1 . The results of the combined meta-analysis showed moderate heterogeneity ( I 2  = 57.1%, p  < 0.001), so the random-effects model was used to test for effect sizes. The combined effect size was (SMD: −0.47, 95% CI: −0.64, −0.30, p  < 0.001), indicating that the combined effect size was statistically significant, i.e., non-pharmacological interventions can significantly improve sleep quality in patients with PD.

3.4.2 Subgroup analysis

To further explore the sources of heterogeneity ( I 2 > 50%) among the studies, subgroup analysis were performed to analyze the factors causing heterogeneity. Considering the effects of different intervention durations, intervention frequencies, and intervention period on the outcomes, subgroup analyses were conducted on three factors: intervention duration, intervention frequency, and intervention period. The results of the subgroup analysis showed that among the forms of non-pharmacological interventions to improve sleep quality in patients with PD, intervention periods of ≥6 weeks and ≥5 interventions per week of <60 min are the optimal regimen to improve sleep quality in patients with PD. The results of subgroup analysis are shown in Table 2 .

www.frontiersin.org

Table 2 . Subgroup analysis of the effects of different covariates on sleep quality in patients with PD.

3.5 Network meta-analysis

The net evidence of different nonpharmacological interventions improve sleep quality in PD was shown in Figure 3 . According to the network plot, rTMS, electroacupuncture, BLT, and music therapy were the more common comparisons. Sham control, treadmill training and stretch-balance training formed a closed loop, but this is a three-arm studies. Furthermore, sham control, conventional treatment, treadmill training and rTMS created a closed loop, which indicated both direct and indirect comparisons. There was no evidence of direct comparisons for the other interventions. Table 3 shows the relative effects of the different interventions on sleep quality. The league table shows the pairwise comparisons of 16 non-pharmacological interventions on sleep quality in PD.

www.frontiersin.org

Figure 3 . The network map of non-pharmacological interventions for sleep quality in PD. CBT, cognitive behavioral therapy; BLT, bright light therapy; rTMS, repetitive transcranial magnetic stimulation; CCBT, computerized cognitive behavioral therapy; tDCS, transcranial direct current stimulation.

www.frontiersin.org

Table 3 . League table for non-pharmacologic interventions.

3.6 Rank probability

The SUCRA plot and values are shown in Figure 4 and Table 4 , respectively. The SUCRA values and the plot revealed that the treatments’ comparative efficacy in improving sleep quality was, in order: massage therapy > music therapy > treadmill training > rTMS > multimodal exercise > electroacupuncture > tDCS > stretch-balance training > qigong > Tai Chi > CBT combined with BLT > CCBT > baduanjin combined with qigong > mindfulness > BLT > conventional treatment > sham control > yoga > waitlist.

www.frontiersin.org

Figure 4 . The SUCRA plot based on cumulative probabilities of interventions. CBT, cognitive behavioral therapy; BLT, bright light therapy; rTMS, repetitive transcranial magnetic stimulation; CCBT, computerized cognitive behavioral therapy; tDCS, transcranial direct current stimulation.

www.frontiersin.org

Table 4 . SUCRA values for non-pharmacologic interventions.

3.7 Consistency analysis

The global inconsistency analysis of this NMA showed a p-value of 0.267, indicating no significant inconsistency. The results are summarized in Supplementary Figure S2 . Moreover, the results of the node-splitting analysis showed no inconsistency between direct and indirect comparisons ( p  = 0.267 to 0.996). The results are summarized in Supplementary Table 3A . Therefore, we used the consistency model to perform the NMA. Sham control-conventional treatment-aerobic exercise-rTMS formed a quadratic loops without significant discordance (IF = 0.674, 95% CI = 0.00–2.13, tau 2  = 0.034). The multi-arm trials formed a triangular loop: Sham control-aerobic exercise-stretch-balance training, which was defined as no inconsistency.

3.8 Publication analysis

To assess publication bias, a comparison-adjusted network funnel plot with a random model was constructed for the outcome ( Figure 5 ). The results were roughly distributed around the overall effect, arranged symmetrically around the center line. The included literature was well distributed, the data analysis results were less affected by the publication bias. Egger’s test also showed no publication bias ( p  = 0.269) ( Supplementary Table 3B ).

www.frontiersin.org

Figure 5 . The funnel plot of non-pharmacological interventions for sleep quality in PD.

3.9 Incidence of adverse events

For the incidence of AEs, quantitative synthesis was judged to be inadequate because the number of AEs and the number of patients experiencing AEs were mixed, so qualitative analysis is adopted. Thirteen studies ( Paus et al., 2007 ; Rios Romenets et al., 2013 ; Videnovic et al., 2017 ; Rutten et al., 2019 ; Kraepelien et al., 2020 ; Zhu et al., 2020 ; Nazarova et al., 2022 ; Yu and Chen, 2022 ) reported the incidences of AEs, with five studies ( Cheung et al., 2018 ; Xu et al., 2020 ; Zhuang et al., 2020 ; Buchwitz et al., 2021 ; Zhang et al., 2022 ) reporting no AEs. In Yu and Chen (2022) , there was one case of headache, one case of drowsiness, two cases of memory loss in the ATU group, while there were two cases of headache in the rTMS group, but their symptoms were relieved quickly. Rutten et al. reported mild ocular symptoms, headache, and gastrointestinal distress ( Rutten et al., 2019 ). Rios Romenets et al. (2013) reported that one patient in the intervention group fell asleep while using BLT, resulting in mild facial burns with no serious consequences. In Videnovic et al. (2017) , there were two cases of headache and sleepiness occurred in the BLT group, while there was one case of itchy eyes occurred in the placebo group, and these adverse effects subsided spontaneously. In Paus et al. (2007) , four patients in the intervention group reported minor and transitory side effects, being eyestrain, and feeling of general malaise (two patients each, respectively). In Kraepelien et al. (2020) , there were one patient experienced temporary stomach discomfort during meditation practice and felt depressed. In Zhu et al. (2020) , two participants in the Tai Chi group and one participant in the conventional treatment group reported fatigue and dizziness, and there was one case of muscle spasm in each group. In Nazarova et al. (2022) , there was one case of fall in the acupuncture group.

4 Discussion

A total of 29 RCT studies of non-pharmacological interventions to improve sleep quality in PD were included in this study, and the included studies were subjected to rigorous quality evaluation and risk assessment. To the best of our knowledge, this is the first NMA comparing the efficacy of different non-pharmacologic interventions for improving sleep quality in PD. The SUCRA values revealed that the best non-pharmacological intervention was massage therapy, followed by music therapy and treadmill training. Subgroup analyses further indicated that non-pharmacological interventions of ≥6 weeks duration, ≥5 interventions per week, and less than 60 min per intervention were the best regimen for improving sleep quality in patients with PD. Furthermore, In terms of the incidence of AEs, all non-pharmacologic interventions resulted in few and mild AEs, improving the clinical safety of the treatment and increasing the compliance of patients.

Sleep disorders in PD are caused by a variety of factors, including the degeneration of central sleep regulation, disease progression, medications, comorbidities, and mood disorders ( Gunn et al., 2010 ; Rothman and Mattson, 2012 ; Falup-Pecurariu and Diaconu, 2017 ) particularly, motor symptoms of PD, such as nocturnal hypokinesia, rigidity, tremor exacerbated by overnight wearing-off and painful dystonia, may contribute to sleep onset and/or sleep maintenance difficulties ( Stack and Ashburn, 2006 ; Louter et al., 2013 ). In addition, nonmotor symptoms such as nocturia, pain, depression, anxiety and hallucinations, often lead to sleep fragmentation ( Gómez-Esteban et al., 2006 ; Wailke et al., 2010 ; Yong et al., 2011 ). Various non-pharmacological interventions, as alternative or complementary therapies, can consistently and effectively improve motor and nonmotor symptoms in PD, leading to improved sleep quality.

Existing evidence suggests that massage therapy improves sleep quality in different populations ( Pruthi et al., 2009 ; Nerbass et al., 2010 ). In addition, our findings suggest that massage therapy is the best non-pharmacologic intervention for improving sleep quality in PD with a SUCRA value of 97.3%. Massage therapy refers to a technical operation by pressing, moistening, pushing, holding, kneading and other techniques on specific parts of the human body surface, easy to operate, safe and non-invasive, high acceptance of patients. Moderate pressure applied during massage may enhance vagal activity and inhibit hypothalamic-pituitary-adrenal function by stimulating pressure receptors that eventually signal the limbic system, which in turn leads to lower norepinephrine and cortisol levels and higher serotonin levels ( Field, 2016 ). Decreased norepinephrine levels slow heart rate, lower blood pressure, and improve sleep ( Blagrove et al., 2012 ). Decreased cortisol levels are thought to be associated with improved sleep disorders such as obstructive sleep apnea and insomnia ( Blagrove et al., 2012 ). Increased serotonin levels may improve depressive symptoms, relieve pain ( Field, 2016 ), and thus improve overall sleep quality. In addition, massage therapy transmits stimulation to the autonomic nervous system, activates blood circulation, relieves muscle tension, releases stress, relaxes the body and promotes sleep. However, the number of studies on the effects of massage therapy on sleep quality in PD is limited, and the results of the present study were derived from only one RCT study ( Skogar et al., 2013 ), which had a small sample size and poor methodological quality, limiting the accuracy and generalizability of the conclusions, and the results should be treated with caution. Therefore, more high-quality, large-sample size RCTs are needed in the future to validate the long-term effects of massage therapy on sleep quality in PD.

Music is one of the most-used self-help strategies to promote sleep. A Cochrane review ( Jespersen et al., 2015 ) showed that music therapy was effective in improving subjective sleep quality in adult insomnia patients. Although the causes and mechanisms are unknown, the effects of music therapy on sleep quality can be explained by the following reasons. Firstly, music’s ability to regulate emotional states may help improve patients’ sleep quality. The study showed that listening to music that participants experience as pleasurable elicit increased activity in the mesolimbic reward network of the brain including the dorsal and ventral striatum ( Jespersen et al., 2023 ). Secondly, music is closely related to physiological responses, sedative music induces a relaxation and distraction response, which reduces activity in the neuroendocrine and sympathetic nervous systems, resulting in decreased pain, stress, anxiety, and sleep ( Jespersen et al., 2015 ). In addition, other mechanisms may come into play such as the masking effect of music whereby music facilitates sleep by covering noise from the hospital environment ( Jespersen et al., 2023 ). Our study showed that music therapy ranked second among nonpharmacological interventions with a SUCRA value of 94.2% and was effective in improving sleep quality in patients with PD. However, the relationship between the objective characteristics of music and the subjective preferences of individuals remains unclear, and more RCTs are needed in the future to investigate potential differences in the effects of researcher-selected music and participant-selected music.

Previous systematic reviews have only demonstrated the efficacy of exercise on sleep quality in PD ( Cristini et al., 2021 ), there have been no cross-sectional comparisons for different exercise types, so it would become meaningful to explore comparisons of efficacy between them. Our study showed that treadmill training was the third ranked non-pharmacological intervention with a SUCRA value of 85.7%. Treadmill training, one of the most common forms of aerobic exercise, promotes increased levels of brain-derived neurotrophic factor (BDNF) and induces sleep by creating a state of energy expenditure and increasing basal metabolic rate ( Doris et al., 2018 ). Moderate-intensity treadmill training also reduces resting plasma concentrations of pro-inflammatory cytokines and increases plasma concentrations of anti-inflammatory cytokines, thereby improving sleep quality ( Kapsimalis et al., 2008 ). Sacheli et al. has been confirmed that three-month treadmill training can increase cortico-striatal neuroplasticity and dopamine release to improve sleep quality ( Sacheli et al., 2019 ). In addition, treadmill training is not limited by road conditions, weather, and time, and is suitable for any age and any place, which can stimulate the potential of patients and achieve lasting and stable therapeutic effects ( Schootemeijer et al., 2020 ). In short, treadmill training can be considered an important method to assist in the treatment of sleep disorders in PD.

Our study showed that rTMS can improve sleep quality in PD, which is consistent with the results of previous studies. rTMS is a noninvasive neurostimulation technique with the advantages of noninvasiveness, painlessness, high therapeutic safety, precise clinical efficacy, and few adverse reactions ( Yang et al., 2018 ). rTMS can induce hyperpolarization of cortical neurons, inhibit the excitability and metabolic level of cerebral cortex, and improve sleep state ( Hsu et al., 2007 ). Song et al. (2019) concluded that rTMS can promote the synthesis of pineal gland and secreting melatonin in PD patients, thus effectively regulating the sleep cycle ( Gorfine and Zisapel, 2007 ). And rTMS can regulate the levels of various neurotransmitters (5-HT, NE, DA) and maintain the balance of neurotransmitters in the body, thus improving sleep quality ( Kiebs et al., 2019 ). Due to the small number of included articles, our study did not perform a subgroup analysis of the stimulation frequency of rTMS (e.g., 1 Hz/5 HZ) to further explore the efficacy of different stimulation frequencies on sleep quality in PD. Therefore, more large sample, high quality RCTs are needed in the future to explore the optimal stimulation parameters and stimulation modes for rTMS to improve sleep quality in PD.

Multimodal exercise refers to the intervention of two or more types of exercise ( Suzuki et al., 2012 ), such as aerobic exercise, strength/resistance training, balance/coordination training, flexibility training and other combinations of exercise methods. Its rich environment and exercise types can induce more new neurons and delay disease progression ( Hirsch and Farley, 2009 ; Chirles et al., 2017 ). Several studies have shown that multimodal exercise can improve muscle strength, flexibility and balance, effectively improve motor disorders and ADL in PD patients, reduce depression symptoms, and improve the overall sleep quality of patients with PD ( Suzuki et al., 2012 ; Vogel et al., 2021 ). It is worth noting that multimodal exercise as a combination therapy should be more effective than treadmill training, but this was not the case in our results. The reason for this effect may be that PD patients are mostly elderly patients with severe functional impairment, and even most of them have cognitive dysfunction, which makes it difficult for them to understand and perform complex, high-intensity multimodal exercise, and thus patients’ motivation and exercise compliance are poor, resulting in poor quality of exercise and difficulty in achieving the desired rehabilitation effect ( Schootemeijer et al., 2020 ).

Traditional meta-analyses have shown that electroacupuncture significantly improves sleep quality ( Hsu et al., 2023 ), which is consistent with the results of this study. Electroacupuncture has a weaker effect on insomnia itself, but it shows excellent results in improving accompanying symptoms, especially pain and pain-related insomnia. However, standardized acupoint selection protocols are a major barrier to the spread of acupuncture therapy ( Hsu et al., 2023 ). In addition, our findings showed that the remaining non-pharmacological interventions had no significant effect on sleep disorders in PD. This may be due to the insufficient number of RCTs included in these interventions, small sample size and/or disease progression ( Rothman and Mattson, 2012 ; Louter et al., 2013 ; Cheung et al., 2018 ; Liu et al., 2018 ; Moon et al., 2020 ; Zhu et al., 2020 ; Buchwitz et al., 2021 ). In particular, the relevant guidelines recommend CBT as a first-level recommended treatment for insomnia in the elderly ( Koychev and Okai, 2017 ), but there are few RCTs on the application of CBT to PD sleep disorders, and the evidence is relatively weak. Therefore, readers should be cautious about these results. In the future, more RCTs are needed to verify the efficacy of these non-pharmacological intervention strategies for sleep disorders in PD.

Our study also had some limitations. Firstly, there was heterogeneity in the included studies, such as the duration, frequency, and period of non-pharmacologic interventions. Secondly, we comprehensively searched for non-pharmacological interventions for sleep disorders in PD, but the language was limited to Chinese and English, which may have contributed to selection bias. Thirdly, many comparisons between non-pharmacological interventions include only a small number of RCTs, which may have affected the accuracy of the conclusions. Fourthly, most of the studies included in this network meta-analysis compared nonpharmacological therapies with conventional treatment, placebo or wait-list, while the actual number of head-to-head trials was relatively small, so efficacy comparisons between interventions are often based on indirect comparisons. Finally, the overall quality of most RCTs included in this study was moderate, especially due to the lack of blinding procedures, there was some risk of overestimation. Due to the nature of non-pharmacological interventions, blinding of participants and researchers seems inevitable. In the future, higher quality research is needed to avoid or reduce the risk of overestimation.

5 Conclusion

Our study compared the efficacy of different non-pharmacological interventions for sleep disorders in patients with PD, and our results suggest that massage therapy may be the preferred non-pharmacological intervention for improving sleep disorders in PD, followed by music therapy and treadmill training. However, the results should be interpreted with caution, considering the limitations of our above meta-analysis and the limited number of existing RCT and the small sample size, other potential risks of bias (e.g. lack of blinded procedures), and some differences in study design.

Therefore, further validation with more large sample, high-quality RCTs are needed to ensure the scientificity of the findings. Finally, the results of this study could provide evidence and a reference to healthcare providers and clinicians when choosing effective interventions to improve the quality of life and health status of patients with PD.

Data availability statement

The original contributions presented in the study are included in the article/ Supplementary material , further inquiries can be directed to the corresponding authors.

Author contributions

RT: Data curation, Methodology, Software, Writing – original draft, Writing – review & editing. SG: Data curation, Software, Writing – review & editing. JLi: Data curation, Writing – review & editing. WH: Data curation, Writing – review & editing. JLiu: Conceptualization, Supervision, Writing – review & editing. CL: Conceptualization, Supervision, Writing – review & editing.

The author(s) declare that no financial support was received for the research, authorship, and/or publication of this article.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary material

The Supplementary material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fnins.2024.1337616/full#supplementary-material

Altmann, L. J. P., Stegemöller, E., Hazamy, A. A., Wilson, J. P., Bowers, D., Okun, M. S., et al. (2016). Aerobic exercise improves mood, cognition, and language function in Parkinson’s disease: results of a controlled study. J. Int. Neuropsychol. Soc. 22, 878–889. doi: 10.1017/s135561771600076x

PubMed Abstract | Crossref Full Text | Google Scholar

Bafeta, A., Trinquart, L., Seror, R., and Ravaud, P. (2014). Reporting of results from network meta-analyses: methodological systematic review. BMJ 348:1741. doi: 10.1136/bmj.g1741

Blagrove, M., Fouquet, N. C., Baird, A. L., Pace-Schott, E. F., Davies, A. C., Neuschaffer, J. L., et al. (2012). Association of salivary-assessed oxytocin and cortisol levels with time of night and sleep stage. J. Neural Transm. 119, 1223–1232. doi: 10.1007/s00702-012-0880-1

Buchwitz, T. M., Maier, F., Greuel, A., Thieken, F., Steidel, K., Jakobs, V., et al. (2021). Pilot study of mindfulness training on the self-awareness of motor symptoms in Parkinson’s disease-a randomized controlled trial. Front. Psychol. 12:763350. doi: 10.3389/fpsyg.2021.763350

Caldwell, D. M., Ades, A. E., and Higgins, J. P. T. (2005). Simultaneous comparison of multiple treatments: combining direct and indirect evidence. BMJ 331, 897–900. doi: 10.1136/bmj.331.7521.897

Cao, X.-Y., Zhang, J.-R., Shen, Y., Mao, C. J., Shen, Y. B., Cao, Y. L., et al. (2020). Fatigue correlates with sleep disturbances in Parkinson disease. Chin. Med. J. 134, 668–674. doi: 10.1097/cm9.0000000000001303

Cheung, C., Bhimani, R., Wyman, J. F., Konczak, J., Zhang, L., Mishra, U., et al. (2018). Effects of yoga on oxidative stress, motor function, and non-motor symptoms in Parkinson’s disease: a pilot randomized controlled trial. Pilot Feasibility Stud. 4:162. doi: 10.1186/s40814-018-0355-8

Chirles, T. J., Reiter, K., Weiss, L. R., Alfini, A. J., Nielson, K. A., and Smith, J. C. (2017). Exercise training and functional connectivity changes in mild cognitive impairment and healthy elders. J. Alzheimers Dis. 57, 845–856. doi: 10.3233/JAD-161151

Cipriani, A., Higgins, J. P. T., Geddes, J. R., and Salanti, G. (2013). Conceptual and technical challenges in network meta-analysis. Ann. Intern. Med. 159, 130–137. doi: 10.7326/0003-4819-159-2-201307160-00008

Cristini, J., Weiss, M., De Las Heras, B., Medina-Rincón, A., Dagher, A., Postuma, R. B., et al. (2021). The effects of exercise on sleep quality in persons with Parkinson’s disease: a systematic review with meta-analysis. Sleep Med. Rev. 55, 101–384. doi: 10.1016/j.smrv.2020.101384

Cumpston, M., Li, T., Page, M. J., Chandler, J., Welch, V. A., Higgins, J. P., et al. (2021). Updated guidance for trusted systematic reviews: a new edition of the Cochrane handbook for systematic reviews of interventions. Cochrane Database Syst. Rev. 10:ED000142. doi: 10.1002/14651858.ed000142

Demet, M. E., Chicz-Demet, A., Fallon, J. H., and Sokolski, K. N. (1999). Sleep deprivation therapy in depressive illness and Parkinson’s disease. Prog. Neuropsychopharmacol. Biol. Psychiatry 23, 753–784. doi: 10.1016/s0278-5846(99)00039-1

Doris,, Yu, S. F., Ng, S. S. M., Lee, D. T. F., Choi, K. C., Siu, P. M. F., et al. (2018). The effects of an activity-based lifestyle intervention on moderate sleep complaints among older adults: study protocol for a randomized controlled trial. Trials 19:69. doi: 10.1186/s13063-018-2465-2

Egger, M., Smith, G. D., Schneider, M., and Minder, C. (1997). Bias in meta-analysis detected by a simple, graphical test. BMJ 315, 629–634. doi: 10.1136/bmj.315.7109.629

Elyazed, T. I. A., Al-Azab, I. M. A., and Semary, M. M. E. (2020). Effect of aerobic exercise on depression and insomnia in Egyptian geriatrics Parkinson’s population. Biosci. Res. 15, 1601–1609.

Google Scholar

Falup-Pecurariu, C., and Diaconu, S. (2017). Sleep dysfunction in Parkinson’s disease. Int. Rev. Neurobiol. 133, 719–742. doi: 10.1016/bs.irn.2017.05.033

Crossref Full Text | Google Scholar

Field, T. (2016). Massage therapy research review. Complement. Ther. Clin. Pract. 24, 19–31. doi: 10.1016/j.ctcp.2016.04.005

Frandsen, R., Asah, C., Ibsen, R., Kjellberg, J., and Jennum, P. J. (2020). Health, social, and economic consequences of rapid eye movement sleep behavior disorder: a controlled national study evaluating societal effects. Sleep 44:zsaa162. doi: 10.1093/sleep/zsaa162

Gómez-Esteban, J. C., Zarranz, J. J., Lezcano, E., Velasco, F., Ciordia, R., Rouco, I., et al. (2006). Sleep complaints and their relation with drug treatment in patients suffering from Parkinson’s disease. Mov. Disord. 21, 983–988. doi: 10.1002/mds.20874

Gorfine, T., and Zisapel, N. (2007). Late evening brain activation patterns and their relation to the internal biological time, melatonin, and homeostatic sleep debt. Hum. Brain Mapp. 30, 541–552. doi: 10.1002/hbm.20525

Gunn, D. G., Naismith, S. L., and Lewis, S. J. G. (2010). Sleep disturbances in Parkinson disease and their potential role in heterogeneity. J. Geriatr. Psychiatry Neurol. 23, 131–137. doi: 10.1177/0891988709358591

Hasler, B. P., Smith, L. J., Cousins, J. C., and Bootzin, R. R. (2012). Circadian rhythms, sleep, and substance abuse. Sleep Med. Rev. 16, 67–81. doi: 10.1016/j.smrv.2011.03.004

He, Q., Zhang, P., Li, G., Dai, H., and Shi, J. (2017). The association between insomnia symptoms and risk of cardio-cerebral vascular events: a meta-analysis of prospective cohort studies. Eur. J. Prev. Cardiol. 24, 1071–1082. doi: 10.1177/2047487317702043

Higgins, J. P. T., and Green, S. (2011). Cochrane handbook for systematic reviews of interventions. 2nd Edn. Chichester, United Kingdom: John Wiley & Sons.

Higgins, J. P. T., Thompson, S. G., Deeks, J. J., and Altman, D. G. (2003). Measuring inconsistency in meta-analyses. BMJ 327, 557–560. doi: 10.1136/bmj.327.7414.557

Hirsch, M. A., and Farley, B. G. (2009). Exercise and neuroplasticity in persons living with Parkinson’s disease. Eur. J. Phys. Rehabil. Med. 45, 215–229

Hsu, W. Y., Hsu, C.-M., Hung, S.-C., and Hung, S.-Y. (2023). Acupuncture improves sleep disorders and depression among patients with Parkinson’s disease: a meta-analysis. Healthcare 11:2042. doi: 10.3390/healthcare11142042

Hsu, J.-L., Jung, T.-P., Hsu, C.-Y., Hsu, C. Y., Hsu, W. C., Chen, Y. K., et al. (2007). Regional CBF changes in Parkinson’s disease: a correlation with motor dysfunction. Eur. J. Nucl. Med. Mol. Imaging 34, 1458–1466. doi: 10.1007/s00259-006-0360-7

Hutton, B., Salanti, G., Caldwell, D. M., Chaimani, A., Schmid, C. H., Cameron, C., et al. (2015). The PRISMA extension statement for reporting of systematic reviews incorporating network meta-analyses of health care interventions: checklist and explanations. Ann. Intern. Med. 162, 777–784. doi: 10.7326/m14-2385

Jespersen, K. V., Hansen, M. H., and Vuust, P. (2023). The effect of music on sleep in hospitalized patients: a systematic review and meta-analysis. Sleep Health 9, 441–448. doi: 10.1016/j.sleh.2023.03.004

Jespersen, K. V., Koenig, J., Jennum, P., and Vuust, P.Cochrane Developmental, Psychosocial and Learning Problems Group (2015). Music for insomnia in adults. Cochrane Database Syst. Rev. 2015:CD010459. doi: 10.1002/14651858.cd010459.pub2

Jun, Y., Yongzheng, W., and Caiqiong, L. (2022). Clinical efficacy of high frequency rTMS combined with selegilan hydrochloride in the treatment of PD with excessive daytime sleepiness sleep disorder and its correlation with DTI. Chin. J. Rehabil. 37, 592–597. doi: 10.3870/zgkf.2022.10.004

Kapsimalis, F., Basta, M., Varouchakis, G., Gourgoulianis, K., Vgontzas, A., and Kryger, M. (2008). Cytokines and pathological sleep. Sleep Med. 9, 603–614. doi: 10.1016/j.sleep.2007.08.019

Kelmanson, I. A., and Adulas, E. I. (2006). Massage therapy and sleep behavior in infants born with low birth weight. Complement. Ther. Clin. Pract. 12, 200–205. doi: 10.1016/j.ctcp.2005.11.007

Kiebs, M., Hurlemann, R., and Mutz, J. (2019). Repetitive transcranial magnetic stimulation in non-treatment-resistant depression. Br. J. Psychiatry 215, 445–446. doi: 10.1192/bjp.2019.75

Koychev, I., and Okai, D. (2017). Cognitive-behavioral therapy for non-motor symptoms of Parkinson’s disease: a clinical review. Evid. Based Ment. Health 20, 15–20. doi: 10.1136/eb-2016-102574

Kraepelien, M., Schibbye, R., Månsson, K., Sundström, C., Riggare, S., Andersson, G., et al. (2020). Individually tailored internet-based cognitive-behavioral therapy for daily functioning in patients with Parkinson’s disease: a randomized controlled trial. J. Parkinsons Dis. 10, 653–664. doi: 10.3233/jpd-191894

Li, X., Chen, S., and Wu, S. (2023). Effects of repetitive transcranial magnetic stimulation on sleep and plasma orexin a content in patients with mid-to-late Parkinson’s disease. Chin. J. Phys. Med. Rehabil. 45, 232–235. doi: 10.3760/cma.j.issn.0254-1424.2023.03.008

Li, L., Tian, Z., and Zhang, X. (2021). Clinical study on electroacupuncture combined with Madopar for sleep disorders in Parkinson disease. J. New Chin. Med. 53, 113–116. doi: 10.13457/j.cnki.jncm.2021.01.030

Liu, C.-F., Wang, T., Zhan, S.-Q., Geng, D. Q., Wang, J., Liu, J., et al. (2018). Management recommendations on sleep disturbance of patients with Parkinson’s disease. Chin. Med. J. 131, 2976–2985. doi: 10.4103/0366-6999.247210

Louter, M., van Sloun, R. J. G., Pevernagie, D. A. A., Arends, J. B. A. M., Cluitmans, P. J., Bloem, B. R., et al. (2013). Subjectively impaired bed mobility in Parkinson disease affects sleep efficiency. Sleep Med. 14, 668–674. doi: 10.1016/j.sleep.2013.03.010

Luo, F., Ye, M., Lv, T., Hu, B., Chen, J., Yan, J., et al. (2021). Efficacy of cognitive behavioral therapy on mood disorders, sleep, fatigue, and quality of life in Parkinson’s diseases: a systematic review and Meta-analysis. Front. Psychiatry 12:793804. doi: 10.3389/fpsyt.2021.793804

Modugno, N., Iaconelli, S., Fiorlli, M., Lena, F., Kusch, I., and Mirabella, G. (2010). Active theater as a complementary therapy for Parkinson’s disease rehabilitation: a pilot study. Sci. World J. 10, 2301–2313. doi: 10.1100/tsw.2010.221

Moon, S., Sarmento, C. V. M., Steinbacher, M., Smirnova, I. V., Colgrove, Y., Lai, S. M., et al. (2020). Can qigong improve non-motor symptoms in people with Parkinson’s disease—a pilot randomized controlled trial? Complement. Ther. Clin. Pract. 39, 101–169. doi: 10.1016/j.ctcp.2020.101169

Nascimento, C. M. C., Ayan, C., Cancela, J. M., Gobbi, L. T., Gobbi, S., and Stella, F. (2013). Effect of a multimodal exercise program on sleep disturbances and instrumental activities of daily living performance on Parkinson’s and Alzheimer’s disease patients. Geriatr. Gerontol. Int. 14, 259–266. doi: 10.1111/ggi.12082

Nazarova, L., Liu, H., Xie, H., Wang, L., Ding, H., An, H., et al. (2022). Targeting gut-brain axis through scalp-abdominal electroacupuncture in Parkinson’s disease. Brain Res. 1790:147956. doi: 10.1016/j.brainres.2022.147956

Nerbass, F. B., Feltrim, M. I. Z., Souza, S. A. de, Ykeda, D. S., and Lorenzi-Filho, G. (2010). Effects of massage therapy on sleep quality after coronary artery bypass graft surgery. Clinics (São Paulo, Brazil) 65, 1105–10. doi: 10.1590/s1807-59322010001100008

Patel, S., Ojo, O., Genc, G., Oravivattanakul, S., Huo, Y., Rasameesoraj, T., et al. (2017). A computerized cognitive behavioral therapy randomized, controlled, pilot trial for insomnia in Parkinson disease (ACCORD-PD). J. Clin. Mov. Disord. 4:16. doi: 10.1186/s40734-017-0062-2

Paus, S., Schmitz-Hübsch, T., Wüllner, U., Vogel, A., Klockgether, T., and Abele, M. (2007). Bright light therapy in Parkinson’s disease: a pilot study. Mov. Disord. 22, 1495–1498. doi: 10.1002/mds.21542

Pruthi, S., Degnim, A. C., Bauer, B. A., DePompolo, R. W., and Nayar, V. (2009). Value of massage therapy for patients in a breast clinic. Clin. J. Oncol. Nurs. 13, 422–425. doi: 10.1188/09.cjon.422-425

Riemann, D. (2019). Sleep, insomnia and neurological and mental disorders. J. Sleep Res. 28:e12892. doi: 10.1111/jsr.12892

Rios Romenets, S., Creti, L., Fichten, C., Bailes, S., Libman, E., Pelletier, A., et al. (2013). Doxepin and cognitive behavioral therapy for insomnia in patients with Parkinson’s disease—a randomized study. Parkinsonism Relat. Disord. 19, 670–675. doi: 10.1016/j.parkreldis.2013.03.003

Rothman, S. M., and Mattson, M. P. (2012). Sleep disturbances in Alzheimer’s and Parkinson’s diseases. Neuromolecular Med. 14, 194–204. doi: 10.1007/s12017-012-8181-2

Rutten, S., Vriend, C., Smit, J. H., Berendse, H. W., van Someren, E. J. W., Hoogendoorn, A. W., et al. (2019). Bright light therapy for depression in Parkinson disease. Neurology 92, e1145–e1156. doi: 10.1212/wnl.0000000000007090

Sacheli, M. A., Neva, J. L., Lakhani, B., Murray, D. K., Vafai, N., Shahinfard, E., et al. (2019). Exercise increases caudate dopamine release and ventral striatal activation in Parkinson’s disease. Mov. Disord. 34, 1891–1900. doi: 10.1002/mds.27865

Salanti, G., Del Giovane, C., Chaimani, A., Caldwell, D. M., and Higgins, J. P. (2014). Evaluating the quality of evidence from a network meta-analysis. PLoS One 9:e99682. doi: 10.1371/journal.pone.0099682

Schootemeijer, S., van der Kolk, N. M., Bloem, B. R., and de Vries, N. M. (2020). Current perspectives on aerobic exercise in people with Parkinson’s disease. Neurotherapeutics 17, 1418–1433. doi: 10.1007/s13311-020-00904-8

Schroeck, J. L., Ford, J., Conway, E. L., Kurtzhalts, K. E., Gee, M. E., Vollmer, K. A., et al. (2016). Review of safety and efficacy of sleep medicines in older adults. Clin. Ther. 38, 2340–2372. doi: 10.1016/j.clinthera.2016.09.010

Shan, J., Wang, Y., and Lai, C. (2022). Clinical efficacy of high frequency rTMS combined with selegilan hydrochloride in the treatment of PD with excessive daytime sleepiness sleep disorder and its correlation with DTI. Journal of Rehabilitation 37, 592–597.

Shi, J., Luo, D., Weng, H., Zeng, X., Zeng, X. T., Lin, L., et al. (2020). Optimally estimating the sample standard deviation from the five-number summary. Res. Synth. Methods 11, 641–654. doi: 10.1002/jrsm.1429

Silveira, H., Moraes, H., Oliveira, N., Coutinho, E. S. F., Laks, J., and Deslandes, A. (2013). Physical exercise and clinically depressed patients: a systematic review and meta-analysis. Neuropsychobiology 67, 61–68. doi: 10.1159/000345160

Skogar, Ö., Borg, A., Larsson, B., Robertsson, L., Andersson, L., Andersson, L., et al. (2013). ‘Effects of tactile touch on pain, sleep and health related quality of life in Parkinson’s disease with chronic pain’: a randomized, controlled and prospective study. Eur. J. Integr. Med. 5, 141–152. doi: 10.1016/j.eujim.2012.10.005

Song, P., Lin, H., Li, S., Wang, L., Liu, J., Li, N., et al. (2019). Repetitive transcranial magnetic stimulation (rTMS) modulates time-varying electroencephalography (EEG) network in primary insomnia patients: a TMS-EEG study. Sleep Med. 56, 157–163. doi: 10.1016/j.sleep.2019.01.007

Stack, E. L., and Ashburn, A. M. (2006). Impaired bed mobility and disordered sleep in Parkinson’s disease. Mov. Disord. 21, 1340–1342. doi: 10.1002/mds.20944

Stefani, A., and Högl, B. (2019). Sleep in Parkinson’s disease. Neuropsychopharmacology 45, 121–128. doi: 10.1038/s41386-019-0448-y

Suzuki, T., Shimada, H., Makizako, H., Doi, T., Yoshida, D., Tsutsumimoto, K., et al. (2012). Effects of multicomponent exercise on cognitive function in older adults with amnestic mild cognitive impairment: a randomized controlled trial. BMC Neurol. 12, 128–135. doi: 10.1186/1471-2377-12-128

Videnovic, A., Klerman, E. B., Wang, W., Marconi, A., Kuhta, T., and Zee, P. C. (2017). Timed light therapy for sleep and daytime sleepiness associated with Parkinson disease. JAMA Neurol. 74, 411–418. doi: 10.1001/jamaneurol.2016.5192

Vogel, O., Niederer, D., and Vogt, L. (2021). Multimodal exercise effects in older adults depend on sleep, movement biography, and habitual physical activity: a randomized controlled trial. Front. Aging Neurosci. 13, 722–799. doi: 10.3389/fnagi.2021.722799

Wailke, S., Herzog, J., Witt, K., Deuschl, G., and Volkmann, J. (2010). Effect of controlled-release levodopa on the microstructure of sleep in Parkinson’s disease. Eur. J. Neurol. 18, 590–596. doi: 10.1111/j.1468-1331.2010.03213.x

Waller, R., and Gilbody, S. (2008). Barriers to the uptake of computerized cognitive behavioral therapy: a systematic review of the quantitative and qualitative evidence. Psychol. Med. 39, 705–712. doi: 10.1017/s0033291708004224

Wang, C. (2022). Influence of comprehensive music therapy in occupational therapy on emotional cognition and sleep in patients with Parkinson’s disease. RARM 3, 176–178.

Wang, F., Sun, L., Zhang, X., Jia, J., Liu, Z., Huang, X. Y., et al. (2015). Effect and potential mechanism of electroacupuncture add-on treatment in patients with Parkinson’s disease. Evid. Based Complement. Alternat. Med. 2015, 1–11. doi: 10.1155/2015/692795

Wu, P., Lee, M., Wu, S., Ho, H. H., Chang, M. H., Lin, H. S., et al. (2021). Effects of home-based exercise on motor, non-motor symptoms and health-related quality of life in Parkinson’s disease patients: a randomized controlled trial. Jpn. J. Nurs. Sci. 19:e12418. doi: 10.1111/jjns.12418

Wu, S., Li, X., and Qi, Y. (2020). The influence of transcranial stimulation on rapid eye movement sleep disorders among persons with Parkinson’s disease. Chin. J. Phys. Med. Rehabil. 42, 50–54. doi: 10.3760/cma.j.issn.0254-1424.2020.01.012

Xiao, C.-M., and Zhuang, Y.-C. (2015). Effect of health Baduanjin qigong for mild to moderate Parkinson’s disease. Geriatr. Gerontol. Int. 16, 911–919. doi: 10.1111/ggi.12571

Xu, Y., Cai, X., Qu, S., Zhang, J., Zhang, Z., Yao, Z., et al. (2020). Madopar combined with acupuncture improves motor and non-motor symptoms in Parkinson’s disease patients: a multicenter randomized controlled trial. Eur. J. Integr. Med. 34:101049. doi: 10.1016/j.eujim.2019.101049

Yang, C., Guo, Z., Peng, H., Xing, G., Chen, H., McClure, M. A., et al. (2018). Repetitive transcranial magnetic stimulation therapy for motor recovery in Parkinson’s disease: a meta-analysis. Brain Behav. 8:e01132. doi: 10.1002/brb3.1132

Yong, M.-H., Fook-Chong, S., Pavanni, R., Lim, L. L., and Tan, E. K. (2011). Case control polysomnographic studies of sleep disorders in Parkinson’s disease. PLoS One 6:e22511. doi: 10.1371/journal.pone.0022511

Yu, X., and Chen, H. (2022). Clinical effect and prognosis analysis of repeated transcranial magnetic stimulation in the treatment of patients with Parkinson’s disease and insomnia. China Foreign Med. Treat. 41, 20–24. doi: 10.16662/j.cnki.1674-0742.2022.04.020

Zhang, H., Gu, Z., An, J., Wang, C., and Chan, P. (2014). Non-motor symptoms in treated and untreated Chinese patients with early Parkinson’s disease. Tohoku J. Exp. Med. 232, 129–136. doi: 10.1620/tjem.232.129

Zhang, W., Nie, K., and Zhang, Y. (2014). Observation on the effect of repetitive transcranial magnetic stimulation in the treatment of insomnia in patients with Parkinson’s disease. J. Nurs. 21, 28–30. doi: 10.16460/j.issn1008-9969.2014.23.012

Zhang, X., Zhuang, S., Wu, J., Wang, L., Mao, C., Chen, J., et al. (2022). Effects of repetitive transcranial magnetic stimulation over right dorsolateral prefrontal cortex on excessive daytime sleepiness in patients with Parkinson’s disease. Sleep Med. 100, 133–138. doi: 10.1016/j.sleep.2022.08.003

Zhu, M., Zhang, Y., Pan, J., Fu, C., and Wang, Y. (2020). Effect of simplified tai chi exercise on relieving symptoms of patients with mild to moderate Parkinson’s disease. J. Sports Med. Phys. Fitness 60, 282–288. doi: 10.23736/s0022-4707.19.10104-1

Zhuang, S., Wang, F.-Y., Gu, X., Wu, J. J., Mao, C. J., Gui, H., et al. (2020). Low-frequency repetitive transcranial magnetic stimulation over right dorsolateral prefrontal cortex in Parkinson’s disease. Parkinson’s Dis. 2020, 1–7. doi: 10.1155/2020/7295414

Zuzuárregui, J. R. P., and During, E. H. (2020). Sleep Issues in Parkinson’s Disease and Their Management. Neurotherapeutics , 17, 1480–1494. doi: 10.1007/s13311-020-00938-y

Keywords: non-pharmacological interventions, sleep quality, network meta-analysis, Parkinson’s disease, randomized controlled trials

Citation: Tang R, Gong S, Li J, Hu W, Liu J and Liao C (2024) Efficacy of non-pharmacological interventions for sleep quality in Parkinson’s disease: a systematic review and network meta-analysis. Front. Neurosci . 18:1337616. doi: 10.3389/fnins.2024.1337616

Received: 13 November 2023; Accepted: 23 January 2024; Published: 21 February 2024.

Reviewed by:

Copyright © 2024 Tang, Gong, Li, Hu, Liu and Liao. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY) . The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

*Correspondence: Jihong Liu, [email protected] ; Chunlian Liao, [email protected]

Disclaimer: All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.

IMAGES

  1. cochrane systematic review funnel plot

    cochrane systematic review funnel plot

  2. cochrane systematic review funnel plot

    cochrane systematic review funnel plot

  3. How to Find Systematic Reviews In Cochrane

    cochrane systematic review funnel plot

  4. cochrane systematic review funnel plot

    cochrane systematic review funnel plot

  5. FAQ SR: Bias & certainty of evidence

    cochrane systematic review funnel plot

  6. Systematic Reviews: an overview

    cochrane systematic review funnel plot

VIDEO

  1. Plot in Cochrane, ON, in just $10/-

  2. Forest Plot vs Funnel plot #research #metaanalysis #systematicreview #publicationbias #analysis

  3. 53. FRCS ortho Audio file- Basic science : Funnel plot

  4. Causality and Machine Learning (3)

  5. A Brief History of Systematic Review

  6. Elaboración de un forest y funnel plot en SPSS

COMMENTS

  1. 10.4.1 Funnel plots

    These regions are included in Figure 10.4.a. Funnel plots of effect estimates against their standard errors (on a reversed scale) can be created using RevMan. A triangular 95% confidence region based on a fixed-effect meta-analysis can be included in the plot, and different plotting symbols allow studies in different subgroups to be identified.

  2. Recommendations for examining and interpreting funnel plot ...

    The recommendations are based on a detailed MEDLINE review of literature published up to 2007 and discussions among methodologists, who extended and adapted guidance previously summarised in the Cochrane Handbook for Systematic Reviews of Interventions. 7 A funnel plot is a scatter plot of the effect estimates from individual studies against ...

  3. 10.4.3.1 Recommendations on testing for funnel plot asymmetry

    Based on a survey of meta-analyses published in the Cochrane Database of Systematic Reviews, these criteria imply that tests for funnel plot asymmetry should be used in only a minority of meta-analyses (Ioannidis 2007b). Tests for which there is insufficient evidence to recommend use. The following comments apply to all intervention measures.

  4. How to read a funnel plot

    A funnel plot is a scatter plot that compares the precision (how close the estimated intervention effect size is to the true effect size) and results of individual studies. It is commonly used in meta-analyses to visually detect publication bias. The term 'funnel plot' refers to the fact that the precision of the estimated intervention ...

  5. PDF Outline Publication and other reporting biases; funnel plots ...

    Assessing Risk Of Bias In Cochrane Systematic Reviews Loughborough UK, March 2012 Publication and other reporting biases; funnel plots and asymmetry tests ... Ioannidis JPA et al. Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials. BMJ 2011;342:d4002 doi: 10.1136/bmj.d4002 20 0. ...

  6. 10.4.2 Different reasons for funnel plot asymmetry

    In interpreting funnel plots, systematic review authors thus need to distinguish the different possible reasons for funnel plot asymmetry listed in Table 10.4.a. Knowledge of the particular intervention, and the circumstances in which it was implemented in different studies, can help identify true heterogeneity as a cause of funnel plot asymmetry.

  7. Detecting small‐study effects and funnel plot asymmetry in meta

    Detecting small‐study effects and funnel plot asymmetry in meta‐analysis of survival data: A comparison of new and existing tests ... Sterne JAC, Egger M, Moher D. Chapter 10: Addressing reporting biases In: Higgins JPT, Green S, eds. Cochrane Handbook for Systematic Reviews of Interventions. 5.1.0, Vol. 10: Cochrane; 2011. www.cochrane ...

  8. The role of the funnel plot in detecting publication and ...

    Evidence-Based Dentistry - The role of the funnel plot in detecting publication and related biases in meta-analysis. ... The Cochrane Handbook for Systematic Reviews of Interventions, ...

  9. Identifying publication bias in meta-analyses of continuous ...

    Egger's test is commonly used to assess potential publication bias in a meta-analysis via funnel plot asymmetry (Egger's test is a linear regression of the intervention effect estimates on their standard errors weighted by their inverse variance). The performance of Egger's and related tests has been extensively studied for binary ...

  10. Assessment of funnel plot asymmetry and publication bias in

    The 9 th issue of the WHO Reproductive Health Library included 105 Cochrane Systematic Reviews. A total of 21 systematic reviews contained meta-analyses with 10 or more trials reporting a binary outcome. In this set of meta-analyses, the number of trials comprising each one ranged from 10 to 83 (median = 13), totaling 379 trials, whose results have been summarized.

  11. Bias in meta-analysis detected by a simple, graphical test

    Funnel plot asymmetry was found in 38% of meta-analyses published in leading general medicine journals and in 13% of reviews from the Cochrane Database of Systematic Reviews Critical examination of systematic reviews for publication and related biases should be considered a routine procedure

  12. The misuse of funnel plots in meta-analyses of proportions: are they

    The authors used funnel plots and Egger's regression test to assess for publication bias in their pooled proportions. ... Of note, the Cochrane Handbook for Systematic Reviews of Interventions mandates the search of at least two databases to provide extensive coverage and reduce publication bias, while the authors only searched one database ...

  13. (PDF) Recommendations for examining and interpreting funnel plot

    Written for review authors, editors, trainers and others with an interest in Cochrane Reviews, the second edition of The Cochrane Handbook for Systematic Reviews of Interventions continues to ...

  14. The appropriateness of asymmetry tests for publication bias in ...

    We therefore appraised almost 7000 meta-analyses in the Cochrane Database of Systematic Reviews to discover the extent to which tests of funnel-plot asymmetry would be inappropriate or nonconcordant. We also examined the appropriateness of the application of asymmetry testing in meta-analyses recently published in print journals.

  15. Assessment of funnel plot asymmetry and publication bias in

    Systematic reviews included in the Reproductive Health Library (RHL), issue No 9, were assessed. Funnel plot was used to assess meta-analyses containing 10 or more trials reporting a binary outcome. A funnel plot, the estimated number of missing studies and the adjusted combined effect size were obtained using the "trim and fill method".

  16. Bias in meta-analysis detected by a simple, graphical test

    In all cases discordance was due to meta-analyses showing larger effects. Funnel plot asymmetry was present in three out of four discordant pairs but in none of concordant pairs. In 14 (38%) journal meta-analyses and 5 (13%) Cochrane reviews, funnel plot asymmetry indicated that there was bias.

  17. The Perils of Misinterpreting and Misusing "Publication ...

    Only 12 of the 41 systematic reviews (26.1%) reported that a minimum of ten studies was required to assess the risk of publication bias. Two of these systematic reviews stipulated this criterion for Egger's test but not for visual inspection of funnel plots (Table 1).

  18. Association of nonpharmacological interventions for cognitive function

    The Cochrane Handbook for Systematic Reviews of Interventions described the calculation methods. The random-effects model was used in pairwise meta-analysis. ... A comparison-adjusted funnel plot was also produced to explore small-study effects and publication bias. ... This systematic review and network meta-analysis was performed based on 28 ...

  19. Forest plots in reports of systematic reviews: a cross-sectional study

    Of the 171 non-Cochrane reviews, 28 (16%) had at least one forest plot. Non-Cochrane reviews with a forest plot had a mean of five plots (median 2, IQR 1-4). The three studies with the largest number of plots had 44, 20 and 9. A total of 3% (4/129) of Cochrane and 3% (5/171) of non-Cochrane reviews had funnel plots.

  20. Detecting small‐study effects and funnel plot asymmetry in meta

    For all methods, a 2-tailed t-test with test statistic and (m pb −2) degrees of freedom can be performed to formally assess whether asymmetry occurs. 22 Hereby, it is common to use a 10% level of significance because the number of studies in a meta-analysis is usually low. 3 EXAMPLES. We illustrate the implementation of aforementioned tests for detecting funnel plot asymmetry in 3 example ...

  21. Effects of game-based physical education program on enjoyment in

    The quality assessment of the included researches is conducted based on the guidelines outlined in the Cochrane 5.1 handbook. Review Manager 5.3 software is employed to synthesis the effect sizes. Additionally, bias is assessed using funnel plots, and to identify potential sources of heterogeneity, subgroup analyses are performed.

  22. Funnel plot

    Funnel plots, introduced by Light and Pillemer in 1984 [1] and discussed in detail by Matthias Egger and colleagues, [2] [3] are useful adjuncts to meta-analyses. A funnel plot is a scatterplot of treatment effect against a measure of study precision. It is used primarily as a visual aid for detecting bias or systematic heterogeneity.

  23. Frontiers

    To assess publication bias, a comparison-adjusted network funnel plot with a random model was constructed for the outcome . The results were roughly distributed around the overall effect, arranged symmetrically around the center line. ... Cochrane handbook for systematic reviews of interventions. 2nd Edn. Chichester, United Kingdom: John Wiley ...