The comparative effectiveness of physical exercise interventions in individuals with chronic non-specific neck pain: protocol for a network meta-analysis

Abstract

Introduction

Neck pain is a global burdensome problem, with a large proportion of neck pain cases becoming chronic. Although physical exercise is a commonly prescribed treatment, the evidence on the effectiveness of isolated exercise interventions remains limited. Traditional pairwise randomised controlled trials (RCTs) and meta-analyses are limited in only comparing two interventions. This protocol describes the design of a network meta-analysis, which enables a comparative investigation of all physical exercise interventions for which RCTs are available. We aim to systematically compare the effectiveness of different types of physical exercise in people with chronic non-specific neck pain.

Methods and analysis

Nine electronic databases (AMED, CINAHL, Cochrane Central Register of Controlled Trials, Embase, MEDLINE, Physiotherapy Evidence Database, PsycINFO, Scopus and SPORTDiscus) were searched for RCTs from inception to 12 March 2019. Titles and abstract firstly, and full-text papers secondly, will be screened by two reviewers. Data will be extracted by two reviewers. The primary outcome measure is effectiveness of the intervention. Methodological quality of included studies will be assessed by two reviewers using the PEDro scale. The overall quality of evidence will be assessed with the Grading of Recommendations Assessment, Development and Evaluation (GRADE) framework, which has been adapted for network meta-analyses. The available evidence will be summarised using a network diagram. A contribution matrix will be presented to allow assessment of direct and indirect evidence. Forest plots will be constructed to visualise effects of all included exercise interventions. Pairwise effect sizes will be calculated by including all evidence available in the network. Effect measures for treatments that have not been compared in a pairwise RCT can be compared indirectly by contrasting effect sizes of comparisons with a common comparator.

Ethics and dissemination

This work synthesises evidence from previously published studies and does not require ethics review or approval. A manuscript describing the findings will be submitted for publication in a peer-reviewed scientific journal.

PROSPERO registration number

CRD42019126523.

Correspondence to Dr Rutger MJ de Zoete; [email protected]

Strengths and limitations of this study

• This protocol describes the design of a network meta-analysis investigating the effects of physical exercise interventions for people with chronic non-specific neck pain.

• Network meta-analysis enables a comparative investigation of all available physical exercise interventions for which randomised controlled trials are available.

• The results from this network meta-analysis will provide directly clinically applicable evidence.

• The overall quality of evidence will be assessed with the Grading of Recommendations Assessment, Development and Evaluation (GRADE), allowing to assess the certainty of evidence for the network meta-analysis.

• As the effects of physical exercise may be different across different neck pain conditions (eg, non-specific neck pain and traumatic neck pain), this network meta-analysis includes only chronic non-specific neck pain and findings might not be transferrable onto other neck pain populations.

Introduction

Neck pain is a global burdensome problem.¹ Up to 70% of the general population will experience neck pain at least once in their lives,^2–4 and in a substantial proportion of people, the pain will become recurrent or chronic.^{5 6} Clinical guidelines recommend physical exercise as a first-line treatment for neck pain.⁷ Different suggestions have been made regarding the type of exercise required in order to improve clinical outcomes, including strengthening, range of motion, motor control, stretching and proprioceptive training.⁸

Several systematic reviews have examined the effectiveness of physical exercise interventions in people with neck pain,^9–12 indicating that combinations of different types of therapeutic exercise have moderate effects on pain. Notably, the inclusion of different exercise interventions in each of these reviews does not allow for an investigation of the efficacy of isolated types of physical exercise. However, for a traditional (meta-analytic) pairwise investigation of these isolated exercise interventions, insufficient randomised controlled trials (RCTs) are available to address each of the different types of physical exercise.

Based on individual RCTs, as well as systematic reviews and meta-analyses, it is difficult to draw conclusions regarding the effectiveness of different physical exercise interventions for people with chronic neck pain.^9–12 Traditional meta-analyses are limited by the availability of pairwise comparisons between interventions.¹³ It is therefore difficult to interpret the entire body of evidence available, with many RCTs available for some interventions, and limited evidence for others. Furthermore, for many types of exercise interventions, no direct comparisons are available, and it is not possible to compare the effectiveness relative to each other.¹⁴ Network meta-analysis (NMA) is a study design that allows for an investigation of the efficacy of different interventions.^{15 16} This approach was recently used to investigate the effectiveness of pharmacological management for depression¹⁷ and physical exercise for chronic low back pain.¹⁸ By creating a network of pairwise RCTs, all direct and indirect evidence can be used in order to determine the effectiveness of interventions.¹⁹ Rather than comparing only two exercise interventions as is traditionally done in systematic review and meta-analysis, NMA enables a comparative analysis of all physical exercise interventions for which RCTs are available. Furthermore, this design allows for creating a ranking of the efficacy of available physical exercise interventions.

This protocol describes the design of an NMA with the aim of systematically comparing the effectiveness of different types of physical exercise in people with chronic non-specific neck pain. The primary research questions is: What is the effectiveness of different types of physical exercise on neck pain intensity and neck disability? The objective is to evaluate the effectiveness of different types of physical exercise, which will allow us to generate a clinically meaningful hierarchy of physical exercise interventions. The secondary research question is: What is the effectiveness of different durations, frequencies and intensities of physical exercise interventions on neck pain intensity and neck disability? Investigating the effectiveness of different durations, frequencies and intensities of exercise will provide valuable information in order to give specific recommendations for therapeutic exercise prescription.

Methods

This protocol follows the Preferred Reporting Items for Systematic Reviews and Meta-Analyses statement for protocols (PRISMA-P).²⁰ The results of the NMA will be reported according to the PRISMA statement and PRISMA extension for network meta-analyses (PRISMA-NMA).^{21 22}

Eligibility criteria

Studies are eligible to be included in the NMA if they are an RCT describing the effects of any physical exercise intervention in adults (age ≥18 years) with chronic non-specific neck pain (symptoms persisting for ≥12 weeks). Neck pain is defined as the pain between the occiput and the first thoracic vertebra as primary complaint. Other terms for non-specific neck pain that may be used are idiopathic neck pain, insidious-onset neck pain, mechanical neck pain and work-related neck pain. As comparator, any physical exercise intervention, control-treatment, sham-treatment, placebo-treatment or no-treatment group, will be included. Studies will be excluded if they report subjects younger than 18 years, non-human subjects or subjects with traumatic neck pain. Studies reporting primary complaints other than neck pain, such as postconcussion syndrome, headache and migraine, will be excluded.

Studies that are available in full-text form and published in the English language will be included, and no date limits will be applied. Studies assessed to have a high risk of bias will be excluded.

Categorisation of studies

Through an iterative process of reviewing relevant RCTs and discussion, identified 10 categories for the current NMA were identified: (A) strengthening, (B) range of motion, (C) stretching, (D) motor control, (E) proprioceptive, (F) strengthening + motor control, (G) strengthening + stretching, (H) comprehensive, (I) general physical activity and (J) no treatment/sham/information. The comprehensive exercise may include combinations of >3 other categories, for example, combining strengthening, range of motion, motor control and stretching exercises.

Information sources

Nine electronic databases (AMED, CINAHL, Cochrane Central Register of Controlled Trials, Embase, MEDLINE, Physiotherapy Evidence Database, PsycINFO, Scopus and SPORTDiscus) were searched for RCTs from inception to 12 March 2019.

Search strategy

A search strategy was developed with a medical librarian; the three-part strategy includes terms to identify studies relating to (1) physical exercise, (2) chronic neck pain and (3) RCTs. The neck pain search terms were consistent with those recommended by the Cochrane Back and Neck review group.²³ The search strategy was developed for the MEDLINE database (online supplementary appendix A), and subsequently adjusted to the requirements of the other databases. The electronic searches will be complemented with manual searches for prospectively identified systematic reviews and meta-analyses, of which the reference lists may include potentially relevant articles. The reference lists of included studies will further be searched to identify any studies missed by the electronic searches.

Selection process

Two reviewers will independently screen titles and abstracts to identify potentially eligible studies. For each identified study, two reviewers will independently review the full-text papers. In either stage, a third reviewer will resolve any disagreements on study inclusion as necessary. Inter-rater agreement will be reported using Cohen’s kappa coefficient (κ). Where studies have been reported in multiple papers, only the paper reporting the most complete analysis of effectiveness will be included (ie, reports of subgroup or secondary analyses will be discarded).

Data collection and management

Two reviewers will extract and record data from the included studies using a standardised extraction table agreed by all authors.

Data items

Extracted data will comprise study characteristics (author, year), participant characteristics (sample sizes, age sex, type of neck pain), type of exercise intervention, duration, frequency and intensity,and timing of follow-up assessment. Means and SDs for primary outcome measures at baseline, and the time point after, and closest to, the end of the treatment will be extracted, accommodating an anticipated variation in treatment duration across studies. Although currently there is no consensus on the appropriate duration of exercise interventions, it is expected that most intervention durations will be between 4 and 12 weeks. With potential differences in treatment durations, this second time point allows for an investigation at a time point ensuring the treatment regimen is completed, and it is likely to be the point of maximal therapeutic effect.

Where studies report more than two physical exercise interventions (or control groups) which independently could be included in this NMA, data from all study arms will be extracted. For example, if one particular RCT describes three treatment arms (A, B, and C), data from all three groups will be extracted.

For primary outcomes where mean±SE are reported, SDs will be calculated using the formula: SD=SE∗n. Where medians and IQRs are reported, the methods described by Wan et al will be used to compute means and SDs.²⁴ Where means and 95% CIs are reported, SDs will be calculated using the formula: ${\displaystyle \mathrm{S}\mathrm{D}=\sqrt{\mathrm{n}}\ast \frac{\left(\mathrm{u}\mathrm{p}\mathrm{p}\mathrm{e}\mathrm{r}95\mathrm{\%}\mathrm{C}\mathrm{I}\mathrm{l}\mathrm{i}\mathrm{m}\mathrm{i}\mathrm{t}-\mathrm{l}\mathrm{o}\mathrm{w}\mathrm{e}\mathrm{r}95\mathrm{\%}\mathrm{C}\mathrm{I}\mathrm{l}\mathrm{i}\mathrm{m}\mathrm{i}\mathrm{t}\right)}{\mathrm{t}}}$, where t is the value from a t-distribution for a 95% CI for a sample distribution with degrees of freedom of the group sample size −1. If insufficient data are available in the paper, an attempt will be made to obtain them from the corresponding author. Extracted data will be tabulated.

Outcomes and prioritisation

The primary outcome measures are the effectiveness of the intervention, as a continuous outcome, measured as the end-point score on pain intensity (eg, Visual Analogue Scale) or neck disability outcomes (eg, Neck Disability Index). Two separate analyses will be conducted for each primary outcome measure.

Risk of bias in individual studies

Two reviewers will assess the methodological quality of the included studies using the PEDro scale, which is a validated tool²⁵ for assessing the risk of bias of RCTs and commonly used to assess physiotherapeutic interventions. A third reviewer will resolve disagreements as required. Inter-rater agreement will be reported using Cohen’s kappa coefficient (κ).

Data synthesis

The characteristics of the included trials (type of neck pain, details of the physical exercise intervention, outcomes) will be summarised and tabulated. The available evidence will be summarised using a network diagram in which each node will represent a class of intervention (as categorised in the inclusion criteria), with node size being proportional to the number of patients receiving the treatment. The effect of pairwise comparisons of two interventions will be shown as edges interconnecting the nodes, where the thickness of the edge lines will represent the weight of pairwise comparisons. A contribution matrix will be presented to show the influence of individual comparisons, and the influence of direct and indirect evidence on the overall summary of effects.

Assessment of transitivity and meta-biases

For the exercise interventions identified in the preliminary search, it is anticipated that all physical exercise interventions are in-principle jointly randomisable, meeting the transitivity assumption. For all comparisons between physical exercise interventions in the network, inferences can be based on direct evidence (pairwise RCTs), indirect evidence (effect B–C derived from A–B and A–C comparisons) or a mixture of direct and indirect evidence. Measures that could potentially modify effects include sex, age and type of neck pain, and distributions of these variables will be inspected in order to meet the transitivity assumption.

Network meta-analysis

Assuming the distribution of effect modifiers are similar across studies, a frequentist NMA will be conducted (proposed closed network geometry is shown in figure 1). Pairwise effect sizes will be calculated by including all evidence available in the network.²⁶ Where secondary outcome data are available, the effectiveness of different durations, frequencies and intensities of physical exercise interventions will be investigated. Effect measures for treatments that have not been compared in a pairwise RCT can be compared indirectly by contrasting effect sizes of comparisons with a common comparator.^{13 27 28} Because previous systematic reviews of exercise for neck pain have shown varying effects, a random effects (DerSimonian and Laird²⁹) model will be used to generate pooled standardised effect sizes. Corrected effect size (Hedges’ g) will be used to allow for the inclusion of smaller studies.³⁰ Network forest plots, interval plots and league tables will be used to present the ranking of mixed (direct and indirect) effect sizes and 95% CIs for all combinations of treatments in the network.

Figure 1. All possible network connections (pairwise comparisons, lines) with 10 nodes (interventions, A-M: (A) strengthening, (B) range of motion, (C) stretching, (D) motor control, (E) proprioceptive, (F) strengthening + motor control, (G) strengthening + stretching, (H) comprehensive, (I) PPA, (J) Yoga/Pilates/Tai Chi/Qigong, (K) PPA + strengthening, (L) balance, (M) no treatment/sham/information. PPA, prescribed physical activity.

Exploring heterogeneity and assessment of inconsistency

Heterogeneity will be reported using 95% prediction intervals and I².Forest plots will be visually examined to identify any obvious inconsistency between direct and indirect treatment effects (loop consistency); any observed inconsistency may indicate that the transitivity assumption has not been satisfied. If significant heterogeneity is detected, the node-splitting approach will be used to evaluate inconsistency one comparison at a time.³¹ Comparison-adjusted funnel plots will be used to visually inspect and assess for small study effects, and assess potential publication bias.³²

Confidence in cumulative evidence

Based on study limitations, imprecision, heterogeneity indirectness and publication bias,³³ the overall quality of evidence will be assessed with the Grading of Recommendations Assessment, Development and Evaluation (GRADE) framework,³⁴ which has been adapted for NMA.³⁵ Each key pairwise comparison within the network will be assessed, from which a conclusion will be drawn regarding the full network’s certainty of evidence.

Statistical analyses

Statistical package R³⁶ will be used for all statistical analyses. The netmeta R-package will be used to conduct and report the NMA. P values will allow for the ranking of treatment efficacy. The netmeta package function forest.netmeta will be used to create a visual network of nodes and connections.

Ethics and dissemination

This work synthesises evidence from previously published studies and does not require ethics review or approval. A manuscript describing the findings will be submitted for publication in a peer-reviewed scientific journal; the results will be presented according to the PRIMSA, and the PRIMSA extension for network meta analyses (PRISMA-NMA) guidelines.^{21 22}

Patient and public involvement

There was no patient or public involvement in this review.

Footnote

Contributors RdZ conceived the content, wrote the paper, and approved the final version. NRA and JHM contributed and approved the final version of the paper. MS conceived the design of the paper and approved the final version of the manuscript.

Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

Competing interests None declared.

Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.

Patient consent for publication Not required.

Provenance and peer review Not commissioned; externally peer reviewed.

References

¹ Martin BI, Deyo RA, Mirza SK, et al. Expenditures and health status among adults with back and neck problems. JAMA 2008; 299: 656–64. doi:10.1001/jama.299.6.656 http://www.ncbi.nlm.nih.gov/pubmed/18270354

² Bronfort G, Evans R, Anderson AV, et al. Spinal manipulation, medication, or home exercise with advice for acute and subacute neck pain: a randomized trial. Ann Intern Med 2012; 156: 1–10. doi:10.7326/0003-4819-156-1-201201030-00002 http://www.ncbi.nlm.nih.gov/pubmed/22213489

³ Fejer R, Kyvik KO, Hartvigsen J. The prevalence of neck pain in the world population: a systematic critical review of the literature. Eur Spine J 2006; 15: 834–48. doi:10.1007/s00586-004-0864-4 http://www.ncbi.nlm.nih.gov/pubmed/15999284

⁴ Hoy D, March L, Woolf A, et al. The global burden of neck pain: estimates from the global burden of disease 2010 study. Ann Rheum Dis 2014; 73: 1309–15. doi:10.1136/annrheumdis-2013-204431 http://www.ncbi.nlm.nih.gov/pubmed/24482302

⁵ Haldeman S, Carroll L, Cassidy JD, et al. The bone and joint decade 2000–2010 Task force on neck pain and its associated disorders. Eur Spine J 2008; 17: 5–7. doi:10.1007/s00586-008-0619-8

⁶ Mäkelä M, Heliövaara M, Sievers K, et al. Prevalence, determinants, and consequences of chronic neck pain in Finland. Am J Epidemiol 1991; 134: 1356–67. doi:10.1093/oxfordjournals.aje.a116038 http://www.ncbi.nlm.nih.gov/pubmed/1755449

⁷ State Insurance Regulatory Authority. Guidelines for the management of acute whiplash-associated disorders–for health professionals. 3 edn. Sydney, 2014.

⁸ O'Riordan C, Clifford A, Van De Ven P, et al. Chronic neck pain and exercise interventions: frequency, intensity, time, and type principle. Arch Phys Med Rehabil 2014; 95: 770–83. doi:10.1016/j.apmr.2013.11.015 http://www.ncbi.nlm.nih.gov/pubmed/24333741

⁹ Louw S, Makwela S, Manas L, Meyer L, et al. Effectiveness of exercise in office workers with neck pain: a systematic review and meta-analysis. S Afr J Physiother 2017; 73: 392. doi:10.4102/sajp.v73i1.392 http://www.ncbi.nlm.nih.gov/pubmed/30135909

¹⁰ Bertozzi L, Gardenghi I, Turoni F, et al. Effect of therapeutic exercise on pain and disability in the management of chronic nonspecific neck pain: systematic review and meta-analysis of randomized trials. Phys Ther 2013; 93: 1026–36. doi:10.2522/ptj.20120412 http://www.ncbi.nlm.nih.gov/pubmed/23559524

¹¹ de Zoete RMJ, Brown L, Oliveira K, et al. The effectiveness of general physical exercise for individuals with chronic neck pain: a systematic review of randomised controlled trials. Eur J Physiother 2019; 43: 1–7. doi:10.1080/21679169.2018.1561942

¹² Miller J, Gross A, D'Sylva J, et al. Manual therapy and exercise for neck pain: a systematic review. Man Ther 2010; 15: 334–54. doi:10.1016/j.math.2010.02.007 http://www.ncbi.nlm.nih.gov/pubmed/20593537

¹³ Lu G, Ades AE. Combination of direct and indirect evidence in mixed treatment comparisons. Stat Med 2004; 23: 3105–24. doi:10.1002/sim.1875 http://www.ncbi.nlm.nih.gov/pubmed/15449338

¹⁴ Mavridis D, Giannatsi M, Cipriani A, et al. A primer on network meta-analysis with emphasis on mental health. Evid Based Ment Health 2015; 18: 40–6. doi:10.1136/eb-2015-102088 http://www.ncbi.nlm.nih.gov/pubmed/25908686

¹⁵ Cipriani A, Higgins JPT, Geddes JR, et al. Conceptual and technical challenges in network meta-analysis. Ann Intern Med 2013; 159: 130–7. doi:10.7326/0003-4819-159-2-201307160-00008 http://www.ncbi.nlm.nih.gov/pubmed/23856683

¹⁶ Li T, Puhan MA, Vedula SS, et al. Network meta-analysis-highly attractive but more methodological research is needed. BMC Med 2011; 9: 79. doi:10.1186/1741-7015-9-79 http://www.ncbi.nlm.nih.gov/pubmed/21707969

¹⁷ Cipriani A, Furukawa TA, Salanti G, et al. Comparative efficacy and acceptability of 21 antidepressant drugs for the acute treatment of adults with major depressive disorder: a systematic review and network meta-analysis. Lancet 2018; 391: 1357–66. doi:10.1016/S0140-6736(17)32802-7 http://www.ncbi.nlm.nih.gov/pubmed/29477251

¹⁸ Owen PJ, Miller CT, Mundell NL, et al. Which specific modes of exercise training are most effective for treating low back pain? network meta-analysis. Br J Sports Med 2019. doi: doi:10.1136/bjsports-2019-100886. [Epub ahead of print: 30 Oct 2019]. http://www.ncbi.nlm.nih.gov/pubmed/31666220

¹⁹ Caldwell DM, Ades AE, Higgins JPT. Simultaneous comparison of multiple treatments: combining direct and indirect evidence. BMJ 2005; 331: 897–900. doi:10.1136/bmj.331.7521.897 http://www.ncbi.nlm.nih.gov/pubmed/16223826

²⁰ Shamseer L, Moher D, Clarke M, et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015: elaboration and explanation. BMJ 2015; 350: g7647. doi:10.1136/bmj.g7647 http://www.ncbi.nlm.nih.gov/pubmed/25555855

²¹ Hutton B, Salanti G, Caldwell DM, et al. The PRISMA extension statement for reporting of systematic reviews incorporating network meta-analyses of health care interventions: checklist and explanations. Ann Intern Med 2015; 162: 777–84. doi:10.7326/M14-2385 http://www.ncbi.nlm.nih.gov/pubmed/26030634

²² Moher D, Liberati A, Tetzlaff J, et al. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med 2009; 6: e1000097–6. doi:10.1371/journal.pmed.1000097 http://www.ncbi.nlm.nih.gov/pubmed/19621072

²³ Chiu TTW, Lam T-H, Hedley AJ. A randomized controlled trial on the efficacy of exercise for patients with chronic neck pain. Spine 2005; 30: E1–7. doi:10.1097/01.brs.0000149082.68262.b1 http://www.ncbi.nlm.nih.gov/pubmed/15626966

²⁴ Wan X, Wang W, Liu J, et al. Estimating the sample mean and standard deviation from the sample size, median, range and/or interquartile range. BMC Med Res Methodol 2014; 14: 135. doi:10.1186/1471-2288-14-135 http://www.ncbi.nlm.nih.gov/pubmed/25524443

²⁵ Yamato TP, Maher C, Koes B, et al. The PEDro scale had acceptably high convergent validity, construct validity, and interrater reliability in evaluating methodological quality of pharmaceutical trials. J Clin Epidemiol 2017; 86: 176–81. doi:10.1016/j.jclinepi.2017.03.002 http://www.ncbi.nlm.nih.gov/pubmed/28288916

²⁶ Guyatt GH, Oxman AD, Santesso N, et al. Grade guidelines: 12. preparing summary of findings tables-binary outcomes. J Clin Epidemiol 2013; 66: 158–72. doi:10.1016/j.jclinepi.2012.01.012 http://www.ncbi.nlm.nih.gov/pubmed/22609141

²⁷ Lumley T. Network meta-analysis for indirect treatment comparisons. Stat Med 2002; 21: 2313–24. doi:10.1002/sim.1201 http://www.ncbi.nlm.nih.gov/pubmed/12210616

²⁸ Salanti G, Higgins JPT, Ades AE, et al. Evaluation of networks of randomized trials. Stat Methods Med Res 2008; 17: 279–301. doi:10.1177/0962280207080643 http://www.ncbi.nlm.nih.gov/pubmed/17925316

²⁹ DerSimonian R, Laird N. Meta-Analysis in clinical trials. Control Clin Trials 1986; 7: 177–88. doi:10.1016/0197-2456(86)90046-2 http://www.ncbi.nlm.nih.gov/pubmed/3802833

³⁰ Hedges LV, Olkin I. Statistical methods for meta-analysis Orlando. Florida: Academic Press, 2014.

³¹ van Valkenhoef G, Dias S, Ades AE, et al. Automated generation of node-splitting models for assessment of inconsistency in network meta-analysis. Res Synth Methods 2016; 7: 80–93. doi:10.1002/jrsm.1167 http://www.ncbi.nlm.nih.gov/pubmed/26461181

³² Chaimani A, Salanti G. Using network meta-analysis to evaluate the existence of small-study effects in a network of interventions. Res Synth Methods 2012; 3: 161–76. doi:10.1002/jrsm.57 http://www.ncbi.nlm.nih.gov/pubmed/26062088

³³ Salanti G, Del Giovane C, Chaimani A, et al. Evaluating the quality of evidence from a network meta-analysis. PLoS One 2014; 9: e99682. doi:10.1371/journal.pone.0099682 http://www.ncbi.nlm.nih.gov/pubmed/24992266

³⁴ Guyatt GH, Oxman AD, Kunz R, et al. GRADE guidelines: 8. Rating the quality of evidence--indirectness. J Clin Epidemiol 2011; 64: 1303–10. doi:10.1016/j.jclinepi.2011.04.014 http://www.ncbi.nlm.nih.gov/pubmed/21802903

³⁵ Puhan MA, Schünemann HJ, Murad MH, et al. A grade Working group approach for rating the quality of treatment effect estimates from network meta-analysis. BMJ 2014; 349: g5630. doi:10.1136/bmj.g5630 http://www.ncbi.nlm.nih.gov/pubmed/25252733

³⁶ Team RC. R: a language and environment for statistical computing. R foundation for statistical computing, 2018.