Oncotarget

Clinical Research Papers:

A quality analysis of clinical anaesthesia study protocols from the Chinese clinical trials registry according to the SPIRIT statement

PDF |  HTML  |  How to cite

Oncotarget. 2018; 9:24830-24836. https://doi.org/10.18632/oncotarget.24982

Metrics: PDF 1277 views  |   HTML 1683 views  |   ?  

Lei Yang, Shouming Chen, Di Yang, Jiajin Li, Taixiang Wu and Yunxia Zuo _

Abstract

Lei Yang1,*, Shouming Chen2,*, Di Yang3, Jiajin Li4, Taixiang Wu5 and Yunxia Zuo6

1Department of Anaesthesiology, West China Hospital, Sichuan University, Chengdu, China

2Department of Anaesthesiology, Second Affiliated Hospital, Zhejiang University School of Medicine, Hangzhou, China

3Department of Anaesthesiology, Sichuan Provincial People’s Hospital, Chengdu, China

4Medical Insurance Office, West China Hospital, Sichuan University, Chengdu, China

5Chinese Clinical Trials Registry, Chengdu, China

6Department of Anaesthesiology, West China Hospital, Sichuan University, Chengdu, China

*These authors have contributed equally to this work

Correspondence to:

Yunxia Zuo, email: [email protected]

Keywords: clinical study; protocol; anaesthesia; SPIRIT; quality analysis

Abbreviations: SPIRIT: Standard Protocol Items: Recommendations for Intervention Trials; WHO ICTRP: World Health Organization International Clinical Trials Registry Platform; ChiCTR: Chinese Clinical Trials Registry; N/A: Not applicable

Received: January 03, 2018     Accepted: February 27, 2018     Published: May 15, 2018

ABSTRACT

Objective: To learn about the overall quality of clinical anaesthesia study protocols from the Chinese Clinical Trials Registry and to discuss the way to improve study protocol quality.

Methods: We defined completeness of each sub-item in SPIRIT as N/A (not applicable) or with a score of 0, 1, or 2. For each protocol, we calculated the proportion of adequately reported items (score = 2 and N/A) and unreported items (score = 0). Protocol quality was determined according to the proportion of reported items, with values >50% indicating high quality. Protocol quality was determined according to the proportion of reported items. For each sub-item in SPIRIT, we calculated the adequately reported rate (percentage of all protocols with score 2 and NA on one sub-item) as well as the unreported rate (percentage of all protocols with score 0 on one sub-item).

Results: Total 126 study protocols were available for assessment. Among these, 88.1% were assessed as being of low quality. By comparison, the percentage of low-quality protocols was 88.9% after the publication of the SPIRIT statement. Among the 51 SPIRIT sub-items, 18 sub-items had an unreported rate above 90% while 16 had a higher adequately reported rate than an unreported rate.

Conclusions: The overall quality of clinical anaesthesia study protocols registered in the ChiCTR was poor. A mandatory protocol upload and self-check based on the SPIRIT statement during the trial registration process may improve protocol quality in the future.


INTRODUCTION

The overall quality of clinical trial reporting is considered to be inadequate [1], with limitations identified in the study design, conduct, and reporting stages [2, 3]. These deficiencies may be avoided by increasing the completeness and transparency of study protocols [46]. Therefore, the Standard Protocol Items: Recommendations for Intervention Trials (SPIRIT) statement was developed in 2013 [7] as an evidence-based guidance for the publication of clinical study protocols. The statement checklist contains 33 items and 51 sub-items under 8 headings. In addition to providing information on how to complete a study protocol, the SPIRIT statement also provides recommendations on important aspects of study design, conduct, and reporting [8].

It is a requirement for clinical trials to be registered on the World Health Organization International Clinical Trials Registry Platform (WHO ICTRP) (http://www.who.int/ictrp) [9, 10]. Of which the Chinese Clinical Trials Registry (ChiCTR) (www.chictr.org.cn) is a primary registry for clinical trials conducted in China as well as in other countries. Registered anaesthesia studies in ChiCTR have increased significantly in recent years. Despite researchers were encouraged to upload their study protocols during registered process in ChiCTR, there were no quality control of these protocols. Anaesthesia studies focus on anaesthesia technique as well as perioperative monitoring methods, which ensure patients’ safe and comfort following a surgical or medical procedure. Quality assessment of randomized controlled trial was performed in endodontics [11], but there is no similar quality assessment in anaesthesiology.

Therefore, to learn about the overall quality of clinical anaesthesia study protocols and to discuss the way to improve study protocol quality, we identified clinical trials in clinical anaesthesia from the ChiCTR and analyzed the protocols based on the 51 sub-items of the SPIRIT checklist.

RESULTS

A total of 204 registered studies was retrieved from the ChiCTR, of which 126 study protocols were available for assessment. Among the 126 protocols there were 99 randomized controlled trials, 6 non-randomized controlled trials, and 21 observational studies (Figure 1).

Flow chart

Figure 1: Flow chart

Overall quality assessment of protocols

Only 4 protocols (3.2%) were assessed as high quality and 111 protocols (88.1%) were assessed as low quality (Figure 2; Table 1). We identified only 18 protocols registered from the date of the SPIRIT statement to April 2015. Among these, 1 protocol (5.6%) was assessed as high quality, 1 (5.6%) was assessed as moderate quality, and 16 (88.9%) were low quality (Table 1).

Distribution diagram of the quality of protocols (n=126).

Figure 2: Distribution diagram of the quality of protocols (n=126).

1 Adequately reported items: sub-items assessed as having a score of 2 or N/A.

2 Unreported items: sub-items assessed as having a score of 0.

Only 4 protocols (3.2%) were located in the upper left quadrant and were therefore of high quality (>50% of sub-items assessed as adequately reported), and 111 protocols (88.1%) were located in the lower right quadrant and were therefore of low quality (>50% of sub-items assessed as unreported), the remaining 11 protocols (8.7%) were located in the lower left quadrant and were of moderate quality.

Table 1: Quality of protocols before and after the SPIRIT statement was issued

 

Number of high quality protocols

Number of moderate quality protocols

Number of low quality protocols

Total number of protocols

Before SPIRIT# issued

3 (2.8%)

10 (9.2%)

95 (88.0%)

108

After SPIRIT# issued

1 (5.6%)

1 (5.6%)

16 (88.9%)

18

Total

4 (3.2%)

11 (8.7%)

111 (88.1%)

126

# SPIRIT: Standard Protocol Items: Recommendations for Intervention Trials.

Analysis of SPIRIT sub-items

The following 18 sub-items had an unreported rate >90%: roles and responsibilities (5c, 5d), interventions (11c, 11d), data collection and methods (18b), data management (19), statistical methods (20c), data monitoring (21a, 21b), auditing (23), protocol amendments (25), declaration of interest (28), access to data (29), ancillary and post-trial care (30), dissemination policy (31a, 31b, 31c), and informed consent materials (32). A summary of the protocol assessment for each SPIRIT sub-item is shown in Table 2.

Table 2: Summary of protocol completeness assessment for each sub-item of the SPIRIT# checklist (n=126)

 

Item NO.

No. of protocols assessed as

Score 2*

Score 1*

Score 0*

N/A*

Administrative information

Title

1

100 (79.4%)

20 (15.9%)

6 (4.8%)

0

Trial registration

2a

2 (1.6%)

23 (18.3%)

101 (80.2%)

0

2b

1 (0.8%)

23 (18.3%)

102 (81.0%)

0

Protocol version

3

20 (15.9%)

16 (12.7%)

90 (71.4%)

0

Funding

4

13 (10.3%)

2 (1.6%)

111 (88.1%)

0

Roles and responsibilities

5a

29 (23.0%)

28 (22.2%)

69 (54.8%)

0

5b

38 (30.2%)

19 (15.1%)

69 (54.8%)

0

5c

2 (1.6%)

4 (3.2%)

120 (95.2%)

0

5d

1 (0.8%)

2 (1.6%)

123 (97.6%)

0

Introduction

Background and rationale

6a

96 (76.2%)

16 (12.7%)

14 (11.1%)

0

6b

57 (45.2%)

36 (28.6%)

31 (24.6%)

2 (1.6%)

Objectives

7

111 (88.1%)

7 (5.6%)

8 (6.3%)

0

Trial design

8

76 (60.3%)

23 (18.3%)

26 (20.6%)

0

Methods: Participants, interventions, and outcomes

Study setting

9

83 (65.9%)

4 (3.2%)

39 (31.0%)

0

Eligibility criteria

10

112 (82.4%)

8 (6.3%)

6 (4.8%)

0

Interventions

11a

105 (83.3%)

11 (8.7%)

10 (7.9%)

0

11b

29(23.0%)

9 (7.1%)

88 (69.8%)

0

11c

6 (4.8%)

3 (2.4%)

117 (92.9%)

0

11d

5 (4.0%)

5 (4.0%)

116 (92.1%)

0

Outcomes

12

111 (88.1%)

12 (9.5%)

3 (2.4%)

0

Participant timeline

13

34 (27.0%)

1 (0.8%)

91 (72.2%)

0

Sample size

14

28 (22.2%)

60 (47.6%)

38 (30.2%)

0

Recruitment

15

15 (11.9%)

29 (23.0%)

82 (65.1%)

0

Methods: Assignment of interventions (for controlled trials)

Allocation:

 

 

 

 

 

Sequence generation

16a

30 (23.8%)

21 (16.7%)

69 (54.8%)

6(4.8%)

Allocation concealment mechanism

16b

11 (8.7%)

15 (11.9%)

92 (73.0%)

8(6.3%)

Implementation

16c

4 (3.2%)

12 (9.5%)

103 (81.7%)

7(5.6%)

Blinding

17a

20 (15.9%)

8(6.3%)

33 (26.2%)

65 (51.6%)

17b

6 (4.8%)

1 (0.8%)

54 (42.9%)

65 (51.6%)

Methods: Data collection, management and analysis

Data collection methods

18a

11 (8.7%)

31 (24.6%)

84 (66.7%)

0

18b

3 (2.4%)

7(5.6%)

116 (92.1%)

0

Data management

19

8 (6.3%)

4 (3.2%)

114 (90.5%)

0

Statistical methods

20a

59 (46.8%)

13 (10.3%)

54 (42.9%)

0

20b

6(4.8%)

8 (6.3%)

112 (88.9%)

0

20c

3 (2.4%)

4 (3.2%)

119 (94.4%)

0

Methods: Monitoring

Data monitoring

21a

6(4.8%)

2 (1.6%)

118 (93.7%)

0

21b

3 (2.4%)

1 (0.8%)

122 (96.8%)

0

Harms

22

33 (26.2%)

10 (7.9%)

83 (65.9%)

0

Auditing

23

1 (0.8%)

1 (0.8%)

124 (98.4%)

0

Ethics and dissemination

Research ethics approval

24

61 (48.4%)

10 (7.9%)

55 (43.7%)

0

Protocol amendments

25

10 (7.9%)

2 (1.6%)

114 (90.5%)

0

Consent or assent

26a

78 (61.9%)

9 (7.1%)

39 (31.0%)

0

26b

1 (0.8%)

2 (1.6%)

51 (40.5%)

72 (57.1%)

Confidentiality

27

14 (11.1%)

3 (2.4%)

108 (85.7%)

0

Declaration of interests

28

1 (0.8%)

3 (2.4%)

122 (96.8%)

0

Access to data

29

7 (5.6%)

5 (4.0%)

114 (90.5%)

0

Ancillary and post-trial care

30

4 (3.2%)

4 (3.2%)

118 (93.7%)

0

Dissemination policy

31a

4 (3.2%)

3 (2.4%)

119 (94.4%)

0

31b

0

2 (1.6%)

124 (98.4%)

0

31c

0

1 (0.8%)

125 (99.2%)

0

Appendices

Informed consent materials

32

3 (2.4%)

0

123 (97.6%)

0

Biological specimen

33

1 (0.8%)

0

49 (38.9%)

76(60.3%)

# SPIRIT: Standard Protocol Items: Recommendations for Intervention Trials. *A score of 2 indicated that all important information was adequately reported in the protocol for that sub-item; a score of 1 indicated that some important information was inadequately reported; a score of 0 indicated that no information related to the sub-item was reported. N/A indicated that the item was not applicable to the specific protocol.

The following 16 sub-items had an adequately reported rate greater than the unreported rate: title (1), background and rationale (6a, 6b), objectives (7), trial design (8), study setting (9), eligibility criteria (10), intervention (11a), outcomes (12), blinding (17a, 17b), statistical methods (20a), research ethics approval (24), consent or assent (26a, 26b), and biological specimen (33) (Figure 3).

Adequately reported rate1 and unreported rate2 for each SPIRIT# sub-item (n=126).

Figure 3: Adequately reported rate1 and unreported rate2 for each SPIRIT# sub-item (n=126).

1 Adequately reported rate was the percentage of protocols with a score of 2 or NA for one sub-item among all protocols assessed.

2 Unreported rate was the percentage of protocols with a score of 0 for one sub-item among all protocols assessed.

# SPIRIT: Standard Protocol Items: Recommendations for Intervention Trials. The adequately reported rate was lower than the unreported rate for 35 of the 51 sub-items.

DISCUSSION

The completeness of protocol was one of the factors which determined clinical study quality. Protocol upload was required during registered process in ChiCTR, but not in other registry. So the anaesthesia study protocols in ChiCTR had a certain representation. Unfortunately, the overall quality of most anaesthesia study protocols submitted to the ChiCTR was poor.

To improve study quality, it has been a requirement of most clinical journals that clinical trials are registered in international databases, although current trial registry requirements appear to be insufficient [12]. Submission of a detailed protocol could help to prevent inadequate trial conduct and unnecessary protocol amendments [13]. However, this is not a mandatory requirement during the trial registration process and only 61.7% of anaesthesia study protocols in our review were available in the ChiCTR. This indicates a lack of awareness among some researchers regarding the importance of the study protocol. We suggest that a mandatory requirement to provide the protocol and perform a self-check of completeness according to the SPIRIT statement during the registration process may improve the quality of clinical studies.

The SPIRIT statement was issued at the end of 2013 [7, 14]. However, we did not observe a significant improvement in the quality of protocols registered after this date (Table 1). Information related to the sub-items in the Administrative information, Introduction, Ethics and dissemination, and Appendices sections should be reviewed and confirmed during the study planning stage. Missing information within these sections may thus be the result of a lack of awareness among researchers about the SPIRIT statement. Therefore, to increase protocol completeness and quality, information on the SPIRIT statement and other relevant programs such as the Research to Publication initiative [15] should be widely disseminated.

Regarding the detailed content of the SPIRIT statement in our study, 16 sub-items were found to have an adequately reported rate greater than the unreported rate, with the remaining 35 sub-items requiring further information (Figure 3). Among these 35 sub-items, both the Participant timeline and Recruitment fields were frequently overlooked. Insufficient recruitment is a common feature of clinical trials, [1619] and can prolong the study period, increase research costs, or reduce the statistical power. Increased focus on participant timelines and recruitment strategies during the study planning stage has been shown to improve recruitment and minimize loss to follow-up [8]. However, in the current study, only 27.0% of protocols adequately reported the participant timeline and only 11.9% adequately reported recruitment. In addition, data collection, management, and monitoring have been identified as important determinants of study quality and participant safety [20, 21]. However, we found that these items were adequately reported in only 10% of protocols. Increased use of the SPIRIT guidelines will allow more researchers to recognize the benefits of establishing detailed protocols prior to study initiation. Meanwhile, adequate clinical research training on methodology, epidemiology, and statistics may improve the completeness of protocols and subsequently enhance the quality of clinical trials.

The limitation of our study is that only 61.8% of study protocol were available. Due to the low percentage, the result of our study may be biased. But even if we assumed that the rest unavailable protocols were high quality, the overall quality of anaesthesia protocols was still poor.

MATERIALS AND METHODS

We searched the ChiCTR for studies registered from 2005 to April 2015 using the term ‘anaesthesia’ in the title or ‘anaesthesiology department’ in the affiliated institution name. Study protocols were obtained from ChiCTR, and two of the authors (DY and SC) independently assessed the completeness of protocols based on the 51 sub-items of the SPIRIT checklist [7]. Any disagreement was resolved by discussion with the third author (LY).

We defined three levels of completeness for each sub-item, scored as 2, 1, and 0. A score of 2 was given where all important information was adequately reported in the protocol for the sub-item. A score of 1 meant that some of the important information was inadequately reported. A score of 0 indicated that none of the important information was reported. N/A (not applicable) was used where the sub-item was not applicable to the protocol.

Quality analysis was performed in two steps:

1. Overall quality assessment of protocols

For each protocol, we calculated the proportion of adequately reported items (sub-items with a score of 2 or N/A) and unreported items (sub-items with a score of 0). Protocols were considered high quality if the proportion of adequately reported items was more than 50% or low quality if the proportion of unreported items was more than 50%; the remaining protocols were considered moderate quality.

2. Analysis of SPIRIT sub-items

We analyzed the reporting rate for each sub-item in the SPIRIT checklist by calculating the adequately reported rate (percentage of scores = 2 or NA for a sub-item across all protocols) as well as the unreported rate (percentage of scores = 0 for a sub-item across all protocols).

CONCLUSIONS

The quality of anaesthesia study protocols in the ChiCTR was assessed as poor in the current study. This issue might be addressed by introducing a mandatory requirement to provide the study protocol during the registration process along with performing a self-check of the protocol according to the SPIRIT guidelines. Awareness of the SPIRIT statement and adequate clinical research training should also be ensured within the clinical anaesthesia research community.

Author contributions

LY performed the search strategy and a quality analysis, and provided major editing of the manuscript; SC performed major quality analysis and provided assistant in editing of the manuscript; DY performed a quality analysis. TW and YZ conceived the study, retrieved the protocols and provided assistant in editing of the manuscript. JL performed the descriptive statistical analysis. LY and SC contributed equally to the article.

ACKNOWLEDGMENTS

We thank Clare Cox, PhD, from Liwen Bianji, Edanz Group China (www.liwenbianji.cn/ac), for editing the English text of a draft of this manuscript.

CONFLICTS OF INTEREST

The authors have no conflicts of interest.

FUNDING

This work was supported by the 1.3.5 Project for Disciplines of Excellence, West China Hospital, Sichuan University.

REFERENCES

1. Hopewell S, Dutton S, Yu LM, Chan AW, Altman DG. The quality of reports of randomised trials in 2000 and 2006: comparative study of articles indexed in PubMed. BMJ. 2010; 340:c723. https://doi.org/10.1136/bmj.c723.

2. Hróbjartsson A, Pildal J, Chan AW, Haahr MT, Altman DG, Gøtzsche PC. Reporting on blinding in trial protocols and corresponding publications was often inadequate but rarely contradictory. J Clin Epidemiol. 2009; 62:967–73. https://doi.org/10.1016/j.jclinepi.2009.04.003.

3. Chan AW, Hróbjartsson A, Haahr MT, Gøtzsche PC, Altman DG. Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. JAMA. 2004; 291:2457–65. https://doi.org/10.1001/jama.291.20.2457.

4. Siegel JP. Editorial review of protocols for clinical trials. N Engl J Med. 1990; 323:1355. https://doi.org/10.1056/NEJM199011083231920.

5. Lassere M, Johnson K. The power of the protocol. Lancet. 2002; 360:1620–22. https://doi.org/10.1016/S0140-6736(02)11652-7.

6. Chan AW. Bias, spin, and misreporting: time for full access to trial protocols and results. PLoS Med. 2008; 5:e230. https://doi.org/10.1371/journal.pmed.0050230.

7. Chan AW, Tetzlaff JM, Altman DG, Laupacis A, Gøtzsche PC, Krleža-Jerić K, Hróbjartsson A, Mann H, Dickersin K, Berlin JA, Doré CJ, Parulekar WR, Summerskill WS, et al. SPIRIT 2013 statement: defining standard protocol items for clinical trials. Ann Intern Med. 2013; 158:200–07. https://doi.org/10.7326/0003-4819-158-3-201302050-00583.

8. Chan AW, Tetzlaff JM, Gøtzsche PC, Altman DG, Mann H, Berlin JA, Dickersin K, Hróbjartsson A, Schulz KF, Parulekar WR, Krleza-Jeric K, Laupacis A, Moher D. SPIRIT 2013 explanation and elaboration: guidance for protocols of clinical trials. BMJ. 2013; 346:e7586. https://doi.org/10.1136/bmj.e7586.

9. Dickersin K, Rennie D. Registering clinical trials. JAMA. 2003; 290:516–23. https://doi.org/10.1001/jama.290.4.516.

10. DeAngelis CD, Drazen JM, Frizelle FA, Haug C, Hoey J, Horton R, Kotzin S, Laine C, Marusic A, Overbeke AJ, Schroeder TV, Sox HC, Van Der Weyden MB, and International Committee of Medical Journal Editors. Clinical trial registration: a statement from the International Committee of Medical Journal Editors. JAMA. 2004; 292:1363–64. https://doi.org/10.1001/jama.292.11.1363.

11. Lucena C, Souza EM, Voinea GC, Pulgar R, Valderrama MJ, De-Deus G. A quality assessment of randomized controlled trial reports in endodontics. Int Endod J. 2017; 50:237–50. https://doi.org/10.1111/iej.12626.

12. Manheimer E, Anderson D. Survey of public information about ongoing clinical trials funded by industry: evaluation of completeness and accessibility. BMJ. 2002; 325:528–31. https://doi.org/10.1136/bmj.325.7363.528.

13. Tetzlaff JM, Moher D, Chan AW. Developing a guideline for clinical trial protocol content: delphi consensus survey. Trials. 2012; 13:176. https://doi.org/10.1186/1745-6215-13-176.

14. Chan AW, Tetzlaff JM, Altman DG, Laupacis A, Gøtzsche PC, Krleža-Jerić K, Hróbjartsson A, Mann H, Dickersin K, Berlin JA, Doré CJ, Parulekar WR, Summerskill WS, Groves T, et al. SPIRIT 2013 Statement: defining standard protocol items for clinical trials. Chin J Evid Based Med. 2013; 13:1501–07.

15. Research to publication. http://www.rtop.bmj.com/ (accessed Jun 8, 2017).

16. Campbell MK, Snowdon C, Francis D, Elbourne D, McDonald AM, Knight R, Entwistle V, Garcia J, Roberts I, Grant A, Grant A; STEPS group. Recruitment to randomised trials: strategies for trial enrolment and participation study. The STEPS study. Health Technol Assess. 2007; 11:iii, ix-105.

17. Pich J, Carné X, Arnaiz JA, Gómez B, Trilla A, Rodés J. Role of a research ethics committee in follow-up and publication of results. Lancet. 2003; 361:1015–16. https://doi.org/10.1016/S0140-6736(03)12799-7.

18. Decullier E, Lhéritier V, Chapuis F. Fate of biomedical research protocols and publication bias in France: retrospective cohort study. BMJ. 2005; 331:19. https://doi.org/10.1136/bmj.38488.385995.8F.

19. Wise P, Drury M. Pharmaceutical trials in general practice: the first 100 protocols. An audit by the clinical research ethics committee of the Royal College of General Practitioners. BMJ. 1996; 313:1245–48. https://doi.org/10.1136/bmj.313.7067.1245.

20. Lane SJ, Heddle NM, Arnold E, Walker I. A review of randomized controlled trials comparing the effectiveness of hand held computers with paper methods for data collection. BMC Med Inform Decis Mak. 2006; 6:23. https://doi.org/10.1186/1472-6947-6-23.

21. DeMets DL, Pocock SJ, Julian DG. The agonising negative trend in monitoring of clinical trials. Lancet. 1999; 354:1983–88. https://doi.org/10.1016/S0140-6736(99)03464-9.


Creative Commons License All site content, except where otherwise noted, is licensed under a Creative Commons Attribution 4.0 License.
PII: 24982