Ensuring Inclusion of Research Reports in Systematic Reviews

Marcel P. Dijkers

doi:10.1016/j.apmr.2009.04.026

Ensuring Inclusion of Research Reports in Systematic Reviews

Marcel P. Dijkers

Research output: Contribution to journal › Article › peer-review

3 Scopus citations

Abstract

Dijkers MP. Ensuring inclusion of research reports in systematic reviews. Rehabilitation is supported by research evidence that is limited in both number and quality. In order for more and better evidence to be published, researchers need to improve their reports of randomized controlled trials and other interventional research. Making it likely that these reports can contribute to the cumulation of scientific knowledge through their inclusion in systematic reviews imposes some special requirements. This article discusses the following steps that investigators should follow: trial registration; reliance on Consolidated Standards for Reporting Trials guidelines in protocol development and report writing; use of recommendations by authoritative groups for the use of specific measures or data sets; adherence in writing to the style and other guidelines offered by the American Medical Association, the American Psychological Association, the International Committee of Medical Journal Editors, and the target journal's instructions for authors; clear declarations of deviations from commonly accepted study methods; and use of informal peer review.

Original language	English
Pages (from-to)	S60-S69
Journal	Archives of Physical Medicine and Rehabilitation
Volume	90
Issue number	11 SUPPL. 1
DOIs	https://doi.org/10.1016/j.apmr.2009.04.026
State	Published - Nov 2009

Keywords

Clinical trials as topic
Peer review
Publishing
Rehabilitation
research

Access to Document

10.1016/j.apmr.2009.04.026

Cite this

@article{ffb81b01f0544f0abc42edbe72a05c7e,

title = "Ensuring Inclusion of Research Reports in Systematic Reviews",

abstract = "Dijkers MP. Ensuring inclusion of research reports in systematic reviews. Rehabilitation is supported by research evidence that is limited in both number and quality. In order for more and better evidence to be published, researchers need to improve their reports of randomized controlled trials and other interventional research. Making it likely that these reports can contribute to the cumulation of scientific knowledge through their inclusion in systematic reviews imposes some special requirements. This article discusses the following steps that investigators should follow: trial registration; reliance on Consolidated Standards for Reporting Trials guidelines in protocol development and report writing; use of recommendations by authoritative groups for the use of specific measures or data sets; adherence in writing to the style and other guidelines offered by the American Medical Association, the American Psychological Association, the International Committee of Medical Journal Editors, and the target journal's instructions for authors; clear declarations of deviations from commonly accepted study methods; and use of informal peer review.",

keywords = "Clinical trials as topic, Peer review, Publishing, Rehabilitation, research",

author = "Dijkers, {Marcel P.}",

note = "Funding Information: A large percentage of the abstracts of clinical trials presented at conferences never sees life as a full-length article published in a peer-reviewed journal. 72,73 There are a number of causes for this file drawer problem: authors are not convinced the results are worth the effort of writing, polishing, and rewriting in response to peer review, especially if there was no support found for the hypothesis underlying the research; peer reviewers and editors have a preference for positive results and are quick to reject articles with negative results, especially if there is no proof that the study was adequately powered; and pharmaceutical companies and other for-profit entities may suppress publication of studies that show their product to be no better than, let alone inferior to, a placebo or the product of a competitor. As a result, the scientific literature tends to be filled with studies with positive results—a phenomenon known as publication bias. While there are several graphical and statistical methods systematic reviewers can use to demonstrate that in a particular area publication bias is very likely to exist, no mechanism to undo it is available, other than access to and a systematic search of the file drawers. A closely related problem is one mentioned previously: publication bias in situ. Selective reporting (whether the selection involves subgroups, time points, outcome measures, or other aspects of the research) has the same impact on cumulated evidence as wholesale nonreporting: a skewing of the evidence. Publication bias in situ is a newly acknowledged problem, 55 and as of now there is no literature to suggest how extensive the issue is and what exactly its implications are for the trustworthiness of the published literature as a whole. This disappearance or selective reporting of evidence has several consequences. One is that the literature is skewed, and that practitioners run the risk of making erroneous decisions because only articles in support of a treatment exist, and none of the ones demonstrating its lack of effectiveness or its equivalence to another, less risky and/or less expensive, intervention. Scientifically, the missing articles fail to do their share in shouldering the cumulation of knowledge, and make no contribution to the design of future, better investigations. A study by Chan et al 74 demonstrates the need for vigilance in this area. These investigators compared 102 protocols as they had been submitted to an IRB with 122 journal reports and principal investigator responses to queries by the authors. They concluded that 50% of the efficacy outcomes specified in the protocols were incompletely reported; 65% of harm outcomes were reported partially; statistically significant outcomes tended to be published more often and more completely; and 62% of the trials had 1 or more primary outcomes changed, omitted, or replaced by a new outcome. A follow-up study concluded that unacknowledged discrepancies between protocols and publications were found for sample size calculations, methods for handling protocol violations and missing data, and primary outcome analyses, overall and in subgroups. 75 The solution that has been developed to ensure that readers, patients, payors, and research sponsors, among others, can determine whether there is research that never made it into the literature or that in its publication has unstated discrepancies from the original design is the mandatory registration of trials. A trials registry requires new research projects to record the research protocol, along with various administrative data elements (primary investigator, sponsor, and so forth) on the registry's website, generally well before the first subject is enrolled. The information can be changed as necessary (adjustment in planned sample size, change in principal investigator, and so forth), but all the original information remains accessible. The registry information is publicly available, so that peer reviewers, editors, systematic reviewers, and others can go back and determine exactly what divergences exist between the manuscript (or the published article) and the original or modified protocol. Systematic reviewers have an easy starting place to determine research reports that are missing from the published literature, and can contact the investigators (or others involved with the project) about the findings and the reasons for not publishing them. Active general purpose registries exist in several countries, including the United States ( ClinicalTrials.gov ), Australia (Australian Clinical Trials Registry), the Netherlands (Netherlands Trials Registry), and Japan (University hospital Medical Information Network [UMIN] Clinical Trials Registry), as well as internationally (International Standard Randomized Controlled Trial Number Register). Mandatory registration of trials is endorsed by the ICMJE, the World Health Organization, the American Association of Medical Colleges, the U.S. Food and Drug Administration, and numerous other professional groups and sponsoring organizations. Many journals, especially high-impact journals, have endorsed trial registration, announcing in their pages that as of a future date, they would no longer publish a trial report if the trial had not been registered in one of the acknowledged registries. The registration number needs to be submitted along with the manuscript, and some journals print it as part of the abstract; one now starts to see in abstracts number strings such as “ISRCTN07836779.” Many rehabilitation journals have not yet signed on to the principle of registration. For instance, the instructions for authors of Archives of Physical Medicine and Rehabilitation have the following statement: “ Archives 's editors have adopted in principle the spirit of the requirement to register clinical trials, at or prior to subject enrollment. At this time, however, it is not a requirement of submission that authors whose studies meet the ICMJE criteria … register their trials or, if they do, comply fully with the Minimal Registration Data Set. In the future and by prior notice, Archives 's editors may make registration and compliance mandatory, that is, a condition prior to peer review. Authors who have registered their trials should provide the trial registration number in the cover letter to their submissions. Registration numbers will appear in all content published in the journal.” 16 Because there is no cost to register trials, it behooves all investigators to register their clinical investigations, even ones that are already under way. It would address any retroactive requirement of registration that might be imposed by journals, and will make studies still in progress easier to find by systematic reviewers. It should be noted that at least the U.S. site, ClinicalTrials.gov , draws no sharp distinctions between clinical trials and other types of studies focused on health. Observational studies and single subject design investigations can be registered without restriction. ",

year = "2009",

month = nov,

doi = "10.1016/j.apmr.2009.04.026",

language = "English",

volume = "90",

pages = "S60--S69",

journal = "Archives of Physical Medicine and Rehabilitation",

issn = "0003-9993",

publisher = "W.B. Saunders",

number = "11 SUPPL. 1",

}

TY - JOUR

T1 - Ensuring Inclusion of Research Reports in Systematic Reviews

AU - Dijkers, Marcel P.

N1 - Funding Information: A large percentage of the abstracts of clinical trials presented at conferences never sees life as a full-length article published in a peer-reviewed journal. 72,73 There are a number of causes for this file drawer problem: authors are not convinced the results are worth the effort of writing, polishing, and rewriting in response to peer review, especially if there was no support found for the hypothesis underlying the research; peer reviewers and editors have a preference for positive results and are quick to reject articles with negative results, especially if there is no proof that the study was adequately powered; and pharmaceutical companies and other for-profit entities may suppress publication of studies that show their product to be no better than, let alone inferior to, a placebo or the product of a competitor. As a result, the scientific literature tends to be filled with studies with positive results—a phenomenon known as publication bias. While there are several graphical and statistical methods systematic reviewers can use to demonstrate that in a particular area publication bias is very likely to exist, no mechanism to undo it is available, other than access to and a systematic search of the file drawers. A closely related problem is one mentioned previously: publication bias in situ. Selective reporting (whether the selection involves subgroups, time points, outcome measures, or other aspects of the research) has the same impact on cumulated evidence as wholesale nonreporting: a skewing of the evidence. Publication bias in situ is a newly acknowledged problem, 55 and as of now there is no literature to suggest how extensive the issue is and what exactly its implications are for the trustworthiness of the published literature as a whole. This disappearance or selective reporting of evidence has several consequences. One is that the literature is skewed, and that practitioners run the risk of making erroneous decisions because only articles in support of a treatment exist, and none of the ones demonstrating its lack of effectiveness or its equivalence to another, less risky and/or less expensive, intervention. Scientifically, the missing articles fail to do their share in shouldering the cumulation of knowledge, and make no contribution to the design of future, better investigations. A study by Chan et al 74 demonstrates the need for vigilance in this area. These investigators compared 102 protocols as they had been submitted to an IRB with 122 journal reports and principal investigator responses to queries by the authors. They concluded that 50% of the efficacy outcomes specified in the protocols were incompletely reported; 65% of harm outcomes were reported partially; statistically significant outcomes tended to be published more often and more completely; and 62% of the trials had 1 or more primary outcomes changed, omitted, or replaced by a new outcome. A follow-up study concluded that unacknowledged discrepancies between protocols and publications were found for sample size calculations, methods for handling protocol violations and missing data, and primary outcome analyses, overall and in subgroups. 75 The solution that has been developed to ensure that readers, patients, payors, and research sponsors, among others, can determine whether there is research that never made it into the literature or that in its publication has unstated discrepancies from the original design is the mandatory registration of trials. A trials registry requires new research projects to record the research protocol, along with various administrative data elements (primary investigator, sponsor, and so forth) on the registry's website, generally well before the first subject is enrolled. The information can be changed as necessary (adjustment in planned sample size, change in principal investigator, and so forth), but all the original information remains accessible. The registry information is publicly available, so that peer reviewers, editors, systematic reviewers, and others can go back and determine exactly what divergences exist between the manuscript (or the published article) and the original or modified protocol. Systematic reviewers have an easy starting place to determine research reports that are missing from the published literature, and can contact the investigators (or others involved with the project) about the findings and the reasons for not publishing them. Active general purpose registries exist in several countries, including the United States ( ClinicalTrials.gov ), Australia (Australian Clinical Trials Registry), the Netherlands (Netherlands Trials Registry), and Japan (University hospital Medical Information Network [UMIN] Clinical Trials Registry), as well as internationally (International Standard Randomized Controlled Trial Number Register). Mandatory registration of trials is endorsed by the ICMJE, the World Health Organization, the American Association of Medical Colleges, the U.S. Food and Drug Administration, and numerous other professional groups and sponsoring organizations. Many journals, especially high-impact journals, have endorsed trial registration, announcing in their pages that as of a future date, they would no longer publish a trial report if the trial had not been registered in one of the acknowledged registries. The registration number needs to be submitted along with the manuscript, and some journals print it as part of the abstract; one now starts to see in abstracts number strings such as “ISRCTN07836779.” Many rehabilitation journals have not yet signed on to the principle of registration. For instance, the instructions for authors of Archives of Physical Medicine and Rehabilitation have the following statement: “ Archives 's editors have adopted in principle the spirit of the requirement to register clinical trials, at or prior to subject enrollment. At this time, however, it is not a requirement of submission that authors whose studies meet the ICMJE criteria … register their trials or, if they do, comply fully with the Minimal Registration Data Set. In the future and by prior notice, Archives 's editors may make registration and compliance mandatory, that is, a condition prior to peer review. Authors who have registered their trials should provide the trial registration number in the cover letter to their submissions. Registration numbers will appear in all content published in the journal.” 16 Because there is no cost to register trials, it behooves all investigators to register their clinical investigations, even ones that are already under way. It would address any retroactive requirement of registration that might be imposed by journals, and will make studies still in progress easier to find by systematic reviewers. It should be noted that at least the U.S. site, ClinicalTrials.gov , draws no sharp distinctions between clinical trials and other types of studies focused on health. Observational studies and single subject design investigations can be registered without restriction.

PY - 2009/11

Y1 - 2009/11

N2 - Dijkers MP. Ensuring inclusion of research reports in systematic reviews. Rehabilitation is supported by research evidence that is limited in both number and quality. In order for more and better evidence to be published, researchers need to improve their reports of randomized controlled trials and other interventional research. Making it likely that these reports can contribute to the cumulation of scientific knowledge through their inclusion in systematic reviews imposes some special requirements. This article discusses the following steps that investigators should follow: trial registration; reliance on Consolidated Standards for Reporting Trials guidelines in protocol development and report writing; use of recommendations by authoritative groups for the use of specific measures or data sets; adherence in writing to the style and other guidelines offered by the American Medical Association, the American Psychological Association, the International Committee of Medical Journal Editors, and the target journal's instructions for authors; clear declarations of deviations from commonly accepted study methods; and use of informal peer review.

AB - Dijkers MP. Ensuring inclusion of research reports in systematic reviews. Rehabilitation is supported by research evidence that is limited in both number and quality. In order for more and better evidence to be published, researchers need to improve their reports of randomized controlled trials and other interventional research. Making it likely that these reports can contribute to the cumulation of scientific knowledge through their inclusion in systematic reviews imposes some special requirements. This article discusses the following steps that investigators should follow: trial registration; reliance on Consolidated Standards for Reporting Trials guidelines in protocol development and report writing; use of recommendations by authoritative groups for the use of specific measures or data sets; adherence in writing to the style and other guidelines offered by the American Medical Association, the American Psychological Association, the International Committee of Medical Journal Editors, and the target journal's instructions for authors; clear declarations of deviations from commonly accepted study methods; and use of informal peer review.

KW - Clinical trials as topic

KW - Peer review

KW - Publishing

KW - Rehabilitation

KW - research

UR - http://www.scopus.com/inward/record.url?scp=71749114417&partnerID=8YFLogxK

U2 - 10.1016/j.apmr.2009.04.026

DO - 10.1016/j.apmr.2009.04.026

M3 - Article

C2 - 19892076

AN - SCOPUS:71749114417

SN - 0003-9993

VL - 90

SP - S60-S69

JO - Archives of Physical Medicine and Rehabilitation

JF - Archives of Physical Medicine and Rehabilitation

IS - 11 SUPPL. 1

ER -

Ensuring Inclusion of Research Reports in Systematic Reviews

Abstract

Keywords

Access to Document

Fingerprint

Cite this