Standards for Quality Research
Quality Matters
This article discusses principles and standards for quality research, the basis for these standards, and strategies for reporting quality research. The terms quality research and quality evidence are related concepts that have been at the center of much debate in academic, professional, and public policy circles. These debates are prevalent in the multidisciplinary fields of health, education, disability, and social welfare (Gersten, Baker, & Lloyd, 2000; Shavelson & Towne, 2002). To some extent the debates stem from the widespread belief that the quality of scientific research is often uneven and lacking in credibility, making it difficult to make a confident, concrete assertion or prediction regarding evidence for improving practice or consumer outcomes (Levin & O'Donnell, 1999; Mosteller & Boruch, 2002; Shavelson & Towne, 2002). The debate is also due, in part, to the lack of consensus on the specific standards for assessing quality research and standards of quality for assessing evidence (Gersten et al., 2000; Mosteller & Boruch, 2002). For example, several researchers have contended that some of the current peer review processes and standards for assessing quality are not well suited for research in the disability arena (Gersten et al., 2000; NCDDR, 2003; Spooner & Browder, 2003).
While this article is concerned with the topic of quality research, it is important to differentiate it from quality evidence. The term evidence or evidence-based, as it relates to research-based knowledge, pertains to the summative collection of research on a specific topic that answers specific and important questions (e.g., questions regarding relationships, why problems exist or persist, or what is the best decision for policymaking) (Raudenbush, February 2002; Shavelson & Towne, 2002). While research quality pertains to the scientific process, evidence quality pertains more to a judgment regarding the strength and confidence one has in the research findings emanating from the scientific process (Mosteller & Boruch, 2002; Shavelson & Towne, 2002). According to Lohr (2004), "The level of confidence one might have in evidence turns on the underlying robustness of the research and the analysis done to synthesize that research." Commonly cited criteria for evaluating systems to rate the strength of bodies of evidence include (West, King, & Carey, 2002):
- Quality: the aggregate of quality ratings for individual studies, predicated on the extent to which bias was minimized in the study designs
- Quantity: the number of studies, the sample size, the study design's statistical power to detect meaningful effects, and magnitude of the effects found or the effect size
- Consistency: for any given topic, the extent to which similar findings are reported using similar and different study designs
Thus, more often than not, quality research is a precursor to quality evidence. Typically, the overall study design, the specific research questions, methods, coherence, and consistency of findings influence the type and quality of evidence produced. Furthermore, the literature suggests that in general, a quality evidence-base typically requires more than a single research study. In rare cases, one study can provide convincing evidence, such as Fischl et al.'s 1987 study on the efficacy of AZT (azidothymidine, or zidovudine) on patients with HIV/AIDS. Because of successful findings and the lack of alternative treatments, the study was stopped early and the Food and Drug Administration approved AZT for treatment of AIDS in a mere 6 months after an analysis of interim data revealed 19 deaths in the placebo group compared to one in the AZT group (Bartlett, 2001; Fischl et al., 1987). At the time, AZT was the only therapy available to counter the opportunistic infections caused by the virus (Dumbrell, 2002; Fischl et al., 1987).
References
Bartlett, J. G. (2001). HIV: Twenty Years in Review. Hopkins HIV Report, 13(4), 8—9.
Fischl, M. A., Richman, D. D., Grieco, M. H., Gottlieb, M. S., Volberding, R. A., Laskin, O. L., et al. (1987). The efficacy of azidothymidine (AZT) in the treatment of patients with AIDS and AlDS-related complex. A double-blind, placebo-controlled trial. New England Journal of Medicine, 317, 185—191.
Gersten, R., Baker, S., & Lloyd, J. W. (2000). Designing high-quality research in special education: group experimental design. The Journal of Special Education, 34(1), 2—18.
Shavelson, R. J., & Towne, L. (Eds.). (2002). Scientific research in education. Washington, DC: National Research Council, National Academy Press.
Levin, J. R., & O'Donnell, A. M. (1999). What to do about educational research's credibility gaps? Issues in Education, 5(2), 177—230.
Lohr, K. N. (2004). Rating the strength of scientific evidence: Relevance for quality improvement programs. International Journal for Quality in Health Care, 16(1), 9—18.
Mosteller, F., & Boruch, R. (Eds.). (2002). Evidence matters: Randomized trials in education research. Washington, DC: The Brookings Institute.
NCDDR. (2003). Evidence-based research in education. The Research Exchange, 8(2), 16.
Raudenbush, S. (February 2002). Identifying scientifically-based research in education. Invited speaker at the Scientifically Based Research Seminar, U.S. Department of Education, Washington DC.
Shavelson, R. J., & Towne, L. (Eds.). (2002). Scientific research in education. Washington, DC: National Research Council, National Academy Press.
Spooner, F., & Browder, D. M. (2003). Scientifically Based Research in Education and Students with Low Incidence Disabilities., Research & Practice for Persons with Severe Disabilities (Vol. 28, pp. 117—125): TASH.
West, S., King, V., & Carey, T. (2002). Systems to rate the strength of scientific evidence. Rockville, MD: Agency for Healthcare Research and Quality.
Adapted from Focus: Technical Brief Number 9, What Are the Standards for Quality Research?