Standards for Quality Research
Quality research most commonly refers to the scientific process encompassing all aspects of study design; in particular, it pertains to the judgment regarding the match between the methods and questions, selection of subjects, measurement of outcomes, and protection against systematic bias, nonsystematic bias, and inferential error (Boaz & Ashby, 2003; Lohr, 2004; Shavelson & Towne, 2002). Principles and standards for quality research designs are commonly found in texts, reports, essays, and guides to research design and methodology. Some scholars, however, suggest the philosophical underpinning and purpose of research methods that are designed specifically to generate rich qualitative data calls for a different characterization of these standards (Spencer, Ritchie, Lewis, & Dillon, 2003).
For example, comparing research methods that are primarily designed to gather qualitative data and research methods that are primarily designed to gather quantitative data, parallel assessments for quality can be framed in terms of credibility (parallels with internal validity), transferability (parallels with external validity), dependability (parallels with reliability), and confirmability (parallels with objectivity) (Boaz & Ashby, 2003; Ragin, Nagel, & White, July 2003).
In this manner, standards for quality research, whether primarily designed to gather quantitative or qualitative data, typically emphasize the traits of objectivity, internal validity, external validity, reliability, rigor, open-mindedness, and honest and thorough reporting (Ragin et al., July 2003; Shavelson & Towne, 2002; Wooding & Grant, 2003).
The National Research Council (2002) and others (Gersten et al., 2000; Greenhalgh, 1997; Ragin et al., July 2003) have described standards that shape scientific understanding and that are frequently used to frame the discourse on the quality of research. This has lead to the term scientifically based research being used in some settings to address research quality. Frequently mentioned standards for assessing the quality of research include the following:
- Pose a significant, important question that can be investigated empirically and that contributes to the knowledge base
- Test questions that are linked to relevant theory
- Apply methods that best address the research questions of interest
- Base research on clear chains of inferential reasoning supported and justified by a complete coverage of the relevant literature
- Provide the necessary information to reproduce or replicate the study
- Ensure the study design, methods, and procedures are sufficiently transparent and ensure an independent, balanced, and objective approach to the research
- Provide sufficient description of the sample, the intervention, and any comparison groups
- Use appropriate and reliable conceptualization and measurement of variables
- Evaluate alternative explanations for any findings
- Assess the possible impact of systematic bias
- Submit research to a peer-review process
- Adhere to quality standards for reporting (i.e., clear, cogent, complete)
While there is no consensus on a specific set or algorithm of standards that will ensure quality research, the more research studies are aligned with or respond to these principles, the higher the quality of the research (Feuer & Towne, 2002; Shavelson & Towne, 2002). This suggests that achieving only one or two standards is typically insufficient to assert quality. For example, some scholars suggest that while standards such as peer review and standardized reporting are important benchmarks, research should not be judged solely by whether or not it is published in the leading journals (Boaz & Ashby, 2003). In addition to the items listed, another quality assessment strategy that is often mentioned is bibliometric analysis, the citing of research by other authors. Bibliometric analysis is premised on the notion that a researcher's work has value when it is judged by peers to have merit sufficient for acknowledgement in a new text or article. While journal publication and bibliometric analysis provide quantitative data, it is a faulty assumption that all "research" that is published in journals or cited by others is accurate, reliable, valid, free of bias, nonfraudulent, or of sufficient quality (Boaz & Ashby, 2003). Further, bibliometric analysis is primarily a measure of quantity and can be artificially influenced by journals with high acceptance rates (COSEPUP, 1999).
References
Boaz, A., & Ashby, D. (2003). Fit for purpose? Assessing research quality for evidence based policy and practice. London: ESRC UK Centre for Evidence Based Policy and Practice.
COSEPUP. Evaluating federal research programs: Research and the Government Performance and Results Act. Committee on Science, Engineering and Public Policy. Washington, DC: National Academy Press.
Feuer, M., & Towne, L. (2002). The logic and the basic principles of scientific based research. Presentation given at a seminar on the use of scientifically based research in education, Washington, DC. Retrieved March 1, 2005, from http://www.ed.gov/nclb/methods/whatworks/research/page_pg11.html.
Gersten, R., Baker, S., & Lloyd, J. W. (2000). Designing high-quality research in special education: group experimental design. The Journal of Special Education, 34(1), 2—18.
Greenhalgh, T. (1997). How to read a paper: Assessing the methodological quality of published papers. British Medical Journal, 315, 305—308.
Lohr, K. N. (2004). Rating the strength of scientific evidence: Relevance for quality improvement programs. International Journal for Quality in Health Care, 16(1), 9—18.
Shavelson, R. J., & Towne, L. (Eds.). (2002). Scientific research in education. Washington, DC: National Research Council, National Academy Press.
Spencer, L., Ritchie, J., Lewis, J., & Dillon, L. (2003). Quality in qualitative evaluation: A framework for assessing research evidence. London: National Centre for Social Research.
Ragin, C. C., Nagel, J., & White, P. (July 2003). Workshop on scientific foundations of qualitative research. National Science Foundation, Arlington, VA.
Wooding, S., & Grant, J. (2003). Assessing research: The researchers' view. Cambridge, England: RAND Europe.
Adapted from Focus: Technical Brief Number 9, What Are the Standards for Quality Research?
[On-line]: http://www.ncddr.org/kt/products/focus/focus9/
[On-line PDF 183 KB] http://www.ncddr.org/kt/products/focus/focus9/Focus9.pdf