ADVANCING RESEARCH, IMPROVING EDUCATION

Funded by the U.S. Department of Education.

National Center for the
Dissemination of Disability Research

NCDDR's WORKSHOP for NIDRR Grantees

Conducting Systematic Reviews of Randomized and Non-Randomized Studies to Inform Evidence-Based Practice and Policy

The Beacon Hotel, Washington DC
December 5, 2007

Chad Nye, PhD
Professor of Communication Sciences & Disorders
Executive Director, Center for Autism & Related Disabilities (CARD)
University of Central Florida

Herb Turner, PhD
President and Principal Researcher
ANALYTICA, Inc.

Researchers need to use the best available evidence to make recommendations for practitioners and consumers. This workshop explains how research designs other than randomized controlled trials (RCTs) yield evidence that could be included in systematic reviews. A systematic review "strives to comprehensively identify, appraise, and synthesize all the relevant studies on a given topic" (Petticrew & Roberts, 2006, p. 19).

The workshop included lecture, interactive discussion/small groups, sample exercises, and a focus on disability and rehabilitation research. This workshop was expanded from a half-day workshop sponsored by the NCDDR on April 25, 2007 in order to provide time for more interaction, small group discussion, additional examples/exercises, and addressing specific participant ideas/concerns related to systematic review challenges in the disability and rehabilitation arena.

Agenda

7:30-8:00 am

Coffee, Rolls, Pre-assessment

8:00-8:15 am

Welcome and Introductions (Joann Starks)

8:15-8:45 am

Overview/Introduction to Disability and Rehabilitation Evidence Production 

8:45-9:15 am

Nuts and Bolts of Systematic Reviews and the Campbell Collaboration Review Process 

  1. 8 Steps in a Systematic Review     
  2. C2 Registration Form for a Review     
  3. Parent Involvement Review Protocol 
  4. Review Procedures
  5. Q & A

9:15-9:45 am

Types of Reviews Three Types of Evidence-based Reviews 

  1. Scoping Reviews
  2. Rapid Evidence Assessment Reviews
  3. Systematic Reviews

9:45-10:00 am

Recap with Q & A

10:00-10:15 am

Break

10:15-10:45 am

Information Retrieval Process 

  1. Preliminary, Main, and Final Searches: Purpose
  2. Selection of Information Retrieval Tools: Decisions
    1. Electronic
    2. Hand Search
    3. Grey Literature
    4. Invisible College

  3. Saving, Managing & Documenting Search Results

  4. Q & A

10:45-11:30 am

Evaluating Reviews of Research

  1. Different Types of Reviews
  2. Rationale & Criteria for Study Coding
    1. Inclusion/Exclusion
    2. Coding Categories
  3. Codebook Development
  4. Practice Coding
    1. Study, Participants, and Interventions
    2. Designs & Outcomes
  5. Q & A

11:30 am-12:15 pm

Exercise: Evaluating Reviews of Intervention

12:15-1:15 pm

Lunch

1:15-1:45 pm

Review of Evaluation Exercise

1:45-2:15 pm

Assessing the Quality of the Evidence 

  1. Rigor of Research
  2. Relevance of Research to Persons with Disabilities
  3. Q & A

2:15-2:30 pm

Break

2:30-3:00 pm

A Taste of the Importance of Effect Sizes 

  1. Calculating and Synthesizing Effect Sizes
    1. ES Calculations and Interpretation
    2. Forest Plot: Example/Small Groups
    3. Group Designs
  2. Analyzing and Reporting Effect Size
    1. Sensitivity/Homogeneity Analysis for Effect Sizes
    2. Testing Moderator Variables and Sub-groups
  3. Q & A

3:00-3:30 pm

Systematic Reviews for Single Subject Designs    

  1. Review Issues for Small N Studies/Lack of RCTs
  2. Coding for Single Subject Design
  3. Calculating Effect Size for Single Subject Design Studies
  4. Reporting a Systematic Review with Multiple Study Designs
  5. Q & A

3:00-3:30 pm

Wrap-up and Next Steps

  1. Other NCDDR Training Options
  2. Post-Assessment

Selected Materials

Reading List     

What is an 'Effect Size'? (Coe, 2000)     

Effects of parent involvement in isolation or in combination with peer tutoring on student self-concept and mathematics achievement, (Fantuzzo, et al., 1995)


The NCDDR is housed at SEDL.

Copyright ©2008 SEDL

Terms of Use
About SEDL | Contact SEDL | SEDL store

arrow  GO TO SEDL WEBSITE 		NCDDR is funded by
The National Institute on Disability and Rehabilitation Research (NIDRR)
Project Number: H133A060028
U.S. Department of Education