Available: Microsoft Word download file [252KB] and PDF file [244KB]

NIDRR logo

Annual Portfolio Assessment Expert Review Pilot

Employment Portfolio
Panel Summary Report

Prepared by:
New Editions Consulting, Inc.
6858 Old Dominion Dr., Suite 230
McLean, VA 22101
January 4, 2006

Skip Table of Contents

Table of Contents

Executive Summary

The National Institute on Disability and Rehabilitation Research (NIDRR) conducted its Annual Portfolio Assessment Expert Review (APAER) Process Pilot for the Employment Portfolio on September 29 and 30, 2005, in Washington, D.C.

The APAER process was developed by NIDRR to assess the agency's progress in meeting Federal performance requirements under the Government Performance and Results Act (GPRA) and the Program Assessment Reporting Tool (PART). The PART is a systematic method of assessing and improving program performance across the Federal government, instituted by the Office of Management and Budget (OMB). The APAER process was designed as an external expert review of NIDRR grantee accomplishments using a three-year cycle, in which one-third of the overall research agenda is reviewed each year. NIDRR's Employment Portfolio was the first review completed under the 2005 pilot phase of this new process.

A panel of nine researchers, consumers, clinicians, policy experts and administrators, reviewed NIDRR's employment portfolio, based on reports from 16 grantees with active awards in 2004, across three program mechanisms: Rehabilitation Research and Training Centers (RRTCs), Disability and Rehabilitation Research Projects (DRRPs), and Field Initiated Projects (FIPs). The 16 awards that were reviewed were grouped into two topical clusters based on the focus of the award: addressing systems level issues and addressing individual level issues.

The review covered:

Since this was a pilot process, data were also collected to assess the feasibility of the process design and implementation.

Findings and Recommendations

The panel provided frank evaluations and recommendations to NIDRR in three areas: (1) the Employment portfolio performance, (2) NIDRR management, and (3) the APAER process. Overall, the panel expressed concern about performance of the portfolio and the soundness of the evaluation and review process. Given the pilot nature of the 2005 APAER process, care must be taken in interpreting and applying the findings contained in this report on NIDRR's Employment portfolio.

Portfolio Performance. Panelists identified a number of outputs and outcomes that appeared to be important to improving employment services, including projects that focused on employment for culturally diverse populations and aging populations, and projects focused on telework and other important accommodations.

Overall, the panel found the research ideas in the portfolio to be of very high quality. However, they expressed concern about: (1) the apparent lack of scientific rigor behind the identified outputs, (2) the lack of sufficient information on the methodologies used by grantees; (3) the lack of evidence supporting many of the claims made by grantees in their Supplemental Information Reports, and (4) the lack of peer reviewed publications.

In the area of knowledge translation, the panel was uncertain about the adequacy of consumer-oriented outputs. Some panelists believed the number of outputs was good, but could not rate the quality of these products based on the information provided. Many reviewers felt that grant activities had the potential to impact individuals with disabilities, but that these activities did not seem connected to the research.

The panel concluded, that in terms of progress toward NIDRR's capacity-building performance measures, the Employment portfolio was limited. Many awards were judged lacking in the use of multidisciplinary teams and no controlled studies were identified. The panel could not judge the adequacy of awards to diverse individuals and institutions, nor the integration of research and training.

Management of the Portfolio. Because persons with disabilities continue to be unemployed and underemployed in disproportionately high numbers, the panel recommended that:

  1. NIDRR should consider whether this portfolio area is adequately funded.

  2. NIDRR should reconceptualize the role of the FIPs and how results from FIPs could translate into other, larger, more comprehensive funding mechanisms within NIDRR. They felt that even though the science was exploratory, the push to disseminate often led to findings being presented as too definitive.

  3. NIDRR should consider more cross-award and cross-agency work.

  4. NIDRR should strengthen its unique role in distributing information to consumers.

  5. NIDRR should provide more focus on employers and labor/demand side issues, as well as on empowerment and school-to-work transition.

  6. NIDRR should consider using an advisory group to help determine priorities.

The panel also suggested that NIDRR examine the unintended bias to continue funding the same centers vs. its role in funding new researchers.

Critique of the APAER Process. In general, the panel commended NIDRR on its efforts to evaluate the Employment portfolio. However, panelists expressed many concerns about the process and their ability to accurately evaluate the portfolio, including: (1) the quality of the grantee reports, suggesting that perhaps grantees did not understand how to complete the online form, were confused by the terminology, or simply did not put much effort into the process; (2) receipt of under-processed data for review; and (3) the general lack of clarity in the definition of terms (multidisciplinary, publication, short-term outcome and intermediate outcome). Panelists suggested that project officers might provide more guidance to grantees in preparing reports.

Annual Portfolio Assessment Expert Review Pilot
Employment Portfolio
Panel Summary Report

The National Institute on Disability and Rehabilitation Research (NIDRR) conducted its Annual Portfolio Assessment Expert Review (APAER) Process Pilot for the Employment Portfolio on September 29 and 30, 2005, in Washington, D.C. This summary report presents a brief description of the APAER process and the findings and recommendations of the expert panel.

Section 1: Overview of the APAER Process

The APAER process was developed by NIDRR to assess its progress in meeting Federal performance requirements under the Government Performance and Results Act (GPRA) and the Program Assessment Reporting Tool (PART), and to capture progress of its grantees using a three-year cycle, with one-third of NIDRR's portfolio reviewed every three years. A team of experts reviewed NIDRR's Employment portfolio as a pilot of this new process. This first year is a baseline year for the Employment portfolio. Through this pilot, NIDRR intends to gather data to compare its baseline performance with results at the end of a specified time period in 2013.

The challenges to NIDRR of APAER were numerous and included establishing an integrated and methodologically sound portfolio assessment and independent expert review process that would:

1.1 APAER Purpose

The APAER was intended to provide NIDRR with a programmatic level, independent assessment of three objectives:

1.2 Pilot Goals and Design Challenges

The specific goals of the 2005 APAER pilots were to test:

In designing and conducting this pilot, NIDRR experienced a number of challenges. The key challenges follow.

1.3 Procedures

NIDRR's annual and long-term performance measures served as the basis of the process.

1.3.1 Portfolio Composition

For APAER, NIDRR grouped eligible awards into portfolios according to the applicable outcome arena of the Logic Model presented in its 2005 Long-Range Plan. For the research and development arena, NIDRR further subdivided these awards into five domains according to its Long-Range Plan. The criteria for inclusion of an award in the Employment portfolio were: employment related focus; active in 2004 and had completed at least one full year of work; and one of the following eligible program funding mechanisms—Rehabilitation Research and Training Center (RRTC), Disability and Rehabilitation Research Project (DRRP), or Field Initiated Project (FIP).

For the Employment portfolio pilot, NIDRR identified 18 eligible awards. To facilitate the review process and reduce reviewer burden, NIDRR divided the awards into two clusters with a similar number of projects in each – awards addressing systems level issues and awards addressing individual level issues.

1.3.2 Inputs to the Process

NIDRR developed an online supplemental information form for grantees to report retrospectively on outputs and outcomes between 2002-2004 corresponding to selected GPRA performance measures. Grantees had six weeks to complete the form, and submit electronic documentation of outputs and outcomes. They provided information on the multidisciplinary teams of investigators, study descriptions, a roster of fellows/trainees and doctoral students, journal publications, and nominations of up to two "best" accomplishment nuggets for each of three arenas: scientific publications, short-term outcomes with evidence, and intermediate outcomes with evidence. NIDRR requested that participating grantees provide feedback on this pilot process.

Individual reports, developed by grantees as they completed their online data entry, served as the basis for the individual level review. The response rate was 100% for the 9 system level projects; 75% (7) of the 9 individual level projects responded.

Panelists received electronic copies of these reports seven business days prior to the APAER meeting. To help prepare for the review, panelists received a background report describing the APAER process, providing information about NIDRR, the employment domain, and the aggregate data from the individual grantees. The aggregate data included information on funding mechanisms (19% of the portfolio were RRTCs, 31% were DRRPs, and 50% were FIPs); first time investigators (four FIPs and one DRRP); years of operation (the average was 2.2 years, and none of the projects that reported were in their 4th or 5th year); the number of awards with at least one previous cycle (three RRTCs, three DRPPs, and three FIPS); study examples; trainees by discipline; number of articles published (a total of 22); data instruments; and the number of nuggets reported as publications (14), short-term outcome nuggets (19), and intermediate outcome nuggets (9). This information was designed to give panelists the broad context at NIDRR and a sense of the accomplishments of the Employment portfolio in general.

Additionally, NIDRR provided panelists with a report to help assess NIDRR management. This report provided NIDRR published funding priorities for a sample year, 2003; a compilation of 81 comments from consumers related to employment needs; an overview of the NIDRR peer review process; selection criteria for each type of funding mechanism; data on peer review scoring for FY 2003 competitions; information on the timeliness of the process; and panel composition.

Reviewers accessed programmatic and logistical information as it became available through a Web site designed for this purpose. (http://www.neweditions.net/APAER2005/)

1.3.3 Panel Composition

NIDRR developed guidelines for panel composition to ensure appropriate representation and expertise. The main criteria were:

Nine reviewers — three researchers, one university administrator/researcher, two economists/policy experts, two clinician researchers, and one representative of a consumer organization — participated on the Employment panel. The expertise of the researchers included medical and vocational rehabilitation, social security and employment statistics, health and disability policy, hearing impairments, self-determination, and community-based mental health programs.

1.3.4 Assessment Process

Panelists participated in two types of assessments: individual grantee and portfolio level performance. Review of the individual grantee reports was intended to assist panelists in judging the overall quality, relevance, and performance results of NIDRR's entire Employment portfolio under the APAER process, rather than the performance of individual grantees.

Individual grantee assessment. The panel received instructions for scoring the individual reports using the Scoring Form for Individual Awards and Accomplishment Nuggets, with an example of a completed form. Using this form, three panelists independently reviewed each report. They were instructed to identify the best published nuggets in the report, based on scoring, and select one nugget at each of three levels: (1) short-term outcomes, (2) intermediate outcomes, and (3) scientific accomplishments. Each panelist scored between five and six grant reports prior to the two-day meeting. At the on-site cluster meeting, NIDRR instructed panelists to:

  1. Discuss individually identified nuggets to generate an inventory of agreed upon accomplishment nuggets.

  2. Give the nugget a brief title to describe it.

  3. Establish whether the accomplishment nugget was an output or outcome.

  4. Rate the nugget and provide a rationale or justification for the rating.

NIDRR asked panelists to rate nuggets on a scale of 1-3, with "1" being a minor contribution and "3" being an outstanding contribution. They used a rating of "0" for "unable to determine." Panelists were instructed to:

Portfolio level review. NIDRR developed a set of questions to guide the APAER portfolio level review and elicit the panel's general assessment of: (1) the performance of the portfolio based on NIDRR's GPRA measures and PART requirements; (2) NIDRR management activities; and (3) the APAER process itself. These broad questions were discussed at both the cluster level and the portfolio level during the meeting.

1.3.5 Meeting Agenda

The panel met for two full days, with the panelists divided into cluster review groups on the afternoon of Day 1.

Dr. Susan Daniels, an independent consultant and former Deputy Commissioner for Disability and Income Security Programs at the Social Security Administration, facilitated the panel meeting. Contract staff provided technical assistance to panelists and logistical support. NIDRR staff participated as presenters, cluster liaisons, and observers. As cluster liaisons, NIDRR staff were available in each room to facilitate the process, primarily by reminding panelists of their charge and the recommended steps to follow. No NIDRR staff were present during the closing session of the meeting.

The meeting opened with introductions and presentations by NIDRR staff. Steven Tingus, Director of NIDRR, provided opening remarks. Dr. Richard Melia, Director of the Research Sciences Division, provided an overview of NIDRR's mission, projected long-range outcomes, three primary goals and research objectives, and highlights from NIDRR's new Long-Range Plan.

Ms. Shelly Reeves and Dr. Edna Johnson, NIDRR project officers, presented background information on NIDRR's Employment portfolio and research directions. Some of the recent employment accomplishments include: 1) research with the Social Security Administration on ways to use research findings to increase benefits and the numbers of unemployed individuals with disabilities returning to work; 2) strategies for job coaching; and 3) best practices for supported employment. Ms. Reeves and Dr. Johnson indicated that input from this APAER panel on gaps in projects was essential for shaping the future of approaches to employment for persons with disabilities.

Dr. Margaret Campbell, Coordinator of Evaluation, reviewed the concept of the 2005 pilot, its relationship to PART, and how APAER was designed. She explained the procedures for the meeting, presented definitions for accomplishments, outputs, and outcomes, and clarified NIDRR's expectations of the panel. Prior to breaking into clusters, a brief discussion was held to clarify the process.

Section 2: Portfolio Performance

2.1 Cluster Level Results and Critique

After meeting separately, each cluster reported its findings to the entire panel, including the number of outputs, examples of the best accomplishment nuggets, and impressions of the quality of these nuggets. The aggregated data from these reports follow.

The two clusters used two different approaches in their cluster level review to address questions. Panelists in the individual level employment issues cluster agreed to settle issues about terminology and adequacy of data as they arose during the review process. When that group was uncertain about the evidence provided by a grantee, they discussed their interpretations and decided if they could rate the particular accomplishment. In contrast, the cluster focused on systems level issues was guided by the meeting facilitator and NIDRR Coordinator of Evaluation to help the group clarify its rating of accomplishments as outputs or outcomes. Consequently, that cluster ranked more of the accomplishment nuggets as either outputs or outcomes.

Panelists were originally instructed to only consider published accomplishments as outcomes nuggets. However, during the meeting, panelists also considered Web site products, the impact of grantees' technical assistance, and other reported accomplishments, such as influence on employment and rehabilitation policies and procedures.

2.1.1 Results

The APAER Employment Panel examined reports from 88% of the awardees in the Employment portfolio. Fifty percent of the total employment awards reviewed were FIPs, 31% were DRRPs, and 19% were RRTCs.

Selected common elements across the individual level and systems level clusters are shown in Tables 1 and 2. Cluster composition is presented in Table 1, and types of accomplishment nuggets are listed in Table 2. In general, panelists reported that they did not have enough information to give scores other than "0" to the identified nuggets.

Table 1
Composition of Clusters

Cluster 1
Individual
Level		 Cluster 2
Systems
Level
By Number of Grantees
# of Grant Reports Reviewed 7 9
Total # of Grantees 9 9
Percentage Grantees Reporting and Reviewed 78% 100%
By Program Mechanism
# of FIPs 4 4
# of RRTCs 0 3
# of DRRPs 3 2

Table 2
Type of Accomplishment Nuggets

Cluster 1
Individual
Level		 Cluster 2
Systems
Level
Academic Publications 5 8
Non-academic/consumer-oriented publications or products 5 10
Discoveries, advancing knowledge and understanding 8 11
Tools 2 6
Interventions 8 6
Total 28 41

Note: Numbers reported reflect a compilation of potential nuggets identified individually by reviewers prior to attending the panel meeting and may not reflect cluster or panel discussion.

2.1.2 Inventory of Accomplishment Nuggets

Some accomplishments were not classified due to insufficient information. Notes from individual and group review were compiled into a comment section. At the cluster level meetings, the panelists decided which of these nuggets merited joint review.

In some cases, the grantee reporting was premature in that a product or accomplishment had not yet been completed, but rather was a work in progress. These were included in the inventory to provide examples of expectations for future accomplishments.

2.1.3 Cluster Level Reports to Panel

During the APAER panel assessment, highlights of findings and impressions from each cluster meeting were reported. Key areas of agreement from each cluster are summarized below.

Systems Level Cluster.The panel found it could not make many solid evaluations about scientific rigor and productivity based on the available data. Given NIDRR's expectations, the panel felt that the level of funding for these awards was low. The panel felt the cluster was narrow and conservative in scope and needed more emphasis on school-to-work transition issues. Panelists suggested NIDRR encourage interdisciplinary participation from non-traditional professions such as business, information technology and medicine to enhance the research.

Individual Level Cluster. This cluster included four FIPs and two DRRPs. The panel determined it could not assess the scientific rigor and productivity of the awards in this cluster due to insufficient information. They noted a disconnect between the goal of the award activities and the reported accomplishments for many awards. Several panelists had the sense that awardees were doing more than was reported. They felt that the funding level for FIPs was low and suggested NIDRR clarify the purpose of FIPs. This panel also suggested that transition research is a gap in the portfolio. They commented that there is little attention given to individuals who have severe disabilities because it is more difficult and costly to conduct research with this population. The panel felt the awards needed more consumer involvement - including employers.

APAER Process. After the presentations, panelists presented some overarching concerns about the assessment process.

2.2 Portfolio Level Results

After the cluster reports, the moderator asked panelists to consider all of the information presented as they evaluated the portfolio as a whole in relationship to specific NIDRR long-term performance measures and the PART criteria for R&D investment for quality, relevance and performance. Using a nominal process to stimulate discussion, the facilitators guided a thoughtful review of the Employment portfolio. To start the discussion, panelists jointly identified the best accomplishments from the portfolio:

Outcomes:

  1. A service delivery model for individuals who are deaf or hard of hearing that is now under Congressional review.

  2. A policy brief on perspectives on disabilities (by race and ethnicity) for the Ticket to Work program.

Outputs:

  1. Publication of a book on disability law and policies.

  2. Publication of a report and compendium on barriers to employment according to race and ethnicity for individuals who are blind, with documentation of a high number of sales.

  3. Tutorials that were developed and used to train individuals who are blind to use Excel and Microsoft products.

  4. Survey finding that interagency initiatives were the only employment policies that benefited individuals who are deaf or hard of hearing.

  5. A literature review of self-employment among people with disabilities.

  6. An investigation of business, employment, and economic development of persons with disabilities in rural areas, including development of a program for vocational rehabilitation counselors, with research conducted by multiple sites.

  7. An accessible e-learning authoring system.

Activities:

  1. A qualitative study of persons with mental illness who got jobs on their own.

  2. Broad dissemination of research on service delivery to persons with mental illness for performance improvements.

  3. Development of a sample for a cohort study regarding individuals who are deaf and aging.

When asked to make an overall assessment of the Employment portfolio in four areas, the panelists determined that:

NIDRR developed a series of questions to elicit feedback on the progress of the portfolio on selected GPRA and internal long-term performance measures that relate to the NIDRR strategic goals and objectives in capacity building, research and development and knowledge translation.

Panelists used response cards for "agree," "disagree," and "uncertain" to indicate their judgment for each question. These responses were not meant to serve as quantitative data, although in some instances the responses were counted as an indication of majority opinion. Key agreed upon points or themes are presented below. Comments that reflect one panelist's opinion are indicated as such.

The first NIDRR strategic goal is to increase capacity to conduct and use high-quality and relevant disability and rehabilitation research. The following four questions address specific objectives for this area.

Overall, are there an adequate number of awards in the Employment portfolio that include multidisciplinary teams of investigators from relevant fields and sub-fields to strengthen NIDRR's capacity to solve problems in creative ways using state-of-the-art methods?(PM 7.1.1)

Agreed. A few panelists who agreed, suggested that there might be a difference in interpreting the question. Since the projects reviewed were very narrow, they considered whether staffing across the current set of projects was adequate to meet the needs of these underfunded projects. They suggested that, within the current market, NIDRR is getting the teams it expects, and that the teams seem to be adequate to solve problems in creative ways. Also, they noted that it would have been helpful to have information on outside consultants.

Disagreed. The majority disagreed with the adequacy of the multidisciplinary teams, indicating that there was not a true representation across disciplines and that sometimes a key discipline was not represented on a team. There was not enough cross-fertilization. With the FIPs, it seemed to be an issue of resources and the difficulty in employing additional staff on a very limited budget. The panel recommended considering incentives for cross-disciplinary participation in competitions.

The panelists noted that there also was a considerable lack of consumer involvement in all aspects of design, implementation, and evaluation.

Uncertain. None.

Overall, does the Employment portfolio have an adequate number of awards that integrate research, education and training? (PM 8.1.2)

Agreed. None

Disagreed. The panel was split between those who disagreed and those who were unable to determine the adequacy of integrating research, education, and training. Those who disagreed indicated that they were not clear how the knowledge from the projects would integrate with education and training at both the individual and systems levels.

Uncertain. Those who were uncertain if the portfolio had an adequate number of awards integrating research, education and training voiced concern that some of the projects are completing accomplishments that did not show up in the grantee reports. They commented that the emphasis on the "nuggets of accomplishments," as defined, would hinder acknowledgement of these accomplishments. They suggested that the data were sometimes incomplete, and that the time horizon for the snapshot was too narrow to expect research, development, and training. There was also a question regarding the relationship between OMB requirements and NIDRR's emphasis on training.

Overall, does the Employment portfolio have an adequate number of awards that include multi-site controlled trials to generate scientific-based knowledge that demonstrates improved individual outcomes and contributes to changes in policy and practice? (PM 8.1.1)

Agreed. None

Disagreed. The panel unanimously disagreed with this question. They did not believe these were adequately represented, and noted that they found no evidence of multi-site controlled trials for either individual or system level grantees. They thought, however, that this was a reflection of the funding amount. They noted that even the DRRPs provided no evidence of a controlled study, let alone a multi-site study.

Several panelists also expressed a belief that some of these projects are conducting but not reporting such studies.

They also were distressed that sometimes projects did not acknowledge NIDRR as a funding source, which they believed should happen even if NIDRR provides only partial funding for specific research.

Uncertain. None

Overall, is there an adequate number of awards in the Employment portfolio to diverse individuals, including first time NIDRR investigators and persons with disability, and to underrepresented institutions; and is the portfolio training an adequate number individuals from diverse disciplines? (PM 8.1.2)

The panel had difficulty answering due to lack of data, and recommended skipping these questions.

The second NIDRR strategic goal is to generate scientifically based knowledge, technologies, and applications to inform policy, change practice and improve outcomes. One question addressed objectives under this goal.

Overall, is the portfolio producing an adequate number of high-quality R&D outputs and outcomes to significantly advance knowledge, improve measurement, and strengthen the scientific basis of disability employment research? (PM 8.2.5)

Agreed. None.

Disagreed. Those who disagreed indicated that the methodology for the research was inadequate, and although there was some movement, the portfolio as a whole did not appear to be advancing knowledge. It seemed that nothing was particularly innovative. One panelist suggested that funding should be invested elsewhere, and that although there were some good ideas, she would want to see more progress next time.

Uncertain. Those who couldn't decide indicated that they did not have access to information on the methodology used by the grantees and very few peer reviewed publications were reported, which might provide some reassurance regarding high quality methodologies. The panel also suggested that "they needed the fullness of time" to determine the quality and significance of the research, and that since the term "significant" was ambiguous, it was difficult to evaluate this. They thought that perhaps "significantly advanced" might be too high a bar, given the timeframe for the grantees.

The third NIDRR strategic goal is to promote the effective use of scientific-based knowledge, technologies, and applications to inform disability and rehabilitation policy, improve practice, and enhance the lives of individuals with disabilities. The following three questions correspond to specific objectives in this area.

Overall, is the portfolio producing an adequate number of high-quality and useful non-academic and consumer-oriented publications and products to increase understanding and awareness among stakeholders and contribute to improvements in disability-related Employment policy, practice, and/or system capacity? (PM 7.2.1)

Agreed. Those who agreed stated that many of the projects demonstrated their value, even if it wasn't directly related to the research or results. They felt that the grantees' activities demonstrated definite potential to positively impact the individuals they were serving. Some were giving the grantees the benefit of the doubt, even though they were not certain that the grantees were producing high quality results. They also noted that these projects historically have developed and disseminated non-academic products.

Disagreed. Those who disagreed acknowledged, however, that some projects demonstrated potential. They also commented that the non-academic publications appeared neither to have been expected as part of a priority, nor planned, but rather appeared almost through serendipity.

Uncertain. Those who were uncertain remarked that, given the way the instructions were interpreted, this is likely an under-reported category. While the number of non-academic publications was high overall, they were uncertain of the quality. These panelists were uncertain regarding NIDRR's expectations, and recommended that a better process might involve first developing a stated purpose for the projects and then evaluating progress against the stated purpose. Also, given that all of the projects had something of this nature, they questioned what was adequate.

Are there any scientific registries and/or collaborations that provide external review (i.e., vetting) of the quality of research methods and findings in the employment area, in general, and disability employment research in particular? (PM 8.2.4; 8.2.6)

The panel did not know of any registries focusing on the disability employment arena. They suggested checking with Statistics Canada and ARCH (a Legal Resource Center for People with Disabilities, also in Canada), the GAO standards on reliability and validity, the NIDRR's Interagency Subcommittee on Disability Statistics, the Research Triangle Institute (RTI), and the Office of Disability Employment Policy (ODEP).

After some discussion, a question was raised about whether there should be any standards and guidelines for disability employment research, and if so, who should develop these standards?

Reviewers noted the importance of the definition of disability and its numerous sub-populations, and suggested caution in relying on secondary data sources. They noted that the work-limited population is an important sub-population, and the focus of some attention. They discussed the controversy that has been generated when ADA data has been used in policy analysis, and suggested that NIDRR:

Section 3: NIDRR Management of Portfolio

Assessments of NIDRR management were offered on the cluster and portfolio levels. At the cluster level, panelists made judgments about each cluster after reviewing, rating and discussing a set of individual grantee reports. The portfolio level assessment was based on reports from both clusters. Discussion focused on portfolio research directions and NIDRR management activities.

3.1 Portfolio Level Results

Facilitators led the group discussion using questions developed by NIDRR. Not all questions were addressed due to time constraints. Group judgments are reported below for research directions and NIDRR management activities.

Overall, is the portfolio producing an adequate number of high-quality and useful non-academic and consumer-oriented publications and products to increase understanding and awareness among stakeholders and contribute to improvements in disability-related Employment policy, practice, and/or system capacity? (PM 7.2.1)

Agreed. None

Those who disagreed expressed a common concern that the FIPs are not adequately funded. Overall, the panel believed that the size of the grants was too small for what they were trying to accomplish, making it difficult to have appropriate multidisciplinary teams and facilitate replicability. The panelists thought that there was an engineering rather than a social science bias within NIDRR, suggesting that grantees may not necessarily be drawing on the best expertise. They also noted that lack of funding for research related to employer-based, labor/demand issues, as well as projects building on the value of self-determination in employment. Several panelists reflected on the lack of school-to-work transition projects, even though this was a prominent goal in the 1999 Long-Range Plan. The panel also had several recommendations regarding increasing funding for the FIPs and reexamining the purpose of these projects, including improving the alignment between the FIPs and the larger centers.

Uncertain. None

Overall, are the funding priorities and specific topics included in this portfolio relevant to and/or aligned with the state-of-the-science in the relevant fields and sub-fields? (Source: OMB Objectives for R&D)

Agreed. None.

Disagreed. The panel indicated that there was no specific focus on self-determination, which should be highlighted because of its importance both in terms of motivating people to work and in the broader scheme of helping plan for careers. They also commented that:

The panel recommended that NIDRR be more directive, and use more of a contract-based model, with a more specific statement of the problems. However, NIDRR should move away from dictating methodologies, leaving that to the creativity of the researchers and being careful not to constrain the researchers. They also noted that the priorities did not identify the stage of knowledge development, which could be useful. In a similar vein, they emphasized that they would welcome and encourage more project officer involvement in the portfolio in guiding and reporting the accomplishment nuggets.

The panel recommended that NIDRR consider more opportunities for dialogue and collaboration between traditional NIDRR researchers and other researchers. They noted that "quality of the ideas" in the grants was high. They also urged NIDRR to find ways to increase communication and sharing among its researchers in the employment arena.

Relative to differences in budget, does the overall performance of this portfolio compare favorably to other federal programs with a similar purpose and goals?

Agreed. Those who answered "yes" stated that within the limitations imposed by inadequate funding, NIDRR was accomplishing what it can. They believed that other agencies were "just as dysfunctional," and commended NIDRR for distributing materials to its audience.

Disagreed. Those who disagreed compared NIDRR to NIH and foundations. They recognized that NIDRR typically provides funding at a higher level than foundations; however, foundations can sometimes do more and might really be getting "a bigger bang for the buck." They mentioned both NIMH and SAMHSA as agencies that seem to have larger budgets and address a larger breadth of research with more advanced standards for methodology. While they were uncertain of the impact of the larger budgets, they believed that budget differences made comparison difficult.

Uncertain. Panelists who were unable to determine the adequacy suggested that most other agencies do not have portfolio systems, and that it may be both difficult and unfair to compare NIDRR to other agencies, especially given the differences in budgets.

The panel recommended dropping this question because of the limited information provided and difficulties in making fair comparisons.

3.2 Closing Discussion

During this final session, the panel discussed issues emerging from earlier discussions relating to a continuum of NIDRR interests, a process to identify critical gaps, and topics that should be discontinued. They also provided recommendations regarding NIDRR management of activities. NIDRR staff were not present. The panel began by commending NIDRR on its willingness to be innovative and bring in experts to assist with this review.

Does NIDRR have a process to identify critical gaps?

The panel recommended that NIDRR consider an advisory group that would help determine priorities for funding and funding mechanisms, and suggested that:

The panel recommended that NIDRR consider using standing panels to provide feedback to assist project directors in revising and enhancing the quality of the proposals submitted to NIDRR. The panel also recommended that:

Suggested Topics to Discontinue

The panel was concerned that the difficulties they experienced in evaluating the quality of some of the accomplishments reported by grantees may be reflective of the type of projects being funded, and recommended that NIDRR focus grantees on products that have proven validity. The panel also suggested that NIDRR:

Does the panel have recommendations in terms of NIDRR management of activities?

The panel recommended that NIDDR continue its unique role in funding disability-related portfolios, continue to be an agent of change, and continue its important role in funding new researchers. They also suggested that NIDRR:

The panel also provided initial comments concerning the content and scientific undertakings related to the Employment portfolio, encouraging NIDRR to:

Section 4: Feedback on APAER Process

Panelists at both the cluster and portfolio levels offered assessments of the pilot APAER process. At the portfolio level, after both clusters reported, panelists were asked to assess the entire portfolio process. At the cluster level, panelists made judgments based on their experience reviewing, rating and discussing a set of individual awards.

4.1 Portfolio Level Results

The panel observed that APAER provides NIDRR with a means to improve its Employment portfolio. They felt that portfolio review is the first step in a necessary but difficult process, and that despite the complexity, NIDRR's broad audience is a strength.

Having completed the portfolio review, the panel perceived a disconnect between the goals of the projects and what was reported. They indicated that the information they received was inadequate to provide a valid evaluation of this portfolio, and that the data reviewed were underprocessed. They suggested that more of the work should be completed by the grantee, and provide a rationale for why the proposed outcomes were important. They reported what they believed to be structural issues with APAER, including the need for more active management of research programs, with project officers facilitating more interaction with grantees.

The panel also discussed terminology, and recommended that future reviews include more terms in a glossary, and changing certain terms, such as "adequate/inadequate," which they felt were not accurate in conveying how projects fit together and fill the gaps in the existing research.

The panelists suggested that the APAER process could be improved by:

Sending a summary to all panelists for reaction after the review, then revising the process, and sending it to selected panelists for review, to then introduce a refined process next year.

4.2 Cluster Level Comments

In their cluster level reports, panelists identified several concerns with how the data nuggets were identified and reported, indicating that:

Section 5: Summary of Discussion

5.1.Implications for the Employment Portfolio

According to the APAER panelists, improving employment outcomes for persons with disabilities remains challenging and yet is crucial to both improving their overall quality of life and reducing their dependence on governmental support. These long-term challenges emphasize the potential impact of any discoveries NIDRR and its grantees make in this arena. Therefore, the panel believed that whatever steps NIDRR can take to improve the Employment portfolio are noteworthy. One of the primary implications of the review of this portfolio is that enhanced communication from NIDRR, cross-agency collaboration, and sharing among researchers conducting similar investigations is highly recommended by the panel. Another is that an advisory group and standing panels could be instrumental in both increasing the quality of research initiatives and the quality of proposals that are funded.

Limitations of the Findings. As panelists expressed many times throughout the APAER review, better data from the grantees could have improved the validity of the outcomes obtained in this review. Panelists felt restricted by confusion over terminology, grant reports that were incomplete, and inconsistencies in the data review process. Panelists wanted more information from grantees on how they conducted their research as well as the impact of their findings. Because of these difficulties, it may be that the grantees have completed more high-quality accomplishments than were identified through APAER.

5.2 Implications for the APAER Process

Panelists made several recommendations for improving NIDRR's APAER process, including enhancing the role of the project officer, simplifying the terminology and the APAER process, and providing panelists better access to publications of grantees.

Limitations with the Employment Pilot. The newness of the portfolio level assessment approach for Federal research agencies contributed to some difficulties in both conceptualizing the process and preparing materials for rating both cluster-level and portfolio-level accomplishments at NIDRR. A number of difficulties were identified in advance of the APAER meeting and others were identified on-site, suggesting that caution be used in interpreting the findings. These difficulties included:

5.3 Grantee Comments

Grantees were given the opportunity to provide feedback on the supplemental information form and the online data collection process. Regarding the online data collection process, grantees indicated that the form helped them figure out how to communicate accomplishments in a way that supports GPRA requirements. However, they had several recommendations for simplifying and improving the process, including providing alternative reporting methods (optional text fields) to allow for providing a fuller report and explanations, improving the online navigation, and changing some of the default formats. They recommended that project officers work more closely with them ahead of time so that they have a better understanding of NIDRR's reporting requirements.