PDF iconPDF version: 461 KB | wordMS Word: file download 568 KB

National Institute on Disability and Rehabilitation Research

Final Draft

Annual Performance
Assessment Expert Review

Health and Function Research


Prepared by
New Editions Consulting, Inc.
6858 Old Dominion Dr., Suite 230
McLean, VA 22101


Table of Contents

Executive Summary

Section 1: Purpose of the Annual Performance Assessment Expert Review
Overview of the APAER Process

Section 2: Panel Discussion
Relevance
Quality
Performance
Other Topics

Section 3: Appendices
Appendix A: Agenda
Appendix B: Panel Member Biographical Information
Appendix C: Noteworthy Grantee Accomplishments Selected by the Panel
Appendix D: Evaluation Results for 2006 Health and Function APAER
Appendix E: Results of Accomplishments Ratings

Executive Summary

The Annual Performance Assessment Expert Review (APAER) is the National Institute on Disability and Rehabilitation Research (NIDRR) response to the government accountability emphasis on results based on performance goals and measures. The 2006 APAER review covered NIDRR’s health and function domain, consisting of 128 awards, or about half of the total number of NIDRR awards, and was intended to provide NIDRR with an independent assessment of:

  1. The quality and relevance of NIDRR-funded research and the extent to which research outputs and outcomes are contributing to the agency's long-term performance goals and measures under the Government Performance and Results Act (GPRA).
  2. The quality and relevance of the agency’s management of research directions and award decisions.
  3. The strengths and weaknesses of the health and function domain, including recommendations to ensure accomplishment of NIDRR’s goals and objectives.

While NIDRR has conducted formative and summative program reviews of its large center grants since 1995, the APAER is a new process focusing on portfolio-level, rather than individual grantee level results. Such portfolio-level assessment is new to the Federal government, particularly as it is applied to research and development as opposed to service programs. This approach to assessing the results of R&D investments is still evolving across the government and no consensus exists on how best to implement it. NIDRR is one of a few Federal agencies currently experimenting with how to apply this method of evaluation to meet PART requirements and assess progress under GPRA measures. Simultaneously, NIDRR initiated a new automated performance reporting system from which data for APAER was gathered. As a result, information was available for the current reporting period only.  For some grantees, the reporting period was less than one year. 

Footnotes have been inserted into this report at NIDRR’s request in order to clarify inaccuracies in panel members’ statements regarding how NIDRR manages peer review and award decisions and what is allowed within Department or OMB policies.

The Expert Panel consisted of 13 members with expertise in research relating to the topics of the health and function domain, including stroke, spinal cord injury, traumatic brain injury, prosthetics, physical medicine and rehabilitation, assistive technology related to function, burn rehabilitation, neuromuscular disorders, and consumer interests. The panel met October 18–19, 2006 and made the following assessments:

Relevance

Quality of Peer Review Procedures

Resulting Portfolio of Awards

Performance

Gaps

Panel Comments on the APAER Process

Section 1: Purpose of the Annual Performance Assessment Expert Review

The Annual Performance Assessment Expert Review (APAER) is NIDRR’s response to the government accountability emphasis on results based on performance goals and measures. While NIDRR has conducted formative and summative program reviews of its large center grants since 1995, the APAER is a new process focusing on domains in the NIDRR’s long-range plan. The APAER review is intended to provide NIDRR with an independent assessment of:

Overview of the APAER Process

The Expert Panel met October 18–19, 2006.  The panel consisted of 13 members with expertise in research relating to the topics of the health and function domain, including stroke, spinal cord injury, traumatic brain injury, prosthetics, physical medicine and rehabilitation, assistive technology related to function, burn rehabilitation, neuromuscular disorders, and consumer interests.  Panel members were drawn from leading university and medical research centers, the Department of Veterans Affairs, the Department of Defense, the National Institutes for Health, and a consumer organization.  The meeting agenda and panel member biographies are in Appendices A and B, respectively.

For performance goals that cannot be measured in quantitative terms, the Office of Management and Budget (OMB) has stated that agencies may satisfy performance reporting requirements via the Research and Development Investment Criteria of Quality, Relevance, and Performance.  NIDRR designed the APAER process to meet those requirements, adapting the exemplary approach used by the National Science Foundation.  The 2006 APAER was the first review of NIDRR performance in the area of health and function.

Relevance

Relevance to agency mission and the field is assessed.  The purpose is to ensure that the research funded in the health and function domain is relevant to NIDRR’s mission and goals. 

Quality

Quality is ensured through the peer review process for grants. Decisions to award or decline grant proposals are based on the informed judgment of NIDRR program officers following peer review2 . The Expert Panel examined competition data for proposals in health and function.  The panel also assessed the overall quality and balance of the resulting health and function portfolio.

Performance

The Office of Management and Budget requires R&D programs to maintain a set of high priority, multi-year R&D objectives with annual performance measures and milestones that show how one or more outcomes will be reached. Metrics should be defined not only to encourage individual program performance but also to promote broader goals such as innovation, cooperation, education, and dissemination of knowledge, applications, or tools. The Expert Panel provided a retrospective performance review by examining the performance information associated with the health and function domain.

Awards Included in the Health and Function Portfolio

The health and function research portfolio consists of 128 awards, or about half of the total number of NIDRR awards. Grants from the technology domain were reassigned to health and function if they focused on one or more of the following:

  1. Bodily aspects of health and function, including restoration, at the impairment and/or physiological level of motor (e.g., mobility/walking, reaching, grasping, lifting), speech, sensory, cognitive functioning and mental health.
  2. The environment in which health care and rehabilitative services are delivered or in which rehabilitation research is conducted – including access to healthcare services, telerehabilitation, and accessible medical instrumentation.
  3. Health promotion behaviors including adaptation, wellness, exercise, and nutrition. 
  4. Basic activities of daily living (ADLs), such as bathing, dressing, toileting, eating, and transferring.
  5. Better understanding or measurement (including diagnosis or assessment) of the underlying mechanisms (e.g., physiological, medical, economic) that cause or contribute to impairment, disease, or lack of access to healthcare services, or the design principles that contribute to the development of devices, products and systems that restore function or maximize positive health outcomes.
Data Sources

The data presented to the panel were submitted by grantees through the new automated Annual Performance Report.  This new report collected information for the current reporting period only: July 1, 2005 through May 31, 2006. 

Data limitation:  Because the Annual Performance Report was new this year, information was available for the current reporting period only.  For some grantees, the reporting period was less than one year.   

There were two primary sources of data for the panel consideration: project abstracts and project accomplishments. 

Project Abstracts were organized by funding mechanism. Included were all active health and function awards as of December 1, 2005.  

Project Accomplishments were organized by NIDRR health and function goal, knowledge translation goal, capacity building goal, intermediate outcomes, research cited by others, and recognition of principal investigator.  NIDRR asked its current health and function grantees to submit up to five outputs or outcomes that the grantee considered to be the most important that occurred or matured during the current reporting period (July 1, 2005 through May 31, 2006).  Nominated accomplishments were required to be related to the objectives of the award, although in many cases they originated in a previous funding cycle.  NIDRR Project Officers reviewed the accomplishments for appropriateness and completeness, and the grantees revised their submissions as needed. 

Report Template

The panel was given a report template with questions to guide the discussion.  Panel consensus was not required in addressing the following questions.

Relevance of the Health and Function Domain

Quality of Peer Review Procedures

Health and Function Portfolio of Awards

Performance

Other Topics

Background Material for the Panel

The panel received a background briefing report two weeks before the meeting. The report included:

NIDRR Presentation

NIDRR Director Steven Tingus welcomed the panel.  Deputy Director Kelly King described the OMB requirements and objectives of the APAER process.  Ruth Brannon, Acting Director of the Research Sciences Division, presented an overview of NIDRR health and function research. 

Section 2: Panel Discussion

This section summarizes the panel discussion of the APAER questions.  Dr. Pamela Duncan served as Panel Chairperson.  The panel was not required to reach a consensus on its answers or suggestions.  Comments are organized by APAER questions:  Relevance, Quality, Performance, and Other Topics.

Footnotes have been inserted into this report at NIDRR’s request in order to clarify inaccuracies in panel members’ statements regarding how NIDRR manages peer review and award decisions and what is allowed within Department or OMB policies.

Relevance

According to the Office of Management and Budget, R&D investments must have clear plans and must be relevant to national priorities, agency missions, relevant fields, and customer needs. Review committees should assess program objectives and goals on their relevance to national needs, customer needs, agency mission, and the field of study the program strives to address.  The main question for the panel was:  Does the agency's research address topics which fulfill the agency's mission?

The mission of NIDRR is to generate new knowledge and promote its effective use to improve the abilities of people with disabilities to perform activities of their choice in the community, and also to expand society’s capacity to provide full opportunities and accommodations for its citizens with disabilities.  [NIDRR 2005-2009 Long Range Plan]

Does NIDRR’s health and function research address topics which fulfill the agency’s mission?

Yes, based on NIDRR’s mission statement and the published priorities reviewed by the panel.  The panel commended NIDRR on the important role it plays in rehabilitation research.

Research Priorities

The panel noted that NIDRR has a very broad mission in addressing the needs of people with disabilities.  The priorities statements seemed very broad.  While the priorities addressed the agency’s overall mission, the panel felt that the priorities should be more specific.  The agency should develop a strategic plan identifying specific unmet needs or specific areas of focus in each of the three research domains (Health and Function; Employment; and Community Participation), and identify long-range goals.  For example, capacity building should be defined with specific numbers or measures of success, perhaps tracking  whether NIDRR-funded research centers have become independent in seeking major funding beyond NIDRR.

Specific focus areas should be reflected throughout NIDRR’s funding mechanisms.  This would make it easier for the panel to assess the relevance of the health and function portfolio. 

The panel also noted that NIDRR should be more specific in its definition of “new knowledge,” highlighting the role the agency has played in supporting the development of important new measures.

Randomized Controlled Trials

The Model Systems centers provide an excellent infrastructure for collaborative multi-center studies, including randomized controlled trials (RCTs). The panel praised NIDRR and the centers for developing innovative collaborations. These collaborations position the centers to be competitive for funding from other agencies.  The panel recommended that NIDRR continue to manage its research portfolio to provide a diverse array of research methodologies appropriate to relevant and important questions in the field of health and function, insuring that costly RCTs do not absorb an inappropriately large share of limited research funds. The panel suggested that NIDRR coordinate and collaborate with other entities to support elements of larger research projects that reflect NIDRR’s specific niche and priorities.

Dissemination and Knowledge Translation

The panel noted that NIDRR’s dissemination role is very important. Yet information on NIDRR research is not easily found.  For example, the background briefing material provided a wealth of information on NIDRR research that is not readily available through conventional literature searches.  The panel suggested that NIDRR improve its Web site to highlight its many strengths and to provide more information about its research programs. The panel also suggested that NIDRR either strengthen its requirements for grantee dissemination (with increased funding to each grantee for this purpose), or establish a specific project or center to disseminate NIDRR’s research accomplishments. 

The panel noted the difference between dissemination and knowledge translation, and emphasized that knowledge translation is more than delivering information through publications and presentations.  Knowledge translation emphasizes expert judgment on the value of information for further dissemination, better accountability for outputs produced by researchers, and improved methods for making information available beyond the research community. Based on this approach, the panel felt that NIDRR did not meet its knowledge translation goal. Grantees are being asked to accomplish a lot with limited funding, and knowledge translation was beyond the resources provided. [The panel noted that NIDRR has begun to address this issue this year by funding a knowledge translation center for the model systems].

Does NIDRR’s health and function research align with the current state-of-the-science in the field?

The panel did not have enough information to consider how NIDRR’s health and function research aligns with the current state-of-the-science in the field.  Instead, panel members agreed to provide suggestions that would enable a future APAER panel to answer this question.

Information provided by the grantees

Accomplishments submitted by grantees should articulate the problem or gap that the accomplishment addresses, and the methodology used to address the problem.  Some of the accomplishments were similar to existing products, so the grantees should explain how their work differs.  For example, many consumer resource guides exist, so the grantee should note how its resource guide addresses a specific geographic area or subpopulation.  There should be a clear statement of how the outcomes relate to specific NIDRR goals. 

Information provided by NIDRR

NIDRR should provide information on how its research projects take a unique approach to the field, and how NIDRR fits among other federal agencies funding similar research. The panel suggested that relevance to the current state-of-the-science should be part of the criteria used in the grant review process.  An overview of the state-of-the-science should be provided to the APAER panel to assist in judging whether NIDRR’s research projects align with the field.  

Assessing grantee progress

Due to the short reporting period under the new Annual Performance Report, the panel found it difficult to assess the grantees progress.  One year is not enough time to determine research progress for multi-year awards. The information provided to future panels should show grantee progress or status in their timelines for completing the project.

Leveraging NIDRR funding

The panel believed that many grantees coordinate NIDRR funding with funding from other agencies to conduct their research.  NIDRR should collect this information from grantees to demonstrate NIDRR’s role in these larger projects.

Please comment on the research priorities that guided the development of the health and function domain.

The panel noted that each funding mechanism has its own set of priorities, rather than a set of clear overarching health and function priorities. The emphasis on funding mechanisms rather than program areas puts a burden on grantees, especially Center grantees, to design a broad range of projects that lack the depth that would be possible with program-focused priorities.  Panel members suggested that goals for each of the three domains (Health and Function; Community Participation; and Employment) should determine the priorities across funding mechanisms. 

Relationship between goals and accomplishments

Information provided to future panels should show the relationship between health and function goals, problem areas, priorities, applications, awards, and accomplishments.  The panel also suggested that NIDRR be more specific in its definition of problem areas, and its role in addressing them.

Information for future review

The panel noted that the accomplishment summaries were not consistent in how they presented information on research findings, particularly in publications, making it difficult to determine relevance.  Grantees seemed to emphasize the publication or presentation, rather than the substance of the findings contained in the publication. The panel suggested that additional information be provided to determine:

Quality

The Office of Management and Budget requires that research agencies maximize the quality of the R&D they fund through the use of a clearly stated, defensible method for awarding a significant majority of their funding. A customary method for promoting R&D quality is the use of a competitive, merit-based process.  Programs must also assess and report on the quality of current and past R&D through retrospective expert review.  To address these requirements, the panel assessed NIDRR peer review procedures and the resulting portfolio of awards for quality.

Quality of Peer Review Procedures

Is the peer review mechanism appropriate?

The panel agreed that NIDRR’s peer review process was generally appropriate, and suggested that additional information was needed to provide specific comments. They noted that since the structure was based on funding mechanisms, it was not possible to comment on how peer review addressed the health and function domain.  The panel was concerned that aspects of the peer review process were not well managed, as evidenced by NIDRR’s inability to provide the necessary data to evaluate the process.

Information for future review

The panel suggested that future APAER panels have the rosters of reviewers for each panel, how many applications were reviewed by each panel member, instructions to panels, and lead time for review 3 .

Comments from individual experience

Several of the panel members had served as NIDRR peer reviewers and offered comments about their experiences.  They suggested that the current process was very burdensome in terms of the number of applications and the length of each application that each peer reviewer had to review.  They noted that the peer review process has improved, particularly for the Field Initiated Projects, and approved of the increased use of electronic review procedures.  They suggested establishing standing panels of reviewers, and agreed that a statistician should be included on every panel. 

Are the selection criteria appropriate for the type of program mechanism?

The panel could not determine if the selection criteria were appropriate based on the information provided. 

Information for future review

Information was needed on the scores for all of the applications, including those that were not funded.  The information should provide sufficient detail to determine how many relevant, significant projects were submitted, but not funded. The panel noted that the selection criteria listed in the funding mechanism competition notices did not reflect goals or priorities specific to health and function.

Comments from individual experience

Point system. The scoring system did not always result in the best applicant scoring the highest points.  Panel members suggested that elements considered important, such as dissemination, should have a larger number of points allocated to them.  Also, certain items, such as affirmative action, should be a “check-off” noting its inclusion, rather than having a score. 

Methodology.  Panel members suggested the need for a methodological evaluation of the project; peer review should consider more than just the project design.  For example, how does the applicant demonstrate that it can recruit a sufficient number of subjects for the study?

Relevance and significance.  The panel suggested that applicants be judged in some way by the relevance and significance of their projects to the goals of the health and function domain, and the state-of-the-science.  For example, the peer review panel might first “triage” the projects by their health and function relevance and significance, then go to the next level of scoring.  A future APAER panel could use the triage ratings to determine the relevance of the portfolio.  Field Initiated Projects should also be scored on their appropriateness to the NIDRR mission.

Past performance.  Panel members noted that some applicants provide information on prior performance, but NIDRR does not require it 4.  Prior performance should be a consideration, and NIDRR should provide this information to the peer reviewers. There is a perception that NIDRR continues to fund grantees that have performed poorly in the past.

Application materials.  Panel members noted problems with incomplete application materials.  They suggested that applicants be required to submit all of the materials both electronically and in paper copies.  Also, one panel member suggested that the applicants present the list of expected deliverables and project timelines in graphic formats that make it easier for the reviewers to consider the feasibility of their proposals. Page limits of 50 pages were also suggested.

Is the time to decision appropriate?

Yes, the decision times seem very good.

Discuss issues identified concerning the peer review procedures in the health and function domain.

Improve the peer review process

The panel strongly suggested that NIDRR conduct a comprehensive review of its peer review process for the purpose of reducing the burden on reviewers.  The review should include the selection criteria, the scoring system, and the possible use of standing panels.    

Insufficient notice to applicants

The panel suggested that two months notice for NIDRR’s RFPs is not sufficient; a significantly longer lead time is needed. The panel noted that NIDRR’s proposed priorities are published in advance.  However, information about the priorities or the competitions is not widely disseminated. Also, some organizations are reluctant to start a proposal based on a proposed priority, preferring to wait for the final priority.  These factors are considered to favor insiders. The panel strongly suggested that NIDRR conduct broader outreach to the research community, including publishing a notice of intent as a way of increasing advance notice.

NIDRR efforts

The panel noted NIDRR’s efforts to improve peer review quality, including standing panels for Field Initiated Projects, an electronic system used in the review process, internal NIDRR staff review of the reviewers, and improved instructions and preparation of the reviewers.

Resulting Portfolio of Awards

Are success rates appropriate for the type of priority?

The panel did not have the information needed to determine if the success rates were appropriate for the type of priority.

Information needed

Scores of the non-funded applications should be provided as well as the funded applications, along with data on the range and distribution of all of the scores, and the amount of funding available compared to the amount allocated. This should be reported consistently across all funding mechanisms. 

Flexibility in allocation of funding

The panel and NIDRR staff discussed the agency’s flexibility in allocating dollars among the funding mechanisms, and noted that NIDRR will re-compete a competition if the scores are too low or the projects are not strong, and that there is a turnover of 25% among large center grants.  The panel suggested that NIDRR consider reallocating funding within and among its program mechanisms to provide greater funding amounts to fewer projects to achieve a better balance of high-performing projects. Several panel members suggested an automatic sunset of projects, perhaps at 10 years. 

Burn centers

The panel discussed the 100% success rate for the Burn centers (two centers funded out of two proposals received), and noted that based on the requirement to demonstrate a significant infrastructure, there were few institutions that would qualify.  Several panel members commented that some prospective grantees did not submit applications due to the perception that NIDRR is an insiders’ club.  One panel member suggested that funding for NIDRR’s burn research should be continued, but switched from a model centers-type program to other approaches.

Are awards appropriate in size and duration for the scope of the projects?

No, the awards are too small across all of the funding mechanisms.  The need for rehabilitation research exceeds the amount of funding available. The panel strongly agreed that NIDRR asks its grantees to do too much for the money provided, and suggested that NIDRR limit the scope of work or increase funding to fewer projects.  The panel also suggested that NIDRR improve the inconsistent policies and low amounts for indirect costs. 

Focus for limited funding

The panel commented that none of the funding mechanisms provided enough dollars to adequately address the broad scope of work expected: data collection, dissemination, training, evaluation, and follow-up, in addition to significant research.  The panel agreed that NIDRR grantees accomplish a lot with the funding provided. One panel member cited the challenge for the Spinal Cord Injury Model Systems to keep up with the growing numbers of consumers at the same funding levels.  The panel commented that the concept of a “model” system may be a misnomer, and suggested that NIDRR re-examine their purpose and definition, and whether they should sunset as they meet their goals.

The panel noted the high cost of health and function research and suggested that to advance its goals in this area NIDRR should be more directive in its priorities. For example, centers should have incentives to target only one of the domains, rather than attempting to cover Health and Function, Employment and Community Participation. 

Importance of infrastructure

The panel recognized NIDRR’s unique role and success in building a disability and rehabilitation research infrastructure. They suggested that NIDRR conduct a critical analysis of the research produced by its centers, and reassess its allocation of resources, perhaps to focus on building capacity, rather than funding the research itself.  One panel member noted that the strength of the Department of Veterans Affairs research centers program was in its infrastructure support for the researcher, administrative personnel, pilot projects and dissemination.  Some panel members suggested that an outcome measure for capacity building could be the ability to leverage funding from other sources5 .

Databases

The panel suggested that NIDRR improve its database activities by centralizing them into specific projects, rather than including them in the scope of work for other grantees.  NIDRR should seek the best data managers and epidemiologists.  One panel member suggested coordination with the Centers for Disease Control and Prevention for data on prevention of secondary disabilities and on the built environment.

Does the portfolio have an appropriate balance of funding for centers and awards to individuals?

No, the balance between centers and individual funding does not appear to be appropriate; more awards should be made to individuals, such as the Field Initiated Projects.

The panel commented that it did not have enough information on the Field Initiated Projects (FIP) to determine the quality of the proposals.  The level of funding for each FIP appears to be too low, and the total dollars for the various funding mechanisms to individuals should be increased.  The panel noted the NIDRR’s staff comment that some FIPs are supplements for existing grants. 

The panel suggested that NIDRR use the FIPs to address specific areas or problems, and discussed the challenges of screening proposals based on relevance and significance.  If FIPs serve as pilots, NIDRR should report how many are successful in making the transition to larger studies.  

Does the portfolio have an appropriate balance of awards to new investigators?

Yes, the portfolio’s balance of new principal investigators appears to be appropriate.  However, the panel did not have sufficient information across funding mechanisms. For example, the new Annual Performance Report did not collect new investigator information for Field Initiated Projects, likely to be a source of new investigators. 

The panel suggested that NIDRR define the term “new” investigator.  One panel member noted that the individual may have worked on the research project, but not been previously listed as a principal investigator.  A list of new investigators would be helpful.

Does the portfolio have an appropriate balance of multidisciplinary awards?

The panel answered a very strong Yes.  Based on the abstracts, the panel noted that many disciplines are represented among the projects.  They recognized NIDRR for the exemplary role it plays in promoting a multidisciplinary approach, a hallmark of rehabilitation. This should be reflected in NIDRR’s mission statement.

Does the portfolio have an appropriate balance of innovative projects?

The panel was challenged in answering this question.  Panel members had differing definitions of innovation. The panel agreed that there were some innovative projects in the portfolio; however, they could not be sure if the balance is appropriate.

NIDRR’s role

Based on NIDRR’s limited funding, low indirect amounts, and short application timeframes, the panel noted that NIDRR may not have full access to the potential research community.  Some commented that NIDRR’s current procedures contribute to the perception of a closed shop. Given the panel’s previous comments on NIDRR’s role in funding research infrastructure and capacity building, the number of innovative projects may be appropriate.  The panel commented that if innovation is a goal, then it should be articulated as a priority, and evaluated as a criterion in reviewing applications.

Information for future review

Information for future APAER panels should include a definition of innovation (e.g., innovation in understanding disabilities; relational databases; rehabilitation) and supporting data for each funding mechanism.  Panel members noted that the descriptions of some accomplishments were not sufficient to determine if they were addressing new aspects of existing areas, making some of the accomplishments seem non-innovative.

Does the portfolio have an appropriate balance of projects that integrate research and training?    

Yes, based on the requirements in the scope of work. 

Discussion

The centers do a good job of balancing research and training, and one of the benefits of NIDRR’s infrastructure-building is the support of training.  However, there were no data presented on the trainees. Several panel members noted that NIDRR has trained some stars in the rehabilitation research field.  The panel suggested that NIDRR establish a mechanism to follow the careers of their trainees to determine if they continue in research, succeed as principal investigators, or pursue a clinical career path.  Information for future APAER review should include long term data—at least two to four years—on the trainees.

Please comment on the overall quality of the research and training projects in the health and function portfolio.  Discuss any concerns identified about the quality of the projects or the balance of the health and function portfolio.

The panel was challenged in assessing the overall quality of the portfolio due the shortcomings in NIDRR’s management approach, as evidenced by the insufficient information provided.  

Limitations of the accomplishments presented

The panel commented that most of the accomplishments provided by the grantees did not identify the impact of the accomplishment or contribution to the field, or identify the problem in the state-of-the-science addressed by the project.  Also, the panel noted that many grantees cited publications or presentations as accomplishments, but did not describe the research findings presented. Research accomplishments should demonstrate research productivity, and relate the new findings to the gap or problem addressed.

Impact of dissemination

The panel recognized NIDRR’s role in educating providers and consumers, and that information on publications, presentations, and other dissemination efforts is important to collect.  However, it should be reported in a separate category, not as a research accomplishment.  Also, additional information is needed to assess the value of the dissemination activities.  For example, what measures can the grantees report to demonstrate the impact of workbooks or culturally-sensitive consumer materials?  Several panel members questioned why NIDRR project funding was used to publish a book.

Performance

The Office of Management and Budget requires R&D programs to maintain a set of high priority, multi-year R&D objectives with annual performance measures and milestones that show how one or more outcomes will be reached. Metrics should be defined not only to encourage individual program performance, but also to promote broader goals such as innovation, cooperation, education, and dissemination of knowledge, applications, or tools.

In this section, the panel provided a retrospective performance review by examining the performance information associated with the health and function domain and commenting on how well NIDRR is performing in achieving its health and function goals. 

NIDRR Performance Goals

Goals for Health and Function

Goal 1.  Provide timely and accurate scientific and statistical information that advances knowledge and/or enhances understanding of and strategies for improving health and functional status and access to physical and mental health services, including medical rehabilitation and health promotion, for people with disabilities.  Results under this goal should be applicable for use in a variety of disability and rehabilitation related research, policy and practice settings.

Goal 2.  Create a portfolio of new or revised theories, measures and methods that improve the identification and/or assessment of health-related barriers and facilitators, health status, bodily function, health behaviors, and/or access to health related services.  The theories and tools making up this portfolio should be developed and tested or validated in such a way that they are applicable for use in a variety of disability and rehabilitation related research, policy and practice settings.

Goal 3.  Create a portfolio of interventions, programs, products and/or devices that have been demonstrated to be effective in improving the health status, functional abilities, health behaviors, and health services outcomes of people with a wide range of disabling conditions, including those aging with long-term disabilities.  The initiatives making up this portfolio should be designed in such a way that they can be implemented in a variety of clinical, home and community and/or industry-based settings.  These initiatives can include the: (a) development of new interventions, products or devices, (b) expansion and/or replication of interventions or promising practices, and (c) modification of existing programs, products and devices shown to be effective in other populations but not yet tested in the target disability population.

Outcome Goal for Capacity Building 

Increase capacity to conduct and use high quality and relevant disability and rehabilitation research and related activities designed to guide decision-making, change practice, and improve the lives of individuals with disabilities.

Outcome Goal for Knowledge Translation

Promote the effective use of scientific-based knowledge, technologies, and applications to inform disability and rehabilitation policy, improve practice, and enhance the lives of individuals with disabilities.

Accomplishments

Evidence of NIDRR performance in the area of health and function was presented in the grantee accomplishments. NIDRR asked its current health and function grantees to submit up to five outputs or outcomes that the grantee considered to be the most important that occurred or matured during the current reporting period (July 1, 2005 through May 31, 2006).  The accomplishments were collected as part of a new automated Annual Performance Report. NIDRR defined accomplishments as outputs or outcomes:

Outputs are the direct results of center or project activities. In order to qualify as an output, the direct results of project activities must have been disseminated or delivered to external audiences, and this dissemination must have occurred within the current reporting period. Outputs consist of:

Outcomes are the indirect results or effects of project activities on external audiences or systems outside of the boundaries of the project or study.

NIDRR Project Officers reviewed the accomplishments for appropriateness and completeness, and the grantees revised their submissions as needed.  Of the 128 grantees, 89 submitted one or more accomplishments. The 185 accomplishments were organized by NIDRR goals and other topic areas.

Data Limitation

This was the first time that health and function grants have been reviewed by an APAER panel and the first time the Annual Performance Report system was used. The accomplishments presented were for the reporting period July 1, 2005 through May 31, 2006 and did not reflect the accomplishments over the entire 3 years or 5 years of an award. 

Please comment on noteworthy achievements based on NIDRR awards.

The panel recognized the importance of the research that NIDRR supports. It strongly suggested that more than one year of information was needed to conduct adequate review.  Each panel member selected noteworthy contributions from the 185 accomplishments submitted by grantees. These are compiled in Appendix C.  A few accomplishments, listed below, were cited by several panel members. 

Prosthetics

Several panel members commented on the importance of the prosthetic foot for use in low-income countries for victims of land mines, both for the victims and for broader US policies.

Wheelchairs

Several panel members noted the push-rim activated wheelchair, significant in its impact on preserving upper-limb function, and recognized that NIDRR is the only source of funding for this type of project.  They commented that prevention of secondary conditions was important to track.  One panel member noted the modular pediatric wheelchair that grows with the child is important in helping families with the cost of equipment.

Consumer assessment of providers

Several panel members commented on the importance of the project addressing the needs of people with mobility impairments in the national Consumer Assessment of Healthcare Providers and Systems (CAHPS), since the results of this survey are used by public and private insurers.

Influence of pain

Several panel members commented the value of the research into the impact of pain as a secondary functional limitation, since it has generally been under-researched by the medical community.

Burn

Several panel members noted that the Burn Model System projects have produced several important contributions, for example, the first model for addressing burn scarring.

Traumatic Brain Injury

A panel member noted that most of the research published in the widely-read Journal of Head Trauma Rehabilitation comes from NIDRR-funded projects.  Projects at Mount Sinai, Medical College of Virginia, and Moss Rehab have produced significant contributions.

Multiple Sclerosis

A panel member noted that the project at the State University of New York on addressing fatigue to improve function in people with multiple sclerosis was outstanding.

Please comment on the ways in which health and function projects have collectively affected progress toward NIDRR’s goals.

The panel agreed that health and function projects have contributed positively toward NIDRR’s goals.  The panel recognized that NIDRR has an ambitious research agenda, and its portfolio represents diversity across impairments. However, it was not possible to provide a more detailed response due to the emphasis on funding mechanisms rather than program areas. 

Information for future review

Information for future APAER panels should be sorted and synthesized in relation to the health and function goals.  Grantees should improve their reporting to provide better information about their accomplishments.  Additional information is needed on capacity building goals—on trainees, their accomplishments, and career tracks.

Follow-up

The panel noted that NIDRR has contributed many accomplishments to the rehabilitation field, and suggested that NIDRR improve its documentation, follow-up and outreach.  Resources should be allocated to collect data for evaluation, and grantees should be required to provide follow-up information.  One panel member noted that the information contained in the panel’s background briefing material should be more readily available to the research community.

Please comment on expectations for future performance based on the current set of awards.

Panel members made the following suggestions and comments on future performance:

Dissemination

Rigorous evaluation studies should be developed to determine what dissemination approaches are working.

Quality of the science

The quality of the science should be emphasized across all projects, with measures to demonstrate the quality.  Realistic expectations would be that only a small number of advancements, new theories and innovations are possible, perhaps only one or two each year.

Research careers

NIDRR trainees should become researchers with funding from other sources.  Grantees should contribute to the career development of the next generation of scientists and researchers in the field of disability and rehabilitation research.    

Research continuum

A pipeline of research should be developed, reflecting different levels of funding, from pilot studies to small interventions to collaborative studies, and eventually to practice.  The initial observations made by model systems should be parlayed into larger studies to establish the efficacy and effectiveness of their findings.

Secondary conditions

Risk factors for secondary conditions that may be amenable to intervention should continue to be studied, with research progressing into practice guidelines.

Other Topics

Please comment on any health and function areas in need of improvement, or gaps (if any) within the domain.

Panel members made the following suggestions and comments on gaps or areas in need of improvement in the health and function domain:

Evidence-based outcomes

Evidence-based measures should be developed and applied to health and function research findings; for example, to grade rehabilitation literature results for potential use in drafting clinical guidelines.  A panel member suggested that a retrospective review be conducted to determine if projects complied with their proposed methodology and if the published results are methodologically sound.  Alternatives to randomized controlled trials for outcomes evidence should be developed through a collaborative effort among NIDRR grantees. 

Collaboration

NIDRR should collaborate with other agencies. For example, NIDRR could work directly with the Centers for Disease Control and Prevention on data collection, with the National Institute of Diabetes and Digestive and Kidney Diseases on prevention, with the National Cancer Institute on disabilities resulting from cancer and its treatment, and with the National Institute on Aging on the functional impairments resulting from chronic diseases and aging.

Funding allocation

New projects can be funded by discontinuing old projects.

Traumatic Brain Injury topics

Health and function should be broadened beyond cognitive issues for traumatic brain injury to include, for example, basic function.  Traumatic brain injury in the elderly—often resulting from falls—is under-researched and elderly individuals often do not receive aggressive treatment.

Children

More health and function projects should focus on pediatric rehabilitation and prevention of secondary conditions in children with disabilities.

Please identify agency-wide issues that should be addressed by NIDRR to help improve the domain’s performance.

Panel members made the following suggestions and comments on agency-wide issues to help improve the domain’s performance:

Additional Total Funding

An increase in total funding into the agency is needed and would permit NIDRR to maintain a broader portfolio.

Improve outreach to potential grantees

NIDRR should improve its outreach to the research community to attract new grantees by providing significantly more lead time for RFPs and providing more information on the funding mechanisms.  Funding should be focused on specific priorities, and a more reasonable scope of work, with increased grant amounts and increased amounts allocated to indirect costs, even if it means fewer awards.

Resources within NIDRR

NIDRR should improve its mechanisms for collecting, maintaining, and organizing its portfolio information.  NIDRR should allocate staff resources to monitor and evaluate grantee performance by establishing a tracking system of key measures and improving data system management.  The agency should ensure greater accountability and follow-up by the grantees.

Increase grantee interaction

A steering committee of principal investigators should be established, staffed by NIDRR, to meet periodically during the year aimed at increasing interaction and coordinating efforts, and keeping the projects and their principal investigators on task.  Interaction should also be increased by funding more projects that span multiple centers and systems.

Increase visibility

NIDRR should increase its profile in the research community and to the public to get credit for its role and accomplishments in disability and rehabilitation research.  NIDRR’s niche is important and not met by any other funding agency.

Coordination with other agencies

NIDRR should coordinate its research agenda with other agencies, including joint funding of projects.

Plan for goal accomplishment and grant sunset

The strategic plan should include end dates for accomplishing goals and expectations for projects to end in order to make room for new goals and new projects to meet those goals. Turn-over will generate excitement for NIDRR's programs.



1 Contrary to the awareness of the expert panelists, NIDRR does have standardized procedures for proposal submission that are publicly disseminated in the Notice of Final Priorities published in both the Federal Register and in a Department of Education Technical Review Manual prepared for all grant competitions.

2 While decisions to award or decline grant applications allow for informed input from NIDRR program officers, the NIDRR Director has the primary responsibility for making final funding recommendation to the OSERS Assistant Secretary. In all cases, recommendations for what awards to fund are based on a formal OSERS-wide pre-funding process that relies primarily on the scores and comments of peer reviewers.

3 Department of Education policy prevents NIDRR from publishing or otherwise distributing a roster of the names of peer reviewers who served on a particular grant competition or on a specific panel within a competition. However, NIDRR is allowed to publish a list of the names of peer reviewers who reviewed for the agency in a given year.

4 Current Department of Education policy prevents NIDRR from requiring applicants to include a statement of “past performance” in their grant applications.

5 Current Office of Management and Budget (OMB) policy prevents NIDRR from using leveraging of funding as a performance or outcome measure.

Section 3: Appendices
Appendix A: Agenda
Appendix B: Panel Member Biographical Information
Appendix C: Noteworthy Grantee Accomplishments Selected by the Panel
Appendix D: Evaluation Results for 2006 Health and Function APAER
Appendix E: Results of Accomplishments Ratings


Top

Return to NIDRR Announcements
or

NIDRR's Web site
or

National Center for the Dissemination of Disability Research Home page