This document is also available in MS Word 152KB (.DOC) and Adobe Acrobat 187KB (.PDF) file formats.

Knowledge Translation Planning Panel

Summary of the June 9-10, 2005 Panel Meeting

Prepared by
New Editions Consulting, Inc.
6858 Old Dominion Dr., Suite 230
McLean, VA 22101

July 2005

Table of Contents

Purpose of the Planning Panel

NIDRR is planning to advance its knowledge translation (KT) activities by emphasizing expert judgments on the value of information for further dissemination, better accountability for outputs produced by NIDRR researchers, and improved methods for making this information available beyond the research community.

NIDRR convened an expert panel that met for one and one half days, June 9-10, 2005. The meeting brought together an expert panel of 10 knowledge translation specialists in evidence based practice, mental health, employment, cultural competence, education, and consumer perspectives.

NIDRR asked the expert panel to provide guidance on the scope and direction of NIDRR's future activities in knowledge translation. The six questions for the panel were:

  1. What are the appropriate standards to use for research funded by NIDRR on theories, measures, and methods?
  2. What are the appropriate standards to use for new and improved products, environmental adaptations, interventions, and programs developed by NIDRR grantees?
  3. How can NIDRR structure its KT activities to implement appropriate standards and validation methods?
  4. What activities or programs are appropriate to insure research on theories, measures, and methods reaches the primary audiences of researchers and practitioners?
  5. What activities or programs are appropriate to insure research on new and improved products environmental adaptations, and interventions reaches the primary audience of people with disabilities?
  6. What new knowledge management and information technology would help expedite the dissemination mandate in the Rehabilitation Act?

Background Material for Panel

A month before the meeting, panel members received a background paper designed to explain NIDRR goals regarding knowledge translation, what dissemination activities NIDRR is currently conducting, the mechanisms through which those activities are conducted, and what NIDRR sees as future challenges in developing its knowledge translation plans. The paper included the following sections:

NIDRR Presentation

The session opened with introductions and presentations by Joel Mykelbust and Dick Melia from NIDRR. The presentation included the following topics: overview of the panel process; NIDRR mission; the NIDRR Long Range Plan; performance reporting; knowledge translation; and measuring the quality of NIDRR research. NIDRR Director Steven Tingus welcomed the panel members after lunch.

Panel Discussion

This section summarizes comments and suggestions expressed by panel members during discussion. Dr. Stephen Leff served as panel chair. Since the purpose of the meeting was to share the panel member experiences and expertise with NIDRR, these comments should not be interpreted as panel consensus. Comments have been organized into topic areas. The key points corresponding to each of the questions before the panel are summarized below.

Continuum of Knowledge Translation

Knowledge translation should be conceptualized as a paradigm from start to finish, as a continuum. KT is a conceptual model of how research is created, used, and what impact it has.

What is the conceptual model from start to finish? Most KT starts after a product is developed, but it should start from conceptualization of research. How to get what we produce to be used. The process must include the end-user. Each project must specify what end-user group to reach. Then align with those groups, build trust, and learn their context. It is an evolving process. Each project is different. Evidence needs to be structured to meet needs of consumers. Part of sorting this out is determining the underlying use goal.

Must work with target audience from beginning. Think about products and strategies from beginning. Determine when it is appropriate to disseminate results of a single research study as opposed to placing new research findings in the context of existing research.

Develop a method. What is state of research before this study, what is the state of policy, how does this study affect what we know in the field? Getting researchers to think in this way up front and how the results might or might not affect decision-making is key.

Role of Stakeholders in Knowledge Translation Planning

To use a sports analogy: Do we have the right team in place? What is the form of the question the end-user needs answered? Frame the question and collect data in way that is easily usable for end user.

Think about who the stakeholders are at the research conception stage; don't bring them in after research is designed. The Agency for Healthcare Research and Quality (AHRQ) involves researchers in thinking about dissemination strategies. When is it appropriate for a researcher to be involved in dissemination activity? Researchers are probably not the best disseminators. Using researchers alone to do KT may not be cost effective either. But, depending on the project, researchers may be important members of a KT team. There is sometimes an inherent conflict between researchers and end users. For example, research may show that certain programs can get by with less staffing - not a good outcome for the end user who will face provider staffing shortages.

Consider putting together teams of researchers, communications specialists, and consumers to frame the research process. Researchers can discuss internal validity; communications specialists and consumers can discuss social validity. Consumers can contribute to the study design in this way.

Identify stakeholders groups and let them do some of the work for you. This will help insure that research is getting out to appropriate groups with appropriate methods.

Identify which stakeholders would be interested in the information and determine how to get the information to them. Some providers actually prefer to read consumer material. Start with what you have and see how that's working.

Grantees have advisory committees that meet once a year. Include them in the process as partners.

Utility

In order to useful, information needs to be consumer driven. Insure utility by involving consumers. Demonstrate utility using social validation methods. If a researcher finds an effective intervention that is too expensive to implement, is that good? How big of a change is needed to say that something works? Multiple stakeholder groups should assess utility.

To be useful, information also must be pre-filtered. The naïve user is often overwhelmed. Use social marketing and determine how people obtain and process information. It is critical to user-test any guidelines that are developed. Involve consumers early and often in the process of developing evidence and translation.

Utility also relies on building capacity so users can be good consumers of research. Not all research can be used by all people.

Grantees should be required to specify user groups in advance, so when information is disseminated, there is a clear idea of who the information applies to. The grantee has a responsibility to specify where the research is going. The grantee has the vision to see how this knowledge is usable.

Considerations for Basic Research

For basic research, look at the nature of scientific development in private industry that is concerned with time to market. Industry research is concerned with timeframe. Products developed by basic scientists may sit on a shelf because they are not usable in a product form. An important consideration is: who is making the investment to take scientific results and turn it into a product? What is the timeframe? Want to decrease time to market?

The National Registry of Evidence-based Programs and Practices (NREPP) only identifies programs that work, but most panel members felt that if a study shows no benefit, that finding absolutely should be disseminated.

Pipeline

NIDRR should view KT as a pipeline that needs a constant supply of research. NIDRR needs to develop a pipeline and identify where innovation is needed. Identify where in the process change will occur. Identify areas where more information is needed before deciding how to translate.

First, conceptualize the process. Second, have NIDRR decide how it can affect the pipeline. Have a joint review to sift through evidence and decide where more knowledge development is needed. Third, there are different systems of service NIDRR is trying to affect. Identify key systems and then make the investment to investigate how to affect those systems. For example, with the vocational rehabilitation system for employment, identify people inside and outside the VR system.

To energize this pipeline, there needs to be a culture change in academia. The system of rewards and prestige places a high value on breakthrough, not entrepreneurship. Agencies that fund research must recognize research translation as valuable and reward it. Partnering with other funders would have an impact on recognizing and rewarding KT.

Evidence Grading

There are a number of approaches to evaluating research proposals and evidence developed by different organizations. When these have been applied to programs of interest to NIDRR, these evaluations should be used. However, for interventions that have not been reviewed by other organizations, NIDRR should consider developing its own approaches. NIDRR should consider approaches that can be used with single studies and recognize the importance of synthesizing multiple studies. Do not favor randomized clinical trials to the exclusion of other types of research.

Look at the KT community, think about how to synthesize research and think about using models not as strict as Cochrane. Results that do not meet Cochrane standards may still have an impact. There is a more realistic approach to synthesis evolving. (See recent studies sponsored by the Canadian Health Services Research Foundation and the National Health Service in the UK, published in the July 2005 issue of the Journal for Health Services Research and Policy, available free on-line at www.rsmpress.co.uk/jhsrp.htm.)

The NREPP Model

The Substance Abuse and Mental Health Services Administration (SAMHSA) has developed the National Registry of Evidence-based Programs and Practices (NREPP) NREPP has 20 criteria to grade evidence to make process transparent, as compared to expert evaluation. NREPP has transparency because it has explicit criteria. Programs get scores so they know what to do to get a better score. Neither the peer review process nor grant reviews are transparent.

NREPP has five categories for programs that extend from generally effective to emerging programs and programs of interest. This could be used to identify interventions that would benefit from further investment to move up the ladder.

Evidence grading is a different process. SAMHSA has devoted staff to meet this challenge. Effect size is one NREPP criteria. Once you get beyond drug research, RCTs are not that common. Getting a control group is a problem in mental health.

The What Works Clearinghouse (WWC) approach looks at genre of treatment, whatever literature is out there, and then conducts meta-analyses. NREPP uses an FDA model: reviewing specific programs and approving individual interventions. Evidentiary criteria puts the burden on developer to do replications or team with others. Like the FDA approval process, programs come into NREPP with studies in hand. Researchers are asked to submit evidence as to the hypothesis at the beginning of the study.

Studies that are not RCTs are not treated seriously by WWC. But with the NREPP model, a researcher can get high marks for observational research well done. Substance abuse prevention programs were products that came out of a business model rather than academic model.

NIDRR should consider the NREPP model. It fits with rehabilitation research. Emphasis on RCTs always undervalues the contribution of rehabilitation work.

Training

There must be an emphasis on pre-service training for clinicians and providers. In-service is important, but it is critical to train the next generation of providers who will carry the rehabilitation field forward. Take time to train people on new interventions; it is unrealistic to expect people to do it on their own.

Training is critical for workforce development. The Center for Mental Health Services developed implementation resource kits, or toolkits. It determined evidence based practice in mental health, then developed toolkits that contained manuals; fidelity measures; general organization index; videos for consumers and family members; videos for providers; and outcome measures. The material spent a year in clearance. However, the toolkits assumed a level of preparation in clinical competency that was not justified.

Reaching Practitioners

Practitioners need information to inform clinical encounters. For practitioners to use it, evidence provided must be pre-filtered and linked to the clinical question at hand. Must be just in time, which can be a challenge.

Practitioners sometimes reject new practices because they are accustomed to existing practices or because a new practice will have an economic impact. For example, if they must pay for new technology and learn how to use it, they may be slow to adopt new methods.

Paralyzed Veterans of America uses published evidence rather than research results. The medium of information is important. Bring it to the practitioners using their literature. Continuing education usually involves workshops, but they are not good at getting adoption of new practices. The field is falling down on opportunities for training.

Identify change agents and realize the importance of champions. For example, therapists tend to believe other therapists rather than researchers. Sometimes it's an unwillingness on the part of practitioners to look at new information. Practitioners must see it as their job to keep informed. Determine whether practitioners really use research.

Determine the demand for research. Can't always expect target audiences to go to web based sources. Think about appropriate means to present research, databases are not always needed. Important to get audience feedback. Use experts who understand how people perceive information and use it.

NIDRR should reach out to existing agencies. Delivery systems already exist; leverage them.

Reaching Consumers

Two approaches are needed: educating consumers and giving them confidence to evaluate information. Perhaps we are putting too much of a burden on consumer to know as much as researchers and practitioners do. Drug companies provide information to consumers in a useful way; some of these components are good to model, such as safety and effectiveness of product or using a trusted spokesperson.

NIDRR projects need a consumer constituency, and projects need to invest in building capacity in consumers. In KT, it is especially important to get consumers interested in products coming out of research, so they look for it. Paralyzed Veterans of America guides are written for consumers so they become better advocates with their service providers.

Decide when it is appropriate to direct information to consumers and when to brokers. Identify key brokers. Collaboration with other organizations is necessary.

Specify what you want to accomplish: decision support vs. knowledge support. It is easier to evaluate a KT initiative that focuses on informing a specific set of decisions than an initiative to broaden public and/or policymaker understanding of an issue. How the question is framed is important.

Knowledge Management and Information Technology

Suggestions included: listserv; clinical reminders; PDA tools; electronic health records, web based tools. High tech is not available to all segments of population, so consider other methods. Ask users how they would like information presented. Determine how to provide information online to stakeholders. Ask: what should the database include?

Web Presence

NIDRR must have a united web presence. There has been a proliferation of Web sites, and it may be possible to unify NARIC and NCDDR at this point. NIDRR is a vital enterprise and needs to get its information out, consolidate its web presence, and take leadership in the disability field. A cohesive web presence also prevents dissemination of inconsistent findings and information.

Building Capacity for Knowledge Translation

It is important to do basic KT research. Add to portfolio a mechanism for KT activity and set requirements. Perhaps use external review before proceeding. KT activity grants need to be investigator driven. However, there must be stewardship on the part of an agency on what is needed. Don't know if we are at a point yet to fund a training center. More work on synthesis may be needed first to see what information is ready for translation.

There is not a large pool of persons well versed in KT. An RFP will attract responses from people who don't really know KT. Robert Wood Johnson conducts orientation sessions which are a requirement for all potential authors of research syntheses.

NIDRR needs to look at the current field and build capacity in pre-service and in-service. Moving from dissemination to a KT model is appropriate, but there will be a learning curve by the agency and the field. Conceptualize in the long range plan where NIDRR expects to be in five years.

Academic researchers promote their own careers. Can't wait for them to adopt KT. Is there capacity with NIDRR? Field must be ready to conduct KT research, otherwise must provide training.

To get reviewers and expertise in KT, Canada launched 3 RFAs in KT. Each had an option to do KT research. Then it set up a KT committee for open competition. Now this committee has 15-30 grants in open competition. Canada is now recruiting KT fellows and new investigators under 5-year funding awards from open competition.

Consider a team approach. One member should have demonstrated KT knowledge and expertise. A program that matched researcher and developer in a team has worked to promote innovation in schools. The researcher knows the literature and the developer knows needs. This approach can build capacity.

Diversity

Those developing KT initiatives must be sensitive to issues of cultural and racial diversity. Cultural factors are more important than just race and ethnicity. Disability plus cultural factors have tremendous impact. The simplest approach is to convene focus groups and use sample messages. Recognize certain research findings may require different translation strategies among different groups. Look through the cultural lens. NIDRR also has a legislative mandate to include underserved populations. As NIDRR pursues its KT plans, there is a need for more specific research related to meeting the needs of underserved populations.

New NIDRR RFP

Dick Melia described the new NIDRR RFP for renewing the National Center for the Dissemination of Disability Research, which will serve as a lead center in the area of Knowledge Translation/Knowledge Dissemination and Utilization. This center will ensure that NIDRR constituencies have ready access to high-quality, research-based information that has the potential to improve the lives of individuals with disabilities. The center must:

Presentation of the new RFP prompted the following discussion.

Dissemination Requirements for Field Initiated Research

There can be negative consequences from dropping dissemination requirements from field initiated projects. It could take five years to get something published that would otherwise get published in one year. The research is being conducted by academics doing other things. Industry has full time scientists. Define your timeframe and lean on researchers to make findings available on your timeframe, not theirs.

One strategy to rein in the lag time in publishing results is to require grantees to submit an article within 6 months of the completion of the research. Conduct a two-way exchange during the course of the research with funded researchers and stakeholders. Stakeholders can hear the results before public release.

But including consumers does not always solve the problem. In one mental health study, consumers strictly adopted an academic paradigm and would not share results until the peer reviewed paper was released.

Use of Public Media

Use of public media is a sensitive area now with controversy over federal government placing stories in the news media. Although it is not possible to control what the media does with your information, stakeholders must cultivate relationships nonetheless. The press tends to cover single studies, not synthesis. But reporting single studies confuses the public. Should be getting syntheses into the press rather than single studies. Disability Rehabilitation Research Projects (DRRP) and field initiated research projects should encourage synthesis.

Knowledge Translation Strategies

There is tremendous variation in how much effort should be placed on KT, depending on nature of the project and the target audience. Think about decision rules for individual projects. For example, intervention studies needs KT; a measurement tool may need both laboratory and real life performance studies.

Place KT in relationship to what the agency expects from the research. KT does not supplant PAR. KT is different piece of puzzle. Publishing is only one aspect of KT. If results will be submitted for evidence based review, need to bring in KT at the beginning. Be clear on difference between PAR and KT.

Research involves a continuum from knowledge generation to knowledge transfer. Not all knowledge generation is ready for transfer. There is a difference between KT research and KT implementation. KT should be part of the research portfolio. KT implementation includes marketing and publications. The current NIDRR RUSH project is determining what barriers exist to KT. A good place to start is to find out what those barriers are.

Canada has put out RFAs that provide up to $100,000 for KT over the life of a three year operating grant. Other RFAs ask for a KT plan up front. It is acceptable for a grantee to say that the KT plan is to publish. The plan depends on the nature of grant. Look for commitment to a Web site and consumer involvement. Governance grants provide for governance models that include policy makers at the beginning of research. Results are expedited as soon as they are ready. Canada is about to launch a KT implementation grant: a knowledge to action grant that provides $75,000 in the first year. Researcher must find an additional $25,000 cash or in kind. Needs to be linked to a particular research project. This type of grant provides follow up money and can be used to fund synthesis.

Academic researchers are accustomed to field initiated research. NIDRR needs to envision its pipeline, what kind of teams it wants, and where to place its funding. NIDRR must decide what the team should look like and direct investments accordingly. Must have the big picture in the long range plan.

Not every investment in research has a good return: was this good quality research? It is worth the effort of disseminating? NIDRR's responsibility is to see that funded research is good and that the grantee does a good job with the funding.

Avoid oversaturation of information. Cultivate balance. Researchers need access to everything. For consumers, there is a need to create a demand. How does it relate to disability? We know that getting information is like drinking from fire hydrant; people are overwhelmed by new information.

Be bold and try new things. Get feedback. For example, having researchers with policy experts go together to talk to audiences. Use Web conferencing with knowledge brokers in the field. Use peer review teams of researchers and policy experts for the KT process.

Rehabilitation has a scattered literature, not just published in journals. Consider funding an RFP to establish a gateway from which people could access the information.

AHRQ has developed a Partner's Guide (January 2005) which provides detailed information on the Evidence-based Practice Centers (EPC) program for current and potential partner organizations. It presents background on the program and the roles and responsibilities of its key participants, including AHRQ, the partners, the EPCs and the EPC Coordinating Center. Also covered are the topic nomination process and specification of evidence questions, topic selection criteria, strategies and expectations for report dissemination and resources on evidence-based health care.

As a starting point for strategies, find out how users feel about what NIDRR is doing. Convene focus groups of representatives from organizations to comment on what is already exists.

Stewardship involves needs assessment. The way to approach this is to ask: what do we have, what would we like to have? Then, go to stakeholders, and ask: what would you like irrespective of what we have? Stewardship also involves investing substantial funding in newer things at point of knowledge generation.

A small agency like NIDRR must focus on a few areas and try to make an impact, or it will end up with a scattershot approach. NIDRR faces a complex task with its KT activities because its constituency is so broad and assessing gaps across different populations is difficult. It will need a balanced portfolio - researcher-driven and user-driven research. Field initiated research might not have an obvious utility, but it is needed to get unanticipated innovation.

NIDRR needs fewer studies, but well funded studies. Current funding levels per project are too low. Discourage NIDRR from taking small grant awards and tacking on KT requirements. Consider the possibility of funding an RRTC on KT to investigate capacity of people to access knowledge and use it. NIDRR must be strategic. There must be dollars for KT research. Canada is putting major dollars into KT.

Summary of Key Points for Each Question

The panel summarized its discussion by approving the following key points for each question. Points are combined for Questions 1 and 2.

  1. What are the appropriate standards to use for research funded by NIDRR on theories, measures, and methods?
  2. What are the appropriate standards to use for new and improved products, environmental adaptations, interventions, and programs developed by NIDRR grantees?
    • How does NIDRR intend to use these standards: to allocate resources (e.g., in awarding grants)? or to evaluate research for deciding what interventions to feature in knowledge transfer activities? Different uses dictate different standards.
    • There are different types of research (e.g., single subject designs) designed to answer different types of questions (e.g., how to treat rare disorders in heterogeneous populations). Different standards should be used for the different types of research.
    • Users should be involved throughout the research process. Researchers should use their input to frame questions and answers. NIDRR should consider activities like polling users and holding town meetings to obtain user input.
    • If NIDRR wants to support research that will be translated for users, it should support researchers or research teams that have translational capacity and track records. It should also recognize and reward translational research.
    • NIDRR should balance basic and intervention research. It should view KT activities as a pipeline that needs a constant supply of research.
    • Research should be structured to provide information about specific subgroups. Research should be conducted with diverse populations.
    • There are a number of approaches to evaluating research proposals and evidence developed by different organizations. When these have been applied to programs of interest to NIDRR, these evaluations should be used. However, for interventions that have not been reviewed by other organizations, NIDRR should consider developing its own approaches. NIDRR should consider approaches that can be used with single studies, recognize the importance of synthesizing multiple studies and do not favor randomized clinical trials to the total exclusion of other types of research.
  3. How can NIDRR structure its KT activities to implement appropriate standards and validation methods?
    • NIDRR should consolidate its various databases. It should develop a united web presence. This would be consistent with knowledge management strategies that stress information technologies that span organizational boundaries.
    • Activities should be consumer driven. Products and materials should be designed for and with the input of particular stakeholders.
    • NIDRR's knowledge translation activities should involve some filtering and focus on validated high quality activities, studies or evidence. But the agency should also be prepared to point out when weak findings and poor research are being inappropriately used to guide decisions.
    • NIDRR should support and evaluate different models for translation.
    • KT activities can be expensive. NIDRR should be efficient.
  4. What activities or programs are appropriate to insure research on theories, measures, and methods reaches the primary audiences of researchers and practitioners?
    • Partner with different research agencies and practitioner organizations.
    • Provide support for research on different translational methods.
    • Involve researchers and practitioners in formulating questions and answers.
    • Develop workforce training in evidence-based practices.
    • Identify and support research advocates and "champions".
    • Anticipate resistance to new practices from providers with allegiances to pre-existing ones. Recognize economic bases for resistance.
    • NIDRR should consider disseminating null findings.
  5. What activities or programs are appropriate to insure research on new and improved products, environmental adaptations, and interventions reaches the primary audience of people with disabilities?
    • Involve end-users early and often in all phases of research.
    • Tailor information to user preferences and create demand for research (social marketing).
    • Educate consumers to use research.
  6. What new knowledge management and information technology would help expedite the dissemination mandate in the Rehabilitation Act?
    • Multi-media materials
    • Web-based tools - e.g. like existing databases unified; bulletin boards, e-forums
    • Mediated navigation
    • Listserv
    • Subscription systems
    • Electronic newsletters
    • Electronic medical record
    • Material downloadable to personal data assistants (PDAs)
    • Learning collaborations

  NIDRR Knowledge Translation Meeting Agenda


Thursday, June 9, 2005

8:30-9:00

Continental Breakfast

9:00-9:20

Welcome, Introductions, and Overview of Panel Process: Joel Myklebust, NIDRR Acting Deputy Director

9:20-10:15

Presentation by NIDRR: NIDRR mission, Long-Range Plan, Performance Reporting, KT, Measuring Quality of Research.
Q&A from panel

10:15

Break

10:30

Standards and Validation Methods
Panel Chair, Steve Leff, will lead discussion to generate answers to questions

  • What are the appropriate standards to use for research funded by NIDRR on theories, measures, and methods?
  • What are the appropriate standards to use for new and improved products, environmental adaptations, interventions, and programs developed by NIDRR grantees?

12:00-1:00

Lunch

1:00-1:10

Welcome from Steven Tingus, NIDRR Director

1:10-2:00

Continue discussion of standards and validations methods, if needed.

2:00-4:30

Program Structure and Design Questions
Panel Chair, Steve Leff, will lead discussion to generate answers to questions

  • How can NIDRR structure its KT activities to implement appropriate standards and validation methods?
  • What activities or programs are appropriate to ensure research on theories, measures, and methods reaches the primary audiences of researchers and practitioners?
  • What activities or programs are appropriate to ensure research on new and improved products and environmental adaptations reaches the primary audience of people with disabilities?
  • What new knowledge management and information technology would help expedite the dissemination mandate in the Rehabilitation Act?

4:30-5:00

Review of today's session and plan for tomorrow.

Friday, June 10, 2005

8:30-9:00

Continental Breakfast

9:00-11:00

Working Session: develop options and rationale.

11:00-12:00

Presentation and discussion of draft options with NIDRR staff.

Top

Return to NIDRR Announcements
or

NIDRR's Web site
or

National Center for the Dissemination of Disability Research Home page