Skip Navigationlogo

Advancing Research, Improving Education

 

National Center for the Dissemination of Disability Research

Knowledge Translation: Introduction to Models, Strategies, and Measures

Measures of Knowledge Use

Measures of Knowledge Use

Table of Contents
Knowledge Translation: Introduction to Models, Strategies, and Measures

Background Information

Backer (1993) defined knowledge utilization as "[including] a variety of interventions aimed at increasing the use of knowledge to solve human problems" (p. 217). Backer envisioned that knowledge utilization embraces a number of subtopics such as technology transfer, information dissemination and utilization, research utilization, innovation diffusion, sociology of knowledge, organizational change, policy research, and interpersonal and mass communication.

The use (application) of research knowledge is certainly the goal of knowledge translation. The expected outcome of such application/use would be a positive impact on the health and well-being of the intended beneficiaries (CIHR, 2004; NIDRR, 2005). The inquiry related to the use (or lack thereof) of research knowledge has been the subject of interest of the multidisciplinary communities since the late 1960s (Paisley & Lunin, 1993). This interest has intensified in recent years, in parallel to increased recognition of the difficulties in moving research to be used in practice.

Types of Knowledge Use

There are three main types of use: (1) instrumental use, (2) conceptual use, and (3) symbolic use. As described by Beyer (1997), instrumental use involves applying research results in specific and direct ways; conceptual use involves using research results for general enlightenment; and symbolic use involves using research results to legitimatize and sustain predetermined positions. Instrumental use has been linked to the decision-making process, in which a direct, demonstrable instrumental use of research is meant to solve clearly predefined problems.

Estabrooks (1999) referred to instrumental use as a concrete application of research, in which research is translated into material in usable forms, such as a protocol that is used to make specific decisions. Conceptual use, according to Estabrooks, occurs when the research may change one's thinking but not necessarily one's particular action. In this type of use, the research helps inform and enlighten the decision maker. Symbolic use, in Estabrooks' view, is the use of research as a political tool to legitimatize opposition or practice.

The conceptual structure of research utilization was investigated empirically in a study with 600 registered nurses in western Canada (Estabrooks, 1999). Using Structural Equation Modeling for data analysis, Estabrooks demonstrated that the instrumental, conceptual, and symbolic research utilization influenced the overall research utilization, with more than 70% of the variance explained. The researcher also asserted that research utilization can be measured with relatively simple questions, as demonstrated by the questions used in this study, which measured each type of research utilization with only one question.

Emerging evidence shows that each type of use should be considered separately, as they may be associated with different predicting factors. In a survey of 833 government officials, Amara, Ouimet, and Landry (2004) found that, in general, the three types of research use simultaneously played a significant role in government agencies and were commonly associated with the same set of factors. However, a small number of factors explained the increase of instrumental, conceptual, and symbolic utilization of research in different ways. Higher conceptual use was significantly associated with the qualitative research products and individuals with graduate studies. Higher instrumental use was significantly associated with research products that focused on advancement of scholarly knowledge and adaptation of research for the user's need. Finally, higher symbolic use was associated with the respondents from provincial government agencies (as opposed to those from the federal government agencies) and also the adaptation of research products to users' needs.

Milner, Estabrooks, and Humphrey (2005), in their survey of nurses in Alberta, Canada, also found that different factors predicted different types of research use. Factors such as having a degree in nursing, attitude, awareness, and involvement in research significantly predicted the instrumental research utilization. Having a degree in nursing also predicted the conceptual research utilization, whereas attitude and awareness did not. Localite communication, on the other hand, significantly predicted the conceptual research utilization but not instrumental research utilization. For symbolic research utilization, attitude, awareness, and involvement in research significantly predicted research use. Mass media were also found to significantly predict symbolic research use, though not other types of use. Certainly, more studies in this area are needed to further understand the subject.

These preliminary findings seem to indicate that it may be beneficial to take a precise, analytical approach when investigating knowledge use. To successfully facilitate the use of knowledge (as it is the focus of KT), it may be essential to predetermine specifically the type of use in which the knowledge user is likely to engage (or a specific use that would be the focus of that knowledge translation effort), consider specific factors that likely influence such use, derive appropriate knowledge translation strategies accordingly, and measure that specific type of use when evaluating the outcome.

Considerations in Measuring Knowledge Use

Early research assumed that utilization occurred when an entire set of recommendations was implemented in the form suggested by the researcher (Larsen, 1980). However, Larsen argued that knowledge can be modified, partially used, used in an alternative way, or justifiably not used at all. Complete adoption of knowledge is generally the exception and not the rule. As Larsen further stated, in some settings, knowledge may not be used if implemented in its original form but may work very well if changed to meet the user's circumstances. A recent example is an implementation study of Constraint-Induced Therapy (Sterr, Szameitat, Shen, & Freivogel, 2006), in which the treatment protocol (as previously developed in research) was locally tested and adapted to fit the needs, preferences, and logistics of the users in a local setting.

In addition, what constitutes an effective use is debatable. Often, the focus is on the use of research as it is intended. However, effective use can occur in ways that are not previously considered (SEDL, 2001). The time line in measuring use—whether it is retrospective or prospective—can also influence the results (Conner, 1981).

Framework in Evaluating Knowledge Use

Knowledge use is not a single discrete event occurring at one point in time. Rather, it is a process consisting of several events (Rich, 1991). Therefore, evaluating the use of knowledge can be complex and requires a multidimensional and systematic approach. Having a framework to guide the process when designing activities to evaluate the use of knowledge can be helpful.

An example of a comprehensive framework that can be used to guide the evaluation of knowledge use is that developed by Conner (1980). Conner proposed a conceptual model for research utilization evaluation, with the emphasis on four general aspects that are important for the evaluator to consider: (1) goals, (2) inputs, (3) processes, and (4) outcomes.

The model is represented by the visual scheme in Figure 6.
D

Figure 6. Conner's conceptual model for research-utilization evaluation

Large image of Figure 6

Figure 6. Conner's conceptual model for research-utilization evaluation

(Source: Handbook of Criminal Justice Evaluation [M. W. Klein & K. S. Teilmann, Eds.], Conner, R. F., The evaluation of research utilization, pp. 629–653, copyright © 1980, reprinted with permission from Sage Publications)

According to Conner (1980), the first step of evaluating research utilization is to set up goals one would like to achieve as a result of the research-utilization effort. Without goals, it will be impossible to evaluate the success of such effort. The goals for utilization should be set up at the beginning of the research program. However, goal setting is perceived as a dynamic process in which goal changes can occur during the course of the research as better insight emerges. One of the main reasons for setting goals, according to Conner, is to know who will be the primary and secondary targets of the utilization effort. Once the target groups have been identified, they should be consulted at the outset of the research program to make sure that the information will be realistic for use by them and to learn of the type of information that would provide convincing evidence for them to use.

In this model, the primary inputs for a utilization effort are research findings. The research findings need to be evaluated in two aspects: quality and importance. Quality of the research findings relates to their sufficiently high validity and reliability, which could be verified by either consultation with other researchers or replication of the research in a similar setting. Determination of importance can also be obtained in several ways, such as the potential users' rating on the degree of importance; assessing socially significant implications; and assessing the clarity of recommendation for action—that is, whether it can be understood by the users. According to Conner (1980), other inputs should also be considered in addition to the research findings. The materials to be disseminated should be assessed for their appeal, clarity, and appropriateness for the target users. The people who will direct the utilization effort should be assessed for whether their skills and temperament would be suitable to conduct such an effort. As the model was later updated (Kirkhart & Conner, 1983), a consideration for resources—whether there are sufficient resources pertaining to the quality of materials or personnel involved in the planned change process—was added to the input component.

For the process component, Conner (1980) suggested that there is a need to monitor and document the course of dissemination and utilization efforts, particularly because the course could change from what was originally planned. Documenting the deviations from the plan will help in adjusting the evaluation to reflect the program that was actually implemented rather than one that was planned to be implemented. The monitoring includes documenting who received the information to be utilized; their opinions of the information and their judgment of subsequent use of such information (pattern); their reasons for use or nonuse of the information (rationale); and the organizational arrangement and personal state and situation of the potential users (state of utilizers). The attitudes and capability of individuals adopting the innovation were later added to the process component of the model (Kirkhart & Conner, 1983).

The central question of this evaluation process is whether the results of the research are utilized. Conner (1980) indicated that evaluation of outcomes must occur for the type, level, and timing of the utilization process. This information can be determined in part by analyzing who has used the information (level of utilization), how the information was used (type of utilization), and the various time frames of utilization activities (timing of utilization). Outcomes assessment could focus on one of the two target groups: the actual targets (people who have been the direct target of utilization efforts) and the potential targets (people who are most relevant for utilization of the findings although they may not be the direct targets of the utilization efforts). The focus on potential targets is intended to address the issue of self-selection bias that could perceivably occur should the assessment focus on the actual targets, who tend to be more involved.

Conner (1980) concluded that evaluating research utilization has five benefits:

  1. Determines the effectiveness of the utilization efforts using a critical review process
  2. Increases recognition of the importance of dissemination-utilization process as a distinct activity rather than just being the last step in a research project
  3. Increases attention to utilization goals and objectives, which will help in the development of utilization efforts
  4. Provides more understanding of the process, including what may facilitate and/or hinder research utilization, which will aid in developing conceptual models of the utilization process that can be empirically tested
  5. Increases the amount of interaction between researchers and users through a neutral third party (presumably the people who conduct the utilization efforts)

Methodologies and Focus of Measuring Knowledge Use

Dunn (1983 ) conducted an 18-month project involving a review of existing literature to develop an inventory of concepts, procedures, and measures available for conducting empirical research on knowledge use. Dunn found that there were widely varied linguistic usages that made it difficult or impossible to compare, contrast, and evaluate essential variations in the concepts, methods, and measures in this area. However, three basic dimensions were found to underlie different concepts of knowledge use: (1) composition, a dimension that distinguishes between individual (for decision making) and collective (for enlightenment) uses of knowledge; (2) expected effects, a dimension that contrasts conceptual and instrumental use of knowledge; and (3) scope, a dimension that contrasts processes of use in terms of their generality (general use of knowledge) and specificity (e.g., use of specific recommendations of a program evaluation). As indicated by Dunn, the scope of use may also be viewed as a continuum ranging from general processes of being familiar with or aware of something to specific processes of being able to explain or perform some action.

More than 60 procedures used to study knowledge utilization were identified through this review and were categorized into three main methods of inquiry: (1) naturalistic observation; (2) content analysis; and (3) questionnaires and interviews. Within the questionnaires and interviews, there were three categories of procedures: (1) relatively structured procedures; (2) semistructured procedures; and (3) relatively unstructured procedures. As reported by Dunn (1983), at least 30 different questionnaires and interviews or schedules were employed to study various aspects of knowledge use, and these were the most frequently used method of the studies included in this review. Naturalistic observation was rarely employed. Content analyses were employed with various kinds of documents, including research reports, case materials, and other records of experience.

Recently, Hakkennes and Green (2006) conducted a structured review of 228 original studies published in peer-reviewed journals to identify the outcome measures used to determine effectiveness of strategies aimed at improving development, dissemination, and implementation of clinical practice guidelines. Three types of data were collected from the included studies: (1) the measures used to assess the effectiveness of the intervention; (2) the methods used for such assessment; and (3) the reliability and validity of those outcome measures when reported.

Hakkennes and Green (2006) grouped the outcome measures into three main categories of measurement, at (1) patient level, (2) health practitioner level, and (3) organizational or process level. The outcome measures that focused on patient level were further categorized into those that measure the actual change in health status of the patient, such as mortality, quality of life, and actual symptom change, and those that use surrogate measures of the patient's change in health status, such as patient satisfaction, length of hospital stay, or number of health-care visits and hospitalizations. The outcome measures at the health practitioner level also followed the same pattern of either measuring actual change in health practice of the practitioner, such as their compliance (or noncompliance) with the implemented guidelines, or surrogate measures, such as measurement of the practitioner's knowledge or attitudes. For the organizational or process level, the focus was on measuring change in the health system, such as to cost, policy and procedures, and the time spent by the practitioner. As indicated by the authors, only a small number (20%) of the studies reported the reliability or validity of the outcome measures.

For studies included in the review (Hakkennes & Green, 2006), several methods were used to collect data, including medical record audit; computerized medical records audit; health practitioners survey, questionnaire, or interview; patient survey, questionnaire, or interview; computerized database; log books, department records, or registers; encounter charts, request slips, or diaries; and others (such as clinical examination, results of blood tests, and video- and audiotaping of consultations). The authors reported that approximately half of the included studies used a medical record audit to collect data for the outcomes. Overall, the measures of change in health practitioner behavior were the most common, followed by the measures that assess change at a patient level (either actual measures of change or surrogate measures of change), health practitioners' knowledge and attitudes, and changes at an organizational level.

Examples of Approaches in Measuring Knowledge Use

Several approaches employed to measure knowledge use were reported in the literature. One approach is to measure users' discrete behaviors related to the use of research-based knowledge. Pelz and Horseley (1981) used survey questionnaire items (that were part of an 18-page questionnaire) to measure research utilization by nursing staff. Five items on the questionnaire directly measured research utilization activities, and the mean score of those five items was used as an index of research utilization. A response scale of 0, 1, 2–4, and 5 or more times in the past year was used for each activity. The Cronbach's alpha of this research utilization index (representing the internal consistency) was reported to be .87. The five research utilization activities are as follows:

  1. Reviewing research literature to identify knowledge for use in practice
  2. Evaluating research study to determine its value for practice
  3. Transferring the research knowledge into useful practice activities
  4. Planning for the implementation and evaluation of such practices
  5. Discontinuing or rejecting practice activities because of the research knowledge learned

Vercoe and Hilton (1995), in a study investigating factors affecting acute-care nurses' use of research findings, measured both general use of research and use of specific research findings. For general use, nurses were asked to rate 10 statements using the scale of 1 = not at all, 2 = sometimes, 3 = frequently, and 4 = always. However, the actual statements were not reported. For use of specific research findings, nurses were asked to rate their use of 10 findings on a scale of 1 = never, 2 = sometimes, 3 = always, or, if the finding was inappropriate for their area of practice, not applicable. As reported by the authors, internal consistency of the items was .87. Content validity was tested through peer review.

A different approach is to view knowledge use as a chronological process. In this approach, each step within the process could be measured to gauge the progress on the knowledge use continuum, as opposed to just measuring whether the use of knowledge occurs at the end of the process. Using this approach, Hall, Loucks, Rutherford, and Newlove (1975) developed the Levels of Use Scale to measure levels of use of innovation. The scale addresses only the behavioral aspect of use and does not focus on other aspects of use, such as attitudinal, motivational, or affective.

The Levels of Use Scale consists of eight levels. Decision points were also provided to distinguish each level. The descriptions of those levels are as follows:

As presented by Hall et al. (1975), each level of use of innovation is accompanied by seven categories to describe the range of behaviors within each level. The seven categories are (1) knowledge, (2) acquiring information, (3) sharing, (4) assessing, (5) planning, (6) status reporting, and (7) performing. For example, within Level I (orientation stage), each category was described as follows:

The psychometric properties of the Levels of Use Scale (Hall et al., 1975) were not reported. However, the scale is one of the most comprehensive in measuring use and could conceivably be quite sensitive in detecting small increments of progress in knowledge use. Particularly, the scale would be useful in evaluating steps taken toward implementation even when the full implementation has not yet occurred.

Larsen (1982) developed a utilization scale containing seven ranked stages of knowledge use and non-use. A piece of information (whether it be a research finding, an idea, a suggestion, or a recommendation) is assigned a rank value based on an ordinal scale outlined in the following list. No reliability or validity were reported.

  1. Considered and rejected: Some discussion takes place, but the information is rejected.
  2. Nothing done: No action, not even discussion, is taken.
  3. Under consideration: Information has not been used but is being discussed and considered.
  4. Steps toward implementation: Although information has not been used, the decision to do so has been made and initial planning steps have been taken.
  5. Partially implemented: Certain features of information have been used, whereas others have been disregarded.
  6. Implemented as presented: The information is used in the form in which it was originally presented.
  7. Implemented and adapted: Information is modified or adapted to fit the local situation.

Brett (1987) developed the Nursing Practice Questionnaire (NPQ) to measure use of specific research findings. Brief descriptions of 14 innovations were provided in the questionnaire. Using Rogers' stages of innovation adoption (Rogers, 1983) as a guide, seven questions measuring the nurse's stage of innovation adoption were developed to be used with each of the 14 nursing practices. The seven questions, as later described by Michel and Sneed (1995), are listed here:

  1. Have you read about this nursing practice?
  2. Have you heard about this nursing practice?
  3. Have you observed this practice in use?
  4. Have you learned about this practice from any other source?
  5. If appropriate to the practice setting, do you believe a nurse should use this nursing practice?
  6. How often do you use this nursing practice?
  7. Are you aware of any policies concerning this nursing practice in your workplace?

These questions address the four stages of innovation adoption identified in the Diffusion of Innovation framework: (1) knowledge (awareness), (2) persuasion, (3) decision, and (4) implementation (Rogers, 1983). Each question represents a specific stage of the innovation adoption:

The points given for each stage (unaware = 0; aware = 1; persuasion = 1; used sometimes = 1; used always = 2) are totaled and averaged. The ranges of possible scores assigned to the various stages of adoption are as follows:

As reported by Brett (1987), the test–retest reliability using a 1-week interval was r = .83. The overall internal consistency (Cronbach's alpha) was .95, whereas the coefficients of the individual innovation scales ranged from .68 to .95. Content validity was assumed because the 14 innovations were derived from published nursing research reports using specific criteria. The NPQ, in both its original form and in a modified form, was used in a number of studies (Berta, 1995; Brett, 1987, 1989; Coyle & Sokop, 1990; Michel & Sneed, 1995; Rodgers, 2000).

Also following the stages-of-use approach, Landry, Amara, and Lamari (2001) measured the use of social science research with a scale derived from the Knott and Wildavsky (1980) stages of knowledge utilization. However, this scale measures knowledge utilization from the perspective of the knowledge producers (who were university researchers) rather that of the knowledge users. A question was posed to the respondents as to what had become of their research of the last 5 years. They then were asked to respond to six statements representing the six stages of knowledge utilization using a five-point scale of 1 = never, 2 = rarely, 3 = sometimes, 4 = usually, and 5 = always. No psychometric properties were reported. The six stages are listed here:

  1. I transmitted my research results to the practitioners and professionals concerned (transmission).
  2. My research reports were read and understood by the practitioners and professionals concerned (cognition).
  3. My work has been cited as a reference in the reports, studies, and strategies of action elaborated by practitioners and professionals (reference).
  4. Efforts were made to adopt the results of my research by practitioners and professionals (effort).
  5. My research results influenced the choices and decisions of practitioners and professionals (influence).
  6. My results gave rise to applications and extension by the practitioners and professionals concerned (application).

A similar scale was developed to measure knowledge utilization as reported by policymakers (Landry, Lamari, & Amara, 2003). The items representing the six stages of knowledge utilization are listed as follows. No psychometric properties were reported.

  1. I received the university research pertinent to my work.
  2. I read and understood the university research that I received.
  3. I participated in meetings for discussion and popularization of the aforementioned university research.
  4. I cited university research studies as references in my own professional reports or documents.
  5. I made efforts to favor the use of university research results.
  6. University research results influenced decisions in my administrative unit.

Another approach is to measure not only the behavioral aspect of the knowledge user, but also other aspects related to knowledge use. Anderson, Ciarlo, and Brodie (1980) used a structured interview method to measure knowledge utilization in the behavioral, cognitive, and affective dimensions. The behavioral aspect was measured by examining reported changes in the activities, practices, or policies that resulted from the researchers' feedback based on their program evaluation for the participating organization. The researchers used two questions, one open-ended and one recommendation-cued, to gather information for behavior change. In the open-ended question, they asked the respondents to describe specific instances of any activities undertaken related to the issues being evaluated. In the recommendation-cued question, they read each respondent a list of the recommendations from their evaluation research and then asked the respondent to cite specific instances of activities relevant to the recommendations. Information was then extracted and coded by category of utilization derived by the research team. The authors stated that after each set of interviews was coded, a measure of the interrater reliability of the coding was computed. However, the reliability results were not reported. The coding was nominal in nature, and the evaluation of knowledge use was descriptive rather than statistical.

To measure the cognitive aspect, the researchers compared the rating of the respondents' beliefs about various aspects of the issue they were studying. Those themes were extracted from the coding of the pre-intervention interviews. The participants were asked to rate their agreement with each of the themes earlier in the project and at 1-month and 6-month intervals after the completion of their intervention. In addition to the original themes obtained from the pre-intervention interviews, a list of "data themes," or statements based on the actual data analysis, was added. In some cases, as stated by the authors, an original theme may also become a data theme. For data themes not originally generated in the pre-intervention interview, the researchers examined whether the postevaluation views of the panel were in agreement or in conflict with those themes.

For the affective aspect, respondents were asked to rate their concern about the problem or issue and their satisfaction with the team's solutions. The timing and rating comparisons were the same as the methods outlined in the cognitive change measurement.

Using a similar orientation, Champion and Leach (1989) developed a scale to measure nurses' research utilization in their study to determine variables related to research utilization in nursing. Four sets of items were measured: (1) attitude, (2) availability, (3) support, and (4) research utilization. The attitude items measured feelings about incorporating research into practice. The availability items measured the opportunity a nurse had to access research findings in his or her institution. The support items measured the degree to which a nurse's administrative leaders and professional colleagues encouraged research utilization. The research utilization items measured the degree to which the nurse felt he or she incorporated findings into practice. The respondents were asked to rate each item using a Likert scale ranging from strongly agree (= 5) to strongly disagree (= 1). The questionnaire consists of 38 items, with 21 items measuring attitude, 7 measuring availability, 8 measuring support, and 10 measuring use. Examples of items representing each of the four aspects are provided below:

Attitude:

Availability:

Use:

Support:

As reported by the authors, the content validity was judged by experts. Internal consistency as measured by Cronbach's alpha ranged from .84 to .94.

Another possible approach in measuring research use is to separately measure each type of use, including conceptual, instrumental, and symbolic use. As discussed earlier, preliminary evidence (Amara, Ouimet, & Landry, 2004; Milner, Estabrooks, & Humphrey, 2005) has shown that each type of use may be uniquely influenced by different factors. Therefore, it seems logical to measure each type of use separately. Estabrooks (1999) developed a measurement scale using four questions to measure overall research utilization, direct research utilization (instrumental use), indirect research utilization (conceptual use), and persuasive research utilization (symbolic use).

The overall-research-utilization question was "Overall, in the past year, how often have you used research in some aspect of your nursing practice?" The question was asked four times throughout the questionnaire. Each time, the question was preceded by the definition of overall research utilization as "The use of any kind of research findings (nursing and non-nursing) in any aspect of your work as a registered nurse. Do not count as research things you learned in nursing school where you did your basic nursing training."

The direct-research-utilization question was "Overall, in the past year, how often have you used research findings in this direct way in some aspect of your nursing practice?" This question was preceded by a definition of direct research utilization as "The use of any kind of research findings (nursing and non-nursing) where you directly use the findings in giving patient care and/or client interventions. Do not count as research things you learned in your basic training." Examples of direct research utilization were provided immediately following the definition.

The indirect-research-utilization question was "Overall, in the past year, how often have you used research findings in this nondirect way in some aspect of your nursing practice?" Similar to the previous two areas, the question was preceded by the definition of indirect research utilization as "The use of any kind of research findings (nursing and non-nursing) to change your thinking or your opinions about how to approach certain patient care or client situations. Do not count as research things you learned in your basic training." Examples were also provided.

The persuasive-research-utilization question was "Overall, in the past year, how often have you used research findings in this persuasive way?" The question was preceded by the definition of persuasive research utilization as "The use of research findings (nursing and non-nursing) to persuade others, who are usually in decision-making positions, to make changes in conditions, policies, or practices relevant to nurses, patients/clients, and/or the health of individuals or groups. Do not count as research things you learned in your basic training." As for the other areas, examples were provided.

The response choices for each question included the scale of 1 = never, 2 = on one or two shifts, 3 = (unlabeled), 4 = (unlabeled), 5 = on about half of the shifts, 6 = (unlabeled), 7 = nearly every shift, and 8 = do not know. This measure was used in other studies (Estabrooks, Chong, & Birdsell, 2003; Milner et al., 2005) with slight modifications in administrative methods, scale, and wording of the questions.

Amara et al. (2004), in an approach similar to that of Estabrooks' measurement of research utilization, developed questions that they included in a survey instrument to distinguish the degree of policymakers' instrumental use, conceptual use, and symbolic use of university research. The instrumental use was described as "The use of university research led to concrete action in my field of work." The conceptual use was described as "The use of university research served to shed light on situations and problems in my field of work." The symbolic use was described as "The use of university research served to confirm choices already made in my field of work." These questions were part of a larger questionnaire. The participants were asked to indicate the level of importance of each type of utilization using the Likert-type scale of 0 = does not apply and does not know, 1 = negligible, 2 = somewhat negligible, 3 = moderately important, 4 = very important, and 5 = decisive.

Focusing on the product of research use rather than on users' behavior, Hanney, Gonzalez-Block, Buxton, and Kogan (2003) proposed a draft scale to measure research utilization in policymaking by directly examining the policy of interest in four areas:

  1. Consistency of policy with research findings
  2. Degree of influence of research on policy agenda setting
  3. Degree of influence of research on policy formulation
  4. Degree of influence of research on policy implementation

The first area, consistency of policy with research findings, concerns the agreement between the policy and the body of international research. The policy would be examined, using documentary analysis, on its consistency with research findings by determining the level of agreement between the policy and the findings of the research.

The second area, degree of influence of research on policy agenda setting, relates to the extent that research helps in getting the issue onto policymakers' agendas, such as by showing the existence of a problem, pointing to actions to be taken and having those actions turned into policy, or enlightening policymakers on the importance of the issue. This information would be obtained via interviews, documentary analysis, and questionnaires to determine degree of influence of research findings on the policy agenda setting.

The third area, degree of influence of research on policy formulation, concerns the actual influence that the research had during the policy formulating process. The actual influence, as indicated by Hanney et al. (2003), includes not only the direct use of research findings to develop the policy content, but also other possible impacts, such as the gradual development of insights, theories, concepts, and perspectives in the enlightenment mode (conceptual use). The degree of influence of research on the policy content, either for instrumental impact or enlightenment, would be determined primarily via interviews but also through survey and documentary data.

The fourth area,degree of influence of research on policy implementation, concerns the use of research in policy implementation, either to determine appropriate implementation strategies or to secure support for the policy in terms of financial resources, political commitment, and public opinion. Information in this area will come from interviews and documentary analysis.

For each area, the ranking scale of considerable, moderate, limited, or no indication is used to measure the degree of research utilization in the four areas indicated above.

Next Section: Epilogue

Previous Section: Effectiveness of Knowledge Translation Strategies

Table of Contents

NIDRR Project Number: H133A060028

National Center for the Dissemination of Disability Research
SEDL
4700 Mueller Blvd., Austin, TX 78723
Voice/Text Telephone: 800-266-1832 or 512-476-6861
Fax: 512-476-2286

Copyright ©2008 SEDL

About SEDL | Contact SEDL | SEDL Terms of Use