Southwest Educational Development Laboratory
Volume 5, Number 2, 2000
Smarter Dissemination: A Word from the Director
Using Market Research for the Dissemination and Utilization of Disability Research
Using Market Research for the Dissemination and Utilization
of Disability Research
In choice modeling, the underlying assumption is that any product or service offering can be conceptualized as a bundle of attributes. Each of these attributes may be more or less important to any particular user, and each attribute may be possessed to a greater or lesser degree by any particular product design. (Adapted from McQuarrie, 1996, p. 101)
Choice modeling is a process of analyzing various components of a product or service to determine which factors influence users to choose certain options over others. Users may rate attributes such as: (a) the interest level of each component; (b) the value of each component; (c) possible uses for each component (d) ease of use of each component.
Choice modeling is particularly useful in field testing products--including research information--to develop decisions about necessary changes prior to widespread dissemination or implementation.
Applied to NIDRR-funded projects, a variety of dissemination approaches may be piloted with user audiences to explore the relative values a variety of users place on key components of each approach. This information can help the researcher in selecting the most appropriate attributes to include in dissemination products or activities. In piloting such choices, the researcher can follow the steps used in direct weighting of attribute importance:
Step 1. The researcher provides two or more dissemination options for user choice modeling. This could include development of format options, such as fact sheets, brief reports, full articles, implementation guides, Web pages, etc. for a sample of users to analyze.
Step 2. The sample of users should be familiar with the research topic. Participants rate each format on how effectively it delivers each particular attribute. Attributes can be abstract or concrete (for example ease of use, professional support, usefulness in their setting, media, and content).
Step 3. Users rate the importance of each attribute for each format according to their own perceptions and self-determined potential to produce behavior change.
Step 4. Additional information should be collected, such as previous awareness of the information, and a rank ordering of preferences among the dissemination formats presented. (Adapted from McQuarrie, 1996, p. 106-107)
Though it has been primarily used to analyze components of physical products, choice modeling offers researchers the opportunity to gain user perspectives of the relative usefulness of various forms of dissemination media and content, such as presentations, journal articles, fact sheets, and implementation guides, among other available formats and modes.
Usability tests are pilot or field tests that provide empirical evidence, before it is disseminated, that test participants can effectively use the product or research information. It evaluates interactions between products and users. According to McQuarrie (1996):
People are notoriously poor at the task of accurately describing step-by-step what they do in a certain situation. Such self-knowledge is tacit and inaccessible. It is far more effective to provide an environment in which people perform some action and then to closely observe the behavior (p. 132).
Usability tests measure the interactions between users and the product format in ways that facilitate redesign of the product or correction of design errors. This may be applied to dissemination activities to improve physical products or revise research information to make them more useful to target users. Such testing is particularly relevant to the dissemination of user's guides or step-by-step instructions that can benefit from evaluations of their ability to be understood and appropriately applied by a sample of market research subjects.
Usability testing typically entails a four step procedure:
Using contextual inquiry involves testing of products or disability research information in the settings for which they are designed. Contextual inquiry allows researchers to observe users interacting with products or implementing research information in settings where barriers are likely to be encountered and competing factors are present. It also allows testing of the researcher's instructions, user guides, or other skill building information in environments that do not require generalization of skills from the laboratory to the actual settings where the skills will be used. This approach holds the promise of providing the most accurate information about necessary functional revisions to products or research information (McQuarrie, 1996).
Usability testing differs from choice modeling in that a sample of users are provided opportunities to actually interact with or implement the product. Consumers can interact with research information and formatting chosen and provided by the NIDRR grantee. In testing product formats, the researcher can observe the user actually manipulating the product and determine the relative ease of use and utility of the product first hand. In testing research information it may take a longer time period to observe sample users implementing new service strategies, through observations of the users' accuracy and identifying their concerns during implementation.
Surveys can be developed to measure a user's knowledge, attitudes, and behavior, or a combination of these elements. Measurement of knowledge is important to finding the extent to which users are familiar with research topics, disability issues, products available, or dissemination media. Many surveys include classifying or qualifying questions, where only those with an understanding of the topic can respond. Measurement of behavior may be most important in market research. Both knowledge and attitudes are displayed in the user's behavior.
It is important to ensure you are not testing for knowledge when your interest is in attitudes. Measurement of attitudes involves soliciting the users' opinions about a topic or product. In many cases, respondents may not have formed their opinions prior to the survey, and researchers should be careful not to construct survey questions that explicitly or implicitly bias unformed attitudes.
"If you want to know how long people wait in line, the best way is to use a stopwatch. But, if you want to know how they feel about waiting in line, use a survey" (Simply Better, 1997, p. II-8).
Surveys are also commonly used in market research as a confirmatory technique to assist in decision making after the product has been disseminated. They serve to confirm the level of user satisfaction and to solicit information that may lead to product improvement.
Researchers and educators commonly use short surveys for evaluating presentations or training sessions. However, researchers could expand their use of surveys to conduct follow-up analyses of publications, to gain user perspectives about Web pages, to gather user satisfaction data concerning improvements following implementation of research-based programs, and to acquire a variety of other information about users' perceptions of research-based products or information formats.
Surveys may be implemented in a variety of ways. Telephone surveys are quick, as responses may be gathered at once with limited waiting time. However, respondents may be less committed than when responding
to in-person interviews. Interviewers must be careful not to influence responses. Mail surveys allow participants to work at their own speed, with no interviewer bias. They can gather large amounts
of solid data, if materials are well-prepared. In mail surveys, you rely on the respondent to understand and interpret questions correctly, and usually mailed surveys have lower response rates compared
to other methods, particularly if the survey is long. Also, you do not know what non-respondents are thinking.
The following example demonstrates usability testing of a universal design assessment instrument, using a survey format. The study included tests of the universal usability qualities of four common household products. Users' perceptions of the usefulness of the usability assessment instrument were examined through a survey. The example also provides evidence of built-in marketing features of the study and potential application of the results in the marketing of universal design concepts. This document has been edited for use in this issue of The Research Exchange.
A Field-Initiated Development Project by the Center for Universal Design
Funded by the National Institute on Disability and Rehabilitation Research (NIDRR),
June 1, 1998 to May 31, 2001
From 1994 to 1997, the Center for Universal Design coordinated the development of the Principles of Universal Design (The Center for Universal Design, 1997) under a NIDRR-funded Research and Demonstration Project. The Principles were a major achievement in facilitating understanding of Universal Design, and today they are widely used internationally in design education, research and development. The critical next step in implementing Universal Design is spreading its influence beyond the research community to accelerate its adoption by industry and acceptance by consumers.
Initial stages of the Promoting the Practice of Universal Design project included the development and testing of a set of Universal Design Performance Measures based on the Principles of Universal Design. Two sets of Universal Design Performance Measures were developed, one set for consumers to use in evaluating products for their own use, and another set for practicing professional product designers to use in developing products for diverse consumer markets. The Performance Measures were tested using a combined usability test and survey approach that measured respondents' perceptions of the universal design characteristics of four common consumer products.
The consumer and designer versions of the Universal Design Performance Measures, called "Product Evaluation Surveys" on the test documents, each comprised a set of 29 statements corresponding
to the 29 guidelines in the Principles of Universal Design. The Principles and the two versions of the Survey address the same issues, but each takes a different approach. As an example, the following
table compares Section 2: Flexibility in Use in each of the three documents.
| The Principles of Universal Design | Consumers' Product Evaluation Survey | Designers' Product Evaluation Survey |
|---|---|---|
2A. Provide choice in methods of use. |
2A. I can use this product in whatever way(s) are safe and effective for me. |
2A. The product offers any user at least one way to use it safely and effectively. |
2B. Accommodate right- or left-handed access and use. |
2B. I can use this product with either my right or left side (hand or foot) alone. |
2B. This product can be used by either right- or left-dominant users, including amputees with or without prostheses. |
2C. Facilitate the user's accuracy and precision. |
2C. I can use this product precisely and accurately. |
2C. This product facilitates (or does not require) the user's accuracy and precision. |
2D. Provide adaptability to the user's pace. |
2D. I can use this product as quickly or as slowly as I want. |
2D. This product can be used as quickly or as slowly as the user wants. |
Each statement in the Product Evaluation Surveys provided for six response options: Strongly Agree, Agree, Neither Agree Nor Disagree, Disagree, Strongly Disagree, and Not Important. Respondents were
asked to circle or "X" their responses to each statement.
The Performance Measures were tested with 60 consumer households and 18 designer households. These households were chosen to be as diverse a group as possible in terms of age, abilities, geographic location, race, socioeconomic status, and home/family situation. In order to assess the true universal usability of the Performance Measures, the consumer group included 60 households, 36 of which contained at least one member with an identifiable disability and 24 of which contained no one with a disability. The designer group included 18 households, some containing individuals with disabilities, and representing a range of experience with and attitudes toward universal design.
The final 60 consumer households participating in the project were located in 25 states, and the 18 designer households were located in nine states. Individual household members ranged in age from infants to 87 years old. In each of six age categories, test participants included individuals with disabilities of upper extremities, lower extremities, vision, hearing, cognition, and health (including multiple chemical sensitivity).
Each household tested the Performance Measures through the evaluation of four consumer products: a cordless hand vacuum cleaner, a digital alarm clock, a plastic food storage container, and a set of ten single-serving breakfast cereals. Participants were asked to have everyone in the household use each product, as appropriate, and keep a carefully structured journal documenting everyone's use of and comments about the products. After using the products for a few weeks, the testing participants were asked to complete a set of four Product Evaluation Surveys, one for each product.
The test results provide a wealth of information about both the consumers, and designers, quantified levels of satisfaction and narrative comments regarding all aspects of using each of the sample products. Preliminary test results suggest that consumers did not agree on the usefulness of the Performance Measures (Survey), although data analysis is underway and the commonalities among those who found them useful have not yet been determined. The designers found the Performance Measures to be less useful than the consumers, but preliminary analysis suggests that productively applying the Performance Measures may require a reasonable knowledge base regarding universal design. To address this need, informational materials will be developed in the final period of the project to augment the Evaluative Survey; they will be distributed together.
The process of recruiting participants for the testing activities also served to pre-market the results of the project. In addition to the 78 households serving as testers, more than 250 individuals, who volunteered to take part in the project but could not be included, requested to receive project results when they are available.
The Center for Universal Design (1997). The principles of universal design (Version 2.0). Raleigh, NC: North Carolina State University. Retrieved April 12, 2000 from the World Wide Web: http://www.design.ncsu.edu:8120/cud/univ_design/princ_overview.htm
Web Accessibility | Site Map | Feedback
NIDRR Project Number: H133A990008
Last Updated: Tuesday, 29 January 2008 at 04:12 PM,
Contents © 1996-2005
SEDL Southwest Educational Development Laboratory (SEDL)
211 East Seventh St., Room 400 - Austin, Texas 78701-3253
Voice/Text Telephone: 800-266-1832 or 512-476-6861 - Fax: 512-476-2286
Copyright ©2004 Southwest Educational Development Laboratory