Use of Market Research Panels in Transit (2013)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

45 A market research panel is a sampling technique, and therefore falls within Step 2. Sampling is a critical issue in developing a panel and the appropriate application of panel survey results. Sampling Traditional survey sampling is based on the concept of a random sample. A random sample is designed to provide results that can be projected to a particular target population with a specified level of accuracy. Collecting data using a traditional random sample can be a lengthy process, increas- ing costs and delaying completion of the research; but yields results with a known level of accuracy. A non-probabilistic sample (of which one type is called a “convenience sample”) is not random, which simplifies and shortens the data collection process; however, because the sample is not random, the results cannot be projected to a target population. The Internet, smart phones, and social media are rapidly supplanting traditional modes of com- munications. Many surveys are now conducted over the Internet; but as online panel recruitment and surveying pro- vides a non-probabilistic sample, those survey results should be used for concepts and idea formation, not for estimates of the population. Sources of Error and Bias in Market Research There are many potential sources of error and bias in market research. This synthesis highlights three categories of error as they relate to market research panels. Coverage error is perhaps the most significant issue for studies that use online recruitment and surveying, because certain segments of the population do not have access to, or do not use, the Internet, and therefore cannot participate. Non-response bias occurs when individuals selected as part of a sample either do not respond to the request to complete the survey or decline to answer some or all of the questions. This can be exacerbated in panel surveys, since members must continue to partici- pate from the recruitment stage through multiple surveys. Measurement error is the difference between an observed response and the true response, and has been shown to occur when researchers shift from interview surveys (e.g., intercept or telephone) surveys to self-administered (e.g., paper or online) surveys. A second source of measurement error in panel surveys is called “panel conditioning,” which refers From the late 1950s through the 1990s, market research was conducted primarily through random-digit-dial telephone sur- veys, personal intercept surveys, and the U.S. Postal Service. Transit studies typically employed on-board surveys, tele- phone surveys, and focus groups. However, evolving modes of communication, most notably cell phones, the Internet, and social media, have created a fundamental change in market research. At the same time, transit agencies face increasingly constrained research budgets and shortened timelines, and need alternative market research approaches that provide quality data more quickly and less expensively. Market research panels provide an option for affordable customer research to support transit management decision- making. This synthesis examines the various types of panels, how they can be tailored to address a variety of transit agency needs, and areas of concern that the researcher should be aware of to ensure appropriate use of panel survey data. MARKET RESEARCH CONTEXT Market research is the gathering and evaluation of data regard- ing consumers’ preferences for products and services. In the transit industry, market research supports decision-making in all aspects of planning and operations, including service routing and scheduling, fare policy and implementation, adver- tising and promotion of transit services, vehicle and customer amenities, customer information, and long-term financing and planning. The private sector has applied market research extensively for more than 50 years and has developed a wide variety of techniques that can be used by the transit industry to collect and analyze customer information. Market Research Process Market research is used to identify and define marketing opportunities and problems; to generate, refine, and evaluate marketing actions; to monitor performance; and to clarify the market research process itself. Market research has four steps: (1) planning the study (specifying the data necessary to address the issue); (2) design- ing the research methodology; (3) conducting the research survey(s); and (4) analyzing and reporting the findings and their implications. Although there are variations in this process, it remains essentially the same for all market research studies. chapter five CONCLUSIONS

46 to members’ becoming more attuned to the issue and therefore responding differently than if they had not been on the panel. Adjustments to Reduce Error and Bias Sampling strategies and data weighting are among techniques that can be used to reduce the effects of sampling error in surveys. Online panel surveys may require additional strategies to weed out respondents who are not truly engaged or are only completing the questionnaire to receive an incentive. Adjust- ing a non-probabilistic sample to make it representative of a target population can be time-consuming and expensive, offsetting two of the primary benefits of panel research. As a result, market researchers are experimenting with dual-frame or multi-frame sampling to increase survey coverage and create a more representative sample. MARKET RESEARCH PANELS A survey panel is selected to provide data for the analysis of some specific aspect of a target population. Panel surveys have been used in transportation for many years, but new technologies are changing the definitions and applications of market research panels, expanding them to address a variety of research needs. Types of Market Research Panels The traditional definition of a panel—what the American Marketing Association considers to be a “true panel”— includes the respondents’ being measured over multiple sur- veys with respect to the same variables. Market researchers also employ “omnibus panels,” in which respondents are measured over multiple surveys but on variables that change each time. Omnibus panels developed by an organization solely for its own use are called “client panels.” In addition, market research vendors have developed online access panels that an agency can “buy into” for a specific project. Recently, social media techniques have been combined with client panels to create “online research communities” in which panel members log in to a closed website to participate in surveys and discussions and to interact with other panel members and/or the sponsoring agency. Conducting Panel Research There are four stages in panel research. Stage 1 involves the recruitment of panel members, either through traditional means, such as telephone or in-person solicitation; through agency contact lists; or through online methods, such as pop-up advertisements or a link on the agency website. Stage 2 refers to the profiling and assembling of panel members. With tele- phone or in-person recruitment, profiling the respondents and having them join the panel is done when the potential panel members are first contacted. Agency contact lists require contacting potential panel members to collect basic informa- tion and screening to determine if the person will make a good addition. With online recruitment, there is typically a screen- ing questionnaire, as part of which the respondent agrees to become a member of the panel. Stage 3 covers implementa- tion of the panel survey. This can be done through traditional methods, such as telephone surveys, paper surveys, or focus groups; or online. Stage 4 pertains to panel maintenance. Maintaining panel membership is important to ensure the integrity of the sample and to limit costs. Attrition from any panel is to be expected, but steps can be taken to reduce it, including frequent communication, engaging survey topics, and incentives such as free transit passes or savings bonds. HOW TRANSIT AGENCIES ARE USING PANEL SURVEYS A survey of 31 industry agencies showed that only 10 transit agencies had completed a project using a market research panel. However, interest in the method is growing, with nine other agencies having considered or reporting being in the process of developing a market research panel. Survey Topics for Panel Research Transit agencies have used panel surveys to address a wide variety of topics: rider attitudes and satisfaction, marketing and message development, public awareness of transit issues and input on planning projects, and evaluating the effects of agency actions. Although less frequently cited, other informa- tion gathered includes basic travel behavior characteristics and demographics. One agency that uses a panel for focus group- style research has used it to test products, such as parking pay- ment equipment and the agency website interface for mobile devices; to develop a brand image and messaging; and for long-range transit planning. Benefits of Panel Surveys Those agencies that have considered or conducted panel sur- veys cited as primary benefits of panel surveys the ability to conduct research at a moment’s notice and the ability to do research with specific target markets (both at 63% of respon- dents). Other key benefits, cited by 53% to 58% of respondents, were the ability to track changes in attitudes or behaviors from the same person over time, and that panel surveys are faster and cheaper than traditional research. Concerns with Panel Surveys The primary concern with panel research was that the panel may not be representative of the target population (70% of respondents). The other key concerns, cited by about half

47 when care is taken in selecting the panel members are and when the research question involves concepts and features, not precise estimates of the population. If precise estimates are needed, a survey using random sampling techniques is necessary to provide reliable results, and research has shown that telephone surveying is still superior to online panels in providing a random and representative sample. Step 1, Recruitment People Want to Participate! There was an initial expectation that riders would not want to participate in a panel or complete surveys online. The experience of the participating agencies demonstrates that an agency need not be timid about creating online panels and actively soliciting customer feedback. Although the panel may not be statistically representative of the riding population, many people are eager to sign up and participate in the decisions that will be made regarding their service and how the government is spending their money. The online panel format makes that participation easy and efficient. Methods of Recruitment A variety of methods has been used to recruit panels. The method selected depends on the type of research panel being assembled, the type of data that will needed, and whether and how much vendors will be used to support the panel research effort. Typical methods include random-digit-dial telephon- ing, using existing agency e-mail lists, recruiting and asking for e-mail addresses from other agency surveys (by telephone or on-vehicle), posting a link on the agency website, and hiring a vendor to provide recruiting services. Advertising and promotion of the panel was a key element of open member- ship research panels at New Jersey Transit and Washington State Ferries (WSF). Means of encouraging participation and promoting the web address include posters at terminals and on vehicles, press releases to engage the media, a message on the agency phone lines for callers on hold, and handing out business cards with the information at high activity transit stops. WSF experimented with using Quick Response codes on their posters to take people directly to the website on their smart phone. Probability-based Sampling Has Benefits A probability-based sample is more expensive to develop than a non-probabilistic sample; consequently, replacing panel members who leave through attrition or systematic replacement is also more costly. The benefit is that a panel can be built that represents the general population and survey results can be fully analyzed using market research techniques. New York’s Metropolitan Transportation Authority successfully used prob- ability sampling for their panel. of the respondents, were panel attrition and lack of funding/ staffing to maintain the panel. Use of Vendors Agencies were asked what role vendors had in developing and implementing their panel research programs. Five general tasks were provided: recruiting the panel, maintaining the panel, developing the questionnaire, implementing the survey, conducting the analysis and reporting. Responses ran the full range, from the agency’s having conducted the panel research completely in-house to having contracted out all tasks to a vendor. The survey also asked if the agency used a vendor online access panel or developed their own client panel. Only one agency used an existing commercial online access panel; the other nine agencies developed an in-house panel to meet specific agency goals. LESSONS LEARNED The literature review, industry survey, and case examples provided a wealth of information on the issues surrounding market research panels and their successful application in transportation. Key lessons learned are provided here. Paradigm Shift The issues and concerns with online access panel market research include the lack of grounding in sampling theory. In 1970, 88% of the households in the United States were esti- mated to have a landline telephone. This level of coverage led to the acceptability of using random-digit-dial telephone sur- veys in place of in-person interviewing. The past 10 years has seen a dramatic shift to cell phones and online technologies as daily methods of communication for many households and individuals. Current estimates are that fewer than 70% of U.S. households now have a landline telephone, raising the issue of whether random-digit-dial telephone surveys can adequately represent the population. The implication is profound: random-digit-dial telephone surveying, the mainstay of current market research, no longer provides a probabilistic sample. This shift from probability sampling to non-probability sampling is a paradigm shift of the magnitude that led to the development of sampling theory in 1934. Sampling theory, specifically non-probabilistic sampling, has not caught up with the “wireless” age. As of the writing of this report, no generally accepted method of sampling by means of the Internet has been estab- lished, nor is Internet access pervasive enough that it can replace telephone surveys. Given the current state of the prac- tice, online access panel research can be used successfully

48 Step 2, Assembling and Profiling the Respondents Open Panel Membership Panel membership can be closed, where members must be selected to join the panel; or open, where anyone can join. WSF’s Ferry Riders’ Opinion Group panel is an example of an open membership panel, with on-going recruitment to solicit new members. Open panels can become over-heavy with members who have a high vested interest in the transit system, whether to complain about services or advocate for a certain planning decision. This creates a need to recruit more general and casual riders, not just the “enthusiastic” regular customer, requiring ongoing advertising and promotion of the panel. Screening for a Good Fit Denver’s Regional Transportation District (RTD) recruits just 16 riders each year for its focus group-style panel, making it important to select members who are engaged and able to communicate effectively in the group format. The potential panelists complete an online questionnaire with open-ended questions about the transit system. These questions allow staff to determine which candidates are thoughtful and engaged, while weeding out those who may have a personal agenda and may not be constructive members. The list of potential panel- ists is also screened against a list of persons who have filed an excessive number of complaints against the agency, reducing the likelihood of including “chronic complainers” unlikely to provide useful input or feedback to the agency. Step 3, Conducting the Research Target Samples versus Completed Surveys In Portland, Oregon, TriMet contracted with a vendor to use the vendor’s existing online access panel. The panel members invited to complete the survey were demographically represen- tative of the transit service district population; however, there was significant demographic bias in who completed the sur- vey. As a result, the data did not reliably represent the popula- tion, and could not be used for the intended purpose. At the Minnesota Department of Transportation (MnDOT), the panel participants were recruited to represent the population of the metropolitan area, and the remainder of the state. A comparison of the unweighted results to the statewide profile has shown that the results may not be perfect, but they are adequate for the agency information needs and intended use of results. In both cases, weighting the data would add time and cost to the survey, thereby eliminating the benefit of using the panel approach. This underscores the need for the researcher to understand the goals of the survey at the project planning stage and to tailor the sampling technique to the require- ments, understanding the strengths and limitations of online access panels vis-à-vis traditional survey techniques. Step 4, Panel Maintenance More Communication Is Better Researchers have an expectation that frequent contact with panel members will lead to burn-out among participants, and that they will drop out of the panel as a result. MnDOT’s experience with its online research community demonstrated that regular weekly discussions and activities are the key to continued panelist engagement, high response rates, and quality information. Incentives Are Welcome Not all public agencies are permitted to provide incentives for panel participants, but if allowed, they are very effective. RTD has quarterly panel meetings. To encourage all 16 panel members to show up every time, the agency offers them a monthly pass at each meeting; and participants who attend all four meetings receive an annual pass for the following year. Incentives used by other agencies include $10 cash cards; drawings for U.S. savings bonds of various denomi- nations; and publications about the city created specifically for the panel. WSF provides a noncash incentive: immediate gratification. When respondents complete a quick survey, they immediately get the results-to-date so they can compare their response with other panel members. Variety of Successful Applications Panel surveys have been successfully applied to a variety of research needs. At RTD, a panel is used for quarterly focus groups. The WSF created an open membership panel to gather input from as many ferry customers as possible, and create a public relations benefit. MnDOT created a closed online community that engages in topical discussions as well as surveys, thereby providing quantitative and qualitative feedback. Metropolitan Transportation Authority created a telephone survey panel with daily input that provided a fluid, dynamic research tool—real-time customer response instead of recollected experience. Agency Costs One of the primary benefits cited for using market research panels is that surveys can be conducted more quickly and economically than with traditional methods. The case exam- ples illustrate that in-house client panels typically require a combination of staff and vendor resources. RTD’s program has the lowest total cost, in part because it is conducted com- pletely in-house. Staff resources across all departments were not directly measured; however, the program is streamlined to the point that it requires about 0.15 FTE. (It is important to recognize that the program has been in existence for many years, so this does not include start-up costs.) An on-going

49 panel research program typically has approximately 0.5 FTE assigned to the program and an annual budget of $100,000– $300,000. It is recommended that, if a vendor is used, a long- term contract be negotiated to smooth out initial start-up costs and possibly lower the vendor’s overall charges in return for several years of guaranteed work. Guidance on the Use of Online Panels The American Association for Public Opinion Research Report on Online Panels provides the following guidance to market researchers who are considering online access panels: • A non-probability online panel is appropriate when precise estimates of population values are not required, such as when testing the receptivity to product concepts and features. • Avoid using non-probability online access panels when the research is to be used to estimate population values. There is no theoretical basis for making projections or estimates from this type of sample. • The accuracy of self-administered computer surveys is undermined because it is a non-probability sample. A random-digit-dial telephone survey is more accurate than an online survey because it is a probability sample. • Weighting the results from online access panel surveys has not yet been demonstrated to be consistently effec- tive and can be used to adjust for panel bias. • There are significant differences in the composition and practices of various online access panels, which can affect survey results. Different online access panels may yield significantly different results on the same questionnaire. • The market research industry has developed guidance for organizations to assist with obtaining quality survey research and understanding the limitations of online research panels (see the Bibliography). TOPICS FOR FUTURE STUDY The transit industry is increasingly involved in market panel research, and as such needs to stay abreast of new research techniques, as well as contribute to the field of research. Topics for participation and additional research are summarized here. Monitor Market Research Industry Activities The market research industry has established a “research-on- research” program related to online panel survey techniques. The program includes research on sampling, data collection, self-administered versus surveyor-administered question- naires; measuring and adjusting for different types of bias and error, and adapting to new technologies. At the same time, the industry is publishing guidelines and standards to help users of market research understand how successfully to adapt to new market research techniques. These research programs are underway in the United States and Europe. Transit professionals will benefit from having access to the guidelines and standards resulting from this market research industry work. Special Populations on Market Research Panels The transit industry has both a requirement and an obligation to hear from all of its existing and potential riders. This includes transit-disadvantaged populations, such as the elderly, persons with disabilities, minorities, low-income households, and per- sons with limited English proficiency. In addition, persons with low reading and writing comprehension are likely to have difficulty completing self-administered online surveys. These groups tend to be under-represented because they are less likely to have Internet access or have physical or language barriers that preclude them from participating in the pan- els. Research into how to include these individuals in panel research is important to providing reliable and representative survey results. Multi-frame Sampling This synthesis provides a discussion of the issues and con- cerns surrounding non-probabilistic sampling, especially as it relates to online panels. Survey researchers have been mov- ing away from single-source sampling (such as using only random-digit-dial telephoning) to dual-frame or multi-frame sampling. Both the New Jersey Transit customer satisfaction survey and the Washington State Transportation Commission case example illustrate dual-frame sampling, where online surveys are supplemented with in-person or paper surveys. Additional research is needed to understand how transit agen- cies can use multi-frame sampling to create efficiencies in customer surveying. Legal and Ethical Issues Agencies participating in this synthesis did not cite any con- cerns with legal or ethical issues, because most already con- duct market research and this is an extension of that activity. However, human subjects laws, privacy acts, and freedom of information acts all have implications for market research in the public sector, especially in an age of readily available and shared electronic information. The implications of conduct- ing online and panel research need to be explored, particularly when it is implemented in-house, and guidance provided to agencies to ensure all legal and ethical requirements are met.