Use of Market Research Panels in Transit (2013)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

21 usefulness for gathering feedback from their customers and general public. Results are shown in Table 1. A call center/live operator was the most widely used method for gathering feedback, and rated the most useful of all tech- niques (63% described it as very useful and 30% as somewhat useful). The second most useful method was the agency web- site (50% called it very useful and 40% somewhat useful). Twitter was the third most used communication medium, with 25 of the 31 agencies having a Twitter feed; among those 25, 16% rated it as very useful and 68% as somewhat useful. Facebook and YouTube were being used by 21 of the 31 agen- cies; however, Facebook was rated as more useful, with 24% giving it a neutral rating and only 5% rating it not very useful. Nearly one-quarter of the respondents with a Facebook page gave it a neutral rating (14% described it as very useful, 57% as somewhat useful). YouTube was rated neutral by 38% of the agencies and by another 24% as not useful (19% called it not very useful, 5% not at all useful). Although only 17 of the 31 agencies had a blog, 24% of those that did rated it as very useful for gathering feedback. LinkedIn, Flickr, and third-party sites (e.g., ICanMakeIt Better.com or SeeClickFix.com) were used by fewer than half of the agencies, and were generally rated as being less useful than other methods for gathering feedback from their cus- tomers or the general public. In addition to the question shown in Table 1, agencies were asked the open-ended question, “What other platforms do you use to obtain feedback from your customers and the general public?” Responses included: • Customer surveys/panels (22 agencies) • Public meetings/hearings/open houses/workshops (8 agen cies) • Telephone hotlines/e-mail (3 agencies) • Comment cards/feedback forms (3 agencies) • Community Committees (3 agencies) • Frontline personnel (2 agencies) • Third party provider: ICanMakeItBetter.com (1 agency). Use of Market Research All agencies responding to the survey conduct market research. Phone interviews with several of the agencies that did not respond to the survey indicated that they do not conduct market METHODOLOGY An online survey of selected transit and transportation agencies was conducted to determine the extent of market research activities and the use of panel surveys in transit. The sample of agencies surveyed was not randomly generated, but rather was selected because: (1) They were known to have experience with panel surveys; (2) they were known to conduct market research and were most likely to have experience with panel survey techniques; or (3) they helped provide representation of all sizes of agencies. As such, the responses reflect the experience and views of the participating agencies, and are not necessarily representative of the industry as a whole. The e-mail invitation to participate in the survey and the questionnaire are provided in Appendix B. The participating agencies are listed in Appendix C. It can be noted that there is limited participation in the survey from smaller transit agen- cies. Small and rural agencies have limited budgets and are much less likely to have a market research function; through this synthesis, smaller firms will be able to learn from the experiences from the larger firms and better understand the elements of successful panel research. FINDINGS The survey covered four primary topics: modes operated; methods of gathering customer feedback; use of market research; and experience with panel surveys. Agency Description Agencies were asked what types of transit service(s) they operate (see Figure 1). Of the 31 respondents, 28 were transit operating agencies, one was a regional transit authority, one was a MPO, and one was a state DOT that does not operate transit services. Customer Feedback Methods Feedback can be gathered informally from customers and the general public through a variety of methods. Social media is becoming a more common way of communicating with cus- tomers and the general public. The survey listed nine traditional and social media communication techniques. The respondents were asked if they used each method, and if so, to rate its chapter three SURVEY RESULTS—USE OF PANELS

22 research. Paper surveys distributed on board vehicles were the most common technique, used by all but one agency. Focus groups, telephone surveys, and online surveys from the company website were also common, with about 80% of the agencies reporting using those techniques. It is interesting to note that among these agencies conducting market research, online surveys are now as prevalent a research method as telephone surveys. In-person interviews are used by about two-thirds of respondents. Only 42% used paper surveys dis- tributed through the mail, and 29% had employed panel surveys (see Figure 2). No agencies reported e-mailing the survey or survey link using an agency e-mail list, or linking to a survey from a social media broadcast (e.g., Facebook, Twitter). When asked what barriers the agency faced in conducting market research, 48% of the agencies cited lack of funds for consultants, with 26% citing lack of staff resources. Other barriers to conducting market research reported included lack of management support (13%), lack of overall funding (6%), and lack of technical staff to conduct the research (3%). Of this select sample of 31 agencies, 39% stated that they did not face any barriers to conducting research. No agency responded that it did not have a need for market research (see Table 2). Respondents were asked how much their agency spent annually on all market research, for in-house staff as well as consultants. Combined staff and consultant budgets ranged from under $10,000 annually to $1 million at the larger agen- cies or for special projects. One agency had not conducted market research over the past three years, but had budget allocated to conduct research in the current year. The primary topics of research over the past three years were rider demographics and rider attitudes/satisfaction (90%). Next were trip characteristics (84%) and marketing/message development (77%). Federal requirements, such as the National Transit Database and Title VI, were the topics of market research over the past three years for 13 agencies (see Table 3). FIGURE 1 Which modes does your agency either directly operate or operate using a contractor (n = 31)? Other: ferry system (1), high-occupancy vehicle lanes (1), vanpools (1), not a transit service operator (3). 81% 68% 52% 39% 39% 35% 29% 26% 19% 0% 20% 40% 60% 80% 100% Pe rc en to fA ge nc ie s TABLE 1 HOW WOULD YOU RATE THE USEFULNESS OF EACH OF THESE METHODS FOR GATHERING FEEDBACK FROM YOUR CUSTOMERS AND THE GENERAL PUBLIC? Method Not Used Used Very Useful Somewhat Useful Neutral Not Very Useful Not At All Useful (No.) (No.) (%) (%) (%) (%) (%) Call Center/Live Operator 1 30 63 30 7 0 0 Agency Website 1 30 50 40 7 3 0 Twitter 6 25 16 68 12 4 0 Facebook 10 21 14 57 24 5 0 YouTube 10 21 10 29 38 19 5 Agency Blog 14 17 24 47 18 12 0 Flickr 16 15 0 27 47 20 7 LinkedIn 18 13 0 23 54 23 0 Third Party (e.g., SeeClickFix.com) 21 10 20 10 50 20 0 n = 31.

23 Those agencies that had considered or conducted panel research were asked what they saw as the primary benefits of panel research. More than half responded that they are able to conduct research in-house at a moment’s notice; that they can target specific market segments; that it is faster and cheaper than traditional survey methods; and that they are able to track changes in attitudes or behaviors from the Panel Surveys Respondents were asked if they had experience with panel surveys. Ten agencies had conducted panel research, nine had considered but not conducted panel research at that time, and 12 had neither considered nor conducted panel research (see Figure 3). 97% 87% 81% 77% 68% 42% 29% 0% 20% 40% 60% 80% 100% Pe rc en to fA ge nc ie s FIGURE 2 What are the market research techniques you use to collect information from your customers and the general public (n = 31)? Barriers Percent Lack of funding to pay for research consultants 48 Lack of staff to conduct or oversee market research activities 26 Lack of support from management for market research activities 13 Lack of overall funding 6 Lack of technical skill to conduct market research 3 We do not have any barriers to conducting needed market research 39 We do not have a need for market research 0 n = 31. TABLE 2 WHAT ARE THE BARRIERS TO CONDUCTING MARKET RESEARCH AT YOUR AGENCY (check all that apply)? Purpose of market research Percent Rider demographics 90 Rider attitudes/customer satisfaction 90 Rider trip characteristics (origin, destination, trip purpose, mode of access, etc.) 84 Marketing and message development 77 General public attitudes and awareness of transit/transit issues 71 Public input on transit planning, transit projects 65 Evaluating the effects of an agency action 55 Federal reporting requirements (Title VI, limited English proficiency, etc.) 42 General public support for funding initiatives 35 Other 3 Not applicable 6 n = 31. TABLE 3 OVER THE PAST THREE YEARS, WHAT HAS BEEN THE PURPOSE OF YOUR MARKET RESEARCH (check all that apply)

24 12 9 10 No, I have not considered or conducted panel surveys Yes, I have considered but not conducted panel surveys Yes, I have considered and conducted a panel survey FIGURE 3 Have you considered or conducted panel surveys for market research purposes (n = 31)? Primary Benefits Percent Able conduct the research in-house on a moment’s notice 63 Can do research with specific market segments 63 Faster than traditional survey methods 58 Ability to track changes in attitudes or behaviors from the same person, over time 58 Cheaper than traditional survey methods 53 Provides a public relations benefit with our riders 37 Other 21 Allows non-rider community stakeholders to feel they are involved in transit planning An experiment with new methods to conduct tracking research Specific reactions to changes made over time Saw the panels survey as a way to smooth out our survey research budgets over a number of years. Typically, our large-scale cross-sectional surveys cost approximately $2ñ3 million dollars, which is a big bite out of our total MPO budget; thus, very hard to fund in a single year. Can only afford these large-scale surveys about once a decade. The original thought was to use a smaller continuing annual panel survey and to budget about $250,000/per year instead of $2,500,000 once every 10 years. n = 19. TABLE 4 WHAT ARE THE PRIMARY BENEFITS THAT LED YOU TO CONSIDER A PANEL SURVEY (check all that apply)? quent large-scale survey that could require special funding (see Table 4). The number one concern with panel surveys was that the panel might not be representative of the target population (cited by 70% of the respondents). At least half of the 19 respon- dents cited concerns with panel attrition and lack of funding or staff to maintain the panel. Seven agencies (35%) cited concerns with panel members becoming sensitized to transit issues. Ethical concerns and public relations concerns were only cited as an issue by one agency each. No one cited legal concerns with panel surveys. Four agencies (21%) had no concerns at all with panel surveys (see Table 5). Panel surveys were most likely to be used for rider and general public attitudes and awareness of transit, and marketing and message development. They were least likely to be used to research rider demographics and trip characteristics, which are typically tracked using on-board paper surveys (see Table 6). The nine agencies that had considered a panel survey but not conducted one at that time were asked why they had not Primary Concern Percent Panel may not be representative of my target population 70 Panel respondents “dropping out” (panel attrition) 55 Lack of funding/staffing to maintain the panel 50 Panel members were becoming sensitized to transit issues 35 Providing incentives to engage and keep panel members participating 20 Public relations concerns 5 Ethical concerns raised regarding panels 5 Other: Costs outweigh benefits 5 Legal concerns 0 I do not have any concerns with panel surveys 20 n = 19. TABLE 5 WHAT ARE YOUR PRIMARY CONCERNS WITH PANEL SURVEYS? same person over time. Other benefits cited include a public relations benefit with riders and non-riders, and tracking changes in attitudes or behavior over time (not by individual). One agency reported using a panel approach for smaller annual surveys, which could more easily be accommodated in the budget using revenues, rather than conducting an infre-

25 conducted panel research. The following open-ended responses were received: • Short staff, not a priority for last year • Still deciding whether or not to do it. Top management leaning towards allowing it • Time constraints • Haven’t decided • Start-up was complicated. New product development could be leaked to general public with negative reactions to beta test. • Primarily concerns about attrition and replacing panel members as we were considering it for a survey that would be conducted annually or every couple of years. • The first regional customer satisfaction study is being conducted that could provide a basis for panels imple- mentation in the future. • We plan to implement panels in 2012. We are currently conducting customer satisfaction survey that will provide us with a pool to recruit customer panels. Panel Survey Experience A series of questions was asked of those agencies that had implemented panel survey research. The survey included the following definition of a panel survey: A panel survey is a community of people who have agreed to participate in research projects periodically. Panel members can be recruited and surveyed using traditional techniques (for example, random-digit-dial with a phone survey) or more recently, using on-line recruitment and surveying. Market research companies develop and maintain general panels and also have the ability to create custom panels. Alternately, a panel can be developed and maintained in-house. Number of Panel Surveys Conducted Based on the definition provided, 10 of the agencies par- ticipating in the survey had implemented panel surveys. Two agencies had extensive experience with panel surveys, stretching over 15 years or more. Following are the responses to the question: “How many panel surveys have you conducted over all survey efforts?” • One survey (3 agencies) • Two surveys (2 agencies) • Three surveys (1 agency) • 4 or 8 per year for 20+ years (1 agency) • 20—periodic surveys over 3 years (1 agency) • 200—1 or more weekly surveys for 3 years (1 agency) • Continuous daily surveying for 15 years (1 agency). Benefits of Panel Research The most commonly cited benefit of panel surveys, reported by seven of the 10 respondents that had conducted them, is that it was cheaper than traditional survey methods. Six remarked that the research could be targeted to specific mar- kets, while five agencies believed that panels were faster than traditional survey methods, and could be conducted in-house at a moment’s notice (see Table 7). The Metro Washington Council of Governments (COG), District of Columbia, saw the panel as a way to smooth out its survey research budget over a number of years by con- ducting a continuous panel survey. See the Metro Washington Council of Governments Panel Survey Experience profile for details of their panel effort. Purpose Rider attitudes/customer satisfaction 68 Marketing and message development 58 General public attitudes and awareness of transit/transit issues 53 Public input on transit planning, transit projects 42 Evaluating the effects of an agency action 42 Rider trip characteristics (origin, destination, trip purpose, mode of access, etc.) 26 Rider demographics 21 General public support for funding initiatives 11 Federal reporting requirements (Title VI, limited English proficiency, etc.) 5 Other: 11 Website evaluation, mobile website evaluation, parking payment system evaluation All—but we don't just focus on transit n = 19. Percent TABLE 6 WHAT WAS THE PURPOSE FOR WHICH YOU CONSIDERED OR CONDUCTED A PANEL SURVEY (check all that apply)? Metro Washington Council of Governments Panel Survey Experience The Metro Washington (D.C.) Council of Govern- ments (COG) conducts a large household travel survey approximately every ten years to update the regional transportation planning model. Data needed are origin/

26 Concerns with Panel Research When asked about concerns they had with panel surveys, two of the ten agencies did not cite any. The main concerns of six of the eight remaining agencies were that the panel might not be representative of their target population, panel attrition, and adequate resources to maintain the panel. Other concerns cited by three or fewer agencies included the panel becoming sensitized to transit issues, the necessity of provid- ing incentives to keep panel members engaged and partici- pating in the panel, and ethical concerns (see Table 8). NJ TRANSIT planned to implement an online customer satisfaction and travel behavior study, but was concerned that the panel might not be representative of the riding population. It instituted several practices to ensure participation by all of their riders. See the NJ TRANSIT Panel Survey Experience profile for a discussion of its efforts. destination, mode of travel, trip purpose, time of travel, etc., for all daily travel. Typically, the large-scale cross- sectional household travel diary study costs $2–$3 mil- lion, which is a significant portion of the COG’s budget and difficult to fund in a single year. The plan was to create a panel which would have fewer members than are typically surveyed in the telephone survey, but that would be surveyed annually. The budget would be about $250,000 annually and cost the same over the ten-year period. The panel was recruited and the survey was fielded for six years, from 1998 to 2003. The panel was dis- banded before the proposed ten years for several rea- sons: High turnover of panel members resulted in the study becoming more of a repeated cross-sectional survey with limited travel behavior tracking at the indi- vidual level; the sample was determined to be too small to get the origin/destination data needed to support the regional travel model; there was little, and inconsistent, year-to-year change in the measures of regional travel behavior; proper weighting and statistical analysis of the survey results was exceedingly complex; and panel maintenance costs were high. TABLE 8 WHAT ARE YOUR PRIMARY CONCERNS WITH PANEL SURVEYS? Primary Concerns Considered Panel Surveys Conducted Panel Surveys Total n = 9 n = 10 n = 19 Panel may not be representative of my target population 7 6 14 Panel respondents may “drop out” (panel attrition) 5 5 11 Lack of funding/staffing to maintain the panel 5 4 10 Panel members were becoming sensitized to transit issues 3 3 7 Providing incentives to engage and keep panel members participating 2 2 4 Public relations concerns 1 0 1 Ethical concerns raised regarding panels 2 1 1 Legal concerns 0 0 0 Other 0 1 1 I do not have any concerns with panel surveys 0 2 4 TABLE 7 WHAT ARE THE PRIMARY BENEFITS THAT LED YOU TO CONSIDER A PANEL SURVEY (check all that apply)? Primary Benefits Considered Panel Surveys Conducted Panel Surveys Total n = 9 n = 10 n = 19 Able conduct the research in-house on a moment’s notice 6 5 12 Can do research with specific market segments 5 6 12 Faster than traditional survey methods 5 5 11 Ability to track changes in attitudes or behaviors from the same person, over time 7 4 11 Cheaper than traditional survey methods 2 7 10 Provides a public relations benefit with our riders 4 3 7 Other 1 3 4 NJ TRANSIT Panel Survey Experience In April 2011, NJ TRANSIT initiated a quarterly on-line survey with 41 service attributes designed to track customer satisfaction. An initial panel was developed

27 Use of Vendors The primary tasks of conducting a panel survey are panel recruitment; maintaining the panel; developing the question- naire; implementing the survey; and conducting the analysis and reporting. Those who had conducted panel research were asked how the responsibilities were assigned for each of these tasks: to in-house staff, an outside consultant, or both working together (see Table 9). Overall, there was no clear-cut division of responsibilities. It is clear that each agency develops its methodology and use of vendors based on its own situation, needs, and resources. The 10 agencies that had implemented panel surveys were asked if they used a custom panel developed for their specific needs or purchased an existing panel from a vendor. Nine agencies developed a custom panel, while only one had used an existing panel from a vendor (see Figure 4). The Tri-County Metropolitan Transportation District of Oregon (TriMet) used a vendor-provided panel represent- ing the Portland metropolitan area to determine if a panel survey could be used instead of the annual telephone track- ing study. See the TriMet Panel Survey Experience profile for details on TriMet’s experience with using an existing vendor panel. using customer e-mail lists from all departments in the agency. Recognizing that existing e-mail lists may not be representative of the actual customer base, the agency developed a wide variety of methods to encour- age customers to visit the website, take the survey, and become part of the panel. Panel membership is open to all customers of NJ TRANSIT bus, rail, and light rail service, and Access Link, NJ TRANSIT’s ADA para- transit service. For each wave of the survey, existing panel mem- bers are sent an e-mail asking them to complete that quarter’s customer satisfaction survey. In addition, the following techniques are used to encourage cus- tomers to visit the website, complete the survey, and become part of the panel: A public relations campaign publicizes the survey in media; the agency website features the survey link; posters are placed on buses; flyers are placed on the seats of trains; Access Link places announcements on the telephone reserva- tion system; business cards are handed out at sta- tions and on board vehicles with the survey invitation; and those without Internet access are encouraged to go a free public access resource, such as the local library in their community, to complete the sur- vey and become part of the panel. Even with these efforts, NJ TRANSIT recognizes that the panel will not represent all customers. The panel survey data are supplemented with data collected in the field by NJ TRANSIT employees, who survey customers using tablet computers. There are approximately 16,000 customers taking the survey each quarter. Detailed origin and destina- tion survey data are used to weight the customer sat- isfaction data, based on ridership counts by mode and geographic market. The final sample is slightly skewed toward the peak period commuter, but is sufficiently representative to be used as quantitative findings rather than being limited to qualitative analysis. Tasks In-House No. Consultant No. Both No. Not Applicable No. Recruited the panel 3 5 2 0 Maintained the panel 5 3 2 0 Developed the questionnaire 6 1 2 1 Implemented the survey 4 4 1 1 Conducted the analysis and reporting 5 1 4 0 n = 10. TABLE 9 WHO COMPLETED EACH OF THE FOLLOWING TASKS ON YOUR MOST RECENT PANEL SURVEY? TriMet Panel Survey Experience Every year, TriMet conducts a 20-minute telephone survey of the general population in the Portland, Oregon, metropolitan area to track attitudes and awareness of TriMet services, customer satis faction, and other agency tracking measures. Telephone sur- veying has become increasingly expensive, and with the increase in cell phone usage and corresponding decrease in households with landlines, some demo- graphic populations are difficult to reach. Online panel surveys were looked at as a lower-cost research technique that could replace the telephone survey.

28 Satisfaction with Panel Survey Effort Half of the agencies (five) reported being very satisfied with their panel survey research, with one other being satisfied. One agency was neutral on its experience. One agency was dissatisfied, whereas two were very dissatisfied. The survey did not ask the respondents for details as to why they gave that satisfaction rating (see Figure 5). SUMMARY OF SURVEY RESULTS Survey results and conversations with transit agencies indi- cated that the use of market research panels is in its infancy in the transit industry. Every effort was made to contact all transit agencies that were conducting panel survey research to invite them to participate. However, survey results show that very few transit agencies have used panel survey techniques, and only ten agencies have completed a panel survey. At the same time, nine agencies are considering or in the process of developing a panel survey, indicating that this technique is of increasing interest to the transit industry. Among agencies that have considered or conducted panel research, the primary benefits cited were the ability to con- duct research on short notice and the ability to target specific markets. Other key benefits were the ability to track changes in attitudes or behaviors of the same persons over time, and that it is faster and cheaper than traditional research. The primary concern with panel research was ensuring representation of the target population. Other key concerns were panel attrition and lack of funding/staffing to maintain the panel. Almost all agencies had collected rider information through their panel surveys. Other topics addressed marketing, attitudes of the general public, rider input on transit planning issues, and effects of agency action. Five major tasks for conducting panel research were iden- tified (panel recruitment, panel maintenance, questionnaire 1 9 Used an existing panel from a vendor Developed a custom panel to meet specific agency goals FIGURE 4 For your most recent panel survey, did you use a vendor’s existing panel or develop a custom panel (n = 10)? 5 1 1 1 2 0 1 2 3 4 5 6 Very Satisfied Satisfied Neutral Dissatisfied Very Dissatisfied N um be ro fA ge nc ie s FIGURE 5 How satisfied has your agency been with its panel survey efforts (n = 10)? TriMet contracted with its telephone vendor to conduct an online panel survey in parallel with the household telephone survey. The panel was an existing group recruited by the vendor to mirror the popu lation within the TriMet service district. The telephone question- naire was used for online surveying, with minor modi- fications to adapt to the online format. Results of the online survey were disappointing. Although the target sample was the general popula- tion if the Portland metropolitan area, the respondents were highly skewed toward a specific demographic. Weighting the data to remove the bias would have been expensive and time-consuming, and without the weighting, the data were clearly erroneous. As a result, it was determined that the best use of the methodology would be to contact specific target markets. Once the methodology becomes more reliable, further use of an online panel would be considered.

29 development, data collection, and analysis and reporting). The survey also investigated the split of responsibilities between agencies and vendors. There was no consistency on the how vendors were used—in some cases, vendors were used for all tasks of the panel study, and in others, not used at all. Only one agency used an existing commercial online panel; the other nine agencies developed in-house panels to meet specific agency goals. Agencies with several years’ experience were more satis- fied with panel research than those who had limited, recent experience. This indicates that there may be a learning curve for agencies to become familiar with the technique, under- stand when and why to use a panel, and ensure the panel is set up properly to achieve the maximum benefit. The results of the survey were used to identify case examples. Three agencies, Metro Washington COG, NJ TRANSIT, and TriMet were developed into agency pro- files, shown in this chapter. Four additional agencies were used as detailed case examples, which are provided in chapter four.