State DOTs Connecting Specialized Transportation Users and Rides Volume 1: Research Report (2016)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

24 C H A P T E R 5 Few of the systems contacted in conjunction with this study had considered—or were required by the funding source to identify—how the systems, once implemented, would be deemed a success. Others included only rudimentary evalu- ation criteria such as website hits. The exceptions included a few systems that were designed with a detailed set of evalua- tion criteria in mind; that is, the designers of the systems used the evaluation criteria to “inform” the design of the linkage system. Some of these, also representing the few that have been implemented and are in use, have been included in this report as best-practice case studies, noting that the evaluation crite- ria and processes among these few were quite different. For example, some providers utilize software tools that automati- cally tabulate a wide range of performance data, while others focus on a select number of criteria analyzed through manual collection, such as follow-up phone calls or customer surveys. This chapter provides details regarding evaluation criteria currently used by linkage providers, as well as other potential criteria that could provide important insights into program effectiveness. As stated previously, linkage programs provide a wide range of services using numerous customer inter- action techniques. The services and methods used by each provider affect the applicability of different evaluation cri- teria and collection methodologies. To distinguish between applicable criteria, this section is divided by the continuum of linkage levels introduced in Table 1. An overview of eval- uation criteria by linkage level is shown in Table 4. Level 1: Central Repository Agencies relying primarily on hard-copy or online direc- tories have limited options for evaluating the successfulness of their program. Evaluation criteria primarily focus on the scale of directory distribution. Agencies using hard copies can track the number of directories distributed by staff and the locations or types of organizations that received directo- ries. Staff can also track the number of individuals or orga- nizations that request directories and the demographics of those making the requests. This data can be used to evaluate whether directory distribution has successfully reached spe- cialized transportation riders within a jurisdiction. Agencies with online inventories can use web tools to develop an overview of who views and downloads the special- ized transportation directory. For example, Google® Analytics uses a JavaScript-based tracking code to automatically gener- ate information about users visiting a web page. The software tracks the number of times the web page is accessed as well as how users initially found and accessed the page. Google Analytics can determine how a user interacts with site content, including the amount of time a user spends on each page. As the software learns more about the user, Google can also develop basic demographic information. Web page admin- istrators can access automatically generated reports through an online application. Google Analytics can be used by linkage providers to help understand who accesses online specialized transportation directories. If directories are distributed as a single static PDF file, the software will enable providers to track the number and basic demographic information of users who download a directory. If directories include different pages for each ser- vice provider or type of service, Google Analytics can provide valuable information about which services or providers are most accessed by local customers. This information can be used to help determine whether existing services are meeting customer needs as well as help prioritize pages that should be more frequently updated. Level 1A: Provider Portal The concept behind agencies providing online provider portals is to improve the accuracy of information (without devoting significant staff time) by allowing service providers Evaluation

25 to directly update contact, fare, and regulations information. Evaluation criteria therefore focus on how frequently pro- viders check and/or update information about their services on the portal. Agencies can track which providers frequently access and update their information and which providers either have not submitted or fail to update information about their services, noting that providers with no changes may not need to update their information. Many providers likely maintain similar services over long periods of time, while others frequently change service areas, fares, and other features. Due to this variation, agencies can- not expect all providers to update their portal information at a consistent rate. In order to understand the validity of the information on their portal, agencies should ideally conduct audits of provider pages. These audits would allow agencies to track the percentage of provider pages that contain accu- rate information. Level 2: Matching Assistance The One Call/One Click process allows agencies to col- lect significantly more information about linkage customers and their trip requests. Agencies can therefore develop more advanced evaluations of how successful their programs are at linking riders with services. When customers call a mobility management specialist or access an online portal, they are asked a series of triage questions as well as information about themselves and their trip. Questions typically focus on type of disability, needed accommodations, and eligibility for certain types of services, as well as the specifics of the trip needed. One Call/One Click allows agencies to track how users respond to these questions. Agencies can also track the number of riders who seek gen- eral information about services in their area versus those that request information about a specific trip. Using this infor- mation, agencies can better understand who utilizes linkage services and whether their needs are satisfied by provider ser- vices in the database. Level 3: Trip Planning Assistance By providing trip planning assistance, agencies can learn more about the ways customers use a linkage to find providers that serve their travel needs. Customers provide informa- tion about their trip origin and destination, the date and time of day they wish to travel, and trip purpose. Using this data, along with answers to triage questions, users are pro- vided with specific transit trip information or shown pro- viders that operate applicable services. Agencies can track trip information to help identify common trips and analyze what types of transportation services customers select for their itineraries. Agencies can also cross tabulate trip, triage, and itinerary data to see how different factors affect which services are applicable to customer requests. This analysis is particularly important for understanding which customers are unable to find an applicable trip and how service changes or new services may reduce the number of requests that cannot be completed. Online one-click portals enhance the opportunities to eval- uate the effectiveness of linkages. For example, the 1-Click/CS software includes a number of automatically generated evalu- ation reports that allow participating agencies to understand how customers are using the linkage service. Administrators Table 4. Evaluation criteria by level of linkage functionality. Level Name Functionality Evaluation Criteria 1 Central Repository Centralized repository of data  Number of hard copies requested or distributed  Number of website hits/directory downloads  Google® Analytics/web activity tracker 1A Provider Portal + provider portal  Add number of direct updates  % of providers with updated information 2 Matching Assistance + ways to narrow down service and program options  Number of registered clients  Number of general requests  Number of specific trip requests 3 Trip Planning Assistance + trip planning assistance  Number of specific requests for trip planning  Selection of an option for further trip planning 4 Trip Booking Assistance + trip booking by mobility specialist  Number of trips booked by mobility specialists  Number of booked trips made (needs follow- up) 5 Direct Trip Booking + trip booking by customers  Number of trips booked directly by customers (Number of trips scheduled onto partners’ vehicles)  Number of booked trips served

26 can generate reports on the number of trips generated as well as mode splits, trip information, and answers to triage information. The software also analyzes the platforms riders use to connect to the service. Most of these variables can be cross tabulated, allowing agencies to understand how dif- ferent factors affect how riders use transportation services. 1-Click/CS also collects feedback from users on how closely generated trip information matched their needs. All evalua- tion reports can also be exported to CSV format. Cambridge Systematics works with clients to develop new reports and regularly integrates new evaluation features accessible to all 1-Click/CS customers. Level 4: Trip Booking Assistance Linkage services that provide trip booking can obtain more precise information about the trips taken by their customers. At a basic level, agencies can track the number of trips booked, which services trips are booked on, and general trip informa- tion. Staff can also track how closely a booked trip matches the itinerary initially requested by the customer. This information can be used to identify services gaps within existing services as well as to identify potential new services that will more closely match rider requests. Level 5: Direct Trip Booking The trip booking process enables agencies to more easily track trip planning outcomes. Linkage providers are involved in almost all aspects of the trip planning process, from the initial triage screening to scheduling a pick-up and drop-off. In addition to tracking trip information and triage variables, agencies can track the number of successfully booked trips and the percentage of providers that allow direct trip booking. On systems that have multiple providers that can serve some similar trips, agencies can track factors that affect which ser- vice a customer chooses. This information can be valuable for agencies that are attempting to transition a portion of their ridership base to a different service. Example Evaluation Criteria The most advanced methods on evaluation criteria—and how that influenced the design of the system and collection of data—comes from a loosely knit consortium of VTCLI grant recipients from Atlanta; Broward County and Jacksonville, Florida; Pennsylvania; and Riverside County, California, each of which developed a platform based on the 1-Click/CS software. Table 5 shows an example of evaluation data collected through the 1-Click/CS software from the inception of the systems through July 2015. Each of the four providers listed has used the software in somewhat different ways and for different durations. For example, as discussed in Chapter 2, PennDOT has used FindMyRidePA as a portal for booking paratransit services and therefore can evaluate the total num- ber of bookings made through the service. The Jacksonville Transit Authority, alternatively, used the software to allow customers to view how a major service change would affect their transit trip itineraries. This usage is reflected in a high rate of transit itinerary requests. Note that the PennDOT sys- tem, FindMyRidePA, is the only one of the four that allows customers to directly book trips (Level 5). JTA’s TransPortal system is about to implement Level 5 functionality.

27 Table 5. 1-Click/CS sample evaluation criteria. A R C Si m pl y G et T he re IE U W 21 1 Ve tL in k JT A Tr an sP or ta l PA Fi nd M yR id eP A Agencies 8 6 15 3 Providers 37 45 27 6 Services 38 53 31 8 Total Users* 570 417 1675 799 Registered Users 65 42 110 187 Logins by Registered Users 361 472 643 770 Trips 988 1086 3444 1710 Desktop 686 945 2304 1093 Tablet 32 19 161 41 Phone 24 46 785 117 Kiosk – – – 295 Itineraries 12577 7694 26125 10003 Bike 79 201 146 – Carpool 1004 – – – Drive 81 323 142 – Paratransit 6080 1919 4231 3885 Transit 5333 5251 21606 6148 Selected Itineraries 420 164 950 1321 Bike 4 0 2 – Carpool 40 – – – Drive 2 0 6 – Paratransit 113 20 118 831 Transit 261 144 824 490 Bookings – – – 315 *User has generated at least one trip. Source: 1-Click/CS Sample Evaluation Criteria. 8 July 2015. Raw data. Cambridge Systematics, Boston.