Improving Access and Management of Public Transit ITS Data (2022)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

1   With the proliferation of automated vehicle location (AVL), automated passenger counters (APCs), and automated fare collection (AFC), transit agencies are collecting increasingly granular data on service performance, ridership, customer behavior, and financial recovery. While granular intelligent transportation systems (ITS) data can meaningfully improve transit decision-making, transit agencies face many challenges in accessing, validating, storing, and analyzing these data sets. These challenges are made more difficult in that the tools for managing and analyzing transit ITS data generally cannot, at this point, be shared across transit agencies because of variation in data collection systems and data formats. Multiple vendors provide ITS hardware and software, and data formats vary by vendor. Moreover, agencies may employ a patchwork of ITS that has been acquired and modified over time, leading to further consistency challenges. Standardization of data structures and tools can help address these challenges. Not only can standardization streamline data transfer, validation, and database structuring, it encourages the development of analysis tools that can be used across transit agencies, as has been the case with route and schedule data, standardized in the General Transit Feed Specification (GTFS) format. This report proposes a data structure for storing data from bus and rail ITS, including vehicle locations, passenger counts, and fare collection information. It also describes how the data structure can facilitate a process in which transit agencies receive ITS data from vendors, orga- nize and validate it, and use it to calculate key performance indicators (KPIs). To support that data flow, this report describes requirements that transit agencies, researchers, and consultants can use to develop tools to transform, validate, and analyze ITS data using the data structure. Motivation Transit ITS data can significantly improve public transit system performance by pro- viding detailed and up-to-date information about service performance, ridership, customer behavior, and financial recovery. Yet many transit agencies lack the management and analysis tools to leverage this data. A common structure for ITS data will provide value to the transit industry by: • Promoting open-source and third-party tool development to support transit agency functions including service planning, scheduling and operations, asset management, S U M M A R Y Improving Access and Management of Public Transit ITS Data

2 Improving Access and Management of Public Transit ITS Data financial planning, customer information provision, and perfor- mance monitoring and reporting. • Enabling transit agencies to share data support protocols and prac- tices, such as data dictionaries and documentation, data validation and cleaning practices, and privacy protocols. • Allowing transit agencies to more easily generate accurate KPIs to support improved planning, reporting, and communication. Research Process The research process included a review of existing and past efforts and initiatives and collaborative engagement with stake- holders to ensure that the work products support industry needs. Research components included: • A review of literature, reports, and existing transit data standards. • Twenty interviews with transit agency staff, ITS vendors, and other stakeholders. • A workshop with more than 50 attendees to collect input on the research approach. • Testing workshops with three transit agencies to assess how the pro- posed data structure and tool requirements could support agency data management processes. Research Findings The main findings regarding ITS data needs from the research process are: • Transit agencies see significant value in ITS data, but management and usage practices vary widely. Some transit agencies have highly developed management practices for ITS data, while others have very limited in-house data capabilities, relying primarily on vendor- prepared reports. Of the transit agencies with more advanced data practices, many have multiple processes for extracting, cleaning, validating, and analyzing transit ITS data, designed to meet different goals that arose over time. • Transit agencies, ITS vendors, and stakeholders identified many common challenges related to transit ITS data management, including consistency across data sets, validation, cleaning, security, and privacy. • Based on workshop polling, the most important use cases for transit ITS data include (1) analysis related to boardings and alightings, such as ridership and load analysis, and (2) analysis related to travel times, such as runtime, dwell time, and on-time performance. The research also identified the following factors as key to the successful development of an ITS data standard: • Flexibility in deployment. A successful data standard should also have the flexibility to evolve to accommodate changing services and data types. • Collaborative end-user-oriented development. Successful data standards are developed through collaboration, with stakeholders and researchers identifying the most important use-case(s) to demonstrate the value of the standard and to guide the development process. National Transit Database (NTD) Reporting Congress established public transit data reporting requirements for federal funding recipients in 49 U.S.C. 5335(a) and (b) and created the NTD as a data repository. The Federal Transit Administration (FTA), which manages the NTD, has been collecting data from NTD reporters for over 35 years. The FTA developed requirements for APC system certification to enable transit agencies to use those systems for reporting. Those requirements were updated in 2016, a process that involved public stakeholder comment through the Federal Register. The FTA allows multiple methods for APC certification. While adoption by a transit agency of the proposed data structure would not directly result in FTA-compliant NTD ridership reporting, this report does provide a draft specification for APC data storage. Building on this specification, vendors, consultants, or researchers can develop tools for NTD reporting, helping transit agencies meet this reporting requirement with less effort.

Summary 3   • Consideration of cost and effort of implementation, including how the data standard inte- grates with transit agency operational procedures. Overcoming costs and effort is critical for initial adoption. In the longer term, if widespread adoption occurs, technology can be expected to become compatible with the standard. • Support for KPIs. The potential to leverage open-source tools that support key performance indicator (KPI) reporting and communication with transit passengers would encourage adoption. • Alignment with the GTFS. Widely adopted, GTFS provides many benefits to transit agencies. Developing a standard that builds on or is consistent with this standard is expected to improve adoption. In addition, input from stakeholders and experience with existing data standards reveals the following conditions for successful standard adoption: • Ongoing advocacy and education to support broad standard adoption across the transit industry. • Compatibility with and understanding of the procurement process and procurement cycles. These cycles are expected to significantly impact adoption. • Governance to oversee the evolution of the specification. Governance is an important component of the future of any potential transit ITS data standard. Key Elements The data structure is designed to manage AVL, AFC, and APC data from fixed-route public transit. It is focused on historic data for internal use, with a primary objective of pre- paring the data so that it can be used to estimate KPIs. In particular, the structure aims to support the estimation of ridership and travel time (on-time performance, runtime, dwell time) KPIs. This report also includes tools requirements for the Format Validation, Data Quality, Data Transfer, and Data Analysis tools. Currently, most transit agencies receive ITS data from private vendors. The level of aggre- gation and format of this data varies across agencies. The data structure and related tool requirements are intended to support the management of data both for transit agencies that receive very detailed data (for example, heartbeat vehicle location data, individual time- stamped boarding records) as well as agencies that receive more processed data (such as data assigned to a trip and stop or summarized at a station by time period). Figure 1 depicts the proposed data flow. Figure 1. Data flow. Note: Box colors in the diagram reflect different data file types. The colors are used consistently throughout this report.

4 Improving Access and Management of Public Transit ITS Data Data originates from ITS hardware and software, which are most commonly provided by vendors. The vendors share data (Vendor Outputs in the first blue box) with transit agencies. Vendor Outputs may consist of summary data (such as total boardings at a stop) or may include detailed time-based event data (such as individual timestamped boardings). Some transit agencies may also collect time-based event data from some systems (e.g., AVL) but not others (e.g., AFC) or from some modes (e.g., bus) but not others (e.g., light rail). Given this variation, transit agencies may follow a combination of two different paths through this data flow, dependent on what type of data they receive as Vendor Outputs: • Path if using Event Data Files: The first path, depicted with the solid blue arrow is designed for an agency that receives time-based event data from their vendor(s) and desires to maintain that data in a common format. In this case, the vendor-specific outputs are transformed into the Event Data Files. Event Data Files consist of timestamped event data. In these files, each row refers to a specific event, and there may be multiple events associated with a single stop visit. Alternatively, agencies may choose to maintain existing data formats tailored to agency needs and then use agency-specific tools to transform these files into the Summary Data Files, which they then would use to evaluate the KPIs. For transit agencies that opt to use the Event Data Files, the Data Transfer Tool transforms the Event Data Files into the Summary Data Files. Before the transfer, the data flow includes the application of the Format Validation and Data Quality Tools to ensure that the data provided is in the correct format and that unique identifiers across internal and external reference files match. • Path if using only Summary Data Files: The second path, shown with the dotted blue arrow, can be used by transit agencies that do not receive detailed time-based event data or do not want to transform their Vendor Outputs into the Event Data Files. In that case, the Vendor Outputs are transformed into the Summary Data Files. The Summary Data Files are summarized in three different ways: for each vehicle stop visit, for each vehicle trip, and for each station and time period (for data not associated with a vehicle). Not only do these files reflect the type of data that many transit agencies receive from vendors, but they also organize the data in such a way that it can support the evaluation of KPIs. Transit agencies may opt to use some but not all Event Data Files. In this case, they would follow the first path for some data types and use the second path for other data types. Regardless of the initial path, after the Summary Data Files are produced, the Format Validation Tool and Data Quality Tool are applied to the Summary Data Files to ensure formats are correct, IDs are unique and consistent, and questionable and missing data is flagged. The analysis tools are then applied to analyze the data to generate the KPI Reports. This report includes requirements for the Format Validation, Data Quality, Data Transfer, and Data Analysis tools. Integration with GTFS The data structure provides consistency and interoperability with GTFS by: • Encouraging the use of GTFS identifiers (such as stop_id, stop_ sequence, and trip_id), and • Incorporating GTFS schedule information into the data structure to make connections between observed data and the scheduled system. Note on Implementation: Transit agencies may process Vendor Outputs into the Event or Summary Data Files themselves, or they may request that vendors do the processing. Third parties may also develop tools for this processing; however, Vendor Outputs typically are specific to each agency, which may limit the potential for shared tools. In future procurement processes, transit agencies could require the proposed file formats from vendors. Over time, vendors may adapt their tools and methods to produce data in the proposed formats.

Summary 5   Adoption, Governance, and Extensions The success of the data structure and related tool requirements will be primarily judged on their adoption. As noted, adoption is expected to depend on several aspects: • Benefits to transit agencies. The most notable intended benefits are the ability to estimate KPIs more easily through shared tools that support validation and cleaning of standardized data and the generation of KPI reports. • Costs to transit agencies. Adoption relies on minimizing the costs to transit agencies of converting their data into the data structure and maintaining the data over time. This may require supporting tools as well as funding to help transit agencies process their custom data sets. Incorporating the data structure into future procurement cycles will minimize this cost, but given the length of procurement cycles, this is expected to be a gradual process. • Good governance. As a new data structure is adopted as a standard, it is important to have clear leadership and authority to champion it and guide future updates. Governance is important to providing stability and consistency while also enabling systematic development and adop- tion of future extensions to the data structure to meet transit agencies’ evolving needs. The following are potential paths to support these needs. Tools to Materialize Benefits This report proposes requirements for Format Validation, Data Quality, Data Transfer, and Data Analysis tools intended to enable the data structure to support KPI estima- tion. If these tools are developed, they will incentivize transit agencies to adopt the data structure to leverage these tools. These tools could be developed by consultants, ITS vendors, or researchers. Funding for tool development will be critical. While this funding could come directly from transit agencies or groups of transit agencies (such as through pooled fund studies), financial sup- port from established institutions will likely be important. This could include a follow-on TCRP implementation effort, or funding from a standards development organization such as APTA or AASHTO. Processes to Minimize Costs There is an upfront cost to transit agencies to convert their existing Vendor Outputs or internally defined data structures to the proposed structure. Unlike the standard tools discussed in the prior section, this effort is likely to require customized support for individual transit agencies, as each transit agency currently has an individualized method for storing data. Therefore, researchers, consultants, or ITS vendors may be required to work directly with individual agencies to support this process. Funding pilot efforts or implementation studies through TCRP, APTA, AASHTO, NCHRP, or another organiza- tion would help support this process. Funding could even be offered directly to transit agencies who desire to complete the process internally or manage it through their own procurement processes. Another component of this effort could be the development of guidance for incorpo- rating the data structure into procurement processes. Establishing a community of prac- tice, consisting of individuals or organizations interested in learning about and following developments of the data structure, can also support transit agencies seeking to adopt the structure.

6 Improving Access and Management of Public Transit ITS Data Governance Models Successful governance models typically build on existing institutions and groups. This could include: • Proposing the data structure as an extension to GTFS and adopting the GTFS governance structure. • Establishing a governance group that is associated with an institution such as APTA or AASHTO. • Leveraging an existing community such as Transit ITS Data Exchange Specification (TIDES) Google group (see: https://groups.google.com/g/tidesproject) or the Mobility Data Slack group (see: https://mobilitydata-io.slack.com/) to develop a governance group and to provide a forum for communication about the data structure.