The National Academies Press

Currently Skimming:

Pages 11-52

The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.

From page 11... ... 11 This Roadmap to big data represents an organic, bottom-up approach for transportation agencies that relies on an iterative process to grow big data use cases, pilot projects, and ultimately value for an organization. An agency may embark on this Roadmap and guidance for various reasons. Read the entire page →
From page 12... ... 12 Guidebook for Managing Data from Emerging Technologies for Transportation • Step 3. Secure buy-in from at least one person from leadership for the pilot project. Read the entire page →
From page 13... ... Roadmap to Managing Data from Emerging Technologies for Transportation 13 organization. This vertical organizational outreach will not only help to market the value of the data and the data products to executive management to identify other use cases that can be developed within the test environment but also begin to gain executive support for organizational change. Read the entire page →
From page 14... ... 14 Guidebook for Managing Data from Emerging Technologies for Transportation were also included in the Waze data, they decided to add the Waze data into the snow-and-ice system to see what would happen. Two months later, Kentucky received the first snow of the season. Read the entire page →
From page 15... ... Roadmap to Managing Data from Emerging Technologies for Transportation 15 Step 1. Develop an Understanding of Big Data The first and most crucial challenge in building a modern data system is overcoming a lack of knowledge about big data. Read the entire page →
From page 16... ... 16 Guidebook for Managing Data from Emerging Technologies for Transportation "We have so much data and so much technology giving us data that there is no human that can keep up with it." -- Delaware DOT The last definition is an acknowledgment that storing data in "data silos" has been the key obstacle to getting the data to work in ways to improve businesses, work, and lives. Big Data Characteristics Most people have encountered the five characteristics or "Vs" of big data: volume, variety, velocity, veracity, and value. Read the entire page →
From page 17... ... Roadmap to Managing Data from Emerging Technologies for Transportation 17 type, and processing of the data are (see Veracity 2019) Read the entire page →
From page 18... ... 18 Guidebook for Managing Data from Emerging Technologies for Transportation • Distributed storage. Similar to distributed computing, distributed storage is the technique of storing large amounts of data on a distributed network or cluster of drives/servers. Read the entire page →
From page 19... ... Roadmap to Managing Data from Emerging Technologies for Transportation 19 Common Misconception: Transportation Agencies Only Need to Purchase More Servers to Store Big Data Despite what the name may imply, big data is not a larger version of traditional data. Rather, big data are so radically different from traditional data that they cannot adequately be collected, stored, or analyzed using traditional techniques. Read the entire page →
From page 20... ... 20 Guidebook for Managing Data from Emerging Technologies for Transportation Additional Resources Suggested additional resources for review follow. An Introduction to Big Data Concepts and Terminology. Read the entire page →
From page 21... ... Roadmap to Managing Data from Emerging Technologies for Transportation 21 This paper also identified two additional, broad areas where big data analytical approaches may be able to provide further value, including transportation system monitoring and management and traveler-centered transportation strategies (Burt, Cuddy, and Razo 2014) Read the entire page →
From page 22... ... Drivers for Change Example Big Data Sources Example Use Cases/Pilot Projects An agency faces an issue or problem that requires new data and new methods, as the issue or problem cannot be addressed easily or efficiently with the current systems and data alone. Crowdsourced data: Crowdsourced data generated through mobile apps such as Waze can help address transportation issues or problems easily and efficiently. Read the entire page →
From page 23... ... Roadmap to Managing Data from Emerging Technologies for Transportation 23 • Aligns with or links to organizational goals or objectives. Beyond the needs of the leadership of the business unit, select a pilot project that aligns with, or links to, the overall goals, objectives, or mission of the organization. Read the entire page →
From page 24... ... 24 Guidebook for Managing Data from Emerging Technologies for Transportation • Internal, cross-business unit. Other business units may also have an interest in or use for the data at hand and may be willing to support the pilot test or even offer up potential or future use cases for the data. Read the entire page →
From page 25... ... Roadmap to Managing Data from Emerging Technologies for Transportation 25 anecdotal benefits from other transportation agencies to support the argument. Table 3 lists various example projects, along with their value propositions and associated questions to assist in further developing the pitch. Read the entire page →
From page 26... ... 26 Guidebook for Managing Data from Emerging Technologies for Transportation used by traffic engineers across Louisville to identify signal-timing needs. The project inspired the expansion of the concept and working with multiple data providers to enhance the signal timing of the other 1,000+ signals across the city. Read the entire page →
From page 27... ... Roadmap to Managing Data from Emerging Technologies for Transportation 27 experiences from which one is absent. A strong case can be made by establishing that some agencies have implemented a similar project/approach with success. Read the entire page →
From page 28... ... 28 Guidebook for Managing Data from Emerging Technologies for Transportation Step 4. Establish an Embryotic Big Data Test Environment After gaining support from leadership for the pilot project, the next step is to establish an embryotic big data environment. Read the entire page →
From page 29... ... Roadmap to Managing Data from Emerging Technologies for Transportation 29 environment are of an unpredictable, elastic nature, and that they can peak to levels that are higher than all the computational capabilities found in the organization for a few hours, will be needed. Preparing an estimate of data access needs ahead of time will also make it easier for the IT team to understand and compare the costs involved. Read the entire page →
From page 30... ... 30 Guidebook for Managing Data from Emerging Technologies for Transportation • Resource concerns originating from not knowing the amount of resources needed to establish the environment or how much it will really cost in addition to maintaining the current infrastructure. • Resistance to establishing a new data environment when there is already an infrastructure in place with people maintaining it. Read the entire page →
From page 31... ... Roadmap to Managing Data from Emerging Technologies for Transportation 31 Once the storage service is acquired, data of interest need to be moved to the playground storage. This can sometimes be a problem, as some organizations do not trust that cloud services can store data securely, especially when data are, or are perceived to be, sensitive. Read the entire page →
From page 32... ... 32 Guidebook for Managing Data from Emerging Technologies for Transportation Again, as with the playground storage layer, the processing layer should be able to support multiple data analysis tools as needed to explore the data in the storage layer. Providing a relational database or traditional statistical software to users in the playground will not suffice, as they will not provide the analytical capabilities needed to process the large data sets in parallel or the unstructured and semi-structured data sets stored in the storage layer. Read the entire page →
From page 33... ... Roadmap to Managing Data from Emerging Technologies for Transportation 33 By implementing such a big data test environment, agencies will be able to safely explore new data and achieve many benefits without massive outlay. By investing in the development of team knowledge, giving them access to the type of data environment and responsibilities typically under the authority of IT departments, they will also start to develop a data understanding from within the organization that can begin to foster a culture of data. Read the entire page →
From page 34... ... 34 Guidebook for Managing Data from Emerging Technologies for Transportation Step 5. Develop the Pilot Project Within the Big Data Test Environment/Playground In Step 5, the champions and team will work within the big data environment established in Step 4 to develop the pilot project. Read the entire page →
From page 35... ... Roadmap to Managing Data from Emerging Technologies for Transportation 35 Develop the Project Applying a Data Science Perspective Given the overall goals of the pilot project, it is important that the team approach the project from a data science perspective (as opposed to a more traditional approach) , that is, extracting value from the data. Read the entire page →
From page 36... ... 36 Guidebook for Managing Data from Emerging Technologies for Transportation • For answers, what level of confidence would the team be happy with? • Can the team formulate hypotheses relating to these questions? Read the entire page →
From page 37... ... Roadmap to Managing Data from Emerging Technologies for Transportation 37 As such, this processing and cleaning can take a long time and can be relatively tedious work, but the results are well worth the effort. This step includes (Turner 2019) Read the entire page →
From page 38... ... 38 Guidebook for Managing Data from Emerging Technologies for Transportation Build a Data Science Pipeline This step focuses on developing a prototype analytical pipeline based on the findings of the previous steps. A data science pipeline is a sequence of processing and analysis steps applied to data for a specific purpose. Read the entire page →
From page 39... ... Roadmap to Managing Data from Emerging Technologies for Transportation 39 at start time and dismantled at stop time, leaving many opportunities to update them between runs without ever being perfected. Iteratively Develop/Improve the Project and Associated Outputs The development of the project is likely to be more successful if the project is treated as an iterative process with multiple feedback loops and revisions taking place throughout the development process. Read the entire page →
From page 40... ... 40 Guidebook for Managing Data from Emerging Technologies for Transportation Case Study: Negotiating Technical Contracts for Data Services (Continued) At the time the partnership agreement was negotiated, few employees at LA Metro had any experience with highly technical IT contracts and had to work closely with legal counsel to fully understand and navigate all the details. Read the entire page →
From page 41... ... Roadmap to Managing Data from Emerging Technologies for Transportation 41 Step 6. Demonstrate the Value of the Data to Other Business Units Once the project has developed to a point where it generates real value for the business unit, it is time to share it with others outside the business unit. Read the entire page →
From page 42... ... 42 Guidebook for Managing Data from Emerging Technologies for Transportation When seeking buy-in from other business units, create a storyline for the project that includes the following: • Introduction (characters and setting are introduced) Read the entire page →
From page 43... ... Roadmap to Managing Data from Emerging Technologies for Transportation 43 agencies may iterate through multiple projects before they are ready. The structure, culture, and relationships within each agency will largely drive this decision. Read the entire page →
From page 44... ... 44 Guidebook for Managing Data from Emerging Technologies for Transportation Case Study: Iterative Success and Growth (Continued) quickly gained attention and support from leadership. Read the entire page →
From page 45... ... Roadmap to Managing Data from Emerging Technologies for Transportation 45 Step 7. Demonstrate the Value of the Data to Executive Leadership After one or more iterations of Steps 2 through 6 and growing the number of use cases and pilot projects within the big data environment, it may be time to begin marketing the data, environment, results, outputs, and benefits to higher-level executives within the agency. Read the entire page →
From page 46... ... 46 Guidebook for Managing Data from Emerging Technologies for Transportation Common Misconception: Data Owners Have Less Control Over Their Data After Uploading the Data to a Data Lake Business unit siloes, and associated data siloes, are and will continue to be a barrier to the ability of transportation agencies to leverage data from emerging technologies. Rigid processes around how data are managed and analyzed is a traditional approach to data management that was developed to facilitate fast query speeds and low error rates when working with relational database management systems. Read the entire page →
From page 47... ... Roadmap to Managing Data from Emerging Technologies for Transportation 47 Figure 8. Iterative process to generating interest and buy-in vertically within the organization. Read the entire page →
From page 48... ... 48 Guidebook for Managing Data from Emerging Technologies for Transportation Step 8. Establish a Formal Data Storage and Management Environment Step 8 ends the focus on individual pilot projects and the test environment and begins the establishment of organization-wide big data management. Read the entire page →
From page 49... ... Roadmap to Managing Data from Emerging Technologies for Transportation 49 First, business areas that are the easiest to migrate and that boast the most supportive and enthusiastic teams should be shifted. Then, more difficult business areas should be progressively migrated. Read the entire page →
From page 50... ... 50 Guidebook for Managing Data from Emerging Technologies for Transportation 3. Integration at the data level. Read the entire page →
From page 51... ... Roadmap to Managing Data from Emerging Technologies for Transportation 51 an aging piece of technology to a newer piece of technology. This could entail replacing a part of the data pipeline with a more efficient process or updating data analysis/visualization tools with more feature-rich options. Read the entire page →
From page 52... ... 52 Guidebook for Managing Data from Emerging Technologies for Transportation Case Study: Continued Room for Growth While many organizational units within KYTC have started to understand and experience the benefits of big data architecture, the system still has much room to grow before all the benefits can be fully realized. The democratization of data and end-user engagement remains a barrier to this day, and the reasons are complicated, as with any new technology adopted by any agency. Read the entire page →

From page 11...

... 11 This Roadmap to big data represents an organic, bottom-up approach for transportation agencies that relies on an iterative process to grow big data use cases, pilot projects, and ultimately value for an organization. An agency may embark on this Roadmap and guidance for various reasons.