National Academies Press: OpenBook
« Previous: 7.0 Defining a Culture of Innovation
Page 18
Suggested Citation:"8.0 Guide Validation Purpose and Results." National Academies of Sciences, Engineering, and Medicine. 2018. Research on Creating and Sustaining a Culture of Innovation for Departments of Transportation. Washington, DC: The National Academies Press. doi: 10.17226/25308.
×
Page 18
Page 19
Suggested Citation:"8.0 Guide Validation Purpose and Results." National Academies of Sciences, Engineering, and Medicine. 2018. Research on Creating and Sustaining a Culture of Innovation for Departments of Transportation. Washington, DC: The National Academies Press. doi: 10.17226/25308.
×
Page 19
Page 20
Suggested Citation:"8.0 Guide Validation Purpose and Results." National Academies of Sciences, Engineering, and Medicine. 2018. Research on Creating and Sustaining a Culture of Innovation for Departments of Transportation. Washington, DC: The National Academies Press. doi: 10.17226/25308.
×
Page 20
Page 21
Suggested Citation:"8.0 Guide Validation Purpose and Results." National Academies of Sciences, Engineering, and Medicine. 2018. Research on Creating and Sustaining a Culture of Innovation for Departments of Transportation. Washington, DC: The National Academies Press. doi: 10.17226/25308.
×
Page 21
Page 22
Suggested Citation:"8.0 Guide Validation Purpose and Results." National Academies of Sciences, Engineering, and Medicine. 2018. Research on Creating and Sustaining a Culture of Innovation for Departments of Transportation. Washington, DC: The National Academies Press. doi: 10.17226/25308.
×
Page 22
Page 23
Suggested Citation:"8.0 Guide Validation Purpose and Results." National Academies of Sciences, Engineering, and Medicine. 2018. Research on Creating and Sustaining a Culture of Innovation for Departments of Transportation. Washington, DC: The National Academies Press. doi: 10.17226/25308.
×
Page 23

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

18 8.0 – GUIDE VALIDATION PURPOSE & RESULTS This validation study presents feedback from employees at DOTs who reviewed a draft version of the Guide to Creating and Sustaining a Culture of Innovation within Departments of Transportation, which was prepared as part of NCHRP Project 20-108. The guide is a resource to help DOTs evaluate where they may be on the path to an innovative culture, and provides a springboard of ideas to help them take their organizations to the next level, no matter where they are on the path. The purpose of the validation study is to ensure that information in the guide is as complete, accurate, and organized as possible to meet users’ needs. The research team gathered feedback about the guide from DOTs via three focus group sessions and an online survey. Overall, comments indicate that people are using the guide in a very practical way – as a workbook of sorts – and they would like more usability features to support that concept. They would also like supporting job aids and training tools to help spur as well as anchor innovation culture changes. Participants especially valued the self-assessment and case studies in the guide, and would like to see where peer organizations are on the maturity scale of developing a culture of innovation. There is also an interest in understanding how others have navigated political factors in building a culture of innovation and how they are motivating front-line employees, empowering employees at all levels of the organization, cutting out the noise of daily operations, and maintaining the momentum of culture change. They would like more guidance and best practices on how to evaluate and prioritize pilot projects and determine what is a good idea and what is not. Overall, there is agreement that this guide outlines important steps that will help lead to and sustain a culture of innovation. Key Findings The validation study findings are based on 20 survey responses and three focus group sessions with staff at Caltrans, Massachusetts DOT, and Nebraska DOT. Key findings include: ► The Innovation Guide provides useful information for agencies seeking to sustain innovation efforts and take their efforts “to the next level.” ► The Self-Assessment and Case Study sections in the guide generally meet reviewers’ expectations. ► Reviewers want information distilled via layout techniques such as color-coded sections that match the self- assessment, call-out boxes, charts, and bulleted lists. ► Appendices should be a separate document so the guide is a more approachable length, and the Literature Review should go in an appendix. ► Examples are helpful and should not be buried in the appendix. In particular, respondents would like to see more examples from state DOTs.

19 Focus Group Findings The research team conducted a series of hour-long focus groups to review and discuss the guide content and usability with a diverse set of DOTs, including Massachusetts DOT (MassDOT), Nebraska DOT (NDOT), and California DOT (Caltrans). They were selected to represent a diversity of geographic regions, organizational scales, and levels of maturity in developing an innovation culture. Each focus group was comprised of five to seven representatives from a single DOT, with representatives from each of the following three employee categories: 1. DOT executives and leadership - People who are responsible for setting goals and strategy for the organization. 2. Innovation team leaders - People who are tasked with leading change efforts related to building an innovative culture. 3. Influential team members - People who are not directly responsible for leading the change efforts, but have influence within the organization. These categories were identified as the target audiences for the guide. Information collected in the sessions was qualitative in nature, focusing on reactions to the content in the guide. WHAT THEY LIKED MOST Focus group question: What did you like most about the guide? How would you rate the guide on a scale of 1 to 10? (10 being the best). Most of the participants indicated that they liked the straightforward nature of the guide. The layout was “easy to follow,” they “got a lot out of it,” and it “laid out a clear path on how to move forward.” One user noted that “the guide makes it sound like putting a culture of innovation in place is easy.” Table 1 shows the average rating in each of the focus groups. Caltrans MassDOT NDOT On a scale of 1-10, 10 being the best, how would you rate the guide? 6.5* 7.3 8 Table 1: Average Focus Group Rating. *Only two of the Caltrans participants opted to give the guide a rating. The others had not read the entire guide and were uncomfortable rating it at that point. Specific Responses for Most-Liked Sections ► “The guide is intuitive and the sections were directly accessible. Information is easy to follow and the content is interrelated. The organization is good.” ► “Perfect level of depth, yet succinct with good steps forward.” ► “Helpful culture change ideas and useful stories about successes in other states.”

20 WHAT WAS MOST/LEAST HELPFUL Focus group question: What section(s) stood out to you as being helpful, and which one(s) did you find less useful or relevant to your organization? The areas rated most helpful were the Self-Assessment and Getting Started. These sections helped respondents understand how they may be progressing in the development of an innovative culture and focus on areas of the guide that would be most helpful to them. The areas rated least helpful were the case studies and some of the examples. One respondent noted that while organizations like Google are widely viewed as pillars of innovation, they wanted to hear specific case studies from DOTs. As one person noted, “the most helpful areas were the examples from the other agencies, because DOTs can better relate to each other.” Specific Responses for Most Helpful and Less Helpful Sections ► “Liked the How to Get Started section, agency examples, and quantity of easy-to-apply ideas.” ► “More information needed on risk management and addressing how DOTs are risk averse.” ► “Measuring success of programs is good and you can communicate that.” ► “Liked the lessons learned, timeline to get traction, good public/private contrast and story.” ► “Liked comparing experiences between states.” ► “Using it as a training tool would be good. The total document is overwhelming in size.” ► “Measurement is work, how much is ROI is not as valuable to innovators because it could become an impediment to culture change.” ► “DOT salary levels are a barrier to attraction of innovators.” ► “Survey in the back didn’t hit home.” ► “We want to accept failure as part of the process but it has to be acceptable down the ranks. That’s currently a barrier.” ► “Have a governance structure where leaders can evaluate and propose ideas.” ► “Good communication (is needed) so each person knows why their innovation ideas are adopted or denied.” ► “Discuss continuum of innovation idea generation − Create space for rapid incremental changes − Crowdsourcing ideas doesn’t work. Enable small groups to undertake 6–12 month efforts to recoup so that ideas are sustainable − Create a value index tool with factors like time, cost savings”

21 SUGGESTED IMPROVEMENTS Focus group question: If you could make one improvement to the guide, what would you suggest? Many respondents commented on the length and format of the report. They said that while they felt the information was useful, the 120-page document was intimidating. Several participants recommended that the appendix be a separate volume. They particularly wanted to learn more about what others are doing. They also asked to see more page layout features that would make it easy to read and highlight key information, such as call-out boxes, charts, and bulleted lists to distill the information further. Many respondents commented that they would like to take notes in the guide and would prefer more white space or wider margins for that purpose. Another suggestion was to add discussion materials into the guide that managers at different levels could use to facilitate conversations with employees about innovation and potential process improvements. Specific Responses for Suggested Improvements ► “What are other states doing for their employees?” ► “More clarity about private and public sector.” INFORMATION ON BARRIERS AND LESSONS LEARNED Survey respondents recommended adding more information about the barriers to innovation and more about some of the lessons learned from innovators. How would this information help you in applying the guide? A constant theme throughout the focus groups was that people want to hear more about what is being achieved at other state DOTs. This type of information would help mitigate the risk of trying innovations and allow employees to see potential return on investment while empowering them to act. Specific Responses for Barriers and Lessons Learned ► “Discuss political barriers such as how you overcome some of the external barriers.” ► “DOTs have huge legacy commitments that are a drag on innovation.” ► “Look at organizations that have made more aggressive, larger scope innovations such as Affordable Care Act Implementation, 18F, US Digital Service.” OVERALL EVALUATION Many participants had no specific comment about the work, but commented on the usefulness of the guide and how they wanted to use the information at their own organization or spend more time reading the research and source materials in the reference section. Addressing Risk After the initial focus group discussion, the research team added the following questions for subsequent focus groups: Being more innovative involves taking risks, which can be challenging in the public sector. Do you think the guidance adequately addresses the need for and challenges associated with risk taking? How could it be expanded?

22 Deciding when and where to take risks is an integral part of the innovation process. While the guide addresses this subject, a few participants felt there could be more information on this topic and that it could be presented in a slightly different way. One respondent noted: “Risk taking should be presented in the context of what aspect of the business it may impact. For instance, an internal innovation that requires a financial investment is managed differently than an innovation that affects public transportation. Having led a team involved in the Fast 14 project (replacing 14 bridges in 12 weekends on a road that carries 250,000 cars a day) requires buy-in from a much larger group of people, from the agency head, to the public, the legislature, and even the Governor. Risk management strategies need to be tailored to the context.” Survey Results To gather additional input that would help validate the guide’s usefulness, the research team developed a survey that covered the entire guide, as well as a survey that allowed respondents to provide input on specific sections. This approach was used to capture more feedback in case individuals were unable to review the entire guide. The research team sent an email with a link to the survey to approximately 300 transportation officials in the project’s community of interest, which was comprised of individuals who had responded to the first online survey. The survey was available from June 8 to August 3, 2017. DOT Roles of Respondents The research team recognized that staff with various roles would find different elements of the guide useful, so they asked participants to respond only to survey sections that were relevant to their roles in their DOTs. For this reason, not every respondent completed every section of the survey. Survey respondents were asked to categorize themselves in one of three possible roles, which were presented as the target audience in the guide. Those roles and the number of respondents are provided in Table 2. DOT Role # of Respondents DOT Executives and Leadership People responsible for setting goals and strategy of organization. 5 Innovation Team Leaders People tasked with leading change efforts in terms of innovative culture. 4 Influential Team Members People not directly responsible for leading change efforts, but who have influence in the organization. 2 Table 2: DOT Role of Focus Group Participants.

23 Survey Design The five modules in the draft guide (Leadership, Communication, Empowerment, Recognition, Measurement) were each assessed with questions designed to measure the following concerns: 1. Value of content. a. Content presented new information? b. Suggestions were helpful? 2. Relevance of content to size/type of respondent’s organization. a. Information would be helpful in sustaining an innovative culture at my site? 3. Ease of use. a. Information was easy to read? b. Information was presented in a logical way? 4. Potential impact of suggestions. a. Suggestions would have positive impact at my site if they were put into action? b. My site is already putting a number of these suggestions into action? Overall Survey Findings DOT staff feedback on the guide was positive. The following tables show the percentage of survey respondents who agreed or slightly agreed with the statement about that section of the guide. The team received 3 responses to the Introduction/Self-Assessment sectional survey, 4 responses to the Innovation 101 sectional survey, and 13 responses to the survey that covered the entire guide. The results displayed in Table 3 combine the responses of all three surveys. Introduction Self-Assessment Innovation 101 The information was easy to read. 100% 100% 100% The information was presented in a logical way. 94% 100% 100% The description of the purpose of this section makes sense. 94% 100% N/A The introduction tells me how to use the guide. 81% N/A N/A The description of how to use self-assessment results makes sense. N/A 100% N/A The self-assessment results accurately reflect my organization. N/A 100% N/A The content provided me with new information N/A N/A 82% Table 3: Survey Results by Guide Section

Next: 9.0 Implementing the Research »
Research on Creating and Sustaining a Culture of Innovation for Departments of Transportation Get This Book
×
 Research on Creating and Sustaining a Culture of Innovation for Departments of Transportation
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

TRB's National Cooperative Highway Research Program (NCHRP) Web-Only Document 248: Research on Creating and Sustaining a Culture of Innovation for Departments of Transportation documents the research process and provides key guidance to implement the research produced in

NCHRP Research Report 885

: Guide to Creating and Sustaining a Culture of Innovation for Departments of Transportation. This guide provides insight on encouraging and sustaining a culture of innovation within the organization, its partners, and other stakeholders. A culture of innovation supports agency managers and staff efforts to encourage and accept innovation as a means to enhance the agency’s success. This guide is designed to assist agencies in assessing their culture with respect to innovation, identifying ways to make the organization more adaptable and open to beneficial change, and sustaining the organization’s adaptability to respond effectively to evolving technology, workforce, and public priorities.

READ FREE ONLINE

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  6. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  7. ×

    View our suggested citation for this chapter.

    « Back Next »
  8. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!