National Academies Press: OpenBook

The Transportation Research Thesaurus: Capabilities and Enhancements (2018)

Chapter: Appendix C - TRT Assessment Criteria

« Previous: Appendix B - Immediate Action Plan
Page 77
Suggested Citation:"Appendix C - TRT Assessment Criteria." National Academies of Sciences, Engineering, and Medicine. 2018. The Transportation Research Thesaurus: Capabilities and Enhancements. Washington, DC: The National Academies Press. doi: 10.17226/25087.
×
Page 77
Page 78
Suggested Citation:"Appendix C - TRT Assessment Criteria." National Academies of Sciences, Engineering, and Medicine. 2018. The Transportation Research Thesaurus: Capabilities and Enhancements. Washington, DC: The National Academies Press. doi: 10.17226/25087.
×
Page 78
Page 79
Suggested Citation:"Appendix C - TRT Assessment Criteria." National Academies of Sciences, Engineering, and Medicine. 2018. The Transportation Research Thesaurus: Capabilities and Enhancements. Washington, DC: The National Academies Press. doi: 10.17226/25087.
×
Page 79
Page 80
Suggested Citation:"Appendix C - TRT Assessment Criteria." National Academies of Sciences, Engineering, and Medicine. 2018. The Transportation Research Thesaurus: Capabilities and Enhancements. Washington, DC: The National Academies Press. doi: 10.17226/25087.
×
Page 80
Page 81
Suggested Citation:"Appendix C - TRT Assessment Criteria." National Academies of Sciences, Engineering, and Medicine. 2018. The Transportation Research Thesaurus: Capabilities and Enhancements. Washington, DC: The National Academies Press. doi: 10.17226/25087.
×
Page 81
Page 82
Suggested Citation:"Appendix C - TRT Assessment Criteria." National Academies of Sciences, Engineering, and Medicine. 2018. The Transportation Research Thesaurus: Capabilities and Enhancements. Washington, DC: The National Academies Press. doi: 10.17226/25087.
×
Page 82
Page 83
Suggested Citation:"Appendix C - TRT Assessment Criteria." National Academies of Sciences, Engineering, and Medicine. 2018. The Transportation Research Thesaurus: Capabilities and Enhancements. Washington, DC: The National Academies Press. doi: 10.17226/25087.
×
Page 83

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

77 This appendix identifies the individual criteria used to assess the TRT. The criteria are orga- nized by the five dimensions of the assessment framework described in Figure 1-1 of this report. The criteria are traced to peer-reviewed literature on thesaurus assessment and evaluation. Full citations for the literature are provided in the bibliography for this report. Assessment Criteria for Content (Dimension A) The team proposed 35 criteria to assess the content of the thesaurus. During the analysis, 12 additional criteria were added and 2 were combined. Thesaurus content is defined to include thesaurus terms, definitions, classes, relationships, and core functions that support its man- agement and maintenance. We include both descriptive statistics of terms, relationships and essential management and maintenance attributes, as well as comparative statistics across and within current facets. This dimension of the reference model is one of the richest in terms of peer-reviewed references to assessment criteria. Each of these criteria is traceable to the peer- reviewed literature on thesaurus assessment and evaluation (Aitchison, Gilchrist, and Bawden 2000; Clarke 2001; Cleverdon and Keen 1966; Coates 1973; Davis 1968; Foskett 1972; Höpker 1972; Jones 1977; Jones 1967; Kim 1973; LaBarre 2010; Lancaster 1968; Mai 2006; Mandersloot, Douglas, and Spicer 1970; Miller 1997; Milstead 2001; Rada et al. 1988; Schirmer 1967; Travis and Fidel 1982; Tudhope, Alani, and Jones 2006; Willetts 1975). • Total number of terms in the TRT • Total number of lead in (non-preferred terms) • Total number of hierarchical relationships (paradigmatic relationships) • Total number of associative relationships (syntagmatic relationships) • Total number of structural terms [New] • Percentage total terms by facet [New] • Thesaurus rank of terms by facet [New] • Inclusion in the TRT or not-approved terms • Inclusion in the TRT of candidate terms • Inclusion in the TRT of provisional terms • How many top descriptors are there in the TRT (second-level terms)? • Unique identifiers for individual terms • Terms with definitions • Terms with traceable source and origin • Terms with scope notes • Number of parenthetical qualifiers used in the TRT • Total number of terms added annually since the TRT creation—all facets • Ratio of descriptors to identifiers A p p e n d i x C TRT Assessment Criteria

78 The Transportation Research Thesaurus: Capabilities and enhancements • Terms over three words • Terms with multiple broader terms • Errors in assigning hierarchical relationships • Errors in assigning equivalent relationships • Errors in assigning associative relationships • Inclusion of varied word forms as synonyms (spelling variations, singular/plurals, misspellings) • How are homographs of different meanings treated? • What percentage of general and specific terms linked through hierarchical relationships? • What is the level of specificity of terms in the TRT? (Hopker’s method of analyzing a class of terms) • What is the number of coordinate terms (combining multiple concepts to define a class) found in the TRT? • What is the rate of occurrence of terms no longer used in transportation that are still found in the TRT (e.g., accidents)? • What is the current rate of productivity of candidate terms that were not accepted for inclu- sion in the TRT? • How many top descriptors are there in the TRT (second-level terms) in each facet? • Number of changes in status of terms over time (when terms change from preferred to non-preferred) • Retrospective indexing practices for the TRT term changes and deprecations • Number of terms within each current facet of the TRT • Number of equivalence terms within each current facet of the TRT • Number of hierarchical terms within each current facet of the TRT • Number of associative terms within each current facet of the TRT • Use of stand-alone adjectives and adverbs [New] • Total population of RT and RTL terms [New] • Ratio of RTL to RT relationships [New] • Number of Level 2 terms [New] • Depth of levels below Level 2 [New] • Number of single branching terms and classes [New] • Number of uncontrolled terms [New] • Number of lateral relationships (RTLs) [New] • Number of terms added annually in each facet since the TRT creation Assessment Criteria for Access and Use (Dimension B) The team proposed eight criteria to assess the access and use of the thesaurus. During the analysis, 18 new criteria were added. Thesaurus access and use are defined to include instances of direct access to the thesaurus by different stakeholders; embedded use in search, workflow, and navigation applications; and customized use in a range of other applications. The met- rics for this dimension focus on the current and extensibility of use of the thesaurus in these different contexts. Because this dimension is represented in later generations, there are fewer peer-reviewed references to evaluation criteria. In most cases, the evaluation criteria are drawn from the open or gray literature and shared by recognized experts (Greenberg 2004; Pinto 2008). • Number of additional external systems currently using the TRT as an embedded search or indexing application • Number of additional external systems currently using the TRT terms for search or indexing as manual updates

TRT Assessment Criteria 79 • Number of additional external systems currently using the TRT for search or indexing as a batch upload • Number and nature of other uses of the TRT beyond TRID, TRBPI, TRBRIP • Depth of placement of most frequently used terms (defined through search logs) • Number of information assets assigned to each level of the hierarchy • Number of closely related terms assigned to the same document • Ability to display to the web the full set of terms assigned to a facet • Use of the TRT to select terms for indexing in TRID or other information sources [New] • Use of the TRT to select terms for searching in TRID or other information sources [New] • Use of the TRT to understand the transportation information landscape [New] • Use of the TRT to discover definitions for transportation terms [New] • Index term + keyword searches in TRID (Google Analytics) [New] • Keyword searches in TRID (Google Analytics) [New] • Ratio of three types of searches [New] • Rate of use of TRT terms as index terms in TRID [New] • Frequency of top term use as index terms in TRID [New] • Frequency of Level 2 term use as index terms in TRID [New] • Facet definition and frequency of use of second and lower level term use in TRID for each facet [New] • Display of full thesaurus term record [New] • Use of the TRT search functionality [New] • Use of navigation structures [New] • Ease of navigation of enumerated structure [New] • Ease of navigation of facet structure [New] • Effectiveness of display of KWIC [New] • Effectiveness of display of KWOC [New] Assessment Criteria for Governance Processes (Dimension C) The team used 12 criteria to assess the governance processes supporting the thesaurus. The- saurus governance processes are defined to include engagement by different stake holders, as well as governance processes (e.g., candidate term discovery, review and evaluation, disposition, and tracing). While governance spans generations, it has generally been tailored to the needs of individual organizations. Less is published in the peer-reviewed literature on this dimension. However, there are authoritative sources from the open and gray literature which can be used as guides. • How do subject matter experts provide input into term selection and adoption? Into thesau- rus evaluations and audits? • How are “not accepted” terms suggested by subject matter experts represented in the thesaurus? • How are data managers and stewards involved in discussions of changes to thesaurus terms? • How are information custodians (librarians/information professionals) who are currently using the thesaurus involved in the governance process? • How are technical stewards/architects involved in thesaurus governance and decisions? This includes technical stewards and architects outside of TRB Headquarters. • What are the opportunities for the TRT consumers to suggest terms for inclusion in the TRT? • How are historical terms managed in the TRT? How are not-approved terms managed in the TRT?

80 The Transportation Research Thesaurus: Capabilities and enhancements • Are scope notes created for historical terms that are not approved managed in the TRT? • How often is the TRT evaluated? Who is involved in the evaluation? • How are stakeholders informed of decisions on terms? Can stakeholders search the TRT to discover the status of their suggested terms? • How does the governance group know if a new candidate term has been previously considered? • Are suggestions for terms for structured data access received and considered by the TRT governance group? Assessment Criteria for Compliance and Standards (Governance Tools) (D) The team proposed 31 criteria to assess the governance processes supporting the thesaurus. During the analysis, one new criterion was added. Governance principles and tools are defined to include thesaurus user warrant, thesaurus literary warrant, scope and coverage, availability and adherence to best practices and style guidelines, alignment with thesaurus standards, alignment with relevant information management standards, and adherence to interoperability standards. This dimension has both longstanding evaluation criteria dating back to the 1970s and updated criteria based on changing context of operation (Antelman, Lynema, and Pace 2006; Austin 1974; Blocks, Cunliffe, and Tudhope 2006; Blocks et al. 2002; Cleverdon and Keen 1966; Cleverdon 1960; Farradane 1967; Farradane 1970; Farradane, Russell, and Yates-Mercer 1973; Jones 1971; Keith 1970; Klingbiel 1973a; Klingbiel 1973b; Klingbiel 1974; Lancaster 1972; Miller 1997; Montague 1965; Neville 1972; Pickford 1968; Pickford 1971; Rada et al. 1991; Rada and Bicknell 1989; Rada and Martin 1987; Rolling 1970; Rolling 1965; Salton 1972; Saracevic 1971; Stokolova 1977; Tudhope et al. 2006; Turski 1971; Vickery 1971). • Level of readability of the TRT content (e.g., a mixture of upper and lower case used in terms) • Consistency of term style guidelines with design guidelines in standard search system architectures • Consistency of terms with stated TRT style guidelines • Comparison of the TRT terms to terms in Research in Progress search logs • Comparison of the TRT terms to terms in TRB Publications Index Search Logs • Comparison of the TRT terms to terms in TRID Search Logs • Coverage of terms from Index and Table of Contents in Highway Safety Manual in the TRT • Coverage of terms from Index and Table of Contents in transportation design manuals included in the TRT • Coverage of TRB research publications author-generated keywords in the TRT • Inclusion in TRB author-assigned keywords as lead-in terms in the TRT • The TRT hierarchy levels assigned to author-generated keywords • Inclusion of research terms used in additional authoritative transportation research resources (e.g., research briefs, dictionaries, glossaries) • Number of popular (e.g., non-expert) transportation terms are included in the TRT • Number of Research in Progress index for aviation terms included in the TRT • Number of terms in Research Results Reports (RIP) included in the TRT • Number of RIP index terms that are lead-in terms in the TRT • Rate of match of TRB search log terms to “parts” of a TRT term • Comparison of coverage to other transportation thesauri • Comparison of the TRT coverage to semantically generated terms from TRB publications • Comparison of the TRT facets to topics used by other transportation associations (state, city, regional, local) • Comparison of the TRT facets to TRB Annual Meeting subject classes

TRT Assessment Criteria 81 • Comparison of the TRT terms to terms used by transportation research institutes, organiza- tions, and vendors • Coverage of facets of the field of transportation as defined by peer-reviewed journals • Coverage of future and “hot topic” terms in the TRT. How well positioned is the TRT to sup- port this research? • Facet-level treatment of modes of transportation in the TRT • General search productivity of the TRT terms in the TRB and transportation domains • Scope of coverage of transportation life cycle stages • Currency of coverage of transportation methods and tools • Subject matter expert assessment of depth of coverage of a facet in their area of expertise • Productivity of the TRT terms in related sources of transportation information (Google, Google Scholar, state DOT search systems) • Productivity of the TRT coordinate terms in transportation-related sources • Clearly stated and followed warrant [New] Assessment Criteria for Thesaurus Architecture (Dimension E) The team used 48 criteria to assess the architecture of the thesaurus management applica- tion. Thesaurus architecture is defined to include underlying data models and four levels of functionality including internal TMS functions, architectural integration with other applica- tions, support for communication protocols, and multilingual capabilities. There are a few peer- reviewed references for these criteria (Ganzmann 1990; Hines and Harris 1971). However, this is an emerging aspect of thesaurus management, so most criteria will come from the field of enterprise and application architecture. • Does the current thesaurus management application support batch upload capability? • Does the current thesaurus management application support the definition of extended types of relationships (building upon existing ISO 25964 defined relationships)? • Does the current thesaurus management application support the maintenance and assign- ment of multiple term status values (e.g., candidate, approved, not approved, provisional)? • Does the current thesaurus management application support the creation of multiple catego- rization schemes? • Does the current thesaurus management application allow for “ownership” of terms? • Does the current thesaurus management application support individual terms? • Does the current thesaurus management application support a Create Date attribute? • Does the current thesaurus management application support a Relationship Add capability? • Does the current thesaurus management application support the ability of end users to gener- ate reports? • Does the current thesaurus management application support the ability of non-developers to parameterize reports? • Does the current thesaurus management application support the following export formats (e.g., text, xml, rdf/skos, delimited text, custom html, rtf, csv Cartesian, csv + extra delimit- ers, CSV in columns)? • Does the current thesaurus management application support on-demand generation and updating of thesaurus statistics? • Does the current thesaurus management application support output of search results and reports to both screen displays and print formats? • Describe all of the types of standard reports the current application can generate • Does the current thesaurus management application support Term Update/Edit by thesaurus managers?

82 The Transportation Research Thesaurus: Capabilities and enhancements • In the current thesaurus management application, are there any constraints to the levels of hierarchical relationships supported in the current application? • Does the current thesaurus management application support a Created By attribute? • Does the current thesaurus management application support a network/consortium ver- sion for use by multiple thesaurus stewards? • Does the current thesaurus management application support the definition of different kinds of notes fields—scope notes, editors notes, personal notes, and definitions? • Does the current thesaurus management application support archiving of thesaurus terms? • Does the current thesaurus management application support dynamic advanced search or in context search? • Can the current TRT application manage and assign multiple levels of permissions and privileges? • Does the current thesaurus management application support a global rebalance function? • Does the current application have the ability to support multiple definitions for a single term record? • Does the current application have the capability to identify relationship errors or conflict upon batch or manual entry of new terms (e.g., multiple entries of same term, multiple rela- tionships, incomplete relations, relationing errors, and consistency checks)? • Does the current thesaurus management application support the ability for a term to appear in more than one facet (e.g., category)? • Does the current thesaurus management application support the existence of distinct records for individual terms? • Can the current thesaurus management software display both fully spelled out labels and label abbreviations? • Is there a Global Replace capability in the TRT today? • Are there any limitations or constraints applied to the length of thesaurus terms in the current thesaurus management application? • Level 1—reports capabilities—ability to define label formats in reports • Does the current thesaurus management application support the selection of fields to be included in export reports? • Does the current thesaurus management application support the ability to generate alphabeti- cal listings of terms? • Does the current thesaurus management application support the ability to generate hierarchi- cal listing? • Does the current thesaurus management application support the ability to generate a KWIC report? • Does the current thesaurus management application support the ability to generate a KWOC report? • Does the current thesaurus management application support the ability to select terms by types of relationships? • Does the current thesaurus management application support simple search capabilities? • Does the current thesaurus management application support advanced search capabilities? • Does the current thesaurus management application impose any character limitations on the size or occurrence of scope notes fields? • Does the current thesaurus management application support the direct extraction and use of unique term identifiers for use in other systems? • Does the current thesaurus management application extract full term names for use in auto- mated indexing and automated categorization applications or is translation required? • Does the current thesaurus management application provide a web-based SDK for publish- ing a thesaurus on the web?

TRT Assessment Criteria 83 • Can the TRT in its current form be integrated into other information management archi- tectures today and in the future? • What level of effort or translation is required to use the thesaurus content in other non-TRB applications (e.g., amount of transformation and manipulation required)? • Does the current thesaurus management application support the exchange of updated terms with other users through standard communication protocols (XML, RDF/SKOS)? • Can the case of terms be prespecified for output? Does the current thesaurus management application support the creation of other language versions of individual terms? (i.e., and full functionality of thesaurus for other language versions)? • Is there an upper level constraint for the number or unique identification of terms? Is there any upper limit to the unique ID sequence?

Next: Appendix D - Interview Guide and Data Collection Forms »
The Transportation Research Thesaurus: Capabilities and Enhancements Get This Book
×
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

TRB's National Cooperative Highway Research Program (NCHRP) Research Report 874: The Transportation Research Thesaurus: Capabilities and Enhancements documents the results of a comprehensive assessment of the Transportation Research Thesaurus’s (TRT’s) capabilities and strategies for the TRT’s future development. The TRT is a structured, controlled vocabulary of terms in English, used by TRB and a variety of other organizations to support indexing, search, and retrieval of technical reports, research documents, and other transportation information. The TRT, covering all modes and aspects of transportation, has evolved over a number of years and is continuously being refined and expanded.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  6. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  7. ×

    View our suggested citation for this chapter.

    « Back Next »
  8. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!