Skip to main content

An Assessment of Selected Divisions of the Information Technology Laboratory at the National Institute of Standards and Technology Fiscal Year 2021 (2021) / Chapter Skim
Currently Skimming:

2 Information Access Division
Pages 6-13

The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.


From page 6...
... , including the much-desired 2021 release of VVSG 2.0 for human factors, accessibility, and usability, is an example of a NIST project requiring assembling expertise from multiple computer science divisions and working with multiple outside constituencies, including community outreach via public working groups. The division contributed high-quality human factors technical expertise in improving the accessibility and usability of voting systems with critical work on improving system understandability by poll workers and work on providing equal access to all voters, including voters with disabilities, through the use of universal design methods.
Read the entire page →
From page 7...
... In 2021, TRECvid supported the following six major tracks: ad hoc video search, activities in extended video, instance search, video to text, video summarization, and disaster scene description and indexing. Disaster scene description is an example of IAD focusing its resources in a way that exploits government data sources, focuses on an area with lagging performance not of commercial interest, and builds important national capabilities.
Read the entire page →
From page 8...
... IAD has made commendable recent investments in social science expertise, leading to the very recent publication of a NIST Special Publication on identifying and managing bias in AI,2 and NIST has done definitive work in characterizing demographic differentials in face recognition. However, other examples of differential performance that have been highlighted for voice and language have not yet been addressed; IAD needs to take a more proactive role in providing data that facilitates more such analyses.
Read the entire page →
From page 9...
... The Image Group has organized online and ongoing evaluations whereby participants can submit at any time and a leaderboard tracks performance. The Retrieval and Multimodal Information Groups also need to move in this direction, where appropriate, in areas such as old data sets or already well-defined tasks or opportunities making for data sets openly downloadable.
Read the entire page →
From page 10...
... The Usability Testing Laboratory was probably underutilized during the pandemic, but it will be an important resource moving forward, particularly given that usability is important for trust in AI. IAD researchers provide expertise in multiple disciplines, including mathematics, engineering, computer science, social science, and IT.
Read the entire page →
From page 11...
... The range of outputs provided, such as published papers, data sets, technical briefings, tools, and guidelines, vary properly to meet the needs of individual technology areas so that IAD is positively impacting their advancements to support government and non-government needs. No substantive changes are necessary in these commendable efforts.
Read the entire page →
From page 12...
... For example, bias in AI is becoming increasingly recognized as a key issue in trustworthiness and is an opportunity for IAD. IAD has made commendable recent investments in social science expertise, leading to the publication of a NIST Special Publication on identifying and managing bias in AI, and IAD has done definitive work in characterizing demographic differentials in face recognition.
Read the entire page →
From page 13...
... IAD groups could do more to extend the reach of their metrology expertise to the broader community, such as by consulting, hosting tutorials, or playing an important role in establishing best practices for setting up informative evaluations. Human-agent interaction and human-in-the-loop systems represent a growing area of AI, particularly for speech, language, and multimodal technologies.
Read the entire page →

Key Terms

  • big data
  • nist special
  • language processing
  • differential performance
  • special publication

  • security research
  • qualitative research
  • information access
  • iris recognition
  • managing bias

  • testing laboratory
  • reference architecture
  • providing data
  • iad staff
  • iad researchers

  • social scientists
  • unclassified information
  • multimodal information
  • characterizing demographic
  • design phase

  • valuing qualitative
  • getting community
  • definitive work
  • individual projects
  • community trust

  • usability testing
  • commendable recent
  • ongoing evaluations
  • informative evaluations
  • usable security


    • This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
    More information on Chapter Skim is available.