5

Engineering Behavioral Ontologies

What would it take to realize the benefits ontologies could bring in the behavioral sciences? To answer this question, the committee first examined what is known about the ontological systems that currently exist for the behavioral sciences. We sought to understand how they are developed and used and to develop insights about how the field could take better advantage of the possibilities that ontologies offer and the scientific or sociological impacts the existing ontological systems have had on the behavioral sciences. We also explored what it takes to engineer ontologies that bring the benefits described in Chapters 2 and 4. This chapter describes what we learned about current ontologies and about the socio-cognitive (human) processes, computer-based tools, and institutional and organizational structures through which ontologies operate and are sustained.

EXISTING BEHAVIORAL ONTOLOGIES

To review the current status of ontology design and implementation in the behavioral sciences, the committee commissioned a scoping review of the published literature on behavioral science ontologies (Falzon, 2021). The review was designed to provide an understanding of how and to what extent these ontologies have been studied; the significant findings from this body of work; and any discernable trends or patterns in the behavioral science ontologies that have been studied. The committee also hoped to gain insight into the methods by which ontologies have been created in the behavioral sciences and their strengths and limitations. This scoping review covered the general literature on ontologies, reviews that synthesized

multiple evaluations of ontologies, evaluations of individual behavioral science ontologies, and other relevant published papers. The committee did not define “ontology” for the scoping review because we wanted to obtain a broad view that could subsequently be refined and narrowed as needed for our work.¹ The scoping review search methods are summarized in Box 5-1.

It is difficult to count the number of behavioral science ontologies that have been developed. Broadly construed, ontologies are used for a wide array of purposes in the behavioral sciences and related fields, and the uses range from large-scale efforts to characterize a broad domain to very small and targeted efforts to meet much narrower objectives. In addition, vague and varying definitions of both “the behavioral sciences” and “ontology” made it difficult for the author of the scoping review to clearly demarcate existing behavioral science ontologies.² This is not surprising in light of what the committee found when we examined a few of the ontological systems that are best known in the behavioral sciences to test the definition of “ontology” (see Chapter 3).³

Falzon developed a list of 49 behavioral science ontologies identified in literature reviews, using a combination of subject headings and free text

___________________

¹ The author of the scoping review cautioned that it is not definitive, noting that, although it was wide and covered multiple databases, the search terms used could have excluded relevant studies. She also noted that no formal appraisal of studies was conducted.

² This section is based on Falzon (2021).

³ As noted above, Falzon’s work was not restricted by our definition.

terms to represent the concept of ontologies: see Table 5-1. A comparatively small proportion of the available literature was specific to behavioral science ontologies as defined in the scoping review; a much larger body of work was about biomedical ontologies. However, many of the ontologies not classified as behavioral science do cover behavioral topics. For example, several articles included behavioral ontologies in reviews of ontologies in health care, biomedicine, and other areas (Lokker et al., 2015; Kanopka, 2015; Stancin et al., 2020; Zhu et al., 2015). Falzon grouped the ontologies into five categories: behavioral (n = 16), phenotypes (n = 6), disease and mental health conditions (n = 13), genetics (n = 4), and neuroscience (n = 10). The number of classes in each ontology ranged from 167,138 (the National Cancer Institute Thesaurus) to 41 (EmotionsOnto), though the exact number of classes was unclear or unavailable for many of the ontologies reviewed.⁴

Several publications included in the scoping review described the development of single behavioral science ontologies (Brenas et al., 2019; Gkoutos et al., 2012; Hicks et al., 2016; Jensen et al., 2013; Köhler et al., 2012; Woznowski et al., 2018). These examples are useful for reviewing and understanding how these behavioral science ontologies were conceived, created, updated, and maintained. They are also helpful for understanding the vast array of engineering processes, or lack thereof, currently used to construct a behavioral science ontology.

There were relatively few studies that looked across the landscape of ontologies in the behavioral sciences (Blanch et al., 2017; Larsen et al., 2017; Norris et al., 2019; Hastings and Schultz, 2012; Poldrack and Yarkoni, 2016). Blanch and colleagues, for example, conducted a literature search of ontologies related to human behavior using the search terms “ontology with human behavior and psychology” and found 17 that had rigorous agreed-upon definitions to represent entities and links to other resources and could represent accumulated knowledge in a way that is easily shared by researchers from varied domains (Blanch et al., 2017, p. 182). However, this and other reviews primarily provided narrative descriptions, rather than evaluations of the strengths and limitations of the ontologies.

In contrast, Norris and colleagues (2019) conducted a review focused on behavior change interventions that assessed their quality. The authors identified 15 ontologies that met their selection criteria, covering such areas as cognition, mental disease, and emotions. These ontologies were developed using methods that included expert consultation and reuse of terms from existing taxonomies, terminologies, and ontologies. However, the authors concluded that none of the 15 represented the “breadth and detail of human behaviour change” (Norris et al., 2019, p. 164).

___________________

⁴ For complete reference information on the ontologies identified in the scoping review, see Falzon (2021).

TABLE 5-1 Ontologies Identified in the Surveyed Literature

NOTE: NA means not available.

SOURCE: Falzon (2021, Table 1).

Poldrack and Yarkoni (2016) describe the contributions that formal cognitive ontologies could make to the clarification, refinement, and testing of theories about the delineation of how brain systems relate to mental function. They argue that ontologies could potentially play a valuable role in the behavioral sciences. Similarly, Larsen and colleagues (2017) review the problems an ontology can help to solve and use ongoing work on ontologies related to behavior change to discuss key steps in ontology development.

Several other studies identified by the scoping review discuss best practices and lessons learned. As noted above, Larsen and colleagues (2017) provide multiple rationales for the creation of ontologies in the behavioral sciences and a guideline for how they should be created and could be used to advance the field of behavioral medicine. Norris and colleagues (2021) argue that behavioral science ontology development should involve expert stakeholders. Several other studies provided more formal evaluations of existing ontologies.

As mentioned above, six studies identified in the scoping review were evaluations of a single behavioral science ontology (Brenas et al., 2019; Gkoutos et al., 2012; Hicks et al., 2016; Jensen et al., 2013; Köhler et al., 2012; Woznowski et al., 2018). Other studies not exclusive to the behavioral sciences suggested criteria for evaluation: tools, methods, and software and metrics (see, e.g., Amith et al., 2018; Yao et al., 2011; Franco et al., 2020). The scoping review (Falzon, 2021) did not identify any metanalyses of evaluation approaches, but did identify various criteria in the literature for evaluating the methods and quality of an ontology (broadly understood):

• uses existing taxonomies;
• uses existing terminologies;
• uses existing ontologies;
• user feedback;
• data driven;
• unique uniform resource identifiers;
• clear definitions;
• clear structure;
• logically consistent;
• evaluated;
• maintained;
• domain coverage;
• task orientation;
• computational efficiency; and
• maps to existing technologies.

These characteristics suggest the properties that are identified as important in the available studies (see below for further discussion of evaluation; also see Chapter 4).

The scoping review also revealed a wide array of applications for existing ontologies in research, clinical settings, and education. These applications and functions closely track many of the elements identified in the committee’s use case survey (see Chapter 2):⁵

• support collaborative and multidisciplinary research;
• support efficient knowledge accumulation;
• organization and structuring of evidence;
• enhanced evidence synthesis;
• automation of meta-analysis;
• analysis of raw data and integrating of findings across domains and subject areas;
• refine diagnostic categories;
• interrogate clinical information systems;
• translate research results across disciplines;
• compile lists of behavior change techniques, implementation strategies, and interventions;
• annotation (e.g., automated annotation of radiology images);
• terminology mapping (e.g., mapping terminology to phenotypic clinical data to advance knowledge of genetic diseases);
• use of natural language processing to code text from clinical documents;
• query enhancement (use of search terms to recognize context and provide synonyms and additional terms to enhance the query);
• clarify, refine, and test theories;
• code randomized clinical trials (e.g., in neurosurgery);
• develop educational software;
• personalize and recommend content;
• design curricula; and
• assess the learning process.

The authors of many of the studies discussed in the scoping review offer strong reasons why progress in ontology development and use would be a boost in the behavioral sciences (e.g., Poldrack and Yarkoni, 2016; Hastings and Schultz, 2012; Blanch et al., 2017; Larsen et al., 2017; Norris et al., 2019). The literature also offers a wealth of recommendations, many very detailed, for developing individual ontologies for targeted purposes, as well as varying guidelines for evaluating existing ones.

___________________

⁵ See Falzon (2021, p. 13–14) for specific citations.

More elusive is a solid picture of how many models in current use can be accurately classified as ontologies, rather than sets of concepts that have not been formally specified. The scoping review did not locate any systematic documentation of existing ontologies (broadly understood), and we note that carrying out such a systematic survey would be challenging. As the continuum presented in Chapter 3 demonstrates, ontological systems vary in their degree of formal specification. We noted there that whether a given system meets the definition of ontology is less important than whether it is designed to suit the purpose for which researchers need it. The fact that the Behaviour Change Intervention Ontology (BCIO) did not turn up in the scoping review, despite being a relatively well known system that provides a high degree of formal specification, illustrates the challenge: in response to a query, the author of the scoping review noted that, although the BCIO is mentioned in two of the reviews, the literature on the project itself did not match the search terms used. While other examples could possibly have been missed in the scoping review for similar reasons, this example illustrates that it is probably more useful to apply available resources in support of researchers who wish to pursue ontological rigor than to more systematically survey existing efforts, which are both evolving and idiosyncratic.

It is also difficult to develop a picture of the extent to which existing ontological systems are currently contributing to progress in the behavioral sciences or the gaps and barriers to their development in the behavioral sciences. It was clear from both the scoping review and the committee’s close look at a number of examples that many individuals have worked tirelessly on ontological efforts that apply in some way to the behavioral sciences. It was also clear that efforts to date in the behavioral sciences have not yet by any means taken full advantage of the potential benefits of ontology development. While a systematic survey of existing behavioral ontologies was well beyond the scope of this study, we did identify some trends across the ontological systems we examined, including key examples from biomedicine.

There are comparatively few ontological systems in the behavioral sciences that are widely known and used, and those that exist have had limited impact. For example, despite the efforts that have gone into their development, the Behaviour Change Intervention Ontology and the Cognitive Atlas have received relatively few citations in the academic literature in comparison with ontologies in other fields. Many ontological efforts have been isolated, and it appears that adoption and use of many existing ontological systems has been constrained. Moreover, the developers of these systems in behavioral domains appear to operate largely on their own in identifying or developing the models and practices that might best suit their needs. Few of the examples we saw took advantage of ontology best practices or computational tools or advances. None was sponsored by a central professional organization. Most appeared to have been built

because of the specific interest of their creator, rather than in response to needs identified by the field. None had a sustained source of funding, and the committee heard from a variety of experts that it is difficult to sustain ontology development efforts. (We review the question of resources and support more fully in Chapter 6.)

THE ONTOLOGY ENGINEERING PROCESS: SOCIO-COGNITIVE PRACTICES

Humans must make key decisions about the terms and relationships to be covered in an ontology. Thus, socio-cognitive (social and intellectual) practices (elements or functions) are one of the two basic components of the process of engineering ontologies. Ontology engineering is a creative and inventive process that requires substantial intellectual and social effort, realized through socio-cognitive practices. The work of ontology creation, editing, dissemination, debugging, and understanding all require such practices, and the impacts of these practices at all stages of the ontology life-cycle cannot be fully captured solely by the formal specification at one moment or even over time. This section identifies some of the key socio-cognitive elements that are involved in ontology engineering, including the knowledge humans have, their activities, and their interactions with technology (social interactions, cognitive strategies, language, and patterns of communication). Best practices for some of these have been identified and are likely to be useful in the specific context of the behavioral sciences, though there are as yet few examples of their use in this context.

Creation

The creation of an ontology almost always requires the translation of complex, potentially ambiguous concepts into a formal specification. A first step in ontology creation is the identification of the key notions, as well as key features or attributes of those notions, that will be included in the ontology. Meta-analyses and critical literature reviews can provide valuable insights about which concepts are most important in a particular domain, though they will rarely be sufficient to uniquely determine what should be included in the formal specification. Almost always, experts are needed to determine how the key concepts and features should be formally represented.

A common emphasis on and approach to creating ontologies in the behavioral sciences would likely reduce the chances of unhelpful pluralism. There have been some efforts to develop design patterns to support ontology creation—relatively formal specifications of characteristic patterns (Blomqvist and Sandkuhl, 2005; Hitzler et al., 2016). As a simple example, an ontology design pattern for “inheritance” would provide a template

for “inheritance” relational structures within the ontology, regardless of whether one means inheritance in genes, in culture, in software, or in some other domain. Design patterns have proven quite useful in other disciplines involving context-specific, complex concepts and ideas (e.g., architecture, software engineering, human-computer interaction); thus, they are a potentially promising avenue even if there is currently little empirical evidence about their utility in the behavioral sciences.

Ontologies can be used for multiple purposes (see Chapters 3 and 4). The value of a particular ontology may depend partly on intended uses, and so those uses should be clearly articulated in the creation stage of an ontology. Ontology creation should thus also involve identification of the stakeholders who may be affected by the ontology, the goals and knowledge bases that will depend on it, and key use cases (see Chapter 2). Developers might sometimes identify the goals simply by reflecting on their own needs and practices. More commonly, however, they will need to perform cognitive task analyses of the stakeholders’ relevant scientific practices and papers to determine when an ontology might be useful and how it might be used (Crandall et al., 2006; Hollnagel, 2003). Proper design and use of an ontology requires an understanding of the contexts in which it will be used, whether from personal introspection or structured investigation.

Socio-cognitive practices are also important once an ontology has been created or proposed. The dissemination of an ontology involves not only transmission of the formal specification, but also instruction about ways to use it. The mappings from objects in the world to elements in the ontology can be very complex and sensitive to context, particularly in the behavioral sciences. The formal specification alone is insufficient to determine how to use an ontology. As result, unless there is already very substantial agreement in the scientific community, instruction and training will typically be needed to ensure that people use the ontology properly and not merely hear about it.

Debugging an ontology requires an understanding of what should—and equally importantly, should not—be included in the ontology (i.e., what is important in the domain). The basic idea of debugging an ontology is to determine the logical implications of what is in the current ontology, perhaps augmented with a minimal knowledge base, and then either include the ontology elements that support those implications, if the implications are correct, or revise the ontology if they are incorrect. This stage thus presupposes that there are some basic facts about a domain that are almost universally accepted in the scientific community. There will almost certainly not be an explicit list of such constraints on an ontology; rather, the widely accepted facts or statements will need to be determined using socio-cognitive practices of analysis and collaboration, just as such practices are needed for identification of key features and concepts for the original ontology

creation. This is a critical step, but also one for which best practices have not necessarily been developed. Once some statements have been identified, formal computational tools can be used to automate aspects of debugging (see below). But the relevant constraints for debugging will typically emerge only after careful investigation.

Change and Evolution

Socio-cognitive practices also play a key role in ontology change and evolution. Because ontologies are formal specifications, it is easy to mistake them for relatively static structures. After all, they are expressed, used, and evaluated at fixed moments in time. But ontologies are dynamic: they evolve and change significantly and sometimes rapidly in response to scientific developments and other factors. Moreover, some ontologies will need to evolve in response to a complicated, open world, while others may need to be responsive to pressures from the more “closed” worlds of, say, billing practices. In almost every case, socio-cognitive factors and processes play key roles. We do not attempt to provide an exhaustive taxonomy of ontology change, but instead provide several illustrative examples.

First, ontology change could be required when scientists learn more about the world, including how the world itself might have changed or be changing. There are many instances in which a scientific community recognized that its current shared understanding was faulty or incomplete, and so any shared ontology had to be adapted in light of this new information. For example, the spread of novel diseases (e.g., COVID-19, Zika, etc.) has required additions to existing ontologies for disease classification. These ontology changes frequently involve significant communication and collaboration in the scientific community.

A second type of ontology evolution arises when scientific goals and needs shift, perhaps in response to technological advances. For example, there can be “function creep” of an ontology as it is gradually used for purposes other than the original goals. This creep can produce gradual changes in the ontology to support the new goals, though often without explicit acknowledgment that anything is changing. Ontologies can thus end up serving very different use cases than their designers originally intended.

A third kind of ontology change is prompted by the desire to harmonize or integrate different ontologies, including the desire to bridge ontologies at different levels. This particular driver of ontology change often occurs at the same time as one of the others, such as when the development of new measurement methods enables scientists to start to bridge between different domains or systems. In practice, such integrations almost always require changes to one or more ontologies, or perhaps the creation of a new ontology to replace or even to compete with the previous ones.

This dynamic can be readily seen in proposals to change aspects of mental health ontologies to better ground them in biological bases: that is, there are efforts to change an ontology so that it better integrates with another one. More generally, there are clear challenges to harmonization of medical ontologies with (potential) behavioral ones, though doing so would be valuable. Such integrations require the careful development of mappings between elements of one ontology with those of another, ideally paired with specification of bridge principles to enable expression of one ontology in the formal language of the other. At the least, there must be clear understanding of how elements of one ontology impose constraints on the possibilities for the other. Moreover, these translation efforts can often be guided by novel empirical data collected during the course of the integration process.

Ontology change can also be driven by factors that are external to scientific communities, such as cultural changes in conceptualizations. Social expectations, needs, values, and concepts can all change over time, and so there can be pressure—social, political, psychological—to adjust a scientific ontology so that it is more consistent with those social factors. For example, changes in cultural perceptions of homosexuality, pregnancy, and hysteria each played a role in subsequent changes to disease classifications. The committee does not take a position about whether or when scientific ontologies ought to respond to this type of reason for ontology change; rather, we simply note that some ontology change seems to happen because of social and cultural evolution.

All of these instances of ontology change involve socio-cognitive practices and factors. For example, an ontology might change through open discussion and debate in the relevant scientific communities, as researchers and clinicians frequently work to revise and refine their shared ontologies (and understandings, more generally). There is little consensus about best practices for scientific communities to develop shared languages and understandings. Alternately, one could crowdsource observations about shortcomings of an ontology to help ensure that diverse perspectives are being included, particularly for ontologies with significant social impact. Or methods to integrate or harmonize distinct scientific theories can provide mechanisms to reconcile or adjust different ontologies, at least if the theories are sufficiently well specified that the core ontology is clear. It is important to note that, although ontology change depends on various socio-cognitive interactions and practices, it is not merely a matter of chance or power. Rather, there are structured and semi-structured mechanisms for intelligently updating or revising an ontology.

The Role of Institutions

Organizations and institutions can play critical roles in the creation, editing, dissemination, adoption, and revision of ontologies. In particular,

institutions—including universities, research funders, journals, and conferences—can create, perhaps implicitly, incentives to use or not use ontologies in various ways. A non-ontology example—regarding the growing recognition of the importance of open data—illustrates the importance of the role of institutions. There were significant changes in open data and data-sharing practices once funding agencies started to require that funded projects share their data (or provide clear, good arguments why data sharing was not appropriate or feasible). Similarly, many journals began requiring that data be made publicly available (or provide a justification as to why not) as a condition of publication. These and other institutional changes have had a significant impact on the forms and frequency of data sharing across multiple scientific disciplines. Similarly, advances in ontology creation, dissemination, and use in the behavioral sciences may require institutional changes, including incentives of various types.

The committee could not identify any existing institutional structures or incentives that directly promote the use of shared ontologies in the behavioral sciences. Practicing clinical psychology and psychiatry in an academic setting clearly requires the use of ontologies. Scholarship increasingly requires data sharing, which also requires ontologies (see Chapter 4). These are indirect incentives, but, for example, to the best of our knowledge, no university requires or prioritizes ontology (re)use when considering promotion and tenure decisions. Some journals require the selection of keywords from a fixed list, but they do not require any accompanying formal specification. Journals may also require that the data described in scientific papers be made publicly available, which, as noted above, explicitly requires ontologies, but this is rarely explicit in journal policies. No U.S. government funding agencies in the behavioral sciences currently require the use of ontologies in grant proposals or other contracts. The same appears to be true for a range of different institutions and structures that support research in the behavioral sciences. We also note that many journals, conferences, and promotion committees favor the use of novel theories and theoretical concepts, yet they do not prioritize the use of formal specifications.

Significantly, the committee also could not identify any significant disincentives to the use of ontologies in the behavioral sciences beyond those already noted (e.g., the “toothbrush problem”) or those that apply in any complex, collaborative setting (e.g., the difficulties of team science). Institutions do not seem to be actively blocking the creation, adoption, or use of ontologies, even if they are generally not providing positive support or inducements. Yet, given that ontology creation, adoption, and use all involve some costs to researchers, it may not be surprising that systematic, shared ontology use in the behavioral sciences has been relatively rare despite the ways ontologies can help with persistent scientific challenges.

Evaluation

Determining whether an ontology is—and remains—useful for the purposes it was designed for is a key to its ongoing viability. Two important components of such an evaluation are verification and validation. Verification is the assessment of whether the ontology was built correctly, that is, whether the specification has utility for its intended purpose. Validation is determining whether the ontology correctly models the domain or real-world application for which it was intended. Essentially, verification addresses the intrinsic aspects of the ontology; validation addresses the extrinsic aspects of the ontology. Metrics such as completeness, accuracy, consistency, computational efficiency, and clarity, among others, are used in this process, and the ontologies literature includes many guides to evaluation, including many highly technical ones (see, e.g., Amith et al., 2018; Obrst et al., 2007; Rensselaer Polytechnic Institute, 2013). We also note that Gruber (1995; also see Chapter 3) identified criteria that can be used to guide the development of ontologies to ensure that the ontologies are well suited for reuse for different purposes and across applications: see Box 5-2. The Open Biological and Biomedical Ontology (OBO) Foundry has also provided a set of principles to guide those who commit to its approach (Jackson et al., 2021). These design criteria are all defined with knowledge sharing and interoperability among users of the shared conceptualization as the main purpose.

Such metrics and guides are important detailed resources for ontology developers. The committee did not assess the many available guides, but it did identify three broad criteria for the development of an ontology—logic, validity, and usefulness—that mirror criteria used in many scientific contexts.

1. Logically sound: the ontology contains no contradictions, is technically correct, and is concisely expressed in formal terms. Historically, the verification of the consistency and correctness of an ontology has been based on metrics provided by human raters, but automated tools have recently been developed that can supplement human efforts by automatically identifying errors in the semantics and logical structure of an ontology.
2. Valid: The definitions asserted in the ontology accurately reflect what the ontology is representing and cover that domain as completely as possible. This function is dependent on human judgment, but algorithmic approaches for evaluating the accuracy and completeness of an ontology have provided new ways of assessing the validity of an ontology.
3. Useful: The ontology is usable by a diverse range of stakeholders, including social or behavioral scientists, health practitioners, and ontology developers, such as computer scientists. Assessing the usability of ontological systems is essential for developing ontologies that are easily deployed and adaptable across different contexts, and enable users to work efficiently and accurately.

THE ONTOLOGY ENGINEERING PROCESS: COMPUTATIONAL TOOLS

Since ontologies existed long before there were computers it is important to acknowledge that computational tools are not strictly necessary for ontology engineering. However, the efficiencies they provide, not to mention the capacities they afford for working with large bodies of data, have made them essential in much of behavioral science, and likely for the development and use of behavioral ontologies. Modern scientific ontologies may contain many thousands or even many millions of terms and are correspondingly complex, so technology has become essential for managing them.

Computational tools can never stand in for the human understanding, ingenuity, and social perceptions that go into the development and use of ontologies. They do not offer ready means of helping scientific communities determine what they need or of engaging colleagues in the socio-cognitive tasks detailed above. But technological tools do play an important supportive role in facilitating ontology design and use. Computational tools

allow for automation of many of the operations important to ontologies, which brings valuable efficiency. Operations that can be automated include creating a subset of an ontology, importing terms from other ontologies, or verifying that all definitions in the ontology meet certain design criteria.⁶ It has been proposed that artificial intelligence (AI), specifically, machine learning, could be used to support ontology development, but there is little evidence so far about its broad utility for this purpose.

A comprehensive review of available computational tools is beyond the scope of this report, but three key elements of the life-cycle of an ontology illustrate the contributions of computer technology (Noy et al., 2010):

• creating and editing the ontology;
• disseminating it so that researchers have awareness of and ready access to the ontology; and
• evaluating and debugging the ontology, in the sense of testing the logical implications of the ontologies’ statements and folding findings back into the ontological structure.

Creation and Editing

Technological supports for human ideation and consensus-building are one type of computational tool that can be useful for ontology creation. Perhaps most widely used in the very early stages of ontology development is collaboration software that supports complex graphics, such as Microsoft Whiteboard. People use such tools to brainstorm ideas for what terms should be included in an ontology by suggesting competency questions, that is, questions about distinctions in the world that the ontology ought to be able to resolve (these are analogous to requirements in conventional software engineering). These questions help developers to flesh out the things that the ontology should encompass and stimulate developers to create an initial set of entities and relationships. Developers may also use mind mapping tools (software that supports visual structuring and organizing of complex ideas) or even simple spreadsheets to flesh out nascent ontologies. Ontology visualizations might show the meta-graph of major entity types and their linkage types to guide the selection and incorporation of existing ontologies: see Box 5-3.

Tools that make it easier to view a hierarchy of concepts, add to it, or add properties of those concepts are extremely useful in creating new ontologies and reviewing and editing existing ones. However, the possibilities for visualization on digital screens generally remain limited to tree structures,

___________________

⁶ A good example of this capability is a collection of utilities developed by the open biological and biomedical ontology community for work in bioinformatics (Overton et al., 2015).

such as file directories. The most widely used of the tools developed for ontology editing is Protégé, an open-source system that was originally developed at Stanford University for the modeling of biomedical ontologies, and is now used in many different disciplines (Musen and Protégé Team, 2015).⁷ Other tools are available: see Box 5-3. And there are commercial systems, such as TopQuadrant’s TopBraid Composer⁸ and OWLGrEd.⁹

These are not particularly complex technologies—they support ontology development in approximately the same way that word processing software supports the writing of a novel. A word processing program can make the writing process far more convenient and efficient than it would otherwise be, but it is of no help with the development of character and plot. Similarly, these ontology editing systems do not help elucidate the way people perceive the world.

___________________

⁷ See https://protege.stanford.edu/about.php

⁸ See https://www.topquadrant.com/products/topbraid-composer/

⁹ See http://owlgred.lumii.lv/

¹⁰ See https://cns-iu.github.io/spoke-vis/home and https://spoke.ucsf.edu

¹¹ UFO is an app that unifies most of the semantic similarity measures for between-term and between-entity similarity calculation for all types of biomedical ontologies in OBO format.

Advances in computing and algorithmic innovations can also allow for the processing of far more data than was previously possible and for increasingly sophisticated ways to identify classifications as alternative starting points for ontology creation. Statistical modeling algorithms can be used to automatically identify classes and nonlinear relationships among them, and to automatically organize very large datasets. One type of computational tool involves so-called clustering and dimensionality reduction methods (James et al., 2013; Becht et al., 2019). In different ways, these methods identify potential “groups” (perhaps based on unobserved factors) in the data. Human researchers can then assign labels or terms to the different clusters or factors (as is done for HiTOP; see Chapter 3), thereby yielding a potential set of entities for an ontology.

A concrete example of this approach, developed for characterizing psychological problems across the human life span, starts with behavioral genetic studies and infers a hierarchy of factors or dimensions that can explain psychopathology across people’s lives (Lahey et al., 2017; Kotov et al., 2021).¹² The resulting entities provided a taxonomy of psychopathology and so enabled the researchers to move from a subjective categorization of psychopathology to using data for the identification of dimensions that have a causal influence on psychopathology (elements of an ontology). Historically, clustering and dimensionality reduction methods have provided either a comprehensive set of entities or information about their relationships, but not both. Statistical modeling algorithms have the potential to provide both of those parts of an ontology. In behavioral neuroscience, for instance, AI has, in recent years, largely automated the manual annotation of animals’ behaviors using of video recordings. In this way, AI is used to help identify an entity for formal specification in an ontology (Mathis et al., 2018; Graving et al., 2019; Pereira et al., 2020).

The second automated approach that can aid ontology creation is to use NLP to analyze large amounts of text or documents. NLP methods can organize a body of text-based documents into topics that can be treated as a representation of the body of knowledge associated with those documents. The researchers who developed the National Cancer Institute (NCI) Thesaurus have used NLP technology developed by the National Library of Medicine (NLM), specifically the MetaMap algorithm to index text documents (Aronson, 2001; Sioutos et al., 2007; Cui et al., 2020).¹³ NLM developed MetaMap with the goal of discovering meta-thesaurus concepts referred to in text. MetaMap relies heavily on NLP and other suites of

___________________

¹² Presentation by Benjamin Lahey to the committee during the spring 2021 workshop; see https://www.nationalacademies.org/event/05-24-2021/what-are-ontologies-and-how-are-they-used-in-science-workshop-1

¹³ Presentation to the committee by Lyubov Remennik and Richard Moser during the spring 2021 workshop; see https://www.nationalacademies.org/event/06-29-2021/understanding-ontologies-in-context-workshop-2

algorithms for the statistical analysis of text data. An alternate approach, topic modeling, arguably the best known and popular form of NLP, tackles the problem of organizing text documents into topics or categories (Blei, 2012). The results can again provide an initial basis for the entities of an ontology, though human evaluation is invariably required. Ongoing research is examining how to incorporate relationships between topics into statistical models and into deep neural networks for processing text data (Gan et al., 2015).

As noted above, the people working on ontology engineering hope that formal development methods or the use of design patterns—as in conventional software engineering—will greatly facilitate the early stages of ontology development (Corcho et al., 2006). For this hope to be realized, many of these methods need to be empirically evaluated in the behavioral sciences and also built into general-purpose ontology development tools. This type of development can be guided by existing tools (e.g., ROBOT¹⁴) that recommend ontology design patterns or that detect the use of ontology design patterns in evolving ontologies. Further progress in the development of automated methods based on AI and machine learning that can fuse sensor and text data at scale and enable the identification of relationship between categories identified on the basis of sensor data could provide further support for ontology development. Such advances could also support the process of evaluating and reconciling different possible ontologies that can emerge from this kind of fusion.

Dissemination

Chapters 2 and 4 highlight the critical importance of making ontologies widely available and accessible, and computational tools are particularly valuable for these purposes. Available tools facilitate such tasks as searching for ontologies that contain specific terms and visualizing them. Especially important are application programming interfaces (APIs), which allow programs to access and use information from others; they depend on shared terminology. Using an API, a third-party computer program can locate terms that may be relevant for describing a scientific problem, a dataset, or some other component of interest.

For example, BioPortal, a repository of biomedical ontologies developed at Stanford University, is an open, searchable portal to which anyone can upload an ontology for distribution (Noy et al., 2009). It can recommend ontologies based on queries, analyze text documents to identify what ontology terms might be mentioned in the text, and support mapping across ontologies

___________________

¹⁴ See https://robotframework.org/ also see Jackson et al. (2019).

(Whetzel and NCBO Team, 2018; Ghaazvinian, 2009).¹⁵ BioPortal archives several hundred ontologies that are available to any user. Users who need to identify relevant ontologies among this large collection of entries for a particular purpose can use the BioPortal ontology recommender service to obtain suggestions for ontologies that may be especially appropriate for their needs (Martinez-Romero et al., 2017). Repositories for ontologies in scientific disciplines outside of biomedicine have been created using the underlying BioPortal software, which is completely domain independent. These BioPortal clones exist in various scientific domains, including materials science, ecology, and agronomy. The investigators responsible for these various discipline-specific ontology repositories have come together to form the OntoPortal Alliance.¹⁶ Members of the Alliance develop and share software extensions to the underlying repository, and they are in discussion regarding the engineering of visualization and query components that could transcend the individual ontology repositories maintained by the group’s members.

The Ontology Lookup Service (OLS) is another repository for biomedical ontologies, developed by the European Bioinformatics Institute.¹⁷ This repository is closed, in that only ontologies approved by a team of curators are available through the site, an approach that provides greater assurance of ontology quality at the expense of benefiting from general contributions from the research community.

Evaluation and Debugging

As we note above, tools and technologies have been developed to automate and facilitate evaluation. For example, the widely used ontology software library ROBOT offers a “report” function that runs a series of quality control tests over an input ontology and generates a report file based on the results, suitable for use in an automated workflow (Jackson et al., 2019). Another emerging trend related to ontology evaluation is the increasing use of NLP to generate semantic definitions through a natural language generation task, which can parse the ontologies and generate natural language text so that humans can assess its quality.

No ontology could predict or specify every possible inference that may turn out to be wrong, but debugging an ontology—fine tuning it to eliminate errors—is challenging. In the context of ontology engineering, “debugging” is usually a matter of running a reasoner—software that can make logical inferences from information or axioms—and seeing where the reasoner makes assertions that the developer deems to be wrong and then

___________________

¹⁵ See https://bioportal.bioontology.org/

¹⁶ See https://ontoportal.org/

¹⁷ See https://www.ebi.ac.uk/ols/index

tracking down the axioms in the ontology that led to the spurious inferences (Vrandečić, 2009; Parsia et al., 2017).

People developing ontologies in OWL, the most commonly used ontology language (see Chapter 3) take advantage of reasoners to see the logical implications of specific statements in their ontologies. Reasoners are especially useful with very large ontologies, such as the NCI Thesaurus or SNOMED,¹⁸ to test the classifications, to make sure they are predictable, and to flag errors. Reasoners are also useful when there is uncertainty about whether the statements made by the ontology result in implications that are logically correct.

Potential Directions for the Future

The committee focused on the functions that computational tools currently provide, but there is considerable potential for future systems to provide additional kinds of support for ontology development and use. Ontology development can be quite expensive, since it requires significant people power for the intellectual work. Might technology be of use in engaging a research community in developing and evaluating ontologies collaboratively in ways that are not possible at present? Since human development and evaluation of ontologies are labor intensive, might technology be leveraged to further facilitate ontology development and evaluation, perhaps through visualization and natural language generation?

Research on ideas such as these could bring valuable developments in the long term. For example:

• One topic is improved understanding of the differences between machine and human interpretations of particular terms and between automated evaluations of ontology validity and manual reviews by human experts, as well as of hybrid systems in which humans and machines work together on ontology development, so that computational tools can be appropriately designed for human needs.
• Work could be done on ways to improve online tools to support the iterative life-cycle of ontology development and evaluation, such as ontology creation, refinement, visualization, and evaluation; more tools would be useful for automating ontology development tasks.
• Also valuable would be work on ways to leverage AI, especially machine learning, in the development and evaluation of ontologies, including application-specific efforts that reflect variability among domains.
• Another possibility is development of frameworks and scaffolding for ontology choice, not only display of ontology options as found, for example, in, the BioPortal ontology recommender.

___________________

¹⁸ For more information see https://www.snomed.org/

• Another topic of interest is enhancement of ontology metadata to encode goals, past uses, organizational endorsement, and other information in a more comprehensive manner.
• It would also be valuable to have innovations that can combine AI synergistically with human intelligence to improve the development and evaluation of ontologies.

With regard to AI, current ontologies mostly capture human knowledge, but, in principle, AI-based tools could extract knowledge in an automated fashion from large datasets, such as by incorporating results from prediction models based on machine learning. For example, with data from electronic health records, one can use AI/machine learning to learn about associations between risk factors and outcomes. The finding of an association between depression and the development of dementia, for instance, could be used to populate an initial relationship in an ontology between depression and dementia. Then, AI could be used to discover and represent causal structures, evaluating an ontology from two perspectives, to determine whether the casual structures found in the data exist in the ontology, or to determine whether the asserted knowledge in the ontology agrees with an expert’s knowledge.

Research on AI systems to generate both ontologies and knowledge bases, including novel hypotheses, could address challenges in efficiently creating high-quality, validated ontologies. In the medical field, for instance, physicians’ reports routinely accompany the results of medical tests and images collected from patients. Taken individually, these different data modalities would result in different ontological descriptions (e.g., through knowledge embeddings) of a disease of interest. These ontological descriptions, however, ought to share some levels of similarity. This is not a trivial task. The field currently lacks a principled approach, based on AI (specifically, machine learning) for fusing knowledge embeddings and ontological descriptions obtained using different modalities. There is a need for research on multimodal data fusion and overlay with AI, and its potential use in specific contexts in the behavioral sciences.

Similarly, human-computer interaction and collaboration is currently widely studied, but rarely for the specific topic of ontologies. Research is needed on methods to translate AI-derived knowledge from large and diverse datasets into ontology elements (e.g., incorporating results from prediction models based on machine learning) and the development of novel user interfaces and data visualizations that help human users to understand and use that knowledge.

These ideas may bear fruit in the future, but we emphasize that current technology is already supporting ontology development and use and is currently more than sufficient to support progress in the behavioral sciences.

NEEDED INSTITUTIONAL AND ORGANIZATIONAL SUPPORT

Without a doubt, developing an ontology entails a lot of hard work, community engagement, and iteration. Ontology engineering is therefore a very expensive endeavor and one that requires resources as well as specific actions and processes that are sustained. It also requires continual investment because any ontology will need to evolve as the relevant science changes. Currently, there are no clear road maps for establishing and sustaining an ontology in the behavioral sciences. Many existing, well-used scientific ontologies may not be in a secure financial position, and the situation seems to be even more precarious for ontologies in the behavioral sciences.

There are a few examples of scientific ontologies that endure as robust entities. In nearly all such cases, there is a substantial commitment of government funding (or there is a government mandate to use the ontology) that ensures the durability of these resources. The NCI Thesaurus has been an intramural project at NCI for nearly three decades. The International Classification of Diseases has been managed by the World Health Organization since its inception in 1948. In contrast, financial support for efforts from university laboratories has been extremely precarious (Baker, 2012).

There is also a need for support for the tools and practices of ontology engineering. The committee believes that tools and practices developed in other contexts are likely to be valuable to behavioral scientists as they pursue ontology development, but we acknowledge that there is as yet no empirical demonstrations of how they might work in the behavioral science domain. Iterative evaluation and testing of methods applied in new contexts will need to be integrated in the broader evaluations discussed above. A wide array of changes and advancements can potentially play an important role in supporting greater reliance on ontologies in the behavioral sciences. This section outlines some key gaps and shortcomings that will need to be addressed, which fall into three categories: discovery, capacity, and practice.

Discovery

One significant need is for new information, practices, and content based on novel research and discovery. On the socio-cognitive side, additional research is needed to develop best practices for creating, disseminating, teaching, and using ontologies in the behavioral sciences. While there has been some research on best practices for ontology engineering in other domains, those techniques have not yet been widely used in the behavioral sciences. Translational research could provide important evidence about how methods in ontology engineering may need to be updated and validated for the behavioral sciences.

Similarly, both foundational and translational research is needed for the development of the next generation of computational tools that can advance the capabilities and uses of ontologies. We emphasize that the

potential value of such research is not a reason to delay immediate progress in the development and use of behavioral ontologies. Nevertheless, the research directions listed above are likely to open up new possibilities.

Capacity

Shortfalls in implementation, and the capacity for implementation, of approaches whose value has already been demonstrated also need to be addressed. When what needs to be done is clear—and additional research is not needed—but the resources or capacity to do it are not currently available, progress is hampered. There is a need for additional resources to increase awareness and training regarding ontologies in the behavioral sciences. There are many individuals in library and information sciences, for example, who have experience with ontology creation and use in other domains but may not have opportunities or time to work with behavioral scientists. Other fields have developed mechanisms to train scientists in ontology-relevant approaches, such as informatics, as exemplified in the partnership of the National Institute of Dental and Craniofacial Research with NLM on continuing education training (T15) grants. At the same time, many of the relevant stakeholders in behavioral sciences may not have the computational training or the skills needed to work with ontologies and the relevant tools, and will thus need additional computational training or support. In addition, full utilization of both current and future computational tools will require significant increases in computational resources including computer time, data access, and server storage. Institutions and organizations will also likely require additional resources, particularly if they play increasingly prominent roles in the development, dissemination, and use of ontologies (as suggested by the success of the NCI Thesaurus).

Practices

A final set of needs involves practices and processes that could support wider use of ontologies in the behavioral sciences for which the capacity is already in place. As noted above, there are currently few explicit institutional incentives to use ontologies in the behavioral sciences, whether from journals, conferences, funding agencies, review committees, or other entities. Open data and code have become much more widespread as relevant institutions have required them. The movement toward open science depends on the existence of ontologies to enable comparisons between datasets, and recent trends in expectations for data sharing are changing this situation. Funders and publishers now often require data sharing, which requires the use of standardized metadata, which in turn requires the use of ontologies. But there have as yet been comparatively few community-level efforts to

build consensus about the use of ontologies in the behavioral sciences: there are many ways to encourage this, as we propose in Chapter 6. There are no perfect or completed ontologies, regardless of domain; ontologies are always subject to revision as the scientific community learns and changes. Nonetheless, experiences in other domains have shown that consensus around an ontology is possible, though it requires concerted efforts by researchers and institutions.

CONCLUSIONS

REFERENCES

Brenas, J.H., Shin, E. K., and Shaban-Nejad, A. (2019). Adverse Childhood Experiences Ontology for mental health surveillance, research, and evaluation: Advanced knowledge representation and Semantic Web techniques. JMIR Mental Health, 6(5), e13498. https://doi.org/10.2196/13498

Brochhausen, M., Spear, A.D., Cocos, C., Weiler, G., Martín, L., Anguita, A., Stenzhorn, H., Daskalaki, E., Schera, F., Schwarz, U., Sfakianakis, S., Kiefer, S., Dörr, M., Graf, N., and Tsiknakis, M. (2011). The ACGT Master Ontology and its applications—Towards an ontology-driven cancer research and management system. Journal of Biomedical Informatics, 44(1), 8–25. https://doi.org/10.1016/j.jbi.2010.04.008

Corcho, O., Fernández-López, M., and Gómez-Pérez, A. (2006). Ontological engineering: Principles, methods, tools and languages. In C. Calero, F. Ruiz, and M. Piattini (Eds.), Ontologies for Software Engineering and Software Technology, 1–48. Berlin: Springer Berlin Heidelberg. https://doi.org/10.1007/3-540-34518-3_1

Crandall, B., Klein, G.A., and Hoffman, R.R. (2006). Working Minds: A Practitioner’s Guide to Cognitive Task Analysis. Cambridge, MA: MIT Press.

Cui, L., Abeysinghe, R., Zheng, F., Tao, S., Zeng, N., Hands, I., Durbin, E.B., Whiteman, L., Remennik, L., Sioutos, N., and Zhang, G.Q. (2020). Enhancing the quality of hierarchic relations in the National Cancer Institute Thesaurus to enable faceted query of cancer registry data. Journal of Clinical Ontology Clinical Cancer Informatics, 4, 392–398. https://doi.org/10.1200/CCI.19.00124

Eisenberg, I.W., Bissett, P.G., Canning, J.R., Dallery, J., Enkavi, A.Z., Whitfield-Gabrieli, S., Gonzalez, O., Green, A.I., Greene, M.A., Kiernan, M., Kim, S.J., Li, J., Lowe, M.R., Mazza, G.L., Metcalf, S.A., Onken, L., Parikh, S.S., Peters, E., Prochaska, J.J., Scherer, E.A., Stoeckel, L.E., Valente, M.J., Wu, J., Xie, H., MacKinnon, D.P., Marsch, L.A., and Poldrack, R.A. (2018). Applying novel technologies and methods to inform the ontology of self-regulation. Behaviour Research and Therapy, 101, 46–57. https://doi.org/10.1016/j.brat.2017.09.014

Fairchild, K.M., Poltrock, S.E., and Furnas, G.W. (1988). SemNet: Three-dimensional graphic representation of large knowledge bases. In Guidon, R. (Ed.), Cognitive Science and its Applications for Human-Computer Interaction, 201–234. Hillsdale, NJ: Lawrence Erlbaum Associates, Hillsdale, N.J.

Falzon, L. (2021). Scoping Review of Ontologies in the Behavioral Sciences. Paper prepared for the Committee on Accelerating Behavioral Science Through Ontology Development and Use, National Academies of Sciences, Engineering, and Medicine. https://nap.nationalacademies.org/resource/26464/Falzon-comissioned-paper.pdf

Franco, M., Vivo, J.M., Quesada-Martínez, M., Duque-Ramos, A., and Fernández-Breis, J.T. (2020). Evaluation of ontology structural metrics based on public repository data. Briefings in Bioinformatics, 21(2), 473–485. https://doi.org/10.1093/bib/bbz00

Gan, Z., Chen, C., Henao, R., Carlson, D., and Carin, L. (2015). Scalable deep Poisson factor analysis for topic modeling. In International Conference on Machine Learning, 1823–1832. Proceedings of Machine Learning Research.

Ghaazvinian, A., Noy, N.F., and Musen, M.A. (2009). Creating mappings for ontologies in biomedicine: Simple methods work. AMIA Annual Symposium Proceedings, 2009, 198–202. https://pubmed.ncbi.nlm.nih.gov/20351849/

Gkoutos, G.V., Hoehndorf, R., Tsaprouni, L., and Schofield, P.N. (2015). Best behaviour? Ontologies and the formal description of animal behaviour. Mammalian Genome, 26(9-10), 540–547.

Gkoutos, G.V., Schofield, P.N., and Hoehndorf, R. (2012). The neurobehavior ontology: An ontology for annotation and integration of behavior and behavioral phenotypes. International Review of Neurobiology, 103, 69–87. https://doi.org/10.1016/B978-0-12-388408-4.00004-6

Graving, J.M., Chae, D., Naik, H., Li, L., Koger, B., Costelloe, B.R., and Couzin, I.D. (2019). DeepPoseKit, a software toolkit for fast and robust animal pose estimation using deep learning. eLife, e47994. https://doi.org/10.7554/eLife.47994

Gruber, T.R. (1995). Toward principles for the design of ontologies used for knowledge sharing? International Journal of Human-Computer Studies, 43(5–6), 907–928. https://doi.org/10.1006/ijhc.1995.1081

Hastings, J., and Schultz, S. (2012). Ontologies for human behavior analysis and their application to clinical cata. In E.M. Chesler and M.A. Haendel, Eds. Bioinformatics of Behavior: Part 1, 103, 89–107. Amsterdam, The Netherlands: Elsevier.

He, X., Zhang, R., Rizvi, R., Vasilakes, J., Yang, X., Guo, Y., He, Z., Prosperi, M., Huo, J., Alpert, J., and Bian, J. (2019). ALOHA: Developing an interactive graph-based visualization for dietary supplement knowledge graph through user-centered design. BMC Medical Informatics and Decision Making, 19(Suppl 4), 150. https://doi.org/10.1186/s12911-019-0857-1

Hicks, A., Hanna, J., Welch, D., Brochhausen, M., and Hogan, W.R. (2016). The ontology of medically related social entities: Recent developments. Journal of Biomedical Semantics, 7, 47. https://doi.org/10.1186/s13326-016-0087-8

Hitzler, P., Gangemi, A., and Janowicz, K. (Eds.). (2016). Ontology Engineering with Ontology Design Patterns: Foundations and applications, 25. Amsterdam, The Netherlands: IOS Press.

Hollnagel, E. (Ed.). (2003). Handbook of Cognitive Task Design. Boca Raton, FL: CRC Press.

Jackson, R.C., Balhoff, J.P., Douglass, E., Harris, N.L., Mungall, C.J., and Overton, J.A. (2019). ROBOT: A tool for automating ontology workflows. BMC Bioinformatics, 20(1), 407. https://doi.org/10.1186/s12859-019-3002-3

Jackson, R., Matentzoglu, N., Overton, J.A., Vita, R., Balhoff, J.P., Buttigieg, P.L., Carbon, S., Courtot, M., Diehl, A.D., Dooley, D.M., Duncan, W.D., Harris, N.L., Haendel, M.A., Lewis, S.E., Natale, D.A., Osumi-Sutherland, D., Ruttenberg, A., Schriml, L.M., Smith, B., Stoeckert, C. J., Jr, Vasilevsky, N.A., Walls, R.L., Zheng, J., Mungall, C.J., and Peters, B. (2021). OBO Foundry in 2021: Operationalizing open data principles to evaluate ontologies. Database: The Journal of Biological Databases and Curation, 2021, baab069. https://doi.org/10.1093/database/baab069

James, G., Witten, D., Hastie, T., and Tibshirani, R. (2013). An Introduction to Statistical Learning. New York: Springer.

Jensen, M., Cox, A.P., Chaudhry, N., Ng, M., Sule, D., Duncan, W., Ray, P., Weinstock-Guttman, B., Smith, B., Ruttenberg, A., Szigeti, K., and Diehl, A.D. (2013). The neurological disease ontology. Journal of Biomedical Semantics, 4(1), 42.

Jung, H., Park, H.A., and Song, T.M. (2016). Development and evaluation of an adolescents’ depression ontology for analyzing social data. Studies in Health Technology and Informatics, 225, 442–446.

Kanopka, B.M. (2015). Biomedical ontologies—A review. Biocybernetics and Biomedical Engineering, 35(2), 75–86.

Köhler, S., Doelken, S.C., Rath, A., Aymé, S., and Robinson, P.N. (2012). Ontological phenotype standards for neurogenetics. Human Mutation, 33(9), 1333–1339.

Kotov, R., Krueger, R.F., Watson, D., Cicero, D.C., Conway, C.C., DeYoung, C.G., Eaton, N.R., Forbes, M.K., Hallquist, M.N., Latzman, R.D., Mullins-Sweatt, S.N., Ruggero, C.J., Simms, L.J., Waldman, I.D., Waszczuk, M.A., and Wright, A. (2021). The Hierarchical Taxonomy of Psychopathology (HiTOP): A quantitative nosology based on consensus of evidence. Annual Review of Clinical Psychology, 17, 83–108. https://doi.org/10.1146/annurev-clinpsy-081219-093304

Lahey, B.B., Krueger, R.F., Rathouz, P.J., Waldman, I.D., and Zald, D.H. (2017). A hierarchical causal taxonomy of psychopathology across the life span. Psychological Bulletin, 143, (2), 142–186. https://doi.org/10.1037/bul0000069

Larsen, K.R., Michie, S., Hekler, E.B., Gibson, B., Spruijt-Metz, D., Ahern, D., Cole-Lewis, H., Ellis, R.J., Hesse, B., Moser, R.P., and Yi, J. (2017). Behavior change interventions: The potential of ontologies for advancing science and practice. Journal of Behavioral Medicine, 40(1), 6–22. https://doi.org/10.1007/s10865-016-9768-0

Lokker, C., McKibbon, K.A., Colquhoun, H., and Hempel, S. (2015). A scoping review of classification schemes of interventions to promote and integrate evidence into practice in healthcare. Implementation Science, 10, 27. https://doi.org/10.1186/s13012-015-0220-6

Martinez-Romero, M., Jonquet, C., O’Connor, M.J., Graybeal, J., Pazos, A., and Musen, M.A. (2017). NCBO Ontology Recommender 2.0: An enhanced approach for biomedical ontology recommendation. Journal of Biomedical Semantics, 8(1), 21. https://doi.org/10.1186/s13326-017-0128-y

Mathis, A., Mamidanna, P., Cury, K.M., Abe, T., Murthy, V.N., Mathis, M.W., and Bethge, M. (2018). DeepLabCut: Markerless pose estimation of user-defined body parts with deep learning. Nature Neuroscience, 21(9), 1281–1289. https://doi.org/10.1038/s41593-018-0209-y

Mlecnik, B., Galon, J., and Bindea, G. (2019). Automated exploration of gene ontology term and pathway networks with ClueGO-REST. Bioinformatics (Oxford, England), 35(19), 3864–3866. https://doi.org/10.1093/bioinformatics/btz163

Musen, M., and Protégé Team. (2015). The Protégé Project: A look back and look forward. AI Matters, 1(4), 4–12. https://doi.org/10.1145/2757001.2757003

Nelson, C.A., Butte, A.J., and Baranzini, S.E. (2019). Integrating biomedical research and electronic health records to create knowledge-based biologically meaningful machine-readable embeddings. Nature Communications, 10(1), 3045. https://doi.org/10.1038/s41467-019-11069-0

Nguyen, Q.H., and Le, D.H. (2021). Similarity calculation, enrichment analysis, and ontology visualization of biomedical ontologies using UFO. Current Protocols, 1(4), e115. https://doi.org/10.1002/cpz1.115

Norris, E., Finnerty, A.N., Hastings, J., Stokes, G., and Michie, S. (2019). A scoping review of ontologies related to human behaviour change. Nature Human Behaviour, 3(2), 164–172. https://doi.org/10.1038/s41562-018-0511-4

Norris, E., Hastings, J., Marques, M.M., Mutlu, A., Zink, S., and Michie, S. (2021). Why and how to engage expert stakeholders in ontology development: Insights from social and behavioural sciences. Journal of Biomedical Semantics, 12(1), 4. https://doi.org/10.1186/s13326-021-00240-6

Noy, N.F., Shah, N.H., Whetzel, P.L., Dai, B., Dorf, M., Griffith, N., Jonquet, C., Rubin, D.L., Storey, M.A., Chute, C.G., and Musen, M.A. (2009). BioPortal: Ontologies and integrated data resources at the click of a mouse. Nucleic Acids Research, 37(Suppl-2), W170–W173. https://doi.org/10.1093/nar/gkp440

Noy, N., Tudorache, T., Nyulas, C., and Musen, M. (2010). The ontology life cycle: Integrated tools for editing, publishing, peer review, and evolution of ontologies. AMIA Annual Symposium Proceedings, 2010, 552–556.

Obrst, L., Ceusters, W., Mani, I., Ray, S., and Smith, B. (2007). The evaluation of ontologies: Toward improved semantic interoperability. Semantic Web: Revolutionizing Knowledge Discovery in the Life Sciences, 139–158. https://www.researchgate.net/publication/225855591_The_Evaluation_of_Ontologies

Overton, J.A., Dietze, H., Essaid, S., Osumi-Sutherland, D., and Mungall, C.J. (2015). ROBOT: A command-line tool for ontology development. International Conference on Biomedical Ontologies. http://icbo2015.fc.ul.pt/demo6.pdf

Parsia, B., Matentzoglu, N., Gonçalves, R.S., Glimm, B., and Steigmiller, A. (2017). The OWL Reasoner Evaluation (ORE) 2015 competition report. Journal of Automated Reasoning, 59(4), 455–482. https://doi.org/10.1007/s10817-017-9406-8

Pereira, T.D., Shaevitz, J.W., and Murthy, M. (2020). Quantifying behavior to understand the brain. Nature Neuroscience, 23(12), 1537–1549. https://doi.org/10.1038/s41593-020-00734-z

Poldrack, R., and Yarkoni, T. (2016). From brain maps to cognitive ontologies: Informatics and the search for mental structure. Annual Review of Psychology, 67, 587–612. https://doi.org/10.1146/annurev-psych-122414-033729

Rensselaer Polytechnic Institute. (2013). The general ontology evaluation framework (GOEF): A proposed infrastructure for the ontology development lifecycle. http://ontolog.cim3.net/file/work/OntologySummit2013/2013-03-14_OntologySummit2013_Ontology-Evaluation-Quality-Methodology-2/OntologySummit2013_GOEF_iChoose—JoanneLuciano_20130314.pdf

Rizvi, R., Vasilakes, J., Adam, T.J., Melton, G.B., Bishop, J.R., Bian, J., Tao, C., and Zhang, R. (2020). iDISK: The Integrated Dietary Supplements Knowledge Base. Journal of the American Medical Informatics Association, 27(4), 539–548. https://doi.org/10.1093/jamia/ocz216

Sioutos, N., de Coronado, S., Haber, M.W., Hartel, F.W., Shaiu, W.L., and Wright, L.W. (2007). NCI Thesaurus: A semantic model integrating cancer-related clinical and molecular information. Journal of Biomedical Informatics, 40(1), 30–43. https://doi.org/10.1016/j.jbi.2006.02.013

Stancin, K., Poscic, P., and Jaksic, D. (2020). Ontologies in education—state of the art. Education and Information Technologies, 25(6), 5301–5320.

Vrandečić, D. (2009). Ontology evaluation. In S. Staab and R. Studer (Eds.), Handbook on Ontologies. Springer. https://doi.org/10.1007/978-3-540-92673-3_13

Whetzel, P., and NCBO Team. (2013). NCBO technology: Powering semantically aware applications. Journal of Biomedical Semantics, 4(Suppl 1), S8. https://doi.org/10.1186/2041-1480-4-S1-S8

Win, K.T., Ramaprasad, A., and Syn, T. (2019). Ontological Review of Persuasion Support Systems (PSS) for Health Behavior Change through Physical Activity. Journal of Medical Systems, 43(3), 49. https://doi.org/10.1007/s10916-019-1159-y

Woznowski, P.R., Tonkin, E.L., and Flach, P.A. (2018). Activities of Daily Living Ontology for Ubiquitous Systems: Development and Evaluation. Sensors, 18(7):20.

Wu, H., and Yamaguchi, A. (2014). Semantic Web technologies for the big data in life sciences. Bioscience Trends, 8(4), 192–201. https://doi.org/10.5582/bst.2014.01048

Yao, L., Divoli, A., Mayzus, I., Evans, J.A., and Rzhetsky, A. (2011). Benchmarking ontologies: Bigger or better? PLoS Computational Biology, 7(1), e1001055.

Yu, C., and Shen, B. (2016). XML, ontologies, and their clinical applications. Advances in Experimental Medicine and Biology, 939, 259–287. https://doi.org/10.1007/978-981-10-1503-8_11

Zhu, Q., Kong, X., Hong, S., Li, J., and He, Z. (2015). Global ontology research progress: A bibliometric analysis. Aslib Journal of Information Management, 67(1), 27–54.

This page intentionally left blank.