1

Introduction

Charles Dickens opened his epic novel A Tale of Two Cities with the statement, “It was the best of times, it was the worst of times.” In many ways, the behavioral sciences are in the best of times. Collectively, humans now know more about behavior than at any point in our history as a species. The pace of scientific discovery is unprecedented, with new clinical trials and experimental research published every day. Psychology, neuroscience, cognitive science, and other fields that build understanding of experiences and behaviors are among the most popular undergraduate majors. The number of published papers in these fields is growing at an exponential rate. Behavioral scientists are among respected experts serving on advisory panels in medicine, law, education, defense, business, and public policy.

Yet despite these advances, the behavioral sciences are facing substantial challenges that constrain scientific progress. These challenges, which ironically have been exacerbated by the many research possibilities opened up by rapid advances in computer technology, hamper the integration of findings from individual studies to accumulate bodies of knowledge. They complicate the efficient transmission of this knowledge to the consumers who can use it to benefit individuals and society. Although concerns about the “wealth of information” problem have been noted for decades (e.g., Simon, 1971), the gap between what is known and the capacity to act on what is known has never been larger, and it continues to grow. In this respect, the behavioral sciences are in the worst of times.

This report examines how a stronger commitment to the use of ontologies—formal systems for organizing knowledge—can help to address these and related challenges in the behavioral sciences. The National

Academies of Sciences, Engineering, and Medicine formed a committee to study ways of accelerating the behavioral sciences by improving the development and use of ontologies. The work was supported by four divisions of the National Institutes of Health (the Office of Behavioral and Social Sciences Research, the National Institute on Aging, the National Library of Medicine, and the National Cancer Institute), the National Science Foundation, the American Psychological Association, the Association for Psychological Science, and the Federation of Associations in Behavioral and Brain Sciences. The committee, which included experts in medicine, population health, psychology, psychiatry, biobehavioral sciences, biomedical informatics, neural and cognitive science, library and information science, the history and philosophy of science, computer science, and bioengineering, was charged with reviewing the literature on ontologies in the behavioral sciences and example ontologies in other sciences and developing recommended approaches for improving them; the complete charge is shown in Box 1-1.

CONTEXT

A Long History of “Classification”

Attempts to synthesize and summarize what is known about the world date back thousands of years. Scholars and nonscholars alike have faced the problem of how to organize knowledge and to integrate new observations with what is already known. One of the most influential methods of organizing knowledge about the world was launched during the 4th century BC by Aristotle, who attempted to find order among chaos in the natural world: see Box 1-2 and Figure 1-1. In AD 77 Pliny the Elder published Natural History, an effort to categorize all that was known. To make it easy to retrieve desired information, he provided a detailed table of contents that guided readers to the departments, or categories of knowledge. The Fihrist published by Ibn al-Nadim in 938 was a similar effort to compile and categorize knowledge (in this case a listing of available books) by discipline and topic. Also during the 10th century, scholars in China produced a massive compendium of knowledge, the Taiping Yulan, which organized knowledge under 5,000 headings across 55 categories (Blair, 2010). More modern examples emerged during the Enlightenment: for example, Denis Diderot proposed the Encyclopédie, co-edited with Jean le Rond d’Alembert and originally published between 1751 and 1772, as a means to fundamentally change how humans think and what humans know. This history suggests that the drive to make sense of and communicate about the world is profoundly human.

Philosophers use the term ontology (literally, discourse on being) to describe efforts to classify or group ideas, particularly those related to the nature of existence. Scientists today use the word ontology to refer to efforts to organize knowledge in particular domains. Although there is no

universal definition of a scientific ontology, a valuable working definition is an explicit, formal specification of a shared conceptualization—a systematic set of shared terms and an explication of their interrelationships (Gruber, 1995). For a simple example, an ontology might define and categorize types of ice cream products, distinguishing among those served in vessels, in cones, and on sticks, as shown in Figure 1-2.

Today, machine-assisted methods offer feasible, affordable, and scalable supports to the process of organizing knowledge. Computer technology has changed scientific research by making possible many new methods for collecting, sorting, and analyzing increasingly large datasets. Across scientific

disciplines, the volume of data generated presents tremendous opportunities, but it also amplifies the challenges of structuring, mining, integrating, and reusing information: those challenges demonstrate the need for and applications of ontologies. Using computer technology to develop and maintain ontologies involving potentially vast quantities of data has offered the potential for significant changes in the ways human beings interact with scientific knowledge.

The natural and biomedical sciences have made substantial progress toward the development and application of accepted ontologies. For example, the Gene Ontology, which was developed by a consortium of researchers and is used around the world, has supported advances and new insights in genetics that would not have been possible without a widely shared, aggregate knowledge base (see Appendix A).¹ Classification systems used in other domains, such as the Iconclass project developed by art historians, which classifies types of images for retrieval, demonstrate growing recognition of the potential for using computer technology to

___________________

¹ See du Plessis et al. (2011) and Gene Ontology Consortium (2019) for descriptions of the gene ontology and examples of how the ontology is typically used.

structure large bodies of data.² As in other scientific contexts, ontologies used in the behavioral sciences (e.g., to categorize the components of psychological well-being or to classify thought processes) may help scientists in a variety of ways. In this report we examine how they can help scientists to, for example, link results from diverse research, communicate clearly about complex concepts, more rapidly identify significant knowledge gaps, formulate novel questions, test clear hypotheses, establish whether results can be reproduced, and retrieve and apply scientific knowledge for diverse uses.

___________________

² See https://test.iconclass.org/. Iconclass provides phrases to identify search terms included in the database; a new version of it currently being developed links terms to exemplar images.

Particular Challenges in the Behavioral Sciences

While ontologies have been embraced by many sciences, developing, establishing, and sustaining them has been a significant challenge in the behavioral sciences. Describing and measuring behavioral phenomena is inherently complex, and even in many subfields of study there is a lack of well-established, widely shared definitions for key concepts. The lack of precision in terminology makes it difficult to extract and combine data from diverse research contexts and to understand relationships among phenomena.

Consideration of one example, stress, suggests the scope of the challenge. Stress can be understood as a stimulus (a stressor); a psychological process (e.g., an individual’s experience of how stressful the stimulus is); a biological mechanism (e.g., cortisol secretion) that causes some outcome (such as a physiological response or a subjective feeling); or the outcome itself (e.g., feeling stressed). Scientists use the term in studying phenomena as diverse as living in poverty, getting divorced, or experiencing chronic illness, and they do not uniformly specify precisely what they mean by stress in describing their research (Crosswell and Lockwood, 2020). Exposure to situations that cause stress—and responses to stress—are measured with such tools as self-report questionnaires, interviews, and other psychological measurements, and by identifying people exposed to obvious stressors, such as natural disasters. Assessments of physiological responses such as heart rate and cortisol levels have become more common as available technology has advanced (Bellido et al., 2018). Responses to stress can be psychological and behavioral. Signs of stress can include phenomena that may be difficult to isolate, such as anger and irritability, sleep disturbance, or feeling overwhelmed, or health consequences, from headaches to increased risk of cancer or cardiovascular disease. Many factors may influence the effects of stress exposure, including the context and time span of the exposure, the severity of the stressor, the life stage at which it occurs, and the degree to which it is controllable. Moreover, there is no reliable, specific relationship between the presence of a stressful stimulus and its correlates (e.g., sleep disturbance does not always occur in response to stress and can occur in response to other conditions).

Not surprisingly, given this complexity, there is a very large literature on aspects of stress. Yet it is difficult to be sure what “stress” means across these diverse studies, and thus to make informed inferences or broad conclusions about stressors and their effects from the body of research that has explored this multifaceted set of phenomena.

Researchers in the behavioral sciences seek to describe observed phenomena, to understand the causes for those phenomena, to use this knowledge to improve psychological and physical health outcomes, and to predict who might be affected in the future. Yet the distinctions between describing and explaining observed phenomena are not always clear. Even defining the

boundaries of the behavioral sciences is challenging. Growing evidence from the study of development, for example, has shown how biological and social influences interact even prenatally to shape gene expression as well as physical and cognitive development (NASEM, 2019). This reality points to the critical importance of research that integrates work from across and beyond the behavioral sciences.

Despite a number of efforts, including the U.K.’s Human Behaviour-Change Project and the Cognitive Atlas (Michie et al., 2017; Poldrack et al., 2011), there are few widely used ontologies in the behavioral sciences; see Chapter 5. The committee was asked to explore reasons to believe that advances in the development of ontologies can help to address the challenges behavioral science researchers face and thereby accelerate progress in these fields. Such progress would not just be of academic interest; it would also hasten the development of interventions that promote health and relieve suffering and support practitioners and others who rely on behavioral research.

COMMITTEE’S APPROACH TO ITS CHARGE

The committee’s approach to the charge began with an effort to reach a shared understanding of the meaning of the key terms we were to address, ontologies in the behavioral sciences, and the basic purposes they serve. Though the committee adopted the most widely used definition of ontology (a formal, explicit specification of a shared conceptualization; Gruber, 1995; see Chapter 3), we acknowledge that both ontology and the behavioral sciences are terms that lack precise boundaries. Ontology is not a word used frequently in everyday discourse, perhaps even among scientists. Some people may remember the concept from a college philosophy class but have only a hazy recollection of what it really means. A crisp definition of the behavioral sciences may be similarly elusive, but it was important for the committee to identify parameters to guide our work. Chapter 3 discusses the meaning of ontology in more detail;³ here we review ways to define the behavioral sciences and the general purposes ontologies serve in this context.

The Behavioral Sciences

The development of specific branches of science is a dynamic and evolving process. Indeed, the identification of branches of science across centuries reflects ongoing ontological thinking about categories of knowledge. This shifting landscape makes it hard to derive a consensus definition of any

___________________

³ In this report, we generally use the term “ontology” to refer to a range of systems that have been developed to enumerate the essential entities in a discipline, though not all of the systems may meet the precise definition; see Chapter 3 for a detailed discussion.

branch of science, including the behavioral sciences. The Encyclopedia Britannica defines behavioral science as any discipline that deals with human behavior or actions (Editors of Encyclopaedia Britannica, n.d.), a definition that would include such disciplines such as psychology, anthropology, sociology, economics, law, psychiatry, political science and the behavioral aspects of biology; the American Psychological Association defines behavioral science as including research on nonhuman animals (Wolman, 1989).

Although the behavioral and social sciences are sometimes regarded as synonymous, some important distinctions can be drawn. The behavioral sciences tend to focus on the study of what happens within and between individuals; the social sciences tend to focus on factors within and across social systems and levels—ranging from those closest to the individual to broad societal and cultural ones—that help to explain complex social behavior. Behavioral scientists may consider system-level influences on behavior, but they are primarily interested in the operationalization and generation of behavioral variables (Adhikari, 2016). The distinction is not crisp, however, because researchers in such fields as social psychology, anthropology, and sociology may blend interest in individuals and groups, taking a more behavioral or a more social approach depending on the target of their investigations. Both classical and behavioral economics are concerned with individual decision making, while political scientists are interested in a host of factors that influence voting and other behavior. Researchers across these domains may use experimental methods (manipulating an independent variable to isolate biological, psychological, or social factors that influence behavior), observational methods and modeling, or they may conduct studies designed to describe or classify behaviors. The behavioral sciences are inherently interdisciplinary, and newer fields of study, such as behavioral economics, continue to challenge traditional disciplinary distinctions.

It is also worth noting the contrasts between the social sciences and the natural sciences and the fact that the behavioral sciences have elements of both. The natural sciences can be said to focus primarily on events in the physical world, that is, phenomena that occur and exist regardless of whether humans perceive them; the social sciences can be said to focus on events in the social world, which is constructed by humans. The behavioral sciences, which focus on the individual or individuals as they are relating to one another, have a foot in both the social and natural camps because they encompass study of the relationships among psychological constructs and underlying biological influences on behavior. Within universities, psychology departments are typically housed in colleges of science, but they are sometimes in colleges of social sciences.

Acknowledging the interdisciplinary nature of the behavioral sciences, we adopt a broad definition of the behavioral sciences: “behavioral sciences refer to the social and biological sciences concerned with the study of behavior”

(Cascio, 2015, p. 348). Regardless of the definition, however, the behavioral sciences encompass a landscape that was far too broad for the committee to examine systematically. It would not have been possible to search the literature for discussion of ontologies in even a fraction of the disciplines that fall under the behavioral umbrella to try to understand how ontologies are developed and used in each of those contexts, or to investigate the particular challenges to ontology development and use in each. The committee, therefore, focused its information gathering on the domain of mental health. We explored issues from this domain in detail in order to develop our thinking about how ontologies function and serve science and to develop conclusions and recommendations that could generalize across the behavioral domain.

Use Cases

Ontologies are not built for their own sake, but rather to serve identified shared goals and concerns about human behavior. We explore these goals and concerns throughout the report, but the foundations for this report are twofold: their basic utility in the behavioral sciences and the concept of “use cases” (cited in the study charge)—a term coined in the context of software engineering to refer to situations in which software is usefully applied or to which it responds.

It is important to distinguish an ontology from its use cases. In simplest terms, a use case is a narrative description or story involving someone interacting with a system to achieve a particular goal (Larman, 2004; Leffingwell and Widrig, 2003). Given the utility of use cases for application design, use case modeling has been highly formalized in software engineering.⁴ Use cases are essentially envisioned scenarios (i.e., models) for what a system should do to help someone achieve a goal in a given context. Thus, they are explicitly different from case examples, case studies, or exemplars, which represent fully realized instances of systems or applications.

Use case modeling is a valuable tool for designing an ontology.⁵ It is possible to enumerate and illustrate uses cases in terms of five parameters: (1) actors, (2) behaving in a particular context, (3) using a resource, (4) to achieve an expected outcome, (5) that may affect additional stakeholders.⁶

___________________

⁴ For example, the Unified Modeling Language (UML), a widely used graphical language for visualizing, designing, documenting, and building object-based systems, represents use cases as one of its 13 core modeling diagrams.

⁵ In the context of evidence-based reasoning, the first step in ontology design typically involves determination of the concepts of interest, which inherently involves envisioning the high-level goals that the ontology will be able to serve within a specific domain with respect to individuals, agents, or organizations with clearly defined roles (Tecuci et al., 2016).

⁶ This approach is based on a simplification of the UML methodology described in Randolph (2004).

The committee developed a working formulation of this rubric, using the acronym ACRES: actors, context, resources, expected (outcome), stakeholders. In the context of the behavioral sciences, the resource might be any ontological entity (e.g., a list of concepts and relationships) or ontologically enabled system (e.g., an application that uses an ontology to perform its functions). An example from the biomedical context is the Resource Discovery System (see Tenebaum et al., 2011). The committee’s working definitions of the other terms illustrate the relevance of use case modeling for ontology use and development in the behavioral sciences:

• Actor: Anyone (or anything) who performs a behavior to put a demand on the resource or system. Actors can be people, usually identified by roles (e.g., teacher, student, caregiver, scientist), but can also include organizations (e.g., a hospital, a school board), as well as computer systems (e.g., a software application).
• Context: The conditions that must be true or present before the use case proceeds (e.g., in a classroom; at bedtime; all parties using Hindi as their spoken language). The context sets constraints on the use case (and on the relevant ontologies), such that there is no presumption of a universally appropriate ontology, set of terms, concepts, or relationships.
• Resource: Any ontological entity (i.e., an ontology or its components or derivatives) or any system or object that is proximally informed or enabled by an ontological entity (e.g., a visualization tool that illustrates concepts; a knowledge graph; a search engine).
• Expected outcome: The goal state or preferred state of the actor (e.g., to have graduate students learn a fact; to retrieve a local summary of related claims for mental health services)
• Stakeholder: Any individual or entity with vested interests in the behavior of the resource or system under discussion or who may be affected by it. Since many people are stakeholders by virtue of their roles, an individual may have multiple stakeholder identities. Stakeholders may be the actors (e.g., a teacher who might use a resource) or the people who are affected by the action (e.g., the students affected by a teacher’s use of the resource).

Box 1-3 presents a list of possible stakeholders to show the broad set of use cases for behavioral science ontologies and ontology resources.

To gain insight into the range of potential use cases for ontologies in the behavioral science, the committee conducted an informal self-survey. Participating committee members enumerated possible use cases, specific ways different actors in a given context could use an ontological resource to achieve expected results relevant to a set of stakeholders. This exercise

was not a means to model or design specific ontology applications, but simply to assess the potential of behavioral science ontologies to facilitate important goals.⁷

Despite the small sample involved with this exercise, a considerable diversity of use cases emerged; see Appendix B. Researchers were the most commonly cited actors, but health care providers, policy makers, educators, students, administrators, and the general public were also cited. Identified contexts included research workplaces, health care facilities, online or mobile devices, educational institutions, government offices, and homes. Broader contexts were also noted, such as general conditions (e.g., “during a pandemic”).

___________________

⁷ The 10 committee members who participated in the self-survey generated 225 data elements across 46 use cases. Use cases followed the ACRES structure, such that any use case could be articulated as a narrative sentence. For example, one use case specified “expert practitioners [actors] in doctors’ offices [context] use an ontology of key mental health symptoms and disorders that arise during a pandemic [resource] to improve communication about mental health [expected result] with children and parents [stakeholders].”

Expected results or goals of use cases showed the greatest diversity, including:

• scientific goals (e.g., “theory advancement,” “improving methodological rigor,” “inform research priorities”);
• health-related goals (e.g., “to manage diet,” “to obtain guidance on better sleep,” “to promote masking or social distancing”);
• educational goals (e.g., “to prepare a dissertation,” “to improve diagnostically relevant knowledge”); and
• community-focused goals (e.g., “to educate citizens about better managing their own finances,” “to empower citizens to resolve situations involving law enforcement,” “to learn a new language”).

In terms of beneficiaries, the public was the most frequently cited beneficiary of ontological resources but researchers, health care recipients, students, administrators, health care providers, policy makers, and educators were also noted.

Although this was not a formal investigation, it suggested the diversity of applications of behavioral science ontologies and the importance of ontologies to both actors and beneficiaries not directly involved in science. The committee considered these results a useful standard of comparison for behavioral science ontologies currently in existence—how existing ontologies serve the potential range of use cases it is possible to imagine.

STUDY PROCESS: FOUR KEY QUESTIONS

The committee recognized that some scholars and stakeholders may question the value of developing and using ontologies in the behavioral sciences, while others have high aspirations for their potential benefits. We began our work from an agnostic stance, eager to better understand how ontologies have actually operated in the behavioral sciences, the challenges of developing and sustaining ontologies in this context, and the possibilities they offer for supporting advances in behavioral research. Developing our conclusions and recommendations did not require the committee to take a position on ontology-related controversies in the behavioral sciences, but rather to assess the available evidence to determine how increased attention to ontologies could advance work in this domain. This report provides our answers to four basic questions:

• Why do ontologies matter?
• What exactly are ontologies?

• How do ontologies facilitate science?
• How can the engineering of ontologies in the behavioral sciences be strengthened?

Although the study charge specifies the behavioral sciences, we note that many, if not most of the committee’s answers to these questions, and our conclusions and recommendations, apply beyond the behavioral domain.

Answering these four questions required investigation of methods, philosophy of science, and ontology development and use cases, in addition to analysis of existing evidence about the impacts of behavioral ontologies. The committee sought to understand the conceptual issues that pertain to ontologies, regardless of the mechanisms by which they are developed, and also to understand in detail how technological innovations have influenced the way behavioral ontologies are developed and used. We examined how ontologies themselves have been studied, the history of ontology use in the behavioral sciences, useful examples from other domains, identifiable characteristics or patterns in the ontologies that have proved effective and sustainable, and the challenges that developers and users of ontologies encounter.

The two primary sources of information we could review were (1) published research about individual ontologies or about the role of ontologies and their development and use, both in science in general and in the behavioral sciences, and (2) investigation of example ontologies in the behavioral sciences and related fields. As we will discuss, particularly in Chapters 3 and 5, existing ontologies and related knowledge structures in the behavioral sciences vary in significant ways and are not easily counted or categorized, but there are fewer ontologies in the behavioral domain than in other scientific domains. Thus, the examples we explored are those that are well known or important for varied reasons, rather than a systematically derived sample.

The committee held two public workshops at which invited experts presented information about individual example ontologies and offered historical and philosophical perspectives on their role in the behavioral sciences.⁸ We reviewed published literature and commissioned five papers to deepen our understanding of a range of topics:⁹ see Box 1-4. Subgroups of the committee also held structured discussions with experts about key issues. The workshops and papers allowed us to look closely at example ontologies that reflected a range of contexts; we refer to them throughout the report.

___________________

⁸ The workshop agendas and presentations are available on the project website, at https://www.nationalacademies.org/our-work/accelerating-social-and-behavioral-science-through-ontology-development-and-use

⁹ The commissioned papers are available at https://nap.nationalacademies.org/catalog/26464/ontologies-in-the-behavioral-sciences-accelerating-research-and-the-spread

These sources provided valuable information and insights but, as we discuss in Chapter 5, the existing literature did not offer empirical answers to many of our questions. In order to provide a response to this important study charge that could be useful now, the committee deliberated and arrived at judgments based on the information available. The committee met formally five times, once in person, and also collaborated using Zoom and other technologies throughout the process of digesting the information and developing the report.

GUIDE TO THIS REPORT

The report is structured by the committee’s four questions, listed above. It begins, in Chapter 2, with a look at why ontologies are important in the first place: the scientific problems an ontology can address, the relationship between those problems and challenges in translating behavioral research into practice that can improve health and alleviate suffering, and the ways stakeholders are affected by those challenges. Chapter 3 provides a technical and detailed

discussion of what an ontology is, in the context of the behavioral sciences. Chapter 4 explores how ontologies can advance scientific progress in the behavioral sciences. In Chapter 5 we turn to the question of how ontologies could be used more effectively in the behavioral sciences, beginning with an overview of what is known about existing ones and then exploring the components needed to engineer ontologies. The report closes in Chapter 6 with an overview of the committee’s primary conclusions about the development and use of ontologies, and recommendations for supporting and sustaining efforts to more fully integrate ontologies in the behavioral sciences. Appendix A provides basic information about example ontologies referred to in this report; Appendix B lists the example use cases generated in a committee self-survey; and Appendix C provides biographical sketches of the committee members and staff.

REFERENCES

The Gene Ontology Consortium. (2019). The Gene Ontology resource: 20 years and still GOing strong. Nucleic Acids Research, 47, D331-D338. https://doi.org/10.1093/nar/gky1055

Gruber, T.R. (1995). Toward principles for the design of ontologies used for knowledge sharing? International Journal of Human-Computer Studies, 43(5-6), 907–928. Available: https://doi.org/10.1006/ijhc.1995.1081

Hastings, J. (2021). Formal Representations of Ontologies for Automation of Analyses. Commissioned paper prepared for the Committee on Accelerating Behavioral Science Through Ontology Development and Use, National Academies of Sciences, Engineering, and Medicine. Available: https://nap.nationalacademies.org/resource/26464/Hastings-comissioned-paper.pdf

Hodos, W., and Campbell, C.B.G. (1969). Scala naturae: Why there is no theory in comparative psychology. Psychological Review, 76(4), 337–350. Available: https://doi.org/10.1037/h0027523

Kendler, K.S. (2021). Problems and Solutions in the Implementation of Integrative Pluralism in Psychiatric Genetics Research. Commissioned paper prepared for the Committee on Accelerating Behavioral Science Through Ontology Development and Use, National Academies of Sciences, Engineering, and Medicine. Available: https://nap.nationalacademies.org/resource/26464/Kendler-comissioned-paper.pdf

Larman, C. (2004). Apply UML and Patterns: An Introduction to Object-Oriented Analysis and Design and the Unified Process. https://personal.utdallas.edu/~chung/SP/applying-uml-and-patterns.pdf

Leffingwell, D., and Widrig, D. (2003). Managing Software Requirements: A Use Case Approach (2nd ed.). Boston, MA: Addison-Wesley.

Michie, S., Thomas, J., Johnston, M., Aonghusa, P. M., Shawe-Taylor, J., Kelly, M. P., Deleris, L. A., Finnerty, A. N., Marques, M. M., Norris, E., O’Mara-Eves, A., and West, R. (2017). The Human Behaviour-Change Project: Harnessing the power of artificial intelligence and machine learning for evidence synthesis and interpretation. Implementation Science, 12(1), 121. https://doi.org/10.1186/s13012-017-0641-5

NASEM (National Academies of Sciences, Engineering, and Medicine). (2019). Fostering Healthy Mental, Emotional, and Behavioral Development in Children and Youth. Washington, DC: National Academies Press. https://doi.org/10.17226/25201

Poldrack, R.A., Kittur, A., Kalar, D., Miller, E., Seppa, C., Gil, Y., Parker, D.S., Sabb, F.W., and Bilder, R.M. (2011). The Cognitive Atlas: Toward a knowledge foundation for cognitive neuroscience. Frontiers in Neuroinformatics, 5, 17. https://doi.org/10.3389/fninf.2011.00017

Randolph, G. (2004). Use-cases and personas: A case study in light-weight user interaction design for small development projects. Information Science, 4, 105–116. https://doi.org/10.28945/505

Simon, H.A. (1971). Designing organizations for an information-rich world. In M. Greenberger (Ed.), Computers, Communications, and the Public Interest, 37–52. Baltimore, MD: Johns Hopkins University Press.

Tecuci, G., Marcu, D., Boicu, M., and Schum, D.A. (2016). Knowledge Engineering: Building Cognitive Assistants for Evidence-Based Reasoning. Cambridge, U.K.: Cambridge University Press.

Tenenbaum, J.D., Whetzel, P.L., Anderson, K., Borromeo, C.D., Dinov, I.D., Gabriel, D., Kirschner, B., Mirel, B., Morris, T., Noy, N., Nyulas, C., Rubenson, D., Saxman, P.R., Singh, H., Whelan, N., Wright, Z., Athey, B.D., Becich, M.J., Ginsburg, G.S., Musen, M.A., Smith, K.A., Tarantal, A.F., Rubin, D.L., and Lyster, P. (2011). The Biomedical Resource Ontology (BRO) to enable resource discovery in clinical and translational research. Journal of Biomedical Informatics, 44(1), 137–145. https://doi.org/10.1016/j.jbi.2010.10.003

Wolman, B.B. (Eds.). (1989). Dictionary of Behavioral Science (2nd ed.). Cambridge, MA: Academic Press.

This page intentionally left blank.