6

Analysis of Institute and Center Autoimmune Disease Research Activity

Both extramural and intramural autoimmune disease research occurs across the various Institutes, Centers, and Offices at the National Institutes of Health (NIH), including the Office of the Director (OD). As noted in Chapter 5, total spending for autoimmune disease research was $1,083,000,000 for fiscal year (FY) 2020 (NIH, 2020a), a 2.6 percent increase from FY 2019. NIH’s investment in autoimmune disease research has remained constant at about 2.8 percent of its total obligations between 2008 and 2020 (Figure 6-1).^1,2

From 2015 to 2020, total NIH obligations increased by 40 percent, whereas autoimmune disease spending increased by approximately 34 percent. Over the 2008–2020 period, NIH-funded autoimmune disease research activities totaled approximately $11.7 billion. The highest levels of spending on autoimmune disease occurred in 2009 and more recently in 2020 (Figure 6-2).

As noted in Chapter 5, some Institutes and Centers (ICs) have a greater focus on autoimmune diseases tied to their mission, and thus there is considerable variation in the level of spending on autoimmune disease research activities. The OD also contributes funds to support autoimmune disease research activities. Table 6-1³ shows total Institute, Center, and OD obligations, autoimmune disease spending, autoimmune

___________________

¹ All NIH RePORTER data used in this report was accessed on July 16, 2021.

² See Appendix G for methodology.

³ See Appendix G for methodology.

TABLE 6-1 IC FY 2020 Actual Total Obligations, Autoimmune Disease Spending, and Average Autoimmune Disease Spending for FY 2008–2020

NOTES: ICs are arranged in descending order according to FY 2020 actual total obligations. The Office of the Director (OD) is not an institute but is included because of its funding contributions to autoimmune disease research. FY, fiscal year; IC, Institute or Center; NCATS, National Center for Advancing Translational Sciences; NCI, National Cancer Institute; NEI, National Eye Institute; NHGRI, National Human Genome Research Institute; NHLBI, National Heart, Lung, and Blood Institute; NIAID, National Institute of Allergy and Infectious Diseases; NIAMS, National Institute of Arthritis and Musculoskeletal and Skin Diseases; NICHD, Eunice Kennedy Shriver National Institute of Child Health and Human Development; NIDCR, National Institute of Dental and Craniofacial Research; NIDDK, National Institute of Diabetes and Digestive and Kidney Diseases; NIEHS, National Institute of Environmental Health Sciences; NIGMS, National Institute of General Medical Sciences; NINDS, National Institute of Neurological Disorders and Stroke; OD, Office of the Director.

SOURCES: NIH, 2021b, 2021f.

spending as a percentage of total obligations, and average spending over the 2008–2020 period.

In 2020, the ICs with the highest level of spending on autoimmune disease research were the National Institute of Diabetes and Digestive and Kidney Diseases (NIDDK), National Institute of Allergy and Infectious Diseases (NIAID), and National Institute of Arthritis and Musculoskeletal and Skin Diseases (NIAMS). NIAMS autoimmune disease spending as a percentage of actual total obligations was the highest at 23.6 percent, followed by NIDDK (15.6 percent). Although NIAID had the third highest level of autoimmune disease spending, it was only about 5 percent of its total obligations in FY 2020. These three Institutes also had the highest average spending on autoimmune disease research during the 2008–2020 period. In 2020, of the total $1,125,730,564 spent on autoimmune disease research, NIDDK (30.8 percent), NIAID (24.7 percent) and NIAMS (13.1) spending accounted for almost 70 percent of the total autoimmune disease spending (Table 6-2).⁴

CHARACTERISTICS OF RESEARCH ACTIVITIES SUPPORTED BY ICS

NIH uses 245 activity codes to characterize the funding mechanisms it employs to fund diverse activities. The committee focused on a select number of activity codes to characterize NIH- and IC-funded autoimmune disease research activities. The committee used RePORTER activity code data to assess autoimmune disease research activities by extramural research (R and P grants); cooperative agreements (U grants); and fellowship, training, and training center activities (F, K, and T grants); and intramural research activities (Z grants) (Table 6-3). Figure 6-3 provides NIH autoimmune disease funding for R, U, and P grants for FY 2008–2020 and the average funding for these grants over that period.

Research project (R) grant funding for autoimmune disease research was highest in 2020 and lowest in 2008 and during 2013–2016. Funding for cooperative agreements (U) grants was greater than funding for research program project and center grants (P) during FY 2008–2020. Funding for cooperative agreements increased from 2008 to 2017, dropped in 2018, and began to rise again in 2019 and 2020. Funding for research programs project and centers was slightly higher in 2009–2011 and remained constant after 2011.

Training grants contribute to building the autoimmune disease research workforce and to advancing research. Table 6-4⁵ shows NIH funding for fellowships (F), research career programs (K), and institutional

___________________

⁴ See Appendix G for methodology.

⁵ See Appendix G for methodology.

TABLE 6-2 IC FY 2020 Autoimmune Disease Spending and Percentage of Total NIH 2020 Autoimmune Disease Spending

NOTES: ICs are arranged in descending order according to FY 2020 actual total obligations. The Office of the Director (OD) is not an institute but is included because of its funding contributions to autoimmune disease research. FY, fiscal year; IC, Institute or Center; NCATS, National Center for Advancing Translational Sciences; NCI, National Cancer Institute; NEI, National Eye Institute; NHGRI, National Human Genome Research Institute; NHLBI, National Heart, Lung, and Blood Institute; NIAID, National Institute of Allergy and Infectious Diseases; NIAMS, National Institute of Arthritis and Musculoskeletal and Skin Diseases; NICHD, Eunice Kennedy Shriver National Institute of Child Health and Human Development; NIDCR, National Institute of Dental and Craniofacial Research; NIDDK, National Institute of Diabetes and Digestive and Kidney Diseases; NIEHS, National Institute of Environmental Health Sciences; NIGMS, National Institute of General Medical Sciences; NINDS, National Institute of Neurological Disorders and Stroke; OD, Office of the Director.

SOURCE: NIH, 2021f.

TABLE 6-3 Committee-Reviewed NIH Grant Types, by Activity Code

SOURCES: NIH, 2019; NIH Grants & Funding, 2021.

training programs (T). Research career program grants had higher levels of funding throughout the entire period.

The committee examined the proportion of autoimmune disease spending used for training to total autoimmune disease spending and compared it to other Research, Condition, and Disease Categories (RCDC) spending categories (Table 6-5). The comparison groups are chronic diseases and are more common in women or have a similar prevalence between men and women (cardiovascular).

IC FUNDING OF GRANTS

To explore ICs’ use of grants to further autoimmune disease research, the committee examined IC funding for research (R), cooperative agreement (U), program project/center (P) and training (F, K, T) grants. Figure 6-4 and Figure 6-5 present funding of R grants categorized by ICs with the largest research budgets and smaller research budgets.

For the purposes of this report, ICs are arranged in order by budget level and have been separated into two categories: large ICs and small ICs. Large ICs have FY 2020 actual total obligations greater than $1 billion, and small ICs have actual total obligations of less than $1 billion. Among the larger ICs, NIDDK and NIAID had the highest level of funding for R grants in FY 2020. NIDDK had the highest level of average autoimmune disease research funding compared with the other ICs. Among the smaller ICs, NIAMS had a significantly higher level of funding in 2020 and average of funding for 2008–2020 for R grants to support autoimmune disease research compared to other small ICs. The OD spent an average of $10 million on autoimmune disease research grants, which was comparable to the average funding spent by other smaller ICs.

U grants support research activities that NIH considers to be complex and of high-priority research areas. Unlike independent investigator R grants, U grants require greater involvement from NIH program or

TABLE 6-4 Autoimmune Disease Spending: Fellowship, Research Career, and Training Program Grant Funding per FY 2008–2020 and Average over the Period

NOTE: NIH autoimmune disease activity codes for fellowship (F), research career (K), and training program (T) grants. FY, fiscal year.

SOURCE: NIH, 2021f.

scientific staff. Figures 6-6 and 6-7 show the level of funding for autoimmune disease U grants for large and small ICs.

Among the larger ICs (Figure 6-6), NIDDK had a significantly higher level of funding for U grants for autoimmune disease research activities in FY 2020 and over the FY 2008–2020 period. NIDDK funding levels for U grants over the period and in FY 2020 were more than double that of NIAID, the IC with the next level of U grant funding. Other large Institute funding of U grants was less than $6 million in 2020. In FY 2020, NIAMS, among the smaller ICs (Figure 6-7) had the highest level of funding for U grants ($9 million). During the FY 2008–2020 period, NHGRI had the highest average U grant funding level ($7 million).

P grants support large, multidisciplinary, and long-term research on autoimmune disease. Figure 6-8 shows P grant funding levels of at least $1 million by IC. NIAID and NIAMS had the highest funding levels for P grants and were similar, although NIAID has a larger overall research budget than NIAMS.

ICs can play an important role in supporting the research workforce in their area of focus. Funding can help build the research workforce through fellowships (F), research career programs (K), and institutional training programs (T) grants, in addition to traditional research grants. Table 6-6 shows IC funding of training grants that support the autoimmune disease research workforce. In FY 2020, NIDDK, NIAID, and NHLBI had the highest level of funding for F grants ($2.0 million, $814,000, and $804,000 respectively). NIDDK also had the highest average funding of F grants over the FY 2008–2020 period ($1.6 million).

Funding for K grants was higher than funding for F or T grants. In FY 2020, NIDDK had the highest level of funding of K grants ($11.3 million), followed by NIAMS ($8.2 million). Over the FY 2008–2020 period, NIDDK ($9.8 million) and NIAMS ($7.9 million) had the highest level of average funding for K grants compared to other ICs. The committee notes that OD funding for K grants was $1.1 million in FY 2020, which is more than double than the average funding over the FY 2008–2020 period.

ICs other than NIAMS relied less on the T grant mechanism to support training of autoimmune disease researchers. In FY 2020, NIAMS funded $3.9 million in T grants with an average funding level of $4.2 million over the FY 2008–2020 period. NIAID, NIDDK, and OD used this mechanism at a significantly lower funding level.

TABLE 6-6 Training Grant Funding for Autoimmune Disease Research by IC and FY

NOTE: FY, fiscal year; IC, Institute or Center; NCATS, National Center for Advancing Translational Sciences; NCI, National Cancer Institute; NEI, National Eye Institute; NHGRI, National Human Genome Research Institute; NHLBI, National Heart, Lung, and Blood Institute; NIAID, National Institute of Allergy and Infectious Diseases; NIAMS, National Institute of Arthritis and Musculoskeletal and Skin Diseases; NICHD, Eunice Kennedy Shriver National Institute of Child Health and Human Development; NIDCR, National Institute of Dental and Craniofacial Research; NIDDK, National Institute of Diabetes and Digestive and Kidney Diseases; NIEHS, National Institute of Environmental Health Sciences; NIGMS, National Institute of General Medical Sciences; NINDS, National Institute of Neurological Disorders and Stroke; OD, Office of the Director.

SOURCE: NIH, 2021f.

The committee reviewed NIH spending on autoimmune disease research by type of grant and type of research according to three categories: intramural, investigator-initiated, and solicited. As noted earlier, intramural research funding (Z) is for internal NIH investigator research activities, and R grant funding is the most common independent research grant mechanism. ICs use solicited research funding opportunities to support priority research areas. NIH communicates with the research community about these grant opportunities through a notice of special interest, program announcement, request for application, request for proposal, and other mechanisms. Figure 6-9⁶ shows NIH funding for autoimmune disease research for the average over the study period and FY 2020. Of the three types of grants, investigator-initiated research grants received greater than 70 percent of the funding, however, funding for each type of research in FY 2020 is higher than the average over the study period. Table 6-7 shows funding of intramural, investigator-initiated, and solicited grants by ICs.

Funding for intramural research was highest for NIAID and NIAMS (about $34 million each) in FY 2020 (Table 6-7).⁷ NIAID and NIDDK had the highest level of funding for investigator-initiated research ($187 million and $185 million respectively), followed by NIAMS ($86 million). NIDDK had the highest level of average FY 2008–2020 funding ($127 million per year). Of the ICs reviewed, NIDDK also had the highest level of funding for solicited research grants. NIDDK solicited research funding was 2.7 times higher than NIAID funding for this grant type.

Intramural research activities represent 17.3 percent of NIH funding for autoimmune disease research in FY 2020. A total of 50 intramural principal investigators conduct autoimmune disease research, which may include basic, translational, and clinical studies. However, the committee notes that there is a relative lack of intramural principal investigators who focus on epidemiology and autoimmune diseases. The committee searched the NIH Intramural Research Program database⁸ to identify principal investigators who conduct autoimmune disease research and principal investigators who engage in epidemiology-focused research (N=121). There are three intramural principal investigators who conduct research that overlaps both of these areas. This number is significantly lower than the number of principal investigators in the NCI intramural

___________________

⁶ See Appendix G for methodology.

⁷ See Appendix G for methodology.

⁸ The committee used the NIH Intramural Research Program database available at https://irp.nih.gov/our-research/principal-investigators/focus/epidemiology (accessed January 7, 2022).

research program whose research focuses on both cancer and epidemiology (N=69).

RESEARCH GRANTS BY AUTOIMMUNE DISEASE

NIH introduced the RCDC in 2009 to categorize funded research. RCDC categories include a research area, specific condition, or disease area, but the categories are not exhaustive or mutually exclusive (NIH, 2021a). Funded research as reported in NIH RePORTER may have multiple RCDC disease spending categories depending on the scientific and research focus of the project. Furthermore, diseases or conditions that do not have a specific RCDC spending category (diabetes mellitus type 1, antiphospholipid syndrome, Sjögren’s disease, celiac disease, autoimmune thyroid disease, primary biliary cholangitis) must be searched for using “key terms,” including the name of the condition and all iterations

TABLE 6-7 Intramural, Investigator-Initiated, and Solicited Autoimmune Disease Research by IC, FY 2020 and Average FY 2008–2020

NOTES: National Center for Advancing Translational Sciences (NCATS) average spending is from FY 2012 to 2020 autoimmune disease research categories were determined using funding opportunity announcement (FOA) and combined total cost of funded autoimmune disease research variables. Grants without an FOA were not included in the analysis. FY, fiscal year; IC, Institute or Center; NCI, National Cancer Institute; NEI, National Eye Institute; NHGRI, National Human Genome Research Institute; NHLBI, National Heart, Lung, and Blood Institute; NIAID, National Institute of Allergy and Infectious Diseases; NIAMS, National Institute of Arthritis and Musculoskeletal and Skin Diseases; NICHD, Eunice Kennedy Shriver National Institute of Child Health and Human Development; NIDCR, National Institute of Dental and Craniofacial Research; NIDDK, National Institute of Diabetes and Digestive and Kidney Diseases; NIEHS, National Institute of Environmental Health Sciences; NIGMS, National Institute of General Medical Sciences; NINDS, National Institute of Neurological Disorders and Stroke; OD, Office of the Director.

SOURCE: NIH, 2021f.

of it (e.g., names roughly equivalent to antiphospholipid syndrome include Hughes’ syndrome, lupus anticoagulant, anticardiolipin, etc.).

The committee was interested in examining trends in the number of grants for autoimmune diseases of specific interest identified in Chapter 3. These diseases include inflammatory bowel disease (IBD), multiple sclerosis, type 1 diabetes, rheumatoid arthritis, systemic lupus erythematous (SLE), Sjögren’s disease, psoriasis, celiac disease, autoimmune thyroid disease, antiphospholipid syndrome (APS), and primary biliary cholangitis (PBC). Hashimoto’s disease and Grave’s disease were combined into a single “autoimmune thyroid disease” category. In addition, grants that were not associated with the committee’s specific disease categories or were not disease specific were labeled as “other autoimmune disease” throughout the analysis. RePORTER data includes funding information for each year of a grant (e.g., a 3-year R01 grant is identified as three increments of funding); the committee aggregated funding increments with the same grant number to represent one grant. In total, the committee identified and aggregated 28,148 funding increments into 8,470 autoimmune disease grants.⁹Figure 6-10 shows number of grants by disease category for FY 2008–2020.

Of the autoimmune diseases the committee examined, the four diseases with the greatest number of funded grants over the FY 2008–2020 period were other autoimmune diseases (31.0 percent), IBD (17.6 percent), multiple sclerosis (16.4 percent) and type 1 diabetes (13.2 percent). Figure 6-11 shows the total number of grants by disease category funded by year, between 2008 and 2020. The number of IBD and type 1 diabetes grants increased gradually over time; multiple sclerosis grants declined after 2009. The number of other specific disease grants fluctuated over the period.

Autoimmune disease grants involved 98 distinct activity codes. Figure 6-12 shows the top 10 activity codes. Among these, R01 (35.9 percent) and R21 (12.4 percent) accounted for the largest number of grants. R01 grants are the standard investigator-initiated grants, and R21 grants are research exploratory/developmental grants meant to encourage the development of new research activities in specific program areas.

For R01 grants, other autoimmune disease (light purple) multiple sclerosis (dark purple), and IBD (yellow) were the most funded grants in FY 2020 (Figure 6-13). The number of grants increased for about half of the disease categories after 2016. IBD grants increased steadily over the period. Sjögren’s disease, psoriasis, celiac disease, autoimmune thyroid disease, APS, and PBC had fewer than 25 grants per year.

___________________

⁹ Aggregated grants also include subprojects added to a parent project.

Among R01 research grants, non-competing continuation research grants (R01-5) were the most funded (32.5 percent) followed by new R01-1 grants (22.3 percent). Figure 6-14 shows the number of non-competing R01 grants by disease and Figure 6-15 shows new R01 grants by disease for FY 2008–2020. Non-competing continuation grants for all disease categories decreased in 2016 as a result of an administrative accounting change (Figure 6-15). There were no clear patterns in the trends in the number of new R01 grants funded by disease categories except for the increase in IBD grants over the study period.

The committee also examined the ICs designated as the funding IC for autoimmune disease grants (Figure 6-16). Among the top 10 funding ICs, NIDDK, NIAID, and NIAMS funded the largest number of grants.

Figures 6-17 through 6-19 show trends in autoimmune diseases by the top three funding ICs (NIDDK, NIAID, and NIAMS). Figure 6-17 shows that among NIDDK-funded grants, the disease categories with the greatest number of grants were IBD (yellow), type 1 diabetes (black), and other autoimmune disease (light purple). Figure 6-18 shows NIAID-funded

grants by disease. The top three disease categories with the most grants were other autoimmune diseases (light purple), SLE (dark blue), and multiple sclerosis (dark purple). While NIAID-funded IBD grants (yellow) seemed to increase steadily since 2011, SLE and multiple sclerosis grants decreased in 2015 but have since increased. Figure 6-19 shows NIAMS-funded grants by disease. The top three disease categories with the most grants were rheumatoid arthritis (orange), other autoimmune diseases (light purple), and SLE (dark blue).

IC Collaboration on Research Funding Opportunities

ICs can take steps among themselves to coordinate their research interests and to leverage their mutual expertise and resources.¹⁰ One measure of collaboration is the degree to which ICs engage in joint funding of specific autoimmune diseases research activities. To assess the level of collaboration among ICs, the committee examined co-funded autoimmune disease research grants for FY 2008–2020. To accomplish this, the committee used NIH RePORTER data to identify administrative ICs funding autoimmune disease research grants and corresponding funding ICs that contributed to the total cost of grants during this period.

All 14 ICs collaborated with other ICs¹¹ to co-fund autoimmune disease research activities during the period. Among the large ICs, NIAID (4.23), NHLBI (2.70), and NIDDK (2.62) had the largest average number of joint IC funding collaborations (Table 6-8).¹² These ICs jointly funded research grants with one to six other ICs in a given year and collaborated with eight to nine different ICs over the period. NIDDK collaborated consistently with at least one IC each year and with up to six ICs during a given year. Among the smaller ICs, NIAMS (3.4) had the highest average number of joint IC funding collaborations followed by NIDCR (1.10) and NCATS (1.00). NIAMS collaborated with nine different ICs over the period. NIDCR and NCATS collaborated with four and three different ICs (respectively) during the period. Most collaborations were on P, R, and U grants.

___________________

¹⁰ According to multiple NIH speakers who spoke at the committee’s public information-gathering sessions (Goldmuntz, 2020; Hodge, 2021; McNamara, 2021).

¹¹ ICs of the committee’s focus.

¹² See Appendix G for methodology.

RESEARCH FOCUS OF FUNDED GRANTS

The statement of task directed the committee to evaluate the NIH research portfolio with particular attention to issues such as risk factors, diagnostic tools, barriers to diagnoses, treatments, and prospects for cures. Given the enormity of the task, which would have required that the committee review the 8,470 abstracts associated with the autoimmune disease grants funded from 2008 to 2020, the committee found three ways to approach the task in a more efficient and manageable way. The first approach used research topic modeling, a type of statistical modeling to identify popular “topics” that appeared in the abstracts. In the second approach, committee members utilized RCDC spending categories and RePORTER data to determine which particular spending categories were co-occurring with the autoimmune disease spending category. In the third approach, the committee reviewed a sample of research grant abstracts to identify the primary focus of the research.

The committee, with the aid of a consultant, used latent dirichlet allocation (LDA) statistical modeling to identify research topics in the set of 8,470 NIH research abstracts related to autoimmune disease grants funded between 2008 and 2020. Topic modeling using LDA is a method of fitting a pre-determined number of topics to a set of texts. The LDA algorithm repeatedly selects and assigns words in each document to a topic and assesses the “fit” of the word to the topic by looking at what words are frequently found together across the texts (Blei et al., 2003). In the past, researchers have applied LDA modeling to scientific abstracts to analyze funding patterns and trends in research (Park et al., 2016; Porturas and Taylor, 2021). Based on the coherence score of words¹³ that tended to co-occur together in the dataset, 30 topics were determined to be the optimal number of topics that could be used to fit the data (words) in the abstracts (Bittermann and Fischer, 2018; Park et al., 2016; Porturas and Taylor, 2021). See Appendix H for more detailed information.

One of the outputs of the LDA model is a matrix that shows the proportion of topics assigned to each abstract. For example, 30 percent of the words in an abstract might belong to topic 1, and 70 percent of words in an abstract might belong to topic 2. The topic with the highest proportion within each abstract was assigned to the abstract. Topics were combined

___________________

¹³ Coherence score is a score that calculates whether the words in the same topic make sense when they are grouped together and is an indicator of the quality of the topics being produced (Cristian, 2020).

TABLE 6-8 IC Collaboration, FY 2008–2020

NOTES: Large ICs have FY 2020 actual total obligations greater than $1 billion, and small ICs have actual total obligations of less than $1 billion. All ICs in this table are administrative ICs. FY, fiscal year; IC, Institute or Center; NCATS, National Center for Advancing Translational Sciences; NCI, National Cancer Institute; NEI, National Eye Institute; NHGRI, National Human Genome Research Institute; NHLBI, National Heart, Lung, and Blood Institute; NIAID, National Institute of Allergy and Infectious Diseases; NIAMS, National Institute of Arthritis and Musculoskeletal and Skin Diseases; NICHD, Eunice Kennedy Shriver National Institute of Child Health and Human Development; NIDCR, National Institute of Dental and Craniofacial Research; NIDDK, National Institute of Diabetes and Digestive and Kidney Diseases; NIEHS, National Institute of Environmental Health Sciences; NIGMS, National Institute of General Medical Sciences; NINDS, National Institute of Neurological Disorders and Stroke; OD, Office of the Director.

SOURCE: NIH, 2021f.

based on similarity of words within topics. Box 6-1 shows the final list of 23 topics that fit the data the best.

Figures 6-20 through 6-23 show the frequency of these topic assignments from 2008 to 2020. This graphic is a depiction of the popularity of the research topics in the grants over the period. Trends over time show that the research topics related to IBD, treatment/therapy, lung disease, diagnostic [tools], and imaging have trended upward in contrast to animal model, genetics, and pathogenesis. Compared to the other disease focused topics, type 1 diabetes was prevalent among grants over time (Figure 6-20). Cancer, rheumatoid arthritis, cardiovascular, and multiple sclerosis remained consistent over the time period. In Figure 6-21, immune response (innate immunity) may be disproportionately high because of its relevance to various aspects of autoimmune disease research as well as a lack of a concrete definition for innate immunity. Between 2008 and 2018, the mention of treatment/therapy in autoimmune disease grant abstracts drastically increased, but then decreased sharply (Figure 6-22). However, mention of disease progression, diagnostic, and imaging generally increased during the latter part of the decade. In Figure 6-23, centers/-

core project funding was variable throughout the time period, and there has been a consistent decrease in the mention of training being present in autoimmune disease abstracts, especially between 2015 and 2020.

One of the downsides of using LDA for topic modeling is that the topics must be discovered “latently” within the texts and cannot be inputted into the model algorithm. In other words, this method cannot be used to actively search for specific research topics.

RCDC Spending Categories

The second approach the committee used to evaluate the NIH research portfolio was to determine which other NIH RCDC spending categories co-occur with the autoimmune disease spending category. This can provide insight into themes within NIH’s autoimmune disease research portfolio. The definitions of the categories include all aspects of the topic, including basic, pre-clinical, clinical, biomedical, health services, behavioral, and social research, as defined by NIH scientific experts (NIH, 2020a). However, the definitions are not publicized. Table 6-9 shows the RCDC spending categories that the committee selected for analysis based on the statement of task, concepts, or conditions known to be related to autoimmune disease, and affected populations. Out of the 25 RCDC spending categories the committee analyzed, Genetics (32.1 percent), Biotechnology (23.4 percent), Prevention (22.9 percent), Pediatric (15.9 percent), and Women’s Health (12.9 percent) were the top five spending categories co-occurring with the autoimmune disease spending category in a total of 8,470 grants.

PICO Portal

PICO Portal is an online tool used to review and manually categorize scientific literature. Given the large variation in the number of grants per disease, the committee decided to take random, weighted samples of the selected diseases as follows: a 10 percent sample of diseases with greater than 1,000 grants (“other autoimmune disease,” IBD, multiple sclerosis, type 1 diabetes, rheumatoid arthritis, and SLE), a 50 percent sample of diseases between 100 and 999 grants (Sjögren’s, psoriasis, and celiac disease), and a 100 percent sample of diseases with fewer than 99 grants APS (autoimmune thyroid disease and PBC). This resulted in a total of 1,148 grants. Grants were then deduplicated (because a grant could have been assigned to more than one disease), resulting in a total of 1,127 grants for the committee to manually categorize. 1,127 is approximately 13 percent of the total number of grants (N=8,470).

Research characteristics were chosen to evaluate the portfolio and identify the focus of the research. These characteristics encompass various

TABLE 6-9 Percent of Autoimmune Disease Grants Co-occurring with other RCDC Spending Categories (N=8,470)

SOURCE: NIH, 2021f.

concepts in the statement of task, co-morbid diseases or complications of autoimmune disease, and type and method of research. Examining more than one autoimmune disease aimed to capture grants exploring the co-occurrence of autoimmune diseases. Within PICO Portal, the committee used the tagging feature to categorize the abstracts with one or more of the following research characteristics: animal model, cancer, cardiovascular, depression, diagnostic, environment/infectious agents, epidemiology, genetics, nursing/behavioral health services, pathophysiology/mechanisms, pediatrics, primary prevention, treatment/therapy, in vivo human study, in vivo animal study, in vitro study, and more than one autoimmune disease. The committee predefined specific definitions of the research characteristics in order to tag abstracts objectively.

Figures 6-24 and 6-25 show the number and percentage of abstracts tagged with the research characteristics. More than half (59.1 percent) of the 1,127 abstracts were tagged as having a focus on pathophysiology/mechanisms, followed by genetics (25.3 percent), treatment/therapy (23.2 percent), and animal model (23.2 percent). “Animal model” refers to the development of a new in vivo animal study for use to study autoimmune disease, whereas “in vivo animal study” refers to the use of an already existing animal model. Approximately 22 percent of the abstracts mentioned or focused on more than one autoimmune disease. In addition, the committee tagged abstracts based on the type of study researchers intended to conduct with their grant funding (Figure 6-25). Almost half of the abstracts were identified as being an in vivo animal study (48.4 percent), followed by in vitro (40.8 percent), and in vivo human study (29.3 percent).

It is important to note the discrepancy between the RCDC spending category “prevention” and “primary prevention” as a predefined research characteristic tag in PICO Portal. The committee defined primary prevention, found in 1.3 percent of abstracts, as “research that focuses on preventing an autoimmune disease from occurring in an individual,” and is only one of the three levels of prevention. In contrast, almost one-third of autoimmune disease grants included the RCDC prevention category, suggesting that RCDC may use a different definition that includes other levels of prevention. Alternatively, depression as both an RCDC category and a PICO Portal research characteristic tag both occurred infrequently in autoimmune disease grants (0.5 percent). This points to a potential opportunity to explore depression as a disease that coincides with autoimmune disease.

Study Section Analysis

Investigator-initiated research (R01) grants are the most common research grants funded (Figure 6-9). As noted in Chapter 5, study sections review applications for these grants, with study section peer reviewers responsible for assigning priority scores based on scientific merit. Peer reviewers draw upon their expertise in investigating line(s) of research comparable to those under review to assign scores that distinguish the very best applications from the weaker ones.

The committee used the NIH Center for Scientific Review (CSR) study section search tool to analyze study sections that review R01 grant applications.¹⁴ As many as 100 different study sections may be involved in

___________________

¹⁴ The committee used the online CSR study section tool available at https://public.csr.nih.gov/StudySections, accessed November 24, 2021. The committee entered the name of each of the select autoimmune diseases of committee interest into the study section search field to identify the names of its associated chartered study sections.

reviewing unsolicited investigator-initiated research grant applications focusing on the select autoimmune diseases the committee chose to examine. Given that such a large number of diverse study sections can potentially review autoimmune disease grant applications, CSR should ensure that individual study section rosters include the appropriate expertise, including expertise for reviewing animal and human studies, to provide the proper review of these applications.

Table 6-10¹⁵ provides a snapshot of data generated by the RePORTER Matchmaker tool. The table shows study sections—the names of which appear in the far left column—that reviewed funded grants focusing on the autoimmune diseases of interest to the committee; the table reflects a 2-year period.^16,17 The table identifies those study sections that reviewed the most funded grants for each disease of focus according to a bar graph RePORTER automatically generates to provide this information; the number of grants are indicated in the corresponding cells. One to three study sections reviewed the majority of funded grants for the two years, suggesting that this more limited number of study sections has the correct expertise for review of autoimmune disease grant applications. The committee notes that a more valuable analysis would have been to identify, for a given period, the grant applications that were reviewed for the select diseases regardless of whether they were approved. This would have provided a better baseline for analyzing patterns of review by the study sections. However this information was not available. The most common study sections responsible for the grant applications in this analysis were Gastrointestinal Mucosal Pathobiology; Hypersensitivity, Autoimmune and Immune-Mediated Diseases; Arthritis, Connective Tissue and Skin; Oral, Dental and Craniofacial Science; Cellular and Molecular Immunology B; Clinical Neuroimmunology and Brain Tumor; Innate Immunity and Inflammation; Cellular and Molecular Biology of Glia; Clinical and Integrative Diabetes and Obesity; Cellular Aspects of Obesity and Diabetes; and Hepatobiliary Pathophysiology.

___________________

¹⁵ See Appendix G for methodology.

¹⁶ The committee used the RePORTER Matchmaker search tool available at https://reporter.nih.gov/matchmaker, accessed December 3, 2021. The committee entered the name of each of the select autoimmune diseases of committee interest into the field to yield the number of projects associated with that disease. The user then activated the Active Projects feature and selected 2019 and 2020 in the Fiscal Year dropdown. RePORTER then generated a graph of the study sections that had reviewed the most grants for that autoimmune disease. The study sections appearing in the graph, and the number of grants each study section reviewed, constitute the data included in Table 6-8.

¹⁷ Data of this nature change in the RePORTER system over time. For its analysis, the committee selected active projects; as projects conclude, they are no longer generated when the “active” option is chosen.

TABLE 6-10 Chartered Study Sections That Reviewed Funded Grants (FY 2019–2020) According to Select Autoimmune Diseases*

NIH AUTOIMMUNE DISEASE RESEARCH: INDICATORS OF ACCOMPLISHMENTS

The statement of task directed the committee to include NIH accomplishments in its report. The committee addressed this task by conducting a high-level review of a number of activities and measures that reflect progress in research (clinical trials and patents), and indicators of scientific performance, quality, and influence on the research field (bibliometric indicators). The committee describes the results of its examination below.

Clinical Trials

Results of biomedical research on autoimmune disease can yield opportunities to improve the prevention, detection, and treatment of these diseases. Clinical trials, also known as interventional studies, translate basic science knowledge for use in clinical care (NLM, 2021a). Clinical trials focus on developing, testing, and implementing diagnostics and therapeutics across a wide range of diseases and conditions to evaluate the effects of the interventions on safety and health-related outcomes (NLM, 2021a). The committee examined clinical trial activity by autoimmune disease as one measure of research portfolio advancement.

The committee obtained clinical trial data from clinicaltrials.gov¹⁸ and linked clinical trials to the grants associated with the committee’s autoimmune diseases of interest. Clinical trials were de-duplicated using their Clinical Trial ID and included based on their relevance to the committee’s specific diseases of interest and autoimmune disease as a broader category (“other autoimmune disease”). Ultimately, the committee identified 353 autoimmune disease clinical trials between filing year 2008 and 2020. The “other autoimmune disease” category represents approximately 27 percent of the clinical trials. The remaining 73 percent were associated with one or more autoimmune disease.

Figure 6-26¹⁹ shows the number of clinical trials by disease from filing years 2008–2020. The top five autoimmune disease categories with the most clinical trials over the study period were type 1 diabetes (38.7 percent), other autoimmune disease (26.6 percent), rheumatoid arthritis (10.3 percent), multiple sclerosis (5.8 percent), and IBD (5.5 percent). Each of the remaining autoimmune diseases accounted for less than 5 percent of clinical trials over this period. Clinical trials were fewer for the remaining autoimmune diseases.

___________________

¹⁸ Grants with an RCDC code for “Autoimmune Disease” obtained from NIH RePORTER for Fiscal Year 2008 (October 1, 2008–September 30, 2008) through Fiscal Year 2020 (October 1, 2019–September 30, 2020).

¹⁹ See Appendix G for methodology.

Out of the 353 clinical trials pertaining to autoimmune disease, 20 are phase 4 clinical trials. A phase 4 clinical trial is a “phase of research to describe clinical trials occurring after the Food and Drug Administration (FDA) has approved a drug for marketing. They include postmarket requirement and commitment studies that are required of or agreed to by the study sponsor, and gather additional information about a drug’s safety, efficacy, or optimal use”(NLM, 2021a). Table 6-11 lists select phase 4 clinical trials.

Publications

The research community often looks to publication citations and other bibliometric measures as indicators of scientific performance, quality, and influence on a research field. The committee selected a number of these measures to examine the performance of the autoimmune disease research portfolio from 2008–2020. These indicators include the number of publications associated with funded grants, relative citation ratio, and journal impact factor.

The committee conducted a bibliometric analysis of publications in PubMed associated with autoimmune disease-associated grant numbers in an effort to characterize scientific productivity and impact. Figure 6-27²⁰ shows publications by disease category per year. Of the autoimmune diseases examined, the four diseases associated with the greatest number of publications over the 2008–2020 period were other autoimmune disease

TABLE 6-11 Phase 4 Clinical Trials, Autoimmune Disease Prevention or Treatment

SOURCES: NIH, 2018, 2020b, 2021c, 2021e.

___________________

²⁰ See Appendix G for methodology.

(27.7 percent), IBD (22.5 percent), type 1 diabetes (18.2 percent), and multiple sclerosis (16.1 percent).

Figure 6-28²¹ shows the trends in autoimmune disease publications. Between publication year 2008 and 2020, other autoimmune disease publications (light purple) generally trended upward and had the most publications overall with an average of 3,398 per year. Publications associated with IBD (yellow) had the greatest number of publications among the committee’s diseases of interest but appear to have peaked in 2014. Publications associated with type 1 diabetes (black), multiple sclerosis (dark purple), and SLE (light blue) have also trended downward.

The Relative Citation Ratio (RCR) is a metric developed within the NIH Office of Portfolio Analysis (OPA). RCR “represents the field- and time-normalized citation rate” to the expected citation rate based upon the article’s co-citation network of NIH-funded publications (Hutchins et al., 2016; NIH, 2021d). A publication with an RCR of 1 means that it has the same influence as the average article. The RCR metric can be used to determine the allocation of research funding and the extent to which NIH awardees “maintain high or low levels of influence on their respective fields of research” (Aksnes et al., 2019; Hutchins et al., 2016). RCR is often used under the assumption that citations are a proxy for quality; however, articles can be cited for a variety of reasons and RCR does not take the specific context of a citation into account (NIH Library, 2020). This counters the argument that RCR, alone, is a measure of quality.

Table 6-12²² shows the publications with RCRs by disease. IBD publications had the highest average RCR of 3.029 and are therefore three times as influential as the average article according to this metric. Although PBC publications had the lowest average RCR of 1.894, they are still approximately twice as influential as the average article.

Journal impact factor (JIF) is “a measure of the frequency with which the average article in a journal has been cited in a particular year and it is used to measure the importance or rank of a journal by calculating the times its articles are cited” (University of Illinois Chicago Library, 2016). The scientific impact of an individual article is unrelated to the impact factor of a journal (The University of Texas MD Anderson Cancer Center, 2021), and JIF is a measure of journal quality exclusively. In the context of evaluating NIH’s autoimmune disease portfolio, the committee focused on publications resulting from NIH-funded autoimmune disease research and the quality of the journals in which they are published.

Figure 6-29²³ shows average JIF for the 10 journals with the highest JIF for all projects. Of the 159,416 autoimmune disease-related publications,

___________________

²¹ See Appendix G for methodology.

²² See Appendix G for methodology.

²³ See Appendix G for methodology.

63 percent (101,199 of 159,416) had JIF data. The size of the bubble represents the number of publications in the journal, while the location of the bubble on the y-axis shows the JIF relative to each other. Although Nature (dark gray) published the most articles between 2008 and 2020, Chemical Reviews (brown) has the highest JIF (43.9). In most fields, the impact factor of 10 or greater is considered an excellent score while 3 is flagged as good and the average score is less than 1 (SCI Journal, 2021).

Patents

A patent grants a property right to the inventor and prohibits others from offering, selling, or importing that invention (USPTO, 2018). Patents granted as a result of NIH-funded autoimmune disease research aim to encourage and protect biomedical innovation and may serve as a proxy for knowledge generation and translational value in the field (Kalutkiewicz and Ehman, 2014). The committee examined patents and patent applications by autoimmune diseases.

The committee obtained patent data linked to the grants associated with the committee’s autoimmune disease of interest from the United

States Patent and Trademark Office (USPTO) at uspto.gov. A total of 4,113 patent applications and patents were associated with NIH grants funding research in autoimmune diseases, of which 60 percent (2,470 of 4,113) were patent applications and 40 percent (1,643 of 4,113) were granted applications (patents). All were included in the analysis. A total of 30 percent of the patents and patent applications were associated with only one disease category. The remaining 70 percent were associated with more than one disease category.

Figure 6-30²⁴ shows the number of patent applications by disease for filing years 2008–2020. The top three autoimmune disease categories with the most patent applications were other autoimmune disease (30.3 percent), multiple sclerosis (22.8 percent), and IBD (19.8 percent). Fewer patent applications were filed for the remaining autoimmune diseases.

Figure 6-31²⁵ shows the granted patents by disease for filing years 2008–2020. The top three disease categories with the most patents were other autoimmune disease (29.1 percent), multiple sclerosis (22.7 percent), and IBD (18.4 percent).

The ultimate outcome of clinical and biomedical research is an observable impact on the health and wellbeing of individuals. This downstream effect is a result of fundamental scientific knowledge “applied to develop actionable interventions that are implemented to enhance health, lengthen life, and reduce illness and disability” (NIH, 2014). Publications disseminated throughout the scientific community often report the outcomes of clinical and biomedical research, and the number of publications resulting from autoimmune disease-related research is astounding, as seen by the thousands of relevant autoimmune-related publications produced between 2008 and 2020.

The committee was unable to perform a comprehensive assessment of research accomplishments and successes and therefore was unable to assess clinical and biomedical achievements as a measure of knowledge in the field beyond quantifying publications, patents, and clinical trials. Another way to assess achievements is to determine how research translates into clinical care and treatment of individuals, but this too is an immense undertaking. Instead, the committee provides a number of illustrative research findings associated with the research portfolio that have the potential to improve the health and healthcare of individuals living with autoimmune diseases in Table 6-13.²⁶

___________________

²⁴ See Appendix G for methodology.

²⁵ See Appendix G for methodology.

²⁶ Autoimmune Disease Research Highlights were collected from NIH biennial and triennial reports to Congress, and NIH websites.

TABLE 6-13 Autoimmune Disease Research Highlights

SUMMARY

In reviewing NIH’s research portfolio on autoimmune diseases, the committee employed a general framework that considered the inputs (spending), activities (research conducted), and outputs (clinical trials, patents, and other indicators) of NIH’s investment in autoimmune disease research. Spending on autoimmune diseases as a percentage of overall NIH obligations has remained at only 2.6 percent between 2013 and 2020. This is in marked contrast to increases seen in overall NIH obligations during the same time period. In addition, the distribution of this funding for autoimmune diseases indicates that certain ICs dominate overall spending on research and training and the types of research activities funded. NIH may need to reconsider the current strong focus on investigator-initiated research in order to target areas of high priority and effectively address the long-term, complex, and heterogeneous nature of autoimmune diseases.

Regarding the activities conducted under funded autoimmune disease research grants, the multiple methods the committee used yielded

useful information at different levels of specificity. However, metrics are not available for evaluating the overall research portfolio for autoimmune diseases.

The outputs of NIH’s investment in autoimmune disease research can be measured in a variety of ways, including clinical trials (the translation of research knowledge into interventions), publications, and the number of patents associated with autoimmune disease research grants (an important indicator of innovation). There is a greater percentage of clinical trials from filing years 2008–2020 for such autoimmune diseases as type 1 diabetes, SLE, IBD, and rheumatoid arthritis, with a much smaller percentage seen for other autoimmune diseases. In general, the knowledge associated with existing research grants has been successfully disseminated to the research community through publications in well-regarded scientific journals. In terms of patent applications, filings occur primarily for some autoimmune diseases with far fewer filings for others. In general, findings related to research outputs suggest the need for a nuanced and in-depth examination of why progress via clinical trials and patent filings does not seem to have been made for certain autoimmune diseases.

Given the findings of Chapter 6, the committee concludes that the absence of an overarching strategic plan with concrete metrics makes it difficult to assess the distribution of funding by IC or disease. The committee also concludes that the relatively small percentage of NIH funding devoted to autoimmune disease research and how it is distributed could be a significant factor in hampering scientific progress on diseases that affect more than 23 million people.

REFERENCES

Goldmuntz, E. 2020. Charge to the committee. Paper read at NASEM Committee for the Assessment of NIH Research on Autoimmune Diseases: Meeting 1, November 18, 2020, Webinar.

Hall, J. C., A. N. Baer, A. A. Shah, L. A. Criswell, C. H. Shiboski, A. Rosen, and L. CasciolaRosen. 2015. Molecular subsetting of interferon pathways in Sjögren’s syndrome. Arthritis & Rheumatology 67(9):2437–2446.

Herold, K. C., B. N. Bundy, S. A. Long, J. A. Bluestone, L. A. DiMeglio, M. J. Dufort, S. E. Gitelman, P. A. Gottlieb, J. P. Krischer, P. S. Linsley, J. B. Marks, W. Moore, A. Moran, H. Rodriguez, W. E. Russell, D. Schatz, J. S. Skyler, E. Tsalikian, D. K. Wherrett, A. G. Ziegler, C. J. Greenbaum, and G. Type 1 Diabetes TrialNet Study. 2019. An anti-CD3 antibody, teplizumab, in relatives at risk for type 1 diabetes. The New England Journal of Medicine 381(7):603–613.

Hodge, D. 2021. NIH peer review, a CSR perspective. Paper read at NASEM Committee for the Assessment of NIH Research on Autoimmune Diseases: Meeting 2 (Continuation), February 1, 2021, Webinar.

Hong, S., R. Banchereau, B. L. Maslow, M. M. Guerra, J. Cardenas, J. Baisch, D. W. Branch, T. F. Porter, A. Sawitzke, C. A. Laskin, J. P. Buyon, J. Merrill, L. R. Sammaritano, M. Petri, E. Gatewood, A. M. Cepika, M. Ohouo, G. Obermoser, E. Anguiano, T. W. Kim, J. Nulsen, D. Nehar-Belaid, D. Blankenship, J. Turner, J. Banchereau, J. E. Salmon, and V. Pascual. 2019. Longitudinal profiling of human blood transcriptome in healthy and lupus pregnancy. Journal of Experimental Medicine 216(5):1154–1169.

Hutchins, B. I., X. Yuan, J. M. Anderson, and G. M. Santangelo. 2016. Relative citation ratio (RCR): A new metric that uses citation rates to measure influence at the article level. PLOS Biology 14(9):e1002541.

Kalutkiewicz, M. J., and R. L. Ehman. 2014. Patents as proxies: NIH hubs of innovation. Nature Biotechnology 32(6):536–537. https://www.acadrad.org/wp-content/uploads/2016/06/Patents-as-Proxies-article.pdf (accessed March 3, 2022).

Martin, J. C., C. Chang, G. Boschetti, R. Ungaro, M. Giri, J. A. Grout, K. Gettler, L. S. Chuang, S. Nayar, A. J. Greenstein, M. Dubinsky, L. Walker, A. Leader, J. S. Fine, C. E. Whitehurst, M. L. Mbow, S. Kugathasan, L. A. Denson, J. S. Hyams, J. R. Friedman, P. T. Desai, H. M. Ko, I. Laface, G. Akturk, E. E. Schadt, H. Salmon, S. Gnjatic, A. H. Rahman, M. Merad, J. H. Cho, and E. Kenigsberg. 2019. Single-cell analysis of Crohn’s disease lesions identifies a pathogenic cellular module associated with resistance to anti-tnf therapy. Cell 178(6):1493–1508 e1420.

McNamara, J. 2021. Autoimmune disease research supported by extramural NIAID. Paper read at NASEM Committee for the Assessment of NIH Research on Autoimmune Diseases: Meeting 2 (Continuation), February 1, 2021, Webinar.

Muraro, P. A., M. Pasquini, H. L. Atkins, J. D. Bowen, D. Farge, A. Fassas, M. S. Freedman, G. E. Georges, F. Gualandi, N. Hamerschlak, E. Havrdova, V. K. Kimiskidis, T. Kozak, G. L. Mancardi, L. Massacesi, D. A. Moraes, R. A. Nash, S. Pavletic, J. Ouyang, M. Rovira, A. Saiz, B. Simoes, M. Trneny, L. Zhu, M. Badoglio, X. Zhong, M. P. Sormani, and R. Saccardi, for the Multiple Sclerosis–Autologous Hematopoietic Stem Cell Transplantation (MS-AHSCT) Long-term Outcomes Study Group. 2017. Long-term outcomes after autologous hematopoietic stem cell transplantation for multiple sclerosis. JAMA Neurology 74(4):459–469. https://doi.org/10.1001/jamaneurol.2016.5867.

NIH (National Institutes of Health). 2015. SPIRES. https://dpcpsi.nih.gov/sites/default/files/opa/document/Posters/Tools/15%20Jordan%20SPIRES-OPA-Symposium-Poster-Final-07212015.pdf (accessed December 14, 2021).

NIH. 2018. Predicting response to standardized pediatric colitis therapy (PROTECT). https://clinicaltrials.gov/ct2/show/NCT01536535?id=NCT01536535&draw=2&rank=1 (accessed December 16, 2021).

NIH. 2019. Types of grant programs. https://grants.nih.gov/grants/funding/funding_program.htm (accessed September 3, 2021).

NIH. 2020a. Estimates of funding for various research, condition, and disease categories (RCDC). https://report.nih.gov/funding/categorical-spending#/ (accessed April 12, 2021).

NIH. 2020b. RHYTHM (formerly Escape II Myocardium) (RHYTHM). https://clinicaltrials.gov/ct2/show/NCT01548768 (accessed December 16).

NIH. 2021a. About RCDC. https://report.nih.gov/funding/categorical-spending/rcdc (accessed July 03, 2021).

NIH. 2021b. Actual total obligations by institute and center: FY 2000–FY 2020. https://officeof-budget.od.nih.gov/pdfs/FY21/spending-hist/Actual%20Obligations%20By%20IC%20FY%202000%20-%20FY%202020%20(V).pdf (accessed September 2, 2021).

NIH. 2021c. Clinicaltrials.Gov is a database of privately and publicly funded clinical studies conducted around the world. https://clinicaltrials.gov/ (accessed May 8, 2021).

NIH. 2021d. iCite: Influence module. https://icite.od.nih.gov/user_guide?page_id=ug_infl (accessed December 7, 2021).

NIH. 2021e. New onset type 1 diabetes: Role of exenatide. https://clinicaltrials.gov/ct2/show/study/NCT01269034?id=NCT01269034&draw=2&rank=1 (accessed December 16, 2021).

NIH. 2021f. NIH reporter database.

NIH Grants & Funding. 2021. Activity codes search results. https://grants.nih.gov/grants/funding/ac_search_results.htm (accessed December 7, 2021).

NIH Library. 2020. Week 11: Citation impact. https://www.youtube.com/watch?v=bxv0tzmbwxg (accessed December 7, 2021).

NLM (National Library of Medicine). 2021a. Glossary of common site terms. https://clinicaltrials.gov/ct2/about-studies/glossary (accessed December 14, 2021).

NLM. 2021b. Pubmed.Gov. https://pubmed.ncbi.nlm.nih.gov/ (accessed December 14, 2021).

Park, J., M. Blume-Kohout, R. Krestel, E. Nalisnick, and P. Smyth. 2016. Analyzing NIH funding patterns over time with statistical text analysis. Palo Alto, CA: Association for the Advancement of Artificial Intelligence.

Porturas, T., and R. A. Taylor. 2021. Forty years of emergency medicine research: Uncovering research themes and trends through topic modeling. American Journal of Emergency Medicine 45:213–220.

SCI Journal. 2021. What’s a good impact factor & why it matters (with 2018/2019 impact factor ranking in 27 categories). https://www.scijournal.org/articles/good-impact-factor (accessed December 7, 2021).

Spencer, E. A., S. M. Davis, D. R. Mack, B. M. Boyle, A. M. Griffiths, N. S. LeLeiko, C. G. Sauer, D. J. Keljo, J. F. Markowitz, S. S. Baker, J. R. Rosh, R. N. Baldassano, M. Oliva-Hemker, M. D. Pfefferkorn, A. R. Otley, M. B. Heyman, J. D. Noe, A. S. Patel, P. A. Rufo, P. S. Group, M. Alison Marquis, T. D. Walters, M. H. Collins, S. Kugathasan, L. A. Denson, J. S. Hyams, and M. C. Dubinsky. 2018. Serologic reactivity reflects clinical expression of ulcerative colitis in children. Inflammatory Bowel Diseases 24(6):1335–1343.

The University of Texas MD Anderson Cancer Center. 2021. Q. What is considered a good impact factor? https://mdanderson.libanswers.com/faq/26159 (accessed December 7, 2021).

University of Illinois Chicago Library. 2016. Measuring your impact: Impact factor, citation analysis, and other metrics: Journal impact factor (if). https://researchguides.uic.edu/if/impact (accessed December 7, 2021).

USPTO (United States Patent and Trademark Office). 2018. General information concerning patents. https://www.uspto.gov/patents/basics/general-information-patents (accessed December 7, 2021).

__________________

This page intentionally left blank.