8

Fostering the Data Management Environment

The cost-forecasting framework presented in this report directs the forecaster through a series of questions related to the cost drivers identified in Chapter 4 and summarized in a template in Appendix E. The framework, if used properly, could drive an analysis of the infrastructure and management activities needed at various points in the data life cycle, and the expertise that will need to be engaged. Understanding personnel needs is at least as important as understanding infrastructure costs because personnel costs associated with data and data platform management are likely to dominate total costs.

It is not part of common practice to think about data management budgets beyond the current funding period; however, creating a research and data environment that allows long-term, efficient, and cost-effective data discovery and data reuse requires long-term planning. That planning, in turn, requires that all involved in the scientific endeavor—researchers, research institutions, data curators and managers, data resource hosts, and funding institutions—embrace long-term planning approaches, regardless of the state of the data platform (i.e., primary research, active repository, or long-term preservation) being managed. This chapter presents strategies, actions, and advances that could be applied by members of the biomedical research community to create an environment conducive to long-term cost forecasts. The reader will need to determine how best to apply these based on his or her role in the scientific endeavor and on the data environment in which he or she works.

STRATEGIES

Efficient long-term data management is more likely if data resource managers, cost forecasters, and institutions that support them apply the strategies presented below (in italics).

• Create data environments that foster discoverability and interpretability through long-term planning and investment throughout the data life cycle. Data sharing is not equivalent to data reuse, and developing processes that allow efficient data preservation, archiving, and access to facilitate data reuse could benefit scientific discovery.

Advances in biomedical and information sciences result in larger and more complex data sets. The growing volumes of complex data exacerbate the challenges already faced by those who generate, use, or manage data. Members of the U.S. biomedical research community understand that scientific discovery is a key benefit of

data preservation, aggregation, and access. That community increasingly advocates for the sharing of data to advance the scientific enterprise (e.g., NASEM, 2018). However, it is data reuse—not just data sharing—that is the objective. Making data discoverable and interpretable, and therefore reusable, requires forethought and sustained long-term investment.

• Incorporate data management activities throughout the data life cycle to strengthen data curation and preservation. Up-front costs may be increased, but data value may also increase, and the overall cost of research may be reduced.

There is a need for a cultural change within the biomedical research community related to data management. Curation activities are often left to the end of the funding period when few resources (or interest, time, or energy) remain. Instead, long-term data curation and data management needs ought to be considered throughout the course of research and the management of information resources. Data management and curation needs are vital in all data states, including during primary research.

• Incorporate the expertise and resources needed to create and curate metadata throughout the data life cycle, and in the transition between data states into the cost forecast. Data discoverability and reusability depend on adherence to community-accepted data and metadata standards.

The potential value of data will not be realized without strategic curatorial decisions by knowledgeable experts resulting in metadata that facilitate data discoverability and interpretability. That expertise needs to be anticipated and included in project budgets. Understanding the expertise and resources needed to create and curate metadata during the different data states and in the transition between states is vital and needs to be supported and encouraged by all within the biomedical research community, including institutions and funding agencies. At the individual-institution level, data-preservation efforts more likely will succeed if, for example, researchers are involved in decision-making and preservation efforts. Closer interactions between data librarians and researchers would result in a more efficient enterprise.

• Weigh the benefits, risks (e.g., data loss), and costs (both up-front and anticipated) of data storage and computation options before selecting among options. A service may look attractive from an immediate-financing perspective, but service-provider strategies require vetting and verification, including examination of exit or transition strategies and costs. Long-term costs need to be informed by a provider’s risk-management strategies.

Substantial attention to confidentiality, ownership, and security; to standards, regulatory, and governance concerns; to access control; and to the various disruptors described in Chapter 7 will always be required, regardless of storage and computational options chosen. The risk-management strategy of service providers, and of any evolution that strategy undergoes with time, needs to be understood and addressed. The institution managing an information resource is not absolved from information technology responsibilities if commercial vendors are chosen to provide services.

ACTIONS

Individuals and select institutions within specific biomedical sectors may collaborate to increase the efficiency of data management efforts, but there is little guidance available from funding agencies and the institutions that support biomedical data resources on practices for long-term management and cost forecasting for the biomedical research community. The actions described below, especially if taken by funding agencies and institutions that support data resources, could expand the capacity of data producers and managers to make sound management decisions and cost forecasts.

• Explicitly recognize the value of State 2 data resources (i.e., active repositories) to the enhanced curation, discoverability, and use of data. This recognition is absent among the funding entities, researchers, and institutions supporting research, most of which apply the more traditional data management approach of transitioning data directly from the primary research environment (i.e., State 1) to long-term archiving (i.e., State 3).

Many recent advances in biomedical research have been possible because of new technologies that allow efficient aggregation, search, and compute of data in ways previously not possible. It is the State 2 environment in which analytic tools and data can be brought together to allow the sophisticated data manipulation necessary to produce those advances. However, creating State 2 platforms implies investments in ingesting and validating data, and the number of cost drivers affecting State 2 platforms is greater than for State 3 platforms. Developing and operating a sophisticated State 2 aggregating platform requires an organization, developers, user-interface designers, training and documentation, help desks, and community building. Current storage costs (even total storage costs) are only one—and, in many cases, probably not the dominant—factor in total system costs. Further, many researchers consider preparing data for public sharing (e.g., moving data from the State 1 primary research environment to a State 2 active repository and platform) to be burdensome and a task that provides little personal benefit. The biomedical research community needs to recognize that the long-term benefits of properly supporting State 2 data resources outweigh the costs and short-term burdens of establishing the resources and preparing data for them.

• Structure cost forecasts for State 2 resources around communities and research programs rather than individual research efforts. Because State 2 resources serve communities of researchers, it may not be appropriate to allocate the costs of managing data in a State 2 resource back to the individual data contributor.

State 2 platform costs generally do not track data from an individual researcher or research project, and the present study committee is not able to identify a good analysis of fixed versus incremental costs associated with individual streams of data contributed to active repositories. The committee was unable to find good examples of how State 2 data management costs—such as those incurred to bring data into compliance with community-developed standards—might be allocated back to the individual researchers who contributed the data. Because communities of researchers are involved, cost forecasting in this setting is better structured around communities and research programs rather than individual research efforts.

• Support standardization efforts, including developing tools and methodologies to estimate the cost of standards development, encouraging the use of those tools and standards as part of the funding programs where appropriate, and explicitly supporting metadata preparation. Support could take the form of funding and the provision of tools. Issuing clarifying language about the use of federal funds for data preservation beyond the performance period of the project that collected them would also help assist in the development and promotion of the use of community standards and metadata preparation.

As has been stated throughout this report, data that do not comply with standards or that have not been documented with appropriate metadata are of lesser value because they cannot be easily aggregated with other data or, more simply, may not be able to be found or understood. Existing incentives for researchers to deposit data in useful formats when standards exist are weak, and requirements to do so lack enforcement. Where no standards exist, data may be collected but then must be retrofit to comply with standards that are established. This process may occur years after the data were collected and possibly long after the supporting research grant has run out, and the last expert has moved on to other efforts. Few mechanisms exist to pay for retrofitting data, and perhaps little interest or incentive to do so exists on the part of the researcher, as she may have moved on to other projects or have little training in data management. Even if the researcher could anticipate and accurately forecast the cost of compliance, grants are not structured to allow money to be “held aside” until standards are established. Funding agencies can assist by contributing to tools for estimating the cost of standards development and metadata

preparation, by explicitly funding metadata preparation, and by issuing clarifying language about the use of federal funds to preserve data beyond the end of the grant.

• Identify incentives, tools, and training for adopting good data management practices, including cost-forecasting practices, which facilitate sustainable long-term data preservation, curation, and access. Such activities would benefit the entire biomedical research community, including the institutions and funding entities that support research. To support these endeavors, funding entities need to better understand research-community needs, help the community to define desired outcomes, support training, develop realistic and actionable metrics for success, and provide near-term incentives for success.

The biomedical research community, including the institutions and those that fund research, needs to provide incentives for adopting good data management practices, including good cost-forecasting practices, that facilitate sustainable long-term data preservation, curation, and access. Researchers often lack the skills needed for efficient and effective data management, which translates to a lack of meaningful management and good data stewardship, and little understanding of the real costs of effective management or of how to forecast them. Based on interactions with various stakeholders during the conduct of this study, data management training for researchers is needed and desired. Training could help change the biomedical research culture so that good data management and cost forecasting become the norm in responsible research.

An incentive for researchers to more accurately account for the uncertainties associated with sharing data and future reuse might be for funders to place greater emphasis on such accounting in data management plans (DMPs) in grant proposals (discussed later in this chapter). Researchers would see an immediate benefit (i.e., research funding is contingent on action), and the prompt to take action is coming when the researchers are establishing their processes for research conduct (thus providing a timely prompt). But clear guidance for the researcher is also necessary for DMPs to be meaningful. For example, requests for application for funding sometimes seem to require new data resources to be all things for all stakeholders and even include potentially contradictory requirements. Incorporating better-directed guidance and training of individuals in data management would increase the likelihood of the desired outcomes.

Publishers and journals could also provide incentives, for example, by requiring data citations. Efforts to implement formal data citation across publishers (Cousijn et al., 2018; Fenner et al., 2019) are gaining traction, and most publishers at least informally accept data citations, although fully machine-readable data citations are still rare. Fully actionable data citations, however, require the infrastructure of a State 2 active repository or State 3 long-term preservation archive to ensure compliance with “findable, accessible, interoperable, and reusable” data principles (Wilkinson et al., 2016). Thus, by requiring data citations, publishers and journals can motivate researchers to use such infrastructure more consistently and possibly earlier.

Data management capacity might be increased by incorporating greater detail in, for example, training offered through the Collaborative Institutional Training Initiative (CITI) Program’s Responsible Conduct for Research modules.¹ Requiring independent proof of training as a requirement of receiving awards might improve capacity, as might encouraging multidisciplinary training much like that offered through the Integrative Graduate Education and Research Traineeship (IGERT)² program at scale, perhaps through multiagency support. Research on the normative outcomes of any increase in benefits resulting from improved data management skills could inform future training efforts.

Another incentive for researchers to participate constructively in data management, and especially State 2 resource planning, is providing them the opportunity to influence a superior computational environment. A data science platform could support complex research environments that free the researcher to focus on the science rather than on data collection and management. This capability could effectively reduce a State 1 environment to data

___________________

¹ The website for the CITI Program’s Responsible Conduct for Research modules is https://about.citiprogram.org/en/series/responsible-conduct-of-research-rcr/, accessed December 19, 2019.

² The website for the IGERT award is https://nsf.gov/awardsearch/showAward?AWD_ID=0903629&HistoricalAwards=false, accessed December 19, 2019.

(i.e., signal) capture. This idea underscores the benefit of a closer interaction between the data curators of State 2 platforms and individual researchers, recognizing that there are a variety of approaches to building and managing archives. This approach would co-locate the costs of supporting computing and analytics with an active repository.

• Understand the charges associated with storage and computation in a data resource, regardless of who “pays the bill,” when making decisions about data and workflows. Institutions supporting research might develop mechanisms to inform researchers of the actual costs paid for the services rendered to them and encourage them to limit those costs.

Regardless of who provides the resources, there is a lack of visibility regarding storage costs in individual laboratories, institutions, and community resources. Understanding the charges associated with storage and computation in a data resource is vital for researchers making decisions about their own data and workflows. Researchers are often unaware of costs associated with data management in part because they typically are not responsible directly for those costs. Costs may be invisible to them if borne by their institutions or by a data-resource-platform manager (see Box 3.3). Purchased services (e.g., storage and computing) may be important, although the ability of individual researchers working in a primary research environment to forecast and manage those costs depends on the transparency of the information-technology environment. Mechanisms are needed to inform researchers of the actual costs paid for the services rendered to them, even if they are not directly charged.

ADVANCES FOR PRACTICE

Successful cost forecasting and sustainable management depend largely on an environment that supports decision makers, whether they are researchers, data scientists, data resource managers, or funding agencies. Methodologies for forecasting the life-cycle costs for preserving, archiving, and accessing biomedical data are immature and few tools and resources are available for those to quantify long-term costs with confidence and aid better understanding of uncertainties that can be tracked. Addressing the necessary advances identified below could facilitate the change in culture needed among decision makers to create such an environment. The following activities, likely to be enabled at an agency or research-institution level, could advance practices and drive future improvements in the ability to forecast costs.

• Recognize explicitly that scientific data constitute an asset and that data stewardship requires support. Biomedical research data and data resources are vital to the delivery of good science, and, ultimately, to the public good. The universities and institutions that support or enable research and host data resources, in turn, benefit from the recognition of that support.

Measuring data value in monetary terms is difficult, and yet it is the potential value of data that warrants the financial investments associated with their preservation. Unlike physical infrastructure, biomedical research data and the resources that house them are assets that contribute to the delivery of good science and, ultimately, the public good. The institutions that host or enable that public good will likely benefit from the recognition received for supporting such assets. Even so, there is only so much that can be done on the project or platform level. Currently, it is impossible to look across all data in the distributed biomedicine data enterprise to learn what data sets exist. Persistent metadata repositories are needed that include data set and research object identifiers.

• Systematically collect data on costs associated with the biomedical research data enterprise to allow the translation of the framework outlined in this report into resources and methodologies that would benefit individual researchers and repository institutions. A clear locus of responsibility for compiling this information systematically is necessary.

The true costs of preserving, archiving, and accessing biomedical research data need to be investigated in a systematic way at the funding-program-manager level rather than at the individual researcher or project level. Cost

information at the researcher level could be collected at the outset of many projects when funds are requested, through DMPs, and tracked when progress is evaluated. Information thus collected so far has neither been uniformly integrated into award decisions nor transmitted to other parties involved. Researchers working in a State 1 primary research environment are often required to keep data for a prescribed period of time but are typically not responsible for costs or management beyond this.

Costs in the longer run (e.g., States 2 and 3) generally become an institutional responsibility. But institutional-level planning horizons are often only 1 or 2 years ahead rather than the many years required to realize the promise of current and future repositories. Some federal agencies (e.g., Department of Defense and Department of Energy) sustain a cadre of cost analysts and consider gathering the data needed for estimating costs as an important agency responsibility. Those agencies treat cost estimation as a profession and invest in training, recognizing success, critiquing failures, and encouraging assembly of cost-related data. The biomedical research data-preservation enterprise has become an undertaking that warrants a similar cadre to augment domain expertise and expertise in data science.

• Develop easier mechanisms for creating and maintaining DMPs, automatically incorporating data and metadata into resources, and improving citations for data to work together with other research products. By providing these mechanisms, funders and research institutions could help improve efficiency, return value for stakeholders, and increase the likelihood that stakeholders will make sound data-related decisions.

DMPs (see Appendix B) are typically static documents prepared as a mandatory—but not necessarily influential—part of the funding process. Placing more emphasis on quantified cost forecasts during the development of the DMP and the award process may be one way to incentivize early planning and communication, even if cost forecasts are uncertain. However, placing greater emphasis on cost forecasting at that time does not mean that the forecasts will become more precise estimates, rather they could be considered accurate reflections of uncertainties. Cost forecasts and DMPs need to evolve and be updated as research progresses and as associated data and the resources and technologies available to manage those data evolve. Monitored evolution of a DMP (e.g., at midterm evaluations or at the end of the award period when making payment on awards) might inform eligibility for future funding. Machine-actionable DMP (e.g., Simms et al., 2017; see Appendix B) technologies may address issues related to realistic and evolving data management.

FACTORS FOR SUCCESSFUL ADOPTION OF DATA-FORECASTING APPROACHES

The current system for funding research cannot accommodate data life-cycle cost forecasting. For instance, the quantity, quality, and format of data collected might be uncertain when a proposal is written. They may become increasingly less uncertain a year into the award and when a grant is only partially spent out. During an information-gathering workshop organized by the committee (NASEM, 2020; see Appendix A for agenda), participants described incentives to do cost forecasting, with much discussion of how both incentives (“carrots”) and rules and enforcement (“sticks”) were important. Participants described how developing those rules and educating the community about the value of implementing them was fundamental to cost forecasting becoming a part of the responsible conduct of research (rather than a bureaucratic chore).

The culture change for the biomedical research community described in this report needs to be driven by community engagement. The Behavioral Insights Team³ in the United Kingdom developed principles for encouraging desired outcomes, which might be applicable in the development and management of community repositories (Service et al., 2010). They recommend making processes easy, attractive, social, and timely (the “EAST” principles). People are more likely to engage in desired behavior if doing so is easy. To the greatest extent possible, it should be made easy for researchers and other stakeholders to make good data-related decisions from the onset. Research funders, research institutions, and journals are in positions to offer incentives, but processes need to be

___________________

³ The website for the Behavioral Insights Team is https://www.bi.team/, accessed December 19, 2019.

driven by researchers so as to meet their needs and so they fully understand and agree to the value returned to them for their efforts. The ultimate beneficiaries of such efforts, of course, are the scientific enterprise and our nation’s citizens, whose well-being biomedical science seeks to advance.

REFERENCES

This page intentionally left blank.