Mixed-Method Review of Channels for Communicating Public Health Alerts and Guidance with Technical Audiences During a Public Health Emergency
This appendix provides a detailed description of the methods for and the evidence from the mixed-method review of channels for communicating public health alerts and guidance with technical audiences during a public health emergency, which is summarized in Chapter 6.1
KEY REVIEW QUESTIONS AND ANALYTIC FRAMEWORK
The overarching question that guided this review addresses the effectiveness of different channels for communicating public health alerts and guidance with technical audiences during a public health emergency. To answer this overarching question, the committee sought evidence on several sub-questions related to documented benefits and harms associated with the channels themselves, as well as the engagement of technical audiences in the development of communication plans and channels. The committee also examined the evidence on the factors that create barriers to and facilitators of effective communication with technical audiences (see Box B3-1).
Effective communication channels provide a conduit for information to be transmitted from public health authorities to recipient technical audiences (and in some cases, allow for bidirectional exchange). The objective of this public health emergency preparedness and response (PHEPR) practice is to ensure that technical audiences are aware of and understand up-to-date information about a particular public health threat. As depicted in the analytic framework in Figure B3-1, awareness of current alerts and guidance may influence the behaviors of information recipients (e.g., changes in diagnostic testing protocols, use of
1 This appendix draws heavily on three reports commissioned by the committee: “Data Extraction and Quality Assessment: Methodology and Evidence Tables” by the Brown University Center for Evidence Synthesis in Health; “Communicating Public Health Alerts and Guidance with Technical Audiences: Qualitative Research Evidence Synthesis” by Julie Novak and Pradeep Sopory; and “Information Sharing with Technical Audiences: Findings from After Action Reports and Case Reports” by Sneha Patel (see Appendix C).
personal protective equipment, case reporting), which may in turn improve the response to a public health threat (e.g., through improved situational awareness and coordination of response partners) and reduce associated morbidity and mortality (e.g., by reducing or better managing infections).
EVIDENCE SUPPORTING THE PRACTICE RECOMMENDATION
This section summarizes the evidence from the mixed-method review examining different channels for communicating public health alerts and guidance with technical audiences during a public health emergency. It begins with a description of the results of the literature search and then summarizes the evidence of effectiveness. In formulating its practice recommendation, the committee considered evidence beyond effectiveness, which was compiled using an Evidence to Decision (EtD) framework encompassing balance of benefits and harms, acceptability and preferences, feasibility and PHEPR system considerations, resource and economic considerations, equity, and ethical considerations. The evidence from each methodological stream applicable to each of the EtD criteria is discussed; a synthesis is provided in Table B3-4 later in this appendix and in Chapter 6. Graded finding statements from evidence syntheses are italicized in the narrative below.
Full details about the study eligibility criteria, search strategy, and processes for data extraction and individual study quality assessment are available in Appendix A. Appendix C links to all of the commissioned analyses informing this review.
Results of the Literature Search
The searches of bibliographic databases identified a total of 5,853 potentially relevant citations (deduplicated) for the mixed-method review of channels for communicating public health alerts and guidance with technical audiences during a public health emergency. A search of the gray literature, reference mining, and a call for papers contributed an additional 599 articles. All 6,452 citations were imported into EndNote and were included in title and abstract screening. During screening, 6,279 articles were excluded because their abstracts did not appear to answer any of the key questions or they indicated that the articles were
commentaries, editorials, or opinion pieces. After the abstracts had been reviewed, 173 full-text articles were reviewed and assessed for eligibility for inclusion in the mixed-method review. The committee considered 79 articles for data extraction and ultimately included 61 articles in the mixed-method review. Figure B3-2 depicts the literature flow, indicating the number of articles included and excluded at each screening stage. Table B3-1 indicates the types of evidence included in this review.
1. Determining Evidence of Effect
Two quantitative comparative studies directly addressed the overarching key question regarding the effectiveness of different channels for communicating public health alerts and guidance with technical audiences during a public health emergency. Both studies evaluated types of electronic messaging systems (e.g., email, fax, text messaging) that are used to push information out to target audiences (rather than relying on target audiences to pull information down). A meta-analysis of the evidence for the effectiveness of these communication channels was not feasible, so the committee conducted a synthesis without meta-analysis (as described in Chapter 3).
Consistent with the methods described in Chapter 3, in making its final judgment on the evidence of effectiveness for electronic messaging channels for communicating public health alerts and guidance with technical audiences during a public health emergency, the committee considered other types of evidence that could inform a determination of what works for whom and in which contexts, ultimately reaching consensus on the certainty of the evidence (COE) for each outcome. Including forms of evidence beyond quantitative comparative studies is particularly important when assessing evidence in settings where controlled studies are challenging to conduct and/or other forms of quantitative comparative data are difficult to obtain. As discussed in Chapter 3, descriptive evidence from real-world implementation of practices offers the potential to corroborate research findings or explain differences in outcomes in practice settings, even if it has lesser value for causal inference. Moreover, qualitative studies can complement quantitative studies by providing additional useful evidence to guide real-world decision making, because well-conducted qualitative studies produce deep and rich understandings of how interventions are implemented, delivered, and experienced. Other forms of evidence considered for evaluation of effectiveness included quantitative data reported in descriptive surveys, case reports, and after action reports (AARs) that involved a real disaster or public health emergency.
Of note, some surveys, case reports, and AARs report on passive electronic messaging systems that rely on the information-seeking behavior of the target audience (e.g., websites) and communication channels other than electronic messaging systems (e.g., telephone conferencing, hotlines). In the absence of comparative data from which conclusions regarding effectiveness could be drawn, however, these other communication channels were not included in the committee’s synthesis of quantitative evidence. While it is clear that channels other than electronic messaging systems are being used in practice to communicate public health alerts and guidance with technical audiences, the effectiveness of these channels has not yet been rigorously studied in the PHEPR context.
Evidence from Quantitative Research Studies
Two quantitative comparative research studies examined the effectiveness of different channels for communicating public health alerts and guidance with technical audiences. A randomized controlled trial conducted by Baseman and colleagues (2016) from 2009
|Evidence Typea||Number of Studies (as applicable)b|
|Quantitative noncomparative (postintervention measure only)||0|
|After action reports||29d|
|Parallel (systematic reviews)||N/A|
b Note that sibling articles (different results from the same study published in separate articles) are counted as one study in this table. Mixed-method studies may be counted in more than one category.
c Two surveys containing a qualitative analysis of free-text responses were included in the qualitative evidence synthesis. The two studies were not classified as qualitative research studies and are not included in the qualitative study count for this table. As described in Chapter 3, the findings from these sources were extracted and considered separately in the qualitative evidence synthesis to affirm or question those findings from the more complete qualitative studies.
to 2012 (the REACH trial) compared what then were considered “traditional” and new (mobile) communication strategies for use by public health agencies to transmit time-sensitive health information to health care providers in Washington State and Montana over a 6- to 12-month period. The trial compared email, fax, SMS (text messaging), and no messaging. The investigators assessed 846 providers’ recall of message topics, correcting for the possibility of false recall. All messaging methods were more successful with respect to recall of message topics than no (active) messaging. Email was statistically significantly more effective than either fax or text (45 percent message topic recall versus 38 percent and 37 percent for fax and text, respectively, p <0.05) (Baseman et al., 2016). Accessing available hyperlinks (available through all messaging methods) was associated with greater recall (odds ratio [OR] = 3.9; p <0.001). Text messaging resulted in a statistically significantly greater likelihood of accessing hyperlinks relative to either phone or fax messages (p ≤0.001). A substudy of the REACH trial (N = 528) showed that increases in the number of messages sent to a provider were associated with a decreased likelihood of correct recall of message content. Every increase of one public health message per week resulted in a statistically significant 41.2 percent decrease (p <0.01) in the odds of recalling the study message content, a finding suggestive of alert fatigue (Baseman et al., 2013). The committee was concerned that the comparisons made in this study are outdated and potentially not relevant (in part because simultaneous messages via multiple modalities are common practice today). Also important is that although study messages were time sensitive, they were not real emergency alerts. These concerns were not incorporated into the assessment of risk of bias and methodological quality for this study as they were not considered to be issues of bias or methodological
quality. The trial had no serious methodological limitations, and overall, the study (and each outcome) was deemed to be of good methodological quality.
A retrospective, nonrandomized comparative study by van Woerden and colleagues (2007) evaluated data from 2001 and 2002 to determine whether the number of patients tested for Q fever was affected by sending faxes to primary care physicians about a Q fever outbreak in Wales that occurred in 2002. After the Q fever outbreak was identified at an urban factory, the National Public Health Service used a fax cascade system to alert primary care practices. Physicians were asked to submit serum samples on any patient meeting a clinical case definition of Q fever and having an association with the area where the outbreak appeared to be occurring. The researchers compared the number of Q fever diagnostic tests ordered during the same 2-month period in 2001 and 2002, which included the dates of the fax cascade in mid-September 2002, as well as the preceding 2 weeks. Approximately 565,000 people lived in the analyzed community. The number of requests for Q fever tests during the 2-month period in 2002 was three times higher than the number of requests during the corresponding time period of the preceding year, and the difference in the proportion of the population tested in 2001 and 2002 was statistically significant (p <0.001). The researchers found an association between the timing of the faxes and the significant increase in the number of requests for Q fever tests, in contrast with similar numbers of test requests in the prior 2 weeks in both analyzed years. This study had major limitations related to lack of adjustment for differences between the two analyzed years, as well as poor power. In addition, the researchers did not attempt to account for other factors (such as the local press) that may have impacted the number of tests ordered. Overall, the study (and each outcome) was deemed to be of poor methodological quality.
Other Evidence That May Inform Effectiveness
The results of one survey support the above findings from quantitative comparative studies regarding the effectiveness of electronic messaging systems.2 Argonne National Laboratory surveyed stakeholders of the Illinois Department of Public Health (IDPH) representing a mix of health and nonhealth (e.g., schools, businesses) sectors regarding strengths and weaknesses of IDPH information-sharing practices during the response to the H1N1 outbreak. Only 24 percent of respondents reported fax to be an effective means of communication during such a disease outbreak, while 64 percent reported that the IDPH website provided timely and useful information. Fewer than half of respondents identified the hospital Health Alert Network (H-HAN) as a “useful” communication tool during a disease outbreak, although 72 percent of hospital respondents did so (most other stakeholders lacked access to the H-HAN) (Walsh et al., 2010).
The committee reviewed six case reports containing quantitative data related to electronic messaging systems used for communicating public health alerts and guidance with technical audiences during public health emergencies. One supportive case report, published in a 2013 Morbidity and Mortality Weekly Report, describes how the Iowa Department of Public Health’s multipronged communication strategy featuring multiple electronic messaging channels (HAN, electronic newsletter, website, Twitter) resulted in increased test-
2 As described in Chapter 3, the committee reviewed other evidence that informed the COE (e.g., mechanistic evidence, experiential evidence from case reports and AARs, qualitative evidence) for coherence with the findings from the quantitative research studies and classified that evidence as very supportive, supportive, inconclusive (no conclusion can be drawn on coherence, either because results are mixed or the data are insufficient), or unsupportive (discordant with the findings). The distinction between supportive and very supportive is based on the magnitude of the reported effect and the directness of the evidence to the question of interest.
ing (as compared with the month prior to the alerts) for an infectious agent presumed to be causing an outbreak of cyclosporiasis. One health care provider ordered Cyclospora testing for a patient with undiagnosed and recurring vomiting and diarrhea after reading the electronic newsletter alert, which resulted in appropriate treatment for that patient (CDC, 2013).
A second supportive case report, published by Nagykaldi and colleagues (2006), describes a web-based alerting and surveillance system for influenza-like illness (ILI) in Oklahoma (OKAlert-ILI). The Oklahoma State Department of Health used this system to send a weekly message containing surveillance data for influenza and other infectious diseases to members of the Oklahoma practice-based research network and to collect ILI reports from users. OKAlert-ILI sentinel reports were correlated with culture-positive laboratory influenza test results (Pearson correlation coefficient = 0.827), but there was a 7-day lag for laboratory test results (i.e., the bidirectional ILI system resulted in more timely case reporting).
A third, very supportive case report, published by Lurio and colleagues (2010), describes an alerting system whereby HAN alerts sent from the public health department are received by leadership at the Institute for Family Health (a network of community health centers) and may, if appropriate, be translated into a best practice advisory (BPA) that alerts clinicians and provides diagnostic/treatment guidance through a popup in the electronic health record (EHR). HAN notifications that were translated into BPAs resulted in orders for diagnostic tests:
- In a case of a Legionella outbreak, a BPA was triggered for 142 patients, and orders were activated in 5 instances (3.5 percent). Two orders were submitted for Legionella urine antigen, a test that had not been ordered in the 5 years prior to the alert.
- During an E. coli O157:H7 outbreak, a BPA was triggered for 287 patients and orders for diagnostic tests were activated 65 times (22.6 percent), but no instance of E. coli O157:H7 was identified.
- During a measles outbreak, a BPA was triggered for 198 patients and orders were activated on four occasions (2 percent). Specimens from a potential case identified through the system were sent to the Department of Health but were not confirmed as measles.
The remaining three case reports with quantitative data (Daniel et al., 2005; Gamache et al., 2010; Gotham et al., 2007) are inconclusive with respect to the effectiveness of the communication channels discussed.
Few AARs addressing the use of communication channels report quantitative data, and among those that do, the findings regarding the effectiveness of electronic messaging channels are mixed. Two of the AARs in the sample collected for this mixed-method review report survey results regarding the perceived “effectiveness” of or satisfaction with electronic communication channels used during a public health emergency. In a survey of registered vaccine providers in New Hampshire during the H1N1 outbreak (N = 141), 92 percent of respondents agreed or strongly agreed that the HAN is “an effective vehicle for communicating information to [their] organization” (New Hampshire Department of Health and Human Services and New Hampshire Department of Safety, 2010). In contrast, another AAR from the H1N1 response reports that only 38 percent of survey respondents were satisfied or very satisfied with the HAN, but 93 percent were satisfied or very satisfied with H1N1 information provided by webcasts, while ~90 percent were satisfied or very satisfied with such information provided though the state government website (Wisconsin Division of Public Health, 2010).
Some AARs include information on the number or proportion of individuals who received electronic alerts. For example, a poll launched during a hospital exercise showed that only
70 percent of participating hospitals reported receiving the HAN alert (Chicago Department of Public Health et al., 2011). This and other AARs raise concerns regarding the distribution of electronic alerts, such as messages not being sent to the full recipient list, delays in receipt due to the passive nature of communication (i.e., intended recipients were not actively monitoring for messages), and failure by initial recipients to ensure that the HAN alert is passed along to other stakeholders. Thus, AARs raise the potential of an undesirable effect whereby reliance on electronic messaging methods could interfere with other, traditional modalities and delay responses.
Summary of the Evidence: Technical Audience Awareness
The committee concluded that there is moderate COE that electronic messaging systems such as email, fax, and text messaging are effective communication channels for increasing technical audiences’ awareness of public health alerts and guidance during a public health emergency. Two quantitative comparative studies (Baseman et al., 2016; van Woerden et al., 2007) provide moderate COE regarding the effects of electronic messaging systems on technical audiences’ awareness of public health alerts and guidance during a public health emergency (see Table B3-2). Other forms of evidence, which include one supportive survey, two supportive and one very supportive case reports, and mixed AAR evidence that raises questions about potential undesirable effects (although providing no conclusive data on harms) were insufficient to upgrade the COE but also did not warrant downgrading.
|Quality Assessment||Number of Studies||2|
|Study Infomation||Baseman et al., 2016
Randomized controlled trial (RCT), good methodological quality (▲)
van Woerden et al., 2007
Nonrandomized comparative study (NRCS), retrospective, poor methodological quality (▲)
|Risk of Bias||Not serious|
|Indirectness||Serious (Baseman et al., 2016, study not conducted during a public health emergency)|
|Upgrade for Large Effect, Dose Response, Plausible Confounding||No|
|Summary of Findings||Initial Certainty of the Evidence (COE)||Moderate|
|Other Evidence||One supportive survey, case report evidence (two supportive and one very supportive), mixed after action report (AAR) evidence that raises concerns about undesirable effects|
|COE||Moderate (improves technical audiences’ awareness)|
Summary of the Evidence: Technical Audience Use of Guidance
The committee concluded that there is very low COE that electronic messaging systems are effective communication channels for increasing technical audiences’ use of current public health guidance during a public health emergency. One quantitative comparative study (van Woerden et al., 2007) provides very low COE regarding the effects of electronic messaging systems on technical audiences’ use of current public health guidance during a public health emergency (see Table B3-3). Other forms of evidence, which include a supportive survey, two supportive and one very supportive case reports, and mixed AAR evidence that raises questions about potential undesirable effects (although providing no conclusive data on harms) were insufficient to upgrade the COE but also did not warrant downgrading.
|Quality Assessment||Number of Studies||1|
|Study Infomation||van Woerden et al., 2007
Nonrandomized comparative study (NRCS), retrospective, poor methodological quality (▲)
|Risk of Bias||Very serious|
|Upgrade for Large Effect, Dose Response, Plausible Confounding||No|
|Summary of Findings||Initial Certainty of the Evidence (COE)||Very low|
|Other Evidence||One supportive survey, case report evidence (two supportive and one very supportive), mixed after action report (AAR) evidence that raises concerns about undesirable effects|
|COE||Very low (improves technical audiences’ use of guidance)|
Based on the evidence described above, the committee concludes that there is evidence of differential impact of different technologies employed as electronic messaging systems for communicating public health alerts and guidance with technical audiences during a public health emergency to increase awareness and appropriate use. However, data are insufficient to conclude what technology is best for which audiences in which scenarios.
2. Balance of Benefits and Harms
Synthesis of Evidence of Effect
As discussed in the section on effectiveness above, electronic messaging communication channels may have beneficial effects on technical audiences’ awareness of public health alerts and guidance (moderate COE). However, these effects may be dampened by alert fatigue arising from excessive message volume.
Qualitative Evidence Synthesis
The 10 studies included in the body of qualitative evidence include very little discussion of the benefits and harms per se of communication channels. Instead, the studies largely considered how different channels corresponded to facilitators of and barriers to communication with technical audiences, after incorporating contextual factors. Participants reported what did and did not work well and offered suggestions for consideration. Notably, however,
clear benefits were observed when public health officials engaged in thoughtful and inclusive deliberations with stakeholders about providing guidance. Such deliberations not only facilitated effective communication but also built relationships with health care providers and other stakeholders that were characterized by trust, respect, responsiveness, transparency, and flexibility (Khan et al., 2017; Lis and Resnick, 2018).
Although not specific to particular channels, technical audiences reported undesirable impacts related to the approach taken to communication of alerts and guidance. For example, although some duplication across different channels could be helpful, the volume of messages could quickly pose a burden and discourage rather than encourage the use of arriving guidance (Khan et al., 2017; Markiewicz et al., 2012; Staes et al., 2011). In addition, some important technical audiences could be excluded from existing communication channels. Leung and colleagues (2008) report that some smaller agencies and community-based partners were not included in prepared email listservs or other directories specifying recipients for public health guidance, and therefore received information through other pathways and relationships. Filice and colleagues (2013) point to a notable harm that manifests when no guidance is received or when guidance is incongruent with practice in public health emergencies. Such instances can lead to poor implementation of guidance, as health care providers may adhere to routine practices and institutionally determined adaptations. Additionally, when past experience and lessons learned fail to lead to changes in protocols and practices, disillusionment with future, coordinated efforts for preparedness may result.
Case Report and AAR Evidence Synthesis
Case reports and AARs included in this review do not specifically assess the benefits or unintended consequences of specific modes of communication. However, a few attributes improved timeliness to specific communication mechanisms. For instance, Cavey and colleagues (2009) assert that the use of telephone reporting through a shelter hotline improved timeliness, reporting compliance, accuracy, and staff satisfaction and knowledge. Gamache and colleagues (2010) also identify timeliness as a benefit of sending public health alerts through Health Information Exchange platforms. Similarly, Nagykaldi and colleagues’ (2006) findings from an evaluation of the OKAlert-ILI system discussed earlier indicate more timely and accurate responses to ILI cases. Reports suggest that in-person meetings, teleconferences, and webcasts also improve timeliness by providing for real-time feedback (Delaware Division of Public Health, 2010; Wisconsin Division of Public Health, 2010).
One noted harm relates to the potential for important stakeholders to be omitted from those channels that require enrollment (New Hampshire Department of Health and Human Services and New Hampshire Department of Safety, 2009b). For example, while the HAN is cited as an important direct communication link to technical audiences, several reports point to the need to expand its reach, as not all technical audiences (e.g., first responders, individual providers, emergency medical departments, medical practices, local boards of health) are enrolled in that network (Boston Public Health Commission, 2013; Daniel et al., 2005; Delaware Division of Public Health, 2010; Gamache et al., 2010; Gursky et al., 2003; Massachusetts Emergency Management Agency et al., 2014; New Hampshire Department of Health and Human Services and New Hampshire Department of Safety, 2009b; Ohio Department of Health, 2010).
Although the evidence from case reports and AARs indicates that hotlines can facilitate bidirectional information sharing, hotline staff may experience excess fatigue as a result of stress associated with response efforts (Boston Public Health Commission, 2013). Additionally, Lurio and colleagues (2010) discuss the potential for alert fatigue if alerts are not
targeted and tailored to specific provider types. “Paralysis by analysis” can also result from an overwhelming amount of information being shared in a short time without clear guidance (Metropolitan Medical Response System, 2016). Furthermore, the Delaware Division of Public Health (2010) reports experiencing a loss of “credibility of the public health community” due to frequent and delayed Centers for Disease Control and Prevention (CDC) modifications to recommendations on vaccine distribution resulting from a temporary vaccine shortage.
Descriptive Survey Study Evidence
Results of one survey support the above findings from the syntheses of qualitative and case report and AAR evidence regarding the potential for information overload. Staes and colleagues (2011) surveyed primary care physicians in Utah and found that they often received emails with information on H1N1 from multiple sources, which could result in an excessive amount of email and information overload. One survey also supports the finding that stakeholders may miss public health alerts and guidance as a result of not being enrolled in communication systems. An Argonne National Laboratory (Walsh et al., 2010) survey found that many stakeholders of the IDPH lacked awareness of or access to some communication channels (e.g., conference calls, H-HAN) used to communicate public health information during the H1N1 outbreak.
3. Acceptability and Preferences
Quantitative Study Evidence
Baseman and colleagues (2016) report on health care providers’ preferences regarding channels used by public health authorities to transmit time-sensitive alerts and advisories as part of the REACH trial described earlier in this appendix. Email was the preferred communication channel, while SMS (text messaging) was preferred over fax for alerts but not advisories. There were differences in preferred channels based on age, gender, provider type, and whether the provider read emails on the phone or a computer.
Qualitative Evidence Synthesis
The body of qualitative evidence shows that technical audiences prefer email and fax as channels for communicating public health alerts and guidance (Khan et al., 2017; Ockers, 2011; Revere et al., 2015). Such audiences also stress the importance of and preference for just-in-time dissemination of guidance (Janssen et al., 2006; Leung et al., 2008).
An inclusive, collaborative, and dynamic process of engaging technical audiences in discussions on communication channels and processes for the generation of public health guidance before and during a public health event may improve providers’ acceptance of and implementation fidelity to that guidance (Filice et al., 2013). Another factor that may affect the acceptability of communication strategies is the potential for bidirectional exchange of information between public health agencies and their stakeholders. To this end, public health agencies can share results generated from information submitted by technical audiences to demonstrate its utility and value, thus ensuring that these results can be utilized by those audiences in carrying out their own work and by public health agencies in improving the community’s health (Revere et al., 2015).
Case Report and AAR Evidence Synthesis
The vast majority of case reports and AARs reviewed do not address the acceptability of communication channels or preferences of technical audiences with regard to information sharing, indicating a need for further research on the acceptability and feasibility of specific communication channels to determine how to best improve traditional channels and whether reported innovations (e.g., OKAlert-ILI, shelter hotlines) are replicable. However, it is evident that technical audiences prefer timely, accurate, consistent information that is easy to navigate and bidirectional (i.e., the ability to both send and receive information). Alerts and guidance tailored to specific audiences are also preferred to facilitate translating the information into appropriate action. Additionally, some audiences desire flexibility in their application of the guidance provided. According to the Delaware Division of Public Health (2010), for example, during the 2009 H1N1 response in that state, physicians looked to public health to determine appropriate priority groups for treatment and prophylaxis, but they also wanted the flexibility to reevaluate priority groups based on the data available from the state as the crisis progressed. The AAR suggests further that physicians preferred direct communication from a credible source and that the majority looked to the department of health or their medical society for leadership. Multiple AARs point to an appreciation of webcasts and a direct line of communication to experts (by phone or in person) (County of San Diego, 2018; Minnesota Department of Health, 2013a; Ohio Department of Health, 2010; Wisconsin Division of Public Health, 2010; Wisconsin Hospital Emergency Preparedness Program, 2010).
Descriptive Survey Study Evidence
Six surveys asked about the preferences of technical audiences (mainly health care providers) for communication channels and sources of alerts and guidance during a public health emergency. This survey evidence supports the finding from other evidence streams that email is the preferred channel for communicating this information. Few surveys addressed text messaging and social media, but when queried on these communication channels, technical audiences did not appear to prefer them. Ockers (2011) found that email and fax blast were the most preferred communication channels for vaccine providers in California, Louisiana, Oregon, and Washington, while text messages and social media were least preferred, as many providers reported they could not receive messages by these latter channels. Surveyed health care providers in New York City during the city’s Zika response indicated a preference for receiving alerts and guidance through email (77 percent of respondents), while fewer than 10 percent preferred hard-copy mailings, in-person presentations, online webinars, or conference calls (Quinn et al., 2018). Seidl and colleagues (2010) report that approximately 94 percent of survey respondents found emails from the health incident controller to be somewhat or very useful as sources of information on H1N1, and nearly 90 percent found local health information bulletins useful (Seidl et al., 2010). In a survey of infectious disease physicians, 91 percent reported that emailed and faxed health alerts were useful, versus 43 percent for mailed notifications, 22 percent for social media, and 46 percent for smartphone applications (Santibanez et al., 2016). According to an Argonne National Laboratory report on surveys sent to various public health stakeholders to solicit feedback on the IDPH’s communication during the H1N1 outbreak, survey respondents indicated a preference for receiving information via email and the health department’s websites versus conference calls (Walsh et al., 2010).
Survey evidence also supports findings from the synthesis of case report and AAR evidence regarding public health authorities and medical societies being trusted information sources, although one survey found that primary care providers in Utah preferred institu-
tional sources (73 percent) over public health sources for information and guidance (Staes et al., 2011). Seidl and colleagues (2010) report that during the H1N1 response, communications from the local health authority were favored as a source of updated information. Santibanez and colleagues (2016) report that physicians generally preferred information from professional authorities, including CDC (98 percent of respondents), professional societies (92 percent), and online medical resources such as ProMed mail (23 percent), over public websites (22 percent) and social media (5 percent). Ockers (2011) notes that surveyed vaccine providers most frequently relied on state and local health departments for timely, accurate information about outbreaks and other public health threats, with less reliance on federal agencies, professional societies, and the news media. Among surveyed health care providers in New York City, the HAN was the preferred local source of information for providers (73 percent) (Quinn et al., 2018). Among nonlocal sources, information from CDC (64 percent) was preferred over that from the state health department (47 percent), followed by roughly similar levels of preference (30–35 percent) for public websites, medical journals and other online or point-of-care resources, and professional societies and associations. The majority of respondents to the Argonne National Laboratory (Walsh et al., 2010) survey mentioned above expressed the view that local health departments, but not the state health department, should customize CDC messages and updates to incorporate local information.
4. Feasibility and PHEPR System Considerations
Qualitative Evidence Synthesis
The body of qualitative evidence included in the review suggests that some communication channels are presently more feasible than others. Use of EHRs remains somewhat limited although EHRs are highly pertinent to the delivery of alerts and guidance at the point of individual care. Text-based messages/SMS are already in use. Advances in information technology often push the public health system to examine and adopt new channels (Janssen et al., 2006; Revere et al., 2015). Doing so, however, may raise concerns about compounding the burdens of message volume and availability of resources, such as personal or work devices and technical support. It appears important to weigh the strengths and limitations of any technology as a channel for communication during a public health emergency in context.
Case Report and AAR Evidence Synthesis
Findings on feasibility and PHEPR system considerations from case reports and AARs are discussed in the above section on this evidence stream relative to acceptability and preferences.
5. Resource and Economic Considerations
Qualitative Evidence Synthesis
All communication channels incur ongoing costs. However, as emphasized in one qualitative study, the indirect costs of new technologies related to training and technical support need to be added to their direct cost (Revere et al., 2015).
Case Report and AAR Evidence Synthesis
Adequate resources—from such resources as phones, radios, computers, servers, software platforms, and notification systems to human resources for hotline management, message development, message delivery, bidirectional communication, and many other functions—are critical for successful information sharing with technical audiences. Gamache and colleagues (2010) report that providing public health alerts through community health information exchanges yielded a cost savings to public health agencies relative to the traditional mail-based alerts. The estimated total cost savings was $3,638 for each set of alerts, based on sending 3,085 alerts to providers. Given advancements in technology, however, it may be more relevant to consider cost savings from this channel relative to other electronic channels. The OKAlert-ILI system was funded by a $50,000 health department contract, and has been made available to participating clinicians at no cost (Nagykaldi et al., 2006). The New York State Hospital Emergency Response Data System (HERDS), developed in 2001 by the New York State Department of Health in partnership with health care and public health agencies, reduced costs by implementing the application within an existing infrastructure used by response partner communities and leveraging existing multimillion-dollar investments (Gotham et al., 2007). Costs included development ($130,000) and the system’s annual recurring cost ($200,000).
Beyond the necessary technologies, findings from case reports and AARs suggest that successful information sharing depends on the availability of critical staff, such as liaison officers and subject-matter experts (Minnesota Department of Health, 2014; New Hampshire Department of Health and Human Services and New Hampshire Department of Safety, 2009a). Mathur and Beckermann (2010) discuss the frequent need to adjust communication strategies during Canada’s H1N1 response. Additional management support was brought in, the frequency of teleconferences was increased, target audiences were expanded, and on-site expert support was provided. This ability to adapt based on need was made possible by a strong, adequately resourced and supported team with vaccine expertise. Conversely, reliance on a handful of liaison officers during a full-scale exercise testing information sharing in the context of a novel respiratory illness was shown to be ineffective (Tri-County Health Department, 2017). Staff were overwhelmed by the sheer volume of calls and unable to meet the demand for information. Many partners were unable to get in touch, further highlighting the need for sufficient human resources to support effective information sharing.
Qualitative Evidence Synthesis
The qualitative evidence included in this review provides some indication that small jurisdictions and rural areas are less able to adapt to changes in technology and communication channels relative to other areas (Revere et al., 2015).
Case Report and AAR Evidence Synthesis
Overall, the body of case reports and AARs does not address equity issues associated with different channels for communicating public health alerts and guidance with technical audiences, highlighting an important evaluation gap. The AAR from the 2009 H1N1 response in Washington’s Seattle & King County identifies an opportunity for “improved relationships with smaller and ethnic pharmacies to expand outreach to ethnic and vulnerable populations” (Public Health—Seattle & King County, 2009). Although this opportunity is not directly
related to sharing information with pharmacies, improving relationships with technical audiences serving underserved populations may lead to more targeted and tailored information sharing during a public health emergency.
Wynn and Moore (2012) are the only authors to mention equity explicitly, stating, “especially during a public health emergency, the health care system must show sensitivity to socioeconomic circumstance and use an understanding of the determinants of health when developing emergency mitigation strategies.” The authors go on to describe the bidirectional role of family health teams during the 2009 H1N1 response in Ontario. These teams were able to relay patient needs to public health agencies through communication with primary care providers. The United States may be able to apply similar approaches when developing communication channels for technical audiences to promote greater equity.
7. Ethical Considerations3
In addition to the equity concerns noted above, which are often considered as reflecting ethical values, the primary value of communication using appropriate channels is often considered to be instrumental, meaning that it is important because using appropriate channels to convey information presumably leads to better information delivery, which can facilitate better decision making. In the language of ethical principles, communication using appropriate channels is important because it promotes the principle of harm reduction/benefit promotion. But problems of overcommunication (such as information overload or alert fatigue) are also possible when appropriate communication channels are used, which can lead to worse or delayed decisions. In addition, communication using appropriate channels also has intrinsic value; that is, setting aside whether decision making is improved by better information delivery, communicating with individuals and communities in ways that are most effective for them is important to achieve transparency, which reflects the principle of respect for persons and communities. As in considering the instrumental value of using more effective channels for communication, one should remember that while communication using ineffective channels is obviously disrespectful, overloading effective communication channels is also disrespectful during crises, when recipients have limited time and bandwidth. In sum, selecting appropriate communication channels is ethically important, and so is careful selection of the information to be delivered over those channels.
3 Ethical considerations included in this section were generated through committee discussions, drawing on the ethical principles laid out in Box 3-4 in Chapter 3 and key ethics and policy texts, including the 2009 Institute of Medicine letter report on crisis standards of care (IOM, 2009), the 2008 CDC white paper “Ethical Guidance for Public Health Emergency Preparedness and Response: Highlighting Ethics and Values in a Vital Public Health Service” (Jennings and Arras, 2008), Emergency Ethics: Public Health Preparedness and Response (Jennings et al., 2016), and The Oxford Handbook of Public Health Ethics (Mastroianni et al., 2019).
|What is the effectiveness of different channels for communicating public health alerts and guidance with technical audiences during a public health emergency?|
|Balance of Benefits and Harms
Although only two quantitative comparative research studies evaluate the effectiveness or benefits of specific channels for communicating public health alerts and guidance, case reports and after action reports (AARs) also cite improved audience awareness and the timeliness of messaging as benefits for some communication channels, such as electronic messaging systems (e.g., fax, email, web-based alerting and surveillance systems), teleconferences, and hotlines. Reported harms and undesirable impacts rarely relate to a specific communication channel but arise as a result of how communication is implemented. For example, several evidence sources note the potential for important stakeholders to be left out of the loop if excluded from the systems used to distribute messages (e.g., the Health Alert Network, teleconferences) and/or if contact information is not kept up to date. Also commonly reported as undesirable impacts of public health messaging are alert fatigue and information overload (particularly when guidance is constantly changing), with potential downstream effects of loss of credibility for the public health agency and disillusionment with future preparedness and response efforts. Finally, one study notes that when guidance does not align with what can feasibly be carried out in practice, it may be ignored.
Sources of Evidence
|Acceptability and Preferences
Email and fax have consistently been reported as preferred channels for communicating public health alerts and guidance, although published technology preferences may be outdated given the rapid pace of technology development and adoption. Recent AARs may be useful sources of more current information on preferred communication channels (e.g., webcasts, social media). Technical audiences generally prefer information from local sources (public health or health care institutional sources) or such national authorities as the Centers for Disease Control and Prevention and medical societies. Engaging technical audiences in communication strategies, providing a direct line of communication, offering opportunities for bidirectional exchange, and ensuring information reciprocity (i.e., returning results generated from information submitted by stakeholders to demonstrate the utility and value of the shared information) may improve the acceptability of and responsiveness to messaging. Tailoring guidance to specific audiences, sending just-in-time guidance, and ensuring that guidance is congruent with practice and allows sufficient flexibility in implementation may help enable the translation of information to appropriate action.
Sources of Evidence
|Feasibility and PHEPR System Considerations
Some communication channels are more feasible than others for public health agencies and technical audiences to implement. The widespread use of traditional channels (e.g., email, fax, phone calls) indicates their feasibility, but further research is needed on the acceptability and feasibility of newer channels (e.g., health information exchange– and electronic health record–based alerting, purpose-built bidirectional surveillance and alert systems). For example, while advances in information technology may lead the public health system to examine and adopt new communication channels, the adoption of these new channels may raise concerns about adding to the burden of message volume and about the availability of needed resources, such as personal or work devices, and technical support.
Sources of Evidence
|Resource and Economic Considerations
Resource requirements for communicating public health alerts and guidance with technical audiences include both technology costs (e.g., phones, radios, computers, servers, software platforms) and human resources. Little research has examined the cost-effectiveness of different communication channels. Many public health agencies and technical audiences already have the technology necessary for traditional communication methods, such as email and conference calls. The initial costs for some purpose-built systems may exceed tens or even hundreds of thousands of dollars, which does not include ongoing maintenance costs. However, such systems often have multiple functions of value to public health agencies, including situational awareness and surveillance. Moreover, the indirect costs of new technologies related to training and technical support need to be added to the direct costs. Designated liaisons and communication networks may help amplify messaging and build or maintain trusted relationships, but the human resource costs of these strategies need to be considered.
Sources of Evidence
Equity issues associated with different channels for communicating public health alerts and guidance with technical audiences are rarely raised in research studies and evaluations (e.g., AARs), and represent an important evaluation gap. One such issue is access to technology, which may be a consideration with respect to rural and underserved populations. Improving relationships with technical audiences that serve disadvantaged populations could lead to more targeted and tailored information sharing during a public health emergency, which in turn could help address equity issues.
Sources of Evidence
The primary value of communication using appropriate channels is often considered to be instrumental, meaning that it is important because using appropriate channels to convey information presumably leads to better information delivery, which can facilitate better decision making. In the language of ethical principles, communication using appropriate channels is important because it promotes the principle of harm reduction/benefit promotion. But problems of overcommunication (such as information overload or alert fatigue) are also possible when appropriate communication channels are used, which can lead to worse or delayed decisions. In addition, communication using appropriate channels also has intrinsic value; that is, setting aside whether decision making is improved by better information delivery, communicating with individuals and communities in ways that are most effective for them is important to achieve transparency, which reflects the principle of respect for persons and communities. As in considering the instrumental value of using more effective channels for communication, one should remember that while communication using ineffective channels is obviously disrespectful, overloading effective communication channels is also disrespectful during crises, when recipients have limited time and bandwidth. In sum, selecting appropriate communication channels is ethically important, and so is careful selection of the information to be delivered over those channels.
Source of Evidence
CONSIDERATIONS FOR IMPLEMENTATION
The following considerations for implementation are drawn from the qualitative evidence synthesis, the case report and AAR evidence synthesis, and evidence from descriptive surveys.
8. Engaging Technical Audiences in the Development of Communication Plans, Protocols, and Channels
Qualitative Evidence Synthesis
Six qualitative studies (Filice et al., 2013; Janssen et al., 2006; Khan et al., 2017; Leung et al., 2008; Lis and Resnick, 2018; Markiewicz et al., 2012) support a finding that engaging technical audiences in the development of communication plans, protocols, and channels appears to help in the dissemination of guidance (moderate confidence in the evidence). Additionally, three qualitative studies (Filice et al., 2013; Khan et al., 2017; Lis and Resnick, 2018) indicate that such engagement efforts also improve the usefulness of guidance, especially through prior attention to how the guidance is translated into actionable knowledge (moderate confidence in the evidence). The act and process of engaging technical audiences prior to public health emergencies facilitates relationship building (also coalition building and the identification of liaisons and points of contact), which may in turn improve emergency response by enhancing understanding of institutional needs, sharing of expertise,
dissemination and implementation of guidance, and situational awareness (Filice et al., 2013; Leung et al., 2008; Markiewicz et al., 2012). Health care stakeholders value strong (and especially preexisting) relationships with local public health and other health care entities, and report those relationships as being beneficial to response scenarios. They similarly report the value of partnerships and coalitions as strategies for effective communication (Filice et al., 2013; Khan et al., 2017). Additionally, when technical audiences are engaged in the development of communication channels, plans, and processes, they can foresee the resulting improvements in interdisciplinary coordination during response (Filice et al., 2013; Khan et al., 2017; Lis and Resnick, 2018). Likewise, there is some indication that engaging technical audiences may facilitate the incorporation of lessons learned from past experience into revised plans and protocols (Filice et al., 2013; Lis and Resnick, 2018).
Case Report and AAR Evidence Synthesis
Although few of the case reports and AARs examined in this review address whether technical audiences were engaged in the development of communication plans, protocols, or channels, several AARs point out the need for better stakeholder engagement in the future (Blue Earth County Public Health, 2014; Capitol Region Council of Governments, 2016; Delaware Division of Public Health, 2010; Metropolitan Medical Response System, 2016; Minnesota Department of Health, 2013b; Ohio Department of Public Health, 2010; Public Health—Seattle & King County, 2009; Wisconsin Hospital Emergency Preparedness Program, 2010). Lessons learned from both exercises and real events suggest that insufficient engagement of partners in planning processes may impede effective communication during responses as a result of planning gaps and unclear communication channels and vetting processes.
Because of staffing gaps resulting from a hiring freeze during Delaware’s 2009 response to the H1N1 outbreak, for example, hospitals and the medical community were not engaged earlier in the planning process (Delaware Division of Public Health, 2010). Following the response, the state’s Division of Public Health recognized the need to further engage partners in the planning and decision-making processes because of confusion around the vaccine ordering process. Blast faxes did not reach physicians, decisions made during meetings sometimes changed based on vaccine allocations, and changes were not well communicated to all parties. The Division subsequently proposed establishing a hotline for medical providers to address this issue, using a stakeholder-engaged process. Wisconsin experienced similar challenges during its 2009 H1N1 response (Wisconsin Hospital Emergency Preparedness Program, 2010). Given how quickly information changed, it was difficult for physicians and employees to keep pace with the information, and some questioned the credibility of the guidelines being provided because of the frequent changes. The public health department recommended engaging the Wisconsin Medical Society to issue a mandate or advisory and establish a point of contact at each hospital to whom emails and communications would be directed. Overall, then, engaging appropriate partners during the planning process in anticipation of a dynamic environment may enhance the credibility and effectiveness of messaging.
Evidence suggests that the public health field is moving toward a more inclusive planning approach. However, the effectiveness of communication channels warrants further study (Boston Public Health Commission, 2013; Cavey et al., 2009; Gamache et al., 2010; Gotham et al., 2007; Massachusetts Emergency Management Agency et al., 2014; McKenna et al., 2003). Some jurisdictions have developed new channels based on direct feedback from stakeholders. For instance, Mississippi developed and tested an infectious disease hotline with surveillance and education capabilities in the aftermath of Hurricane Katrina in
response to requests made by shelter staff for “both a reporting system and infectious disease education” (Cavey et al., 2009). The hotline enabled direct verbal communication between shelter staff and hotline managers, allowing for immediate feedback and education for staff unfamiliar with diseases and reporting processes. A satisfaction survey confirmed immediate positive feedback from the system’s users. This example illustrates a truly stakeholder-driven approach based on an identified need.
Gamache and colleagues (2010) highlight the importance of evaluating end-user acceptance of a new data-sharing mechanism designed to deliver public health alerts to Iowa providers by leveraging an existing electronic clinical messaging system within the context of a health information exchange. Although they do not present findings from their evaluation, they emphasize the value of engaging both clinical and public health stakeholders as a means of building trust and establishing infrastructure for a more complex public health decision support process. Similarly, the HERDS, discussed earlier, serves as the infrastructure for linking and exchanging health preparedness and response information in the state (Gotham et al., 2007). A key lesson learned in developing this system was the need for a bottom-up approach to system requirements that cut across jurisdictions and knowledge domains.
Descriptive Survey Study Evidence
Seidl and colleagues (2010) report on a survey-based real-time quality improvement mechanism that enabled public health authorities to tailor communications and better meet stakeholders’ information needs.
9. Considerations for Selection of Communication Channels
Qualitative Evidence Synthesis
Table B3-5 summarizes considerations that can inform the use of the various channels for communicating public health alerts and guidance with technical audiences that are discussed in the qualitative studies included in this review. Both public health and health care stakeholders describe using judgment to select the optimal communication channel depending on the context at hand, such as the level of uncertainty or urgency (Khan et al., 2017; Staes et al., 2011).
Multiple channels facilitate effective communication by attending differentially to contextual dynamics while avoiding message overload. Contextual dynamics include such priorities as access, accuracy, coordination, dissemination, reciprocity, and timeliness (moderate confidence in the evidence). This finding is supported by seven qualitative studies (Garrett et al., 2011; Janssen et al., 2006; Khan et al., 2017; Leung et al., 2008; Ockers, 2011; Revere et al., 2015; Staes et al., 2011). A decision as to which among multiple communication strategies should be utilized needs to balance message content (emergency versus routine communications), delivery (one- versus two-way), and channel (e.g., text, email) with stakeholder preferences and technical capabilities, all the while mitigating the risk of incurring message overload and overlooking important information (Janssen et al., 2006). Participants in one study described utilizing multipronged approaches, first using one method, followed up with another (Khan et al., 2017). Specifically mentioned was following up on email with a phone call for something urgent.
As discussed earlier, some channels allow for bidirectional communication. The decision to use bidirectional messaging strategies is complex, and public health agencies need to manage concerns about and barriers to such strategies to ensure benefits for all parties
|Face to Face||Direct contact through in-person meetings is synchronous (i.e., allows real-time exchange of information), which allows for degrees of nuance and flexibility related to the uptake and understanding of public health guidance (Khan et al., 2017).
In-person meetings between public health personnel and clinicians are useful, especially when there is perceived anxiety or discomfort about particular guidance (Khan et al., 2017).
|Phone Calls||Direct contact through phone calls and teleconferences is synchronous, which allows for degrees of nuance and flexibility related to the uptake and understanding of public health guidance. Such contact is also helpful for very urgent communication (Khan et al., 2017).
In one example, public health epidemiologists, in their role as liaisons, participated in weekly phone calls with the state public health department (Markiewicz et al., 2012). In another example, the use of two-tiered conference calls (a triage call followed by a coordination call) expedited specific decision making for coordinated patient care decisions (Lis and Resnick, 2018). Such two-tiered calls allow for collaborative, cross-agency decision making.
|Regardless of situational context (emergency versus nonurgent) and message recipients (target audience[s]), email is a favored modality for receiving public health messages (Revere et al., 2015). Email is a push-type channel, generally used in the one-way delivery of alerts and guidance to target audiences.
Despite its limitations (see below), email was cited as the preferred channel for communication of public health guidance to frontline staff by emergency department clinician administrators, who judged it the fastest way of presenting information to clinicians (Khan et al., 2017).
Email dissemination relies on an established listserv, prepared in advance. This may be seen as a limitation, as some key people may not be on the list, and/or the list may require constant maintenance to be kept up to date (Khan et al., 2017; Leung et al., 2008).
|Fax||Fax is often used in tandem with email (Ockers, 2011; Revere et al., 2015). Faxes still may arrive when phone calls cannot connect.|
|Internet/Websites/Social Media||One study showed that providers were as likely to seek information from Google as from the Centers for Disease Control and Prevention (Janssen et al., 2006).
Revere and colleagues (2015) note that some health care providers and community-based organizations are currently using social media as a communication channel to some degree; however, this is less so for public health agencies. Evident in the body of studies included in this review is the lag time of research related to emerging technologies.
|Text Messaging/SMS||Text messaging/SMS provides rapid, in-the-field short messages, probably helpful in emergencies but not for mass communications. When information is lengthy, email appears to be better suited and preferred (Revere et al., 2015).
Both public health agencies and their stakeholders have noted multiple values and uses as well as concerns regarding two-way public health text messaging (Revere et al., 2015). For example, this channel can readily provide “eyes on the ground” reports, short polls, and postdisaster check-in on status and availability, and is an alternative when phone lines are out of service. Conversely, there are concerns with this channel, including receipt of text messages on personal phones, restrictive screen space, ease of ignoring messages, limited cell coverage, security, and the inability to forward messages. Whether mobile phones are sufficiently made available or supported by workplaces appears to be understudied (Revere et al., 2015).
|Electronic Health Records||Enabling guidance to arrive directly at the point of individual care and monitoring, electronic health records have the potential to serve as a channel for communicating public health alerts and guidance with health care audiences. However, many issues related to technology, resources, and compatibility with emergency guidance would need to be considered and managed before effective implementation of this channel could occur (Garrett et al., 2011).|
(Revere et al., 2015). Public health participants in some studies discussed using direct contact and bidirectional communication practices to follow up and facilitate closing the communication loop (Khan et al., 2017; Leung et al., 2008; Markiewicz et al., 2012). In another study, community-based partners expressed wanting the option to reply to a message, whether they would actually do so or not (Revere et al., 2015). Yet, despite the benefits of bidirectional communication channels (e.g., ability to receive confirmation of message receipt and information from stakeholders for purposes of surveillance or surge capacity awareness), concerns have been raised regarding burden; management; technology requirements; privacy, security, and Health Insurance Portability and Accountability Act4 considerations; information utility; and the potential for misunderstanding replies. Other concerns include the funding for new technologies, whether the technology is supported by the workplace, and the need to learn how to utilize a new system (Revere et al., 2015).
It should be noted that there is a time lag between the adoption of new communication technologies in the field and evaluation studies. Although text messaging and some Internet-based technologies have existed as communication channels for at least a couple of decades and are being used to some degree for PHEPR messaging, relatively little research has investigated their use or effectiveness for this purpose. Thus, there is an urgent need for more research in this area.
10. Barriers to and Facilitators of Communicating Alerts and Guidance During a Public Health Emergency
Qualitative Evidence Synthesis
Target audiences have noted the difficulties that they experience when there are multiple sources of guidance—international, national, state, and local public health agencies, as well as institutional sources—and often inconsistencies in guidance information resulting from uncoordinated messaging (Filice et al., 2013; Khan et al., 2017; Leung et al., 2008; Staes et al., 2011). In one study, a clinician commented that if “medical office emails duplicated health department ones, reading both to find discrepancies was too time consuming,” and “if [health care institution] recommendations are different than the CDC’s then this difference should be explicitly noted and explained” (Staes et al., 2011, p. 6). Compounding this barrier to accurate and quality guidance is the rapidly changing nature of information during a response and, in turn, the need for rapid dissemination of updated guidance (Filice et al., 2013; Khan et al., 2017; Leung et al., 2008). Additionally, some study participants have identified inconsistencies in channel use as a challenge; they have come to expect certain sources to use certain channels, and when those expectations are not met, timely access may be hindered. Inconsistencies in channel use or differences in channel preferences across institutions and jurisdictions also are frustrating challenges for health care providers (Khan et al., 2017; Staes et al., 2011). Overall, six qualitative studies (Filice et al., 2013; Janssen et al., 2006; Khan et al., 2017; Leung et al., 2008; Markiewicz et al., 2012; Staes et al., 2011) support a finding that source and channel inconsistencies, excessive message volume, guidance and practice incongruences, and poor coordination within and between agencies work against effective communication during emergencies (moderate confidence in the evidence).
Liaisons and institutional points of contact may facilitate message dissemination, congruence between guidance and practice, and coordination efforts during emergencies (moderate
4 Health Insurance Portability and Accountability Act, HR 3103. Public Law 104-901, 104th Cong. (August 21, 1996).
confidence in the evidence), a finding supported by seven qualitative studies (Filice et al., 2013; Janssen et al., 2006; Khan et al., 2017; Leung et al., 2008; Lis and Resnick, 2018; Markiewicz et al., 2012; Staes et al., 2011). Liaisons and institutional points of contact can increase the speed of dissemination of public health messages (Markiewicz et al., 2012), but beyond amplifying message dissemination, they can have additional influences on the communication process. The institutional knowledge afforded by their roles appears to aid in reaching target audiences within organizations, and by identifying key contacts within public health agencies, they gain an advance understanding of the public health bureaucracy during response (Filice et al., 2013; Khan et al., 2017; Leung et al., 2008; Markiewicz et al., 2012). Additionally, they have facilitated reciprocity by promptly meeting needs for bidirectional information sharing between institutions and public health agencies and adapting as needed to changing dynamics (Markiewicz et al., 2012). Hospitals and other health care entities appear to readily designate these individuals as team leaders for preparedness efforts, protocol development and revision, and integration of institutional learning (Filice et al., 2013; Khan et al., 2017; Markiewicz et al., 2012).
In addition to liaisons and points of contact, coalitions, through developed relationships and networks, have facilitated improvements in interdisciplinary communication and coordination during responses. Khan and colleagues (2017) found that a regional coalition helped with challenges related to the coordination of communication across institutions and jurisdictions and differences in work environments across sectors. Another study found that a coalition helped with developing consistent use of communication channels (e.g., teleconferences) and collaborative decision making (Lis and Resnick, 2018). At the same time, two studies found that coalitions can be very time intensive to maintain. Such challenges may be felt most acutely by smaller agencies (Leung et al., 2008; Lis and Resnick, 2018).
Case Report and AAR Evidence Synthesis
Given the often dynamic nature of public health emergencies, the ability to maintain consistent messaging remains a substantial challenge (ASTHO, n.d.; Capitol Region Council of Governments, 2017; Chicago Department of Public Health et al., 2011; Multnomah County Health Department, 2010; New Hampshire Department of Health and Human Services and New Hampshire Department of Safety, 2009b; Texas Department of State Health Services, 2010; Wisconsin Hospital Emergency Preparedness Program, 2010). Consistent with the findings from the qualitative evidence synthesis, several AARs mention how lack of coordination among partners led to conflicting or inconsistent messaging, resulting in confusion and frustration among technical audiences. During the H1N1 outbreak, guidance sometimes changed several times a day, with multiple guidance documents, forms, and instructions being distributed, some remaining valid while others were superseded (New Hampshire Department of Health and Human Services and New Hampshire Department of Safety, 2009b). Providers with clinics in bordering states often received conflicting messages (Wisconsin Hospital Emergency Preparedness Program, 2010). Regions that established joint information centers struggled to avoid conflicting recommendations from states (Multnomah County Health Department, 2010). More coordinated messaging can help prevent information overload, duplication of effort, and conflicting recommendations (ASTHO, n.d.). Some AARs point to other simple solutions for addressing the challenges of a dynamic information environment, solutions that require minimal resources, such as posting of webinar highlights on relevant websites, sharing of meeting notes after conference calls, and color coding of new information in frequently changing guidance documents (Maine Center for Disease Control and Prevention, 2010; Texas Department of State Health Services, 2018).
Amplification of public health guidance through media (including social media) has been shown to help facilitate the dissemination of technical guidance (CDC, 2013; Maine Center for Disease Control and Prevention, 2010), as has engaging medical societies (County of San Diego, 2018; Delaware Division of Public Health, 2010; Wisconsin Division of Public Health, 2010). Delaware’s joint health department and Medical Society of Delaware communication to Delaware physicians during the state’s H1N1 response, for example, was considered an effective strategy for crisis management as “physicians were more likely to use professional channels for getting technical information during a crisis” (Delaware Division of Public Health, 2010).
A commonly cited barrier to access to and the reach and timeliness of public health guidance during an emergency is the lack of preexisting or up-to-date distribution lists (ASTHO, n.d.; Buffalo Hospital and Wright County Public Health, 2013; Texas Department of State Health Services, 2018). For example, many hospitals’ points of contact participating in a 2011 pediatric full-scale mass casualty incident exercise in Chicago did not report receiving the HAN or State of Illinois Rapid Electronic Notification alert (Chicago Department of Public Health et al., 2011). Additionally, it is often unclear just who is on alert distribution lists, especially when there are multiple channels with various permission rights to each (e.g., WebEOC, agency email, Everbridge) (Boston Public Health Commission, 2013). A lesson learned from Hurricane Harvey was the need to develop and maintain standard distribution lists for health care providers, local health departments, executive leadership, and response managers; to predetermine routine communications to be sent to each recipient based on need; and to develop an automated system (e.g., RedSky) to ensure that all necessary recipients receive the appropriate information (Texas Department of State Health Services, 2018). Maintaining these lists and systems as a routine preparedness activity can save valuable time during responses. It is also important to keep in mind that communication systems can fail as a result of technical issues or power outages (Chicago Department of Public Health et al., 2011; Gursky et al., 2003; McKenna et al., 2003; Montana Department of Public Health and Human Services, 2014; Ramsey County Public Health, 2014). Therefore, redundant individual contact information (e.g., cell phone, email, pager) and redundant systems are critical to ensure that technical audiences receive alerts and guidance in a timely manner.
Another similar barrier mentioned in AARs is the lack of access to such platforms as WebEOC across the local, state, and regional levels (Capitol Region Council of Governments, 2016, 2017). Establishing this linkage would enhance information sharing between levels. Additionally, even when use of WebEOC is limited to a specific locality, ensuring that passwords are updated routinely is important to maintaining accessibility (Boston Public Health Commission, 2013).
Unclear vetting processes, roles and responsibilities, and communication channels can also hinder the effectiveness of communication (Florida Department of Health, 2010). During San Francisco’s 2009 H1N1 response, the lack of protocols led to confusion over how reports should be reviewed, who should review them prior to release, and the appropriate target audiences (San Francisco Department of Public Health, 2009). Similarly, during the potential Ebola threat in Connecticut, Regional Emergency Support Function (ESF) #8’s role in the development and vetting process was unclear (Metropolitan Medical Response System, 2016). In the absence of guidance from the health department, ESF #8 developed recommendations based on the information available, which concerned the health department because it had not vetted the guidance before it was disseminated. Vetting processes therefore need to be formally documented and shared to minimize confusion over roles (Boston Public Health Commission, 2013). Furthermore, overly complex processes can hinder the timeliness of alerts and guidance. Findings suggest that simplified review protocols and eas-
ily customizable alerting frameworks are essential for providing timely decision support to technical audiences (Lurio et al., 2010).
Descriptive Survey Study Evidence
Survey findings align with several of the barriers identified in the qualitative and case report and AAR syntheses. Santibanez and colleagues (2016), for example, surveyed infectious disease physicians and found that problems related to not knowing whom to contact at the public health department and the connection to automated answering services impeded communication. Physicians wanted phone numbers for designated points of contact. They also suggested that building trust relationships in advance (e.g., through online joint rounds or in-person updates at periodic meetings) would help overcome these barriers. In addition, Staes and colleagues (2011) found that changing public health guidance was associated with reduced awareness of current guidance.
Surveys also were sources of strategies (often suggested by representatives of technical audiences) for facilitating communication with technical audiences. Note that the following compilation of these strategies, although potentially of use to public health stakeholders, should not be viewed as an exhaustive list and that additional evidence is needed before these strategies can be recommended as evidence-based practices:
- Given the need for increased coordination of messaging, public health authorities should consider routing notifications regarding alerts and guidance through preferred institutional communication channels (Staes et al., 2011).
- As messages received from public health authorities are often communicated to the full clinic staff in person, public health agencies should consider disseminating talking points as an attachment to their notifications (Ockers, 2011).
- When communicating text message alerts, inclusion of the following information should be considered: topic, recommendations, geographic location, signs and symptoms, population affected, and a hyperlink to additional information (Revere et al., 2014).
- When communicating changing information and guidance, including executive summaries at the beginning of informational emails can help quickly highlight new, important information for technical audiences (Seidl et al., 2010).
- Having a public information officer and a pandemic influenza plan was associated with greater odds of health care stakeholders receiving H1N1 information from the local health department, suggesting that these strategies may facilitate the sharing of information with technical audiences (Howard et al., 2012).
REFERENCES FOR ARTICLES INCLUDED IN THE MIXED-METHOD REVIEW
Quantitative Comparative Studies
Baseman, J. G., D. Revere, I. Painter, M. Toyoji, H. Thiede, and J. Duchin. 2013. Public health communications and alert fatigue. BMC Health Services Research 13:295.
Baseman, J., D. Revere, I. Painter, M. Oberle, J. Duchin, H. Thiede, R. Nett, D. MacEachern, and A. Stergachis. 2016. A randomized controlled trial of the effectiveness of traditional and mobile public health communications with health care providers. Disaster Medicine and Public Health Preparedness 10(1):98–107.
Revere, D., I. Painter, M. Oberle, and J. Baseman. 2014. Health-care provider preferences for time-sensitive communications from public health agencies. Public Health Reports 129(6 Suppl 4):67–76.
van Woerden, H. C., M. R. Evans, B. W. Mason, and L. Nehaul. 2007. Using facsimile cascade to assist case searching during a Q fever outbreak. Epidemiology & Infection 135(5):798–801.
References marked with an asterisk (*) are surveys containing a qualitative analysis of free-text responses.
Filice, C. E., F. E. Vaca, L. Curry, S. Platis, N. Lurie, S. Bogucki, and M. N. Shah. 2013. Pandemic planning and response in academic pediatric emergency departments during the 2009 H1N1 influenza pandemic. Academic Emergency Medicine 20(1):54–62. http://doi.org/10.1111/acem.12061.
Garrett, N. Y., N. Mishra, B. Nichols, C. J. Staes, C. Akin, and C. Safran. 2011. Characterization of public health alerts and their suitability for alerting in electronic health record systems. Journal of Public Health Management & Practice 17(1):77–83.
Janssen, A. P., R. R. Tardif, S. R. Landry, and J. E. Warner. 2006. “Why tell me now?” The public and healthcare providers weigh in on pandemic influenza messages. Journal of Public Health Management & Practice 12(4):388–394.
Khan, Y., S. Sanford, D. Sider, K. Moore, G. Garber, E. de Villa, and B. Schwartz. 2017. Effective communication of public health guidance to emergency department clinicians in the setting of emerging incidents: A qualitative study and framework. BMC Health Services Research 17(1).
Leung, C. S., M. M. Ho, A. Kiss, A. V. Gundlapalli, and S. W. Hwang. 2008. Homelessness and the response to emerging infectious disease outbreaks: Lessons from SARS. Journal of Urban Health 85(3):402–410.
Lis, R., and A. T. Resnick. 2018. Coordinated communications and decision making to support a regional severe infectious disease response. Health Security 16(3):158–164.
Markiewicz, M., C. A. Bevc, J. Hegle, J. A. Horney, M. Davies, and P. D. MacDonald. 2012. Linking public health agencies and hospitals for improved emergency preparedness: North Carolina’s public health epidemiologist program. BMC Public Health 12:141.
*Ockers, S. 2011. Communication preferences of health care providers during an emergency: Data from four state surveys conducted before and after the H1N1 influenza mass vaccination campaigns. (Master’s thesis—joint program MPH/PA). Atlanta, GA: Rollins School of Public Health, Emory University.
Revere, D., R. Calhoun, J. Baseman, and M. Oberle. 2015. Exploring bi-directional and SMS messaging for communications between public health agencies and their stakeholders: A qualitative study. BMC Public Health 15:621.
*Staes, C. J., A. Wuthrich, P. Gesteland, M. A. Allison, M. Leecaster, J. H. Shakib, M. E. Carter, B. M. Mallin, S. Mottice, R. Rolfs, A. T. Pavia, B. Wallace, A. V. Gundlapalli, M. Samore, and C. L. Byington. 2011. Public health communication with frontline clinicians during the first wave of the 2009 influenza pandemic. Journal of Public Health Management & Practice 17(1):36–44.
Descriptive Survey Studies
Howard, A. F., H. M. Bush, R. M. Shapiro, and A. Dearinger. 2012. Characteristics of Kentucky local health departments that influence public health communication during times of crisis: Information dissemination associated with H1N1 novel influenza. Journal of Public Health Management & Practice 18(2):169–174.
Ockers, S. 2011. Communication preferences of health care providers during an emergency: Data from four state surveys conducted before and after the H1N1 influenza mass vaccination campaigns. (Master’s thesis—joint program MPH/PA). Atlanta, GA: Rollins School of Public Health, Emory University.
Quinn, C., E. Poirot, A. Sanders Kim, A. L. Viswanath, S. N. Patel, D. M. Abramson, and R. Piltch-Loeb. 2018. Variations in healthcare provider use of public health and other information sources by provider type and practice setting during New York City’s response to the emerging threat of Zika virus disease, 2016. Health Security 16(4):252–261.
Revere, D., M. R. Schwartz, and J. Baseman. 2014. How 2 txt: An exploration of crafting public health messages in SMS. BMC Research Notes 7:514.
Santibanez, S., P. M. Polgreen, S. E. Beekmann, C. Cairns, G. A. Filice, M. Layton, and J. M. Hughes. 2016. Communication between infectious disease physicians and U.S. state and local public health agencies: Strengths, challenges, and opportunities. Public Health Reports 131(5):666–670.
Seidl, I. A., A. J. Johnson, P. Mantel, and P. Aitken. 2010. A strategy for real time improvement (RTI) in communication during the H1N1 emergency response. Australian Health Review 34(4):493–498.
Staes, C. J., A. Wuthrich, P. Gesteland, M. A. Allison, M. Leecaster, J. H. Shakib, M. E. Carter, B. M. Mallin, S. Mottice, R. Rolfs, A. T. Pavia, B. Wallace, A. V. Gundlapalli, M. Samore, and C. L. Byington. 2011. Public health communication with frontline clinicians during the first wave of the 2009 influenza pandemic. Journal of Public Health Management & Practice 17(1):36–44.
Walsh, D., M. Barry, and K. Overly. 2010. Illinois Department of Public Health 2009 H1N1 influenza A: Pandemic communications evaluation survey. Argonne, IL: Argonne National Laboratory. https://digital.library.unt.edu/ark:/67531/metadc1013467/m2/1/high_res_d/990518.pdf (accessed April 9, 2020).
ASTHO (Association of State and Territorial Health Officals). n.d. Addressing communication challenges during an infectious disease emergency response: State experiences from the H1N1 pandemic. https://www.astho.org/Programs/Infectious-Disease/Addressing-Communication-Challenges-During-an-Infectious-DiseaseEmergency-Response (accessed April 9, 2020).
Cavey, A. M., J. M. Spector, D. Ehrhardt, T. Kittle, M. McNeill, P. G. Greenough, and T. D. Kirsch. 2009. Mississippi’s infectious disease hotline: A surveillance and education model for future disasters. Prehospital & Disaster Medicine 24(1):11–17.
CDC (Centers for Disease Control and Prevention). 2013. Notes from the field: Use of electronic messaging and the news media to increase case finding during a cyclospora outbreak—Iowa, July 2013. Morbidity & Mortality Weekly Report 62(30):613–614.
Daniel, J. B., D. Heisey-Grove, P. Gadam, W. Yih, K. Mandl, A. Demaria, Jr., and R. Platt. 2005. Connecting health departments and providers: Syndromic surveillance’s last mile. Morbidity and Mortality Weekly Report 54:147–150.
Gamache, R., K. C. Stevens, R. Merriwether, B. E. Dixon, and S. Grannis. 2010. Development and assessment of a public health alert delivered through a community health information exchange. Online Journal of Public Health Informatics 2(2).
Gotham, I. J., D. L. Sottolano, M. E. Hennessy, J. P. Napoli, G. Dobkins, L. H. Le, R. L. Burhans, and B. I. Fage. 2007. An integrated information system for all-hazards health preparedness and response: New York State health emergency response data system. Journal of Public Health Management and Practice 13(5).
Gursky, E., T. V. Inglesby, and T. O’Toole. 2003. Anthrax 2001: Observations on the medical and public health response. Biosecurity and Bioterrorism 1(2):97–110.
Lurio, J., F. P. Morrison, M. Pichardo, R. Berg, M. D. Buck, W. Wu, K. Kitson, F. Mostashari, and N. Calman. 2010. Using electronic health record alerts to provide public health situational awareness to clinicians. Journal of the American Medical Informatics Association 17(2):217–219.
Mathur, A., and K. Beckermann. 2010. Moving up, moving down: Communications flow to and from mass immunization clinics. Canadian Journal of Infectious Diseases and Medical Microbiology 21(4):209.
McKenna, V. B., J. E. Gunn, J. Auerbach, K. H. Brinsfield, K. S. Dyer, and M. A. Barry. 2003. Local collaborations: Development and implementation of Boston’s bioterrorism surveillance system. Journal of Public Health Management and Practice 9(5):384–393.
Nagykaldi, Z., J. W. Mold, K. K. Bradley, and J. E. Bos. 2006. Bridging the gap between public and private healthcare: Influenza-like illness surveillance in a practice-based research network. Journal of Public Health Management and Practice 12(4):356–364.
Wynn, A., and K. M. Moore. 2012. Integration of primary health care and public health during a public health emergency. American Journal of Public Health 102(11):e9–e12.
Blue Earth County Public Health. 2014. Information sharing or TB contact investigation after action report/improvement plan. Mankato, MN. June 20, 2014.
Boston Public Health Commission. 2013. 2013 Boston Marathon emergency support function 8 (ESF-8) public health and medical planning, response, and recovery operations (April 15–April 26): Final after-action report/improvement plan. Boston, MA. December 20, 2013. https://delvalle.bphc.org/mod/wiki/view.php?pageid=63 (accessed April 9, 2020).
Buffalo Hospital and Wright County Public Health. 2013. Buffalo Hospital closed pod after-action report/improvement plan. Buffalo, NY. November 21, 2013.
Capitol Region Council of Governments. 2016. Ebola virus disease functional exercise after action report. Hartford, CT. May 18, 2016.
Capitol Region Council of Governments. 2017. Ebola virus disease full scale exercise after action report. http://woodcountyhealth.org/ep/documents/2017%20WCHD%20Functional_Full-Scale%20Exercise%20AAR_IP.pdf (accessed April 9, 2020).
Chicago Department of Public Health, Illinois Department of Public Health, and Metropolitan Chicago Healthcare Council. 2011. Illinois hospitals’ pediatric full-scale exercise after action report. Chicago, IL. May 21, 2011.
County of San Diego. 2018. Hepatitis A outbreak after action report. May 2018. San Diego, California. https://www.sandiegocounty.gov/content/dam/sdc/cosd/SanDiegoHepatitisAOutbreak-2017-18-AfterActionReport.pdf (accessed January 23, 2020).
Delaware Division of Public Health. 2010. Novel H1N1 influenza Delaware response April 2009 to March 2010: After action report/improvement plan. Dover, DE. June 1, 2010. https://dhss.delaware.gov/DHSS/DPH/php/files/h1n1aar.pdf (accessed January 23, 2020).
Florida Department of Health. 2010. 2010 Deepwater Horizon oil spill response: ESF 8 after action report and improvement plan. Tallahassee, FL. April 30, 2011. http://www.floridahealth.gov/programs-and-services/emergencypreparedness-and-response/training-exercise/_documents/deepwater-aar.pdf (accessed January 23, 2020).
Maine Center for Disease Control and Prevention. (2010). Maine CDC 2009 H1N1 Influenza Pandemic After Action Summary. https://www.maine.gov/dhhs/mecdc/infectious-disease/epi/influenza/maineflu/documents/piop/AAR%20MECDC%202009%20H1N1%20Pandemic%202.11.pdf (accessed May 23, 2020).
Massachusetts Emergency Management Agency, Massachusetts Department of Public Health, City of Boston, City of Cambridge, Town of Watertown, Massachusetts Bay Transportation Authority Transit Police Department, Massachusetts National Guard, and Massachusetts State Police. 2014. After action report for the response to the 2013 Boston Marathon bombings. https://www.policefoundation.org/wp-content/uploads/2015/05/afteraction-report-for-the-response-to-the-2013-boston-marathon-bombings_0.pdf (accessed January 23, 2020).
Metropolitan Medical Response System. 2016. CT Region 3 ESF-8 Ebola preparedness and response: After action report. Falls Church, VA. October 29, 2015.
Minnesota Department of Health. 2013a. Operation Loon Call 2013: After-action report/improvement plan. St. Paul, MN. June 11, 2013.
Minnesota Department of Health. 2013b. White powder incident November 2013 after action report/improvement plan. St. Paul, MN. February 2014.
Minnesota Department of Health. 2014. DOC FE flash floods 2014. After-action report/improvement plan 2014. St. Paul, MN. May 29, 2014.
Montana Department of Public Health and Human Services. 2014. Big Sky Push II: Medical supplies management and distribution full scale exercise March 21—April 11, 2014. Great Falls, MT. May 22, 2014.
Multnomah County Health Department. 2010. H1N1 Fall 2009–Multco Aug 4–5, 2009–December 8, 2009: After action report/improvement plan. Portland, OR. May 4, 2010.
New Hampshire Department of Health and Human Services and New Hampshire Department of Safety. 2009a. Cities ready initiative operation rapid RX full-scale exercise: After action report. Concord, NH. October 16, 17, 24, 2009.
New Hampshire Department of Health and Human Services and New Hampshire Department of Safety. 2009b. 2009 Spring H1N1 response: After action report/improvement plan. Concord, NH. September 22, 2009.
New Hampshire Department of Health and Human Services and New Hampshire Department of Safety. 2010. New Hampshire July 1, 2009–March 30, 2010 H1N1 response: After action report/improvement plan. Concord, NH.
Ohio Department of Health. 2010. Fall 2009 H1N1 response. ICS operations conducted through September 21, 2009–February 4, 2010. After action report/improvement plan. Columbus, OH. June 29, 2010.
Public Health—Seattle & King County and King County Healthcare Coalition. 2009. H1N1 influenza (swine flu) 2009. King County ESF-8 after action report. April 26, 2009–May 15, 2009.
Ramsey County Public Health. 2014. Operation: Communication woes: After-action report/improvement plan. St. Paul, MN. May 5, 2014.
San Francisco Department of Public Health. 2009. Fall/Winter 2009–2010 H1N1 swine flu response: San Francisco, California. September 28, 2009–March 9, 2010: After action report/improvement plan. August 20, 2010. https://www.sfcdcp.org/wp-content/uploads/2018/01/H1N1-AAR-Executive-Summary.Fall-Winter-2009-2010id639.pdf (accessed January 23, 2020).
Texas Department of State Health Services. 2010. Texas Department of State Health Services response to the novel H1N1 pandemic influenza (2009 and 2010): After action report. The Litaker Group, LLC. Austin, TX. August 30, 2010.
Texas Department of State Health Services. 2018. Hurricane Harvey response: After-action report. Austin, TX. May 30, 2018.
Tri-County Health Department. 2017. Public health emergency dispensing exercise (PHEDEX) after action report and improvement plan. Greenwood Village, CO. June 15–17, 2017.
Wisconsin Division of Public Health. 2010. 2009 H1N1 influenza response after action report and improvement plan. Madison, WI. July 2010.
Wisconsin Hospital Emergency Preparedness Program. 2010. After action report (AAR) for H1N1 influenza. Madison, WI. April 24, 2009–Spring 2010.
Ethics and Policy Text
IOM (Institute of Medicine). 2009. Guidance for establishing crisis standards of care for use in disaster situations: A letter report. Washington, DC: The National Academies Press.
Jennings, B., and J. Arras. 2008. Ethical guidance for public health emergency preparedness and response: Highlighting ethics and values in a vital public health service. https://www.cdc.gov/od/science/integrity/phethics/docs/white_paper_final_for_website_2012_4_6_12_final_for_web_508_compliant.pdf (accessed February 23, 2020).
Jennings, B., J. D. Arras, D. H. Barrett, and B. A. Ellis. 2016. Emergency ethics: Public health preparedness and response. New York: Oxford University Press.
Mastroianni, A. C., J. P. Kahn, and N. E. Kass, eds. 2019. The Oxford handbook of public health ethics. New York: Oxford University Press. https://www.oxfordhandbooks.com/view/10.1093/oxfordhb/9780190245191.001.0001/oxfordhb-9780190245191 (accessed June 3, 2020).