2

Analysis of the Problem

Energy has always been an important consideration for military operations, both in peacetime and during conflict. It is becoming an increasingly important aspect owing to the rapidly exploding and exponentially growing use of data to drive military operations, particularly at the edge of conflict. Part of the challenge at the tactical edge is understanding and addressing the needs of the various users: those in semi-fixed deployments, those in mobile units conducting conventional activities, those who operate in highly variable conditions that require both movement and fast operational changes, and those who are in low-profile special operations activities. Besides having different power needs, each has different logistical support structures. Each, of course, must be supported, but how they are supported must be considered as part of the overall mission planning efforts.

THE ENERGY ECOSYSTEM

It is helpful to have a common structure when considering the challenges associated with the consumption of energy by the U.S. Air Force (USAF) specifically, and the Department of Defense (DoD) writ large. Without a common structure, individuals and groups are likely to view the problem from specific vantage points and miss the larger holistic issues or fail to take into consideration the effect on the entire ecosystem. The illustration in Figure 2.1 provides structure to this discussion.

At the center of the discussion and central to the diagram in Figure 2.1 is the thesis that the explosion of data and data analytics is going to both drive an increase in demand for energy but also, perhaps more importantly, reshape how that

energy is stored, delivered, and dispersed across the globe in support of military operations. When considering the problem, it is imperative to first break down data and data analytics into pieces so that we can better understand how this field will impact energy consumption.

Finding 1: The explosion of data creation, sharing, and use, including data analytics to support near-real-time decision making, will drive an increase in demand for energy.

Finding 2: The increase in demand for energy will influence the energy supply chain and accompanying logistics.

Compute and Store

Data analytics, at its core, has four main components: compute, store, math, and software.¹ Understanding these four components, both how they come together to

___________________

¹ See H.R. Lieberman and L. Robinson, 1967, “The Electronic Digital Computer: How It Started, How It Works and What It Does,” New York Times, January 9, Section National Economic Review, page 136.

form the data capabilities used by the military (and civilian society in general) and how energy is consumed, leads to a better understanding of how solving the energy challenge is possible.

Data analytics is the use of mathematics to solve problems.² Taken alone, this is not new: in reality, it has impacted military operations for millennia. Since the birth of modern computing based in the application of mathematics to cryptography and artillery tables in World War II, which birthed the electronic numerical integrator computer (ENIAC), the importance of data analytics to military operations has grown exponentially. Because these early computers were enormously large and complex, they were permanently housed in large facilities with extensive fixed power systems. Over the course of time, these big computers from the 1940s evolved to the handheld calculators of the 1970s, hardware-driven machines that used energy to execute fixed algorithms. While these machines could compute, they were not very good at storage, which was next to impossible on a handheld calculator, and phenomenally expensive on mainframes.³

The advances of the research community, fueled by the requirements of the space race,⁴ the miniaturization of electronic components, and the application of computation to an increasingly broad swath of human activities, resulted in several leaps forward in computational capabilities. The “personal computer” untethered the ability to execute programs from bulky mainframes, ushering in an era where individuals could store and analyze information at their desks. This began a new arms race⁵ in computing that led to the Desert Storm 1990-era command centers that moved the compute function farther away from fixed national sites and into regional and deployed operational command centers.⁶ Even then, the movement of information between the edge computers was slow and expensive. Thus, they needed to be collocated, which simplified the associated energy challenges even as it gave rise to large operational command centers.⁷

The technological disruption resulting from the commercialization of the Internet, the growth of cellular telephony, and the emergence of mobile computing

___________________

² See Masters In Data Science, 2020, “What Is Data Analytics,” 2U, July, https://www.mastersindatascience.org/learning/what-is-data-analytics/.

³ See K.A. Zimmerman, 2017, “History of Computers: A Brief Timeline,” LiveScience, September 7, https://www.livescience.com/20718-computer-history.html.

⁴ See B. Hayes, 2019, “Moonshot Computing,” American Scientist, May-June, https://www.americanscientist.org/article/moonshot-computing.

⁵ See M. Van Creveld, 1989, Technology and War: From 2000 B.C. to the Present, Macmillan, New York, NY.

⁶ See A. Campden (ed.), 1992, The First Information War: The Story of Communications, Computers, and Intelligence Systems in the Persian Gulf War, AFCEA International, Fairfax, VA.

⁷ See D. Alberts and R. Hayes, 2003, “Power to the Edge: Command and Control in the Information Age,” Command and Control Research Program (CCRP), Washington, DC.

on very small handheld devices solved the next link in the data analytics distribution puzzle by disconnecting the compute and store functions from the device. The iPhone,⁸ for example, could send queries far away and get answers back almost immediately. In many ways, this mirrored the earlier mainframe architectures. A critical distinction was distributing that information to individuals, virtually anywhere in the world.

Cloud computing, in essence, replicated the personal computer (PC) revolution of the 1980s but at an enormous scale.⁹ This enabled individuals using mobile devices to have as much compute and store capability at their fingertips as any nation-state had in the 1970s. These “cloud” sites require very large data centers to support both storage and computation. Because of their size and the emergence of “big data” computational approaches, the sites are energy consumption monsters, located where access to energy was assured. A fundamental feature of these sites is that they are both large and fixed, and so during a large war could be easily targeted and destroyed—it is very difficult to hide a large data center.¹⁰

Finding 3: Cloud computing enables highly distributed and mobile information technology (IT) that is very powerful.

Finding 4: Back-end data centers require an enormous amount of energy, are fixed sites, and are potentially a target during wartime, both physically and virtually.

An additional development that both complicates and enables information technology applications in distributed environments is what is known as the Internet of Things (IoT).¹¹ The IoT is a way of referring to very small and specialized devices, such as thermostats, fitness trackers, and energy meters, that are augmented with intelligence and network enabled.¹² As these “smart” devices are coordinated over networks through data centers, creating the IoT, a systemic synergy is created,

___________________

⁸ See L. Goode and D. Pierce, 2018, “The WIRED Guide to the iPhone,” WIRED, December 7, https://www.wired.com/story/guide-iphone/.

⁹ See R. Polding, 2018, “Cloud Computing: The Invisible Revolution,” Robert Polding, PhD, blog, August 10, https://medium.com/@rpolding/cloud-computing-the-invisible-revolution-52e82d43c465.

¹⁰ See World’s Top Data Centers, 2021, “America’s Size Rankings,” http://worldstopdatacenters.com/americas-size-rankings/.

¹¹ See National Research Council (NRC), 2001, Embedded, Everywhere: A Research Agenda for Networked Systems of Embedded Computers, The National Academies Press, Washington, DC, https://doi.org/10.17226/10193.

¹² See National Academies of Sciences, Engineering, and Medicine (NASEM), 2018, A Primer to Prepare for the Connected Airport and the Internet of Things, The National Academies Press, Washington, DC, https://doi.org/10.17226/25299.

over and beyond what any individual component could contribute. Taking advantage of deployed smart devices, back-end data centers, and nearly ubiquitous communications pipelines, the battlefield is being drenched in devices that are sensing and creating data.¹³ A tipping point is rapidly being reached, however. At first, and most commonly still, this data is transmitted to off-premise computational centers for processing and then sent back to the tactical edge, oftentimes in seconds.¹⁴ As the amount of data produced increases, the bandwidth needs for communications will concomitantly increase. The back-end processing solution is working right now because the ability to transmit this information is still adequate and, more importantly, has not been contested. In a future war, that cannot be assumed.

Finding 5: The integration of smart devices into the tactical edge is resulting in a very large increase in data generated at the tactical edge. As the amount of data being produced increases, the bandwidth required to transmit it back to off-premise computational centers will also increase.

Finding 6: Reliance on communications between tactical-edge centers and off-premise data centers is a vulnerability that could be exploited by an adversary.

This trend is illustrated by Special Operations Command (SOCOM), which is now generating more data inside the tactical bubble than outside the bubble.¹⁵ This constitutes a reversal of the status quo, in which data was fed to the tactical edge from both back-end sensors and computational devices. Combining the data flow reversals with the expected contested transmission of data will force the movement of required compute and store functions all the way to the tactical edge. Not only will this change warfare but it will also profoundly affect how energy is consumed on the battlefield.

Finding 7: The reversal of data flows to and from the tactical edge will force the movement of computational devices to the tactical edge, which will profoundly affect the energy supply chain requirements.

For the past several decades, since Desert Storm, energy consumption has been structured around large, fixed command centers directing far-flung operations. The

___________________

¹³ See D. Alberts et al., 1999, Network Centric Warfare: Developing and Leveraging Information Superiority, CCRP, Washington, DC; D. Alberts et al., 2001, Understanding Information Age Warfare, CCRP, Washington, DC; D. Gombert et al., 2006, Battle-Wise: Seeking Time-Information Superiority in Networked Warfare, NDU Press, Washington, DC.

¹⁴ See Ginny et al., 2021, Smartphone processor architecture, operations, and functions: Current state-of-the-art and future outlook: Energy performance trade-off, Journal of Supercomputing 77:1377–1454, https://doi.org/10.1007/s11227-020-03312-z.

¹⁵ See conversation with Col. Matt Benigni in April 2020, Appendix C.

next several decades are likely to see this function of compute and store distributed across the battlefield, both for operational efficiency and also for the need to survive. This will change how much energy is needed, where the energy is needed, how energy is moved, and how it is stored. In many ways, the past several decades have lulled DoD into a false sense of energy security. Data centers and operational command centers could cheaply and easily be supplied with energy. Breaking this apart will be inherently inefficient and messy. An interesting parallel in history is illustrated by the adoption of the internal combustion engine to the activities of conflict.¹⁶ Then, nations had to acquire the ability to generate appropriate forms of power and then distribute that power to machines around the world.¹⁷ In the current situation, it is not oil that has to be moved but electricity, and the physics of moving and storing electricity is vastly more complicated than liquid fuel.

Finding 8: Movement of electricity to power data-driven operations at the tactical edge is a critical need that is more complicated than moving petroleum products.

Energy Challenges in the Ecosystem

The distribution of compute and store functions across the battlefield, embedded in hundreds of millions of devices, will impact energy consumption in several different ways. The problem includes four primary components: (1) transmission (energy and data); (2) environmental control (hot and cold); (3) protection (how to mitigate the risk to force and mission); and (4) time horizon (now and 10 years from now). This structure is not meant to be the answer to how to fix the challenges that the Air Force will experience concerning the use and distribution of energy given the data-driven operational warfare envisioned in the future. Instead, it is meant to provide a structure through which conversations can begin, so that decisions can be made to solve the problems of the 2030s, during this decade of the 2020s.

Transmission

Transmission can be thought about in two ways: distribution of energy across many devices and distribution of data across many devices. Together, the transmission function of energy to power the devices and energy to move the data

___________________

¹⁶ See G.W. Phillips, 1927, The internal combustion engine and its influence on war material, Royal United Services Institution 72(488):789–807.

¹⁷ The adaptation of internal combustion machines to the conduct of conflict enabled speed over distance in movement but also required the creation of fuel logistics—the movement of fuel in trucks and ships along with the ships, submarines, tanks, and airplanes.

are separate issues that need to be solved in order for both of the components to operate effectively.

Data transmission (moving data from servers to clients, between clients, between servers, etc.) does not normally create a significant increase in energy demand as compared to the computer systems themselves. However, the communications infrastructure’s energy needs cannot be overlooked. Data transmission requires switches, routers, and radios (for cellular, satellite, or other wireless modes), all of which require electric power. Planning should also include energy needs for security devices such as firewalls, intrusion detection systems, and other appliances.

Much like electricity distribution, there will be an internal communications network inside a military installation or forward-deployed base that will require power. Some commercial wireless providers will build their towers and infrastructure inside an installation’s footprint in order to increase coverage. The need for more antennas and higher-powered transmitters, and thus the need for more electric power, will increase as 5G¹⁸ (and eventually 6G, 7G, and xG) technologies are adopted.

The distribution of energy across many devices is the challenge of getting enough and appropriately configured energy to each device in a timely enough fashion that usage is not affected out of established tolerances. When many (where “many” can mean millions of) devices, which are increasingly small and battery powered, are spread across the battlefield, energy must be distributed to each device in order for all of them to be operational.¹⁹ The farther the device is from the supply site, the more challenging it can be to distribute the energy supplies. The problem is exacerbated if there are many different energy configuration standards (such as different battery types required), in that the supply of energy can become logistics-unfriendly.²⁰

Finding 9: Highly distributed small devices require energy to operate. The form of that energy can pose challenges unless the supply of energy is logistics-friendly.

___________________

¹⁸ Owing to the still evolving standards associated with 5G, there is still significant variation in energy need estimations. Standards are not expected to be finalized until 2022. See Department of Homeland Security (DHS), 2020, “Feature Article: 5G Introduces New Benefits, Cybersecurity Risks,” https://www.dhs.gov/science-and-technology/news/2020/10/15/feature-article-5g-introduces-newbenefits-cybersecurity-risks.

¹⁹ See K. Matthews, 2019, “Energy Consumption and IoT Technologies: What to Know,” Information Age, October 29, https://www.information-age.com/energy-consumption-and-iot-technologieswhat-to-know-123485884/.

²⁰ Logistics-friendly power is that which requires the least amount of variation in both form and content, so that the supply chain is not burdened with the problems of accounting and distribution of customized solutions. See NRC, 2014, Force Multiplying Technologies for Logistics Support to Military Operations, The National Academies Press, Washington, DC, https://doi.org/10.17226/18832.

The other transmission problem is the movement of data between the devices. Moving data consumes energy. The default configurations of small devices in fixed base and civilian organizations provide that energy through communications technologies, including fiber, copper, and cellular technologies. This renders the energy usage invisible and also provides users with an assumption of energy availability in communications. When these devices are widespread in deployed or contested environments, neither of these characteristics are potentially true: there may be a differential energy need between the communications nodes and the computational devices that may need to be managed. Small devices may use tiny amounts of energy to process limited amounts of data but not have enough energy to power transmission of the data beyond highly localized nodes. In other words, the need to power the data transmission infrastructure is nontrivial and should not be assumed away as part of the device power challenge.

Finding 10: Small devices spread across a deployed force may need to have separately powered transmission capabilities.

Environmental Control

Environmental control includes both temperature and humidity, both of which can adversely affect both humans and technologies in an operational construct. At extremes of temperatures, human bodies become lethargic and then fail. Similarly, equipment is designed with thermal envelopes, the exceeding of which can cause equipment to fail. Humidity poses its own challenges, in that excess humidity can cause the human body temperature management system to malfunction and cause electronics to short circuit or spark.²¹

Temperature is an interesting challenge. Typically, the analyses regarding temperature are that of excess heat, but conflicts can take place in many different weather conditions, including very hot and very cold and in an area with many microclimates.²² Compute and storage devices may need to be cooled or warmed just to operate. Excessive heat and excessive cold can drain energy from storage devices, such as batteries, and cause power generation and transmission systems, such as diesel generators, to malfunction.²³

___________________

²¹ See Circuitnet, 2020, “Average Temperature/Humidity for an Electronic Assembly Facility?” July 29, https://www.circuitnet.com/experts/40104.html.

²² Microclimates are those that exist as highly localized phenomena that are markedly different from the area climate. See L. Perry, 2021, “Microclimates: What Do They Mean to You?” University of Vermont Extension Department of Plant and Soil Science, https://pss.uvm.edu/ppp/articles/microclimates.html.

²³ See University of Michigan, 2019, “How Extreme Cold Weather Affects Vehicles and Batteries,” https://news.umich.edu/how-extreme-cold-affects-vehicles-and-batteries/.

Finding 11: Storing energy in environmentally appropriate ways and keeping the devices operational through managing the environmental characteristics is a necessary and important analytical element.

Protection

Protection exists in two forms: minimizing risk to force and minimizing risk to mission. This is a fundamental part of the military’s mission but one that is not a primary concern for most commercial sector entities. With some exceptions (such as cyberattacks), the commercial sector in the United States generally does not expect to be actively targeted for wholesale destruction by an adversary. Defense of assets and systems is a fundamental function of military operations and DoD must assume that data transmission paths and centers will be attacked, energy supply lines will be targeted, and detectable electromagnetic emissions will be exploited. All of these put both the force and the mission in jeopardy. Whatever solutions are found to the distribution of energy across the battlefield must take into account measures to reduce the risk to the force and to the mission.

Commercial power providers optimize their generation, transmission, and distribution assets to lower their customer’s costs, to minimize environmental impact, and to maintain a required minimum level of reliability. Threats to a commercial power provider’s infrastructure are normally centered on weather or nature and not on deliberate, targeted attacks. Military planners will need to consider human attacks against their power infrastructure in addition to threats from nature.

Industry normally reacts to consumer demand and typically will not invest in technologies or solutions that will not generate profit or that have a limited market. Currently, however, there is increasing interest for developing new power generation, transmission, and distribution capabilities with a focus on renewable power sources and more efficient power consumption. There will be parts of the solution space where USAF can ride the commercial investment, such as chip manufacturing and storage; however, mobile, fixed, austere energy storage and production may be something that USAF has to invest in on its own to develop because the commercial marketplace is not that big.

Finding 12: Adversaries will target both data capabilities and communications, including the energy used to power such capabilities.

Time Horizon

The Air Force operates with a force that has an average of 10 years of experience and training.²⁴ The information processing technologies powering the Air Force are evolving at an increasingly fast pace, averaging a generational leap forward every 18 months.²⁵ The energy systems powering the information technologies are also evolving quickly, leveraging advances in material science, nanotechnology, and integrated manufacturing.²⁶ These variations in time horizons for the personnel, the information technology (IT), and the power systems require a management approach that is focused on forward-looking design strategies. Such a paradigm shift is not likely to arise from the commercial sector, so the solutions that enable true Joint All-Domain Operations (JADO) must come from within DoD.

Finding 13: The energy needs for deployed forces operating within the JADO construct will require solutions that are not likely to come from the commercial sector.

CURRENT STATE OF THE AIR FORCE

The vision of future Air Force operations is that of a coordinated multidomain environment, fueled by an expectation of ubiquitous information and large data sets powering formidable expert algorithms (AI). What is missing is a treatment of the energy needs associated with realizing this vision. When interviewing people in the offices²⁷ responsible for fleshing out the vision of multidomain operations, the committee asked two specific questions: (1) Do you know if anyone in your office is taking into account the energy needs at the tactical edge to support these data-rich operations? (2) Is this an appropriate time to be looking into this issue? The answers to these questions were interesting and multifaceted. The planning for the future operational environment is still in early days, so not all details have been identified or included in the concept of operations. The need for energy to be included in these planning efforts was acknowledged as very important. Appropriately including energy issues in the planning efforts includes bringing all parties

___________________

²⁴ See NASEM, 2021, Strengthening U.S. Air Force Human Capital Management: A Flight Plan for 2020–2030, The National Academies Press, Washington, DC, https://doi.org/10.17226/25828.

²⁵ See A. Berman and J. Dorrier, 2016, “Technology Feels Like It’s Accelerating—Because It Actually Is,” SingularityHub, March 22, https://singularityhub.com/2016/03/22/technology-feels-like-itsaccelerating-because-it-actually-is/.

²⁶ See S. Aderyani et al., 2020, Comparison of nanoarchitecture to porous media diffusion models in reduced graphene oxide/aramid nanofiber electrodes for supercapacitors, ACS Nano 14(5):5314, doi: 10.1021/acsnano.9b07116.

²⁷ See Appendix B for committee agendas.

together and should include specifically identifying energy needs in the requirements processes.²⁸

There is an organizational disconnect between those dealing with future energy needs and those dealing with future information needs: while unfortunate, this would actually be expected, given the extreme differences between the two topics. Those forecasting, planning, and developing future energy needs tend to be civil, mechanical, and electrical engineers, focused on structural aspects, such as providing reliable and defensible power to very large installations. Those forecasting, planning, and developing future information and computational needs tend to be operators, communicators, intelligence officers, and computer scientists, with a sprinkling of radio frequency (RF) engineers for the specialized problems. Thus, the situation begins with a set of functional mismatches: different languages, different time scales, and different expectations. It simply would not be natural for coordination between the two systems to spring up organically. Those doing the forecasting have the luxury of living in a power-rich environment, where people are used to accessing power easily. Understanding if and how energy strategic coordination is being performed, and, of course, how effectively, becomes ever more important as the expectation of and demand for adequate energy supplies increases.

Planning

The Air Force does have an energy strategic plan.²⁹ This plan identifies three specific goals: “The Air Force Energy Flight Plan reflects the Department’s emphasis on operational energy and energy resiliency while reframing the Air Force’s approach to energy through three goals: Improve Resiliency, Optimize Demand, and Assure Supply.” The Air Force subdivides the energy challenge into two focus areas: operational energy and installation energy. Operational energy is defined as “the energy used by the Air Force that has a direct and immediate impact to missions.” This is codified in U.S. law 10 USC §2924. Installation energy is defined as “the energy used by facilities and non-tactical ground vehicles in support of air, space, and cyberspace missions.”

The context of the strategic plan is naturally skewed by how the Air Force energy needs are currently balanced. As noted in the plan, the Air Force accounts for 48 percent of the total DoD energy expenditures. Within the Air Force, 86 percent of the expenditures are associated with aviation fuel. With facilities’ energy costs

___________________

²⁸ See NASEM, 2020, Energy Challenges and Opportunities for Future Data-Driven Operations in the United States Air Force: Proceedings of a Workshop—in Brief, The National Academies Press, Washington, DC, https://doi.org/10.17226/25872.

²⁹ See Air Force Civil Engineer Center, 2017, “Energy Flight Plan 2017–2036,” https://www.afcec.af.mil/Portals/17/documents/Energy/AFEnergyFlightPlan2017.pdf?ver=2019-12-16-105948-090.

coming in at only 11 percent of the budget, it is understandable that most of the attention would be elsewhere. And indeed, it is. Relatively small issues, such as distributed operational energy for data systems, do not currently warrant inclusion in the plan.

With regards to the energy associated with the use of information, there is a single goal in the document: “Employ best management practices for energy-efficient management of servers and data centers.” There are two other explicit statements regarding data centers, one of which offers outsourcing as the preferred option: “Third-party performance contracts, such as Energy Savings Performance Contracts (ESPCs) and Utility Energy Service Contracts (UESCs), provide the Federal Government with a partnership opportunity to procure energy savings and facility improvements with no up-front capital costs. The Air Force is pursuing ESPCs across its facilities, targeting depots and data centers, and exploring new ways of bundling ESPCs to take advantage of economies of scale.” This overall focus is associated with the goal of optimizing demand, including a push for reducing energy needs. There is a nod to the operating concept for the future Air Force, which “envisions a future in which information technologies permeate almost every object,” but no explicit discussion, analysis, or description about how the underlying energy needs are being addressed or planned for.

The Air Force also has an information dominance strategy.³⁰ The word “energy” does not appear in this document, not even once. The word “power” occurs 19 times, but never in the context of energy needs.³¹ The term “data center” also does not appear in the document. The focus of the document is leveraging information for use, rather than on the infrastructure needed to obtain, curate, manipulate, and exploit information.

Research and Development

The Air Force has several offices focused on researching energy needs and developing advanced technology solutions for Air Force operations. The Materials and Manufacturing Directorate at the Air Force Research Laboratory (AFRL) is responsible for “energy assurance/efficiency” and bringing new technologies into solutions for Air Force operational needs.³²

___________________

³⁰ See Office of the Chief Information Officer, 2017, “Air Force Information Dominance Flight Plan: Operating in, thru, and from Cyberspace,” https://www.safcn.af.mil/Portals/64/documents/20170203-%20IDFPv3.pdf?ver=2017-02-16-072244-507.

³¹ The use of the word “power” in the Air Force Information Dominance Flight Plan document is in terms such as manpower, power projection, and other uses to indicate force strength and projection.

³² See Air Force Research Laboratory (AFRL), 2021, “RX,” https://www.afrl.af.mil/RX/.

Further, the Air Force has asked RAND to research energy-related issues. Their research has provided insight into how to improve energy coordination and usage.³³

ENERGY NEEDS AT THE TACTICAL EDGE

Data is the blood of the warfighting body. The heart that powers the movement of data is the logistics of the energy infrastructure. The data moves through the warfighting body, enabling the operations and coordination of the various parts. If the heart stops beating, the data stops flowing, the parts of the body become damaged and possibly die. What is data, though? Increasingly, it is everything: the information in databases, the structural components of the computational infrastructure, the administrative data (metadata), the algorithms, the control structures, the interface protocols, and the things that most people think of as data: words, videos, images, sounds, smells, movements, seismic events, signals, and telemetry.

Every aspect of operations is impacted by the promulgation of information technologies. As shown in Figure 2.2, the tactical edge is no exception and is increasingly dependent on organic capabilities for sensing and processing. For the Air Force, it is tempting to assume that the energy needs will be met through the auspices of the fuel logistics for aircraft operations. However, it is worth pointing out several facts. First, aircraft fuel contributes to the generation of electricity only when the aircraft engines are activated.³⁴ Further, advanced aircraft require significant ground support capabilities that are powered through separate generation systems.³⁵ Last, the proliferation of information technology within and surrounding operations all require electricity to operate.³⁶ The inexorable conclusion is that energy needs at the tactical edge separate from aircraft fuel are an increasingly important issue.

The focus of this report is on the energy needs for the data processing at the tactical edge. Trends that are influencing the growth of data processing at the tactical edge include the following:

___________________

³³ See RAND, 2021, “USAF+Energy,” https://www.rand.org/search.html?query=usaf+energy.

³⁴ See Skybrary, 2020, “Ground Handling,” September 20, https://www.skybrary.aero/index.php/ Ground_Handling; S. Vora, 2017, “Why Planes Get Hot—and How to Stay Cool When They Do,” New York Times, July 24.

³⁵ See Government Accountability Office, 2019, “F-35 Aircraft Sustainment: DOD Needs to Address Substantial Supply Chain Challenges,” https://www.gao.gov/assets/700/698693.pdf.

³⁶ See Congressional Research Service, 2020, “F-35 Joint Strike Fighter (JSF) Program,” https://fas.org/sgp/crs/weapons/RL30563.pdf.

³⁷ See C. Kawasaki and Pacstar, 2019, “Four Future Trends in Tactical Network Modernization,” U.S Army, January 24, https://www.army.mil/article/216392/four_future_trends_in_tactical_network_modernization.

³⁸ See NASEM, 2018, Multi-Domain Command and Control: Proceedings of a Workshop—in Brief, The National Academies Press, Washington, DC, https://doi.org/10.17226/25316.

³⁹ See R. Uppal, 2019, “Military or Battlefield Internet of Things (MIOT/BIOT) Will Network Sensors, Wearables, Weapons, Munitions, and Platforms for Information Dominance,” International Defense, Security and Technology, August 8, https://idstch.com/cyber/internet-things-battlefield/.

• Application of “artificial intelligence” technologies for operations, support, and management of activities;³⁷
• Doctrinal adoption of multidomain operations, which necessitates active coordination through near-real-time data collection, analysis, and communications;³⁸ and
• Application of information technologies to operations management efforts, including the proliferation of “smart devices.”³⁹

Further, the advanced information technologies can have specific energy needs in terms of power stability and allowable variations in current flow.⁴⁰ For mobile

___________________

⁴⁰ See Manufacturing.net, 2013, “7 Ways Signal Noise Can Impact Your Electrical Equipment,” November 13, https://www.manufacturing.net/industry40/article/13057416/7-ways-signal-noisecan-impact-your-electrical-equipment.

platforms, such as trucks, increasing the number of devices that require power may result in the need to add more batteries or higher output alternators. Within the “battlefield of things,” there will be a multitude of devices without access to mobile platforms.

It is axiomatic that the tactical edge is not in garrison and may even be behind enemy lines. The planning and research for steady and reliable energy for fixed bases is a different problem from planning for deployed operations and dynamic basing. The adoption of intelligent devices, unmanned systems, and artificially intelligent services enables a complexity of deployed forces that minimizes the number of people who need to be involved. Each of these technologies needs energy. As these technologies proliferate, the energy needs will grow. Where the energy comes from, how it is managed logistically, and how demand at the tactical edge and beyond is mediated are all issues that warrant consideration.

Finding 14: There will be energy needs associated with the proliferation of smart devices, AI applications, and unmanned systems.

Batteries serve as extremely useful forms of portable energy, especially for small or distributed devices. They serve an important part in any energy infrastructure, particularly in powering highly mobile electronic information processing systems. There are several major issues in battery technologies of interest in designing tactical-edge architectures: physical dimensions (size and weight), cycle life (how many discharges the battery can sustain), and volatility (propensity to explode either from insult or spontaneously).⁴¹ When choosing a battery solution, finding the optimum mix of small size, light weight, high cycle life, and low volatility can be a challenge, particularly in batteries that are integrated into battle gear, such as clothing or helmets. Additionally, sources of power to charge the battery must be considered, particularly if the batteries used are meant to be recharged on site.

Currently, lithium-ion batteries are chosen owing to their combination of light weight and high cycle power. Problems with volatility, as illustrated by spontaneous ignition, are concerning, but manufacturing continues to improve their safety profile. Unfortunately, lithium-ion batteries have a narrow environmental operating envelope and are unsuitable for ambient temperatures lower than 5°C (41°F) or higher than 55°C (131°F).⁴²

___________________

⁴¹ See T. Moynihan, 2017, “Don’t Blame the Batteries for Every Lithium-Ion Explosion,” WIRED, March 19, https://www.wired.com/2017/03/dont-blame-batteries-every-lithium-ion-explosion/; N. Jenkins et al., 2018, McEvoy’s Handbook of Photovoltaics (Third Edition), Academic Press, Cambridge, MA, https://doi.org/10.1016/B978-0-12-809921-6.00025-2.

⁴² See T. Deng et al., 2018, “Temperature effect and thermal impact in lithium-ion batteries: A review,” Progress in Natural Science: Materials International, 28(6): 653-666, https://doi.org/10.1016/j.pnsc.2018.11.002.

Fuel cells operate much like batteries, except that the energy is supplied continuously by a fuel. Depending on the type of fuel cell, the fuel can be hydrogen or a hydrocarbon. Currently, fuel cell technology fills an important niche in the portable energy segment, providing reliable power in situations where the challenges associated with storing and accessing the fuel do not present a problem. Hydrogen, for example, can be made from natural gas or by separating water into its components. If made at a distance from the fuel cell, it must be transported, typically under pressure. Hydrogen is highly flammable, so safe handling procedures are necessary.⁴³

The U.S. government in general and USAF in particular have invested heavily in portable energy technologies. Because there is broad commercial use of such technologies, including for the electric vehicle market, there is additionally significant investment in the commercial sector.⁴⁴

Finding 15: Portable energy technologies are advancing in capabilities from both commercial and government research and development (R&D) investments.

Energy Dependence and Demand

Dependence on commercial power and alternative energy sources will certainly grow in the coming years and decades.⁴⁵ Commercial energy use planning typically has a 10- to 15-year horizon, which takes into account the time needed for real estate acquisition, permits, construction, and other items that support the power utilities, as well as local, city, or county growth plans. Military installations should be engaged with commercial energy providers to ensure that planning includes known expansion or increases in energy demands for their tenant organizations.⁴⁶ Except in wartime conditions, military installation expansion intentions are well known years in advance, which should allow for sufficient planning time with commercial energy providers. The units that deploy to the tactical edge originate at these installations and can be assumed to deploy with doctrinal energy support infrastructures, as well as with an initial supply of energy.

___________________

⁴³ See Lawrence Livermore National Laboratory (LLNL), 2021, Strategic Latency Unleashed: The Role of Technology in a Revisionist Global Order and the Implications for Special Operations Forces, https://cgsr.llnl.gov/research/book.

⁴⁴ See LLNL, 2021, Strategic Latency Unleashed: The Role of Technology in a Revisionist Global Order and the Implications for Special Operations Forces, https://cgsr.llnl.gov/research/book.

⁴⁵ See International Energy Agency, 2020, Renewables 2020, Paris, https://www.iea.org/reports/ renewables-2020; U.S. Energy Information Administration, 2021, “Short-Term Energy Outlook,” https://www.eia.gov/outlooks/steo/report/electricity.php.

⁴⁶ See Department of the Air Force, 2021, Installation Energy Strategic Plan 2021, January 1, https://www.af.mil/Portals/1/documents/2021SAF/01_Jan/AF_Installation_Energy_Strategic_Plan_15JAN2021.pdf.

A factor that must be considered in military planning is the rapidly increasing energy demand from large data processing companies such as Amazon Web Services (AWS), Microsoft Azure, Google, Facebook, and others. These and other companies are building (and will continue to build or expand) very large data centers that are significant consumers of electric power.⁴⁷ If these data centers are located near military installations, either fixed or deployed, there can be a benefit in that the local utilities will plan and install upgraded facilities, which could result in added capacity for the military users. There can also be a resource competition issue should the local power utilities be unable to provide the additional facilities in a timely manner, or if they are not aware of the military installation’s growing needs and do not plan adequately for them. Other issues may emerge, especially with respect to the construction of new generation facilities or transmission lines, both of which require decades of planning, hearings, permitting, and property or right of way acquisition.

Finding 16: The initial deployment of a unit to the tactical edge will include energy that originates from the fixed installations, including energy from commercial energy provider partners.

Support for Machine Learning and Artificial Intelligence

Machine learning (ML) and AI are the terms used to refer to a set of algorithms and analysis methodologies that enable an automated ability to derive useful patterns from data sets, either as de novo discoveries or as rule matching for encountered situations.⁴⁸ For extremely complex situations, the ability to harness automation to quickly discern patterns is very valuable. Some common uses for ML include product recommendations, event alerts, and predictive medicine.⁴⁹

The application of ML and AI to military operations is increasing in value as the amount of data collected, processed, and used at the tactical edge increases in both volume and type. Ubiquitous sensing from both fixed and mobile devices, such as unmanned aerial vehicles (UAVs), provides a wealth of data in audio, image, video, and signal formats. Time delays from pushing the data to back-end proces-

___________________

⁴⁷ See Energy Innovation, 2020, “How Much Energy Do Data Centers Really Use?” March 17, https://energyinnovation.org/2020/03/17/how-much-energy-do-data-centers-really-use/; R. Battish, 2015, “Parameters to Consider in the Data Center Location Decision,” AreaDevelopment, https://www.areadevelopment.com/data-centers/Data-Centers-Q1-2015/data-center-location-decisionparameters-46734866.shtml.

⁴⁸ See I.H. Witten and E. Frank, 2005, Data Mining: Practical Machine Learning Tools and Techniques (2nd edition), Morgan Kaufman Publishers, Waltham, MA.

⁴⁹ See Educba, 2021, “Uses of Machine Learning,” https://www.educba.com/uses-of-machinelearning/.

sors, even when communications capabilities support that transfer, can adversely affect the ability to act in a timely manner on the resultant intelligence data. Adopting ML capabilities at the tactical edge ameliorates this problem by enabling near-real-time analysis of the data in order to discover important situational aspects.⁵⁰

Energy needs associated with ML and AI can be extensive. First, the data being analyzed must be transferred from the source systems to the ML or AI platform. This process includes storing the data in the system, creating metadata for each data artifact, and cleaning the data to ensure that it is error-free and in a format that is process-friendly. Next, the applications execute the analysis of the data, possibly against previously collected and analyzed data, which means that computer processors operate on the data, moving portions of it in and out of memory as the processing continues. An important distinction exists between data processing for training as opposed to inference, in that training can be very expensive in terms of energy, while inference can be relatively cheap, depending on the application and only if engineered appropriately.⁵¹ Last, the results of the ML operations need to be formatted and output for action. During each of these phases, there may be some or minimal human interaction. For any human interaction, there needs to be an interface that is human-friendly, such as a display or a keyboard. Each of these interfaces requires energy.⁵²

Finding 17: Tactical-edge operational needs are driving the adoption of advanced computational capabilities, such as AI and ML. There are significant energy considerations associated with AI and ML that derive from the sensitivity of electronic components, the human-machine interface requirements, and the increased integration of sensors.

Power for User Interfaces

While USAF facility computing systems will need reliable power in the continental United States (CONUS) and outside the continental United States (OCONUS), the users of the information also need energy support because they will need to be able to view and interact with the data and processes through

___________________

⁵⁰ See Defense Systems Staff, 2020, “Automated Analytics for the Tactical Edge,” Defense Systems, September 16, https://defensesystems.com/articles/2020/09/16/socom-automated-analytics.aspx.

⁵¹ See R. Toews, 2020, “Deep Learning’s Carbon Emissions Problem,” Forbes, June 17, https://www.forbes.com/sites/robtoews/2020/06/17/deep-learnings-climate-change-problem/?sh=1b516f8f6b43.

⁵² See W. Knight, 2020, “AI Can Do Great Things—If It Doesn’t Burn the Planet,” WIRED, January 21, https://www.wired.com/story/ai-great-things-burn-planet/; B. Bailey, 2019, “Power Is Limiting Machine Learning Deployments,” SemiEngineering, July 25, https://semiengineering.com/power-limitations-of-machine-learning/.

peripherals such as keyboards, mice, earphones, microphones, and displays.⁵³ This implies reliable power at the human consumption end, which could be inside an aircraft or could be in a forward-deployed command post. As an example, large displays (or thousands of smaller body-worn displays) and the computers that will create the high-resolution images cumulatively require significant power resources.

For fixed systems such as those in a building, power for computers and displays is obtained from the utility infrastructure and typically consists of very small percentages of the energy used and available. Mobile systems, however, typically rely on battery power, or can draw power from a vehicle or platform that has internal power generation. The number of displays, tablets, keyboards, and other human-machine interfaces will increase, and the supply of batteries, fuel cells, or other forms of portable electric power will also increase. This will require additional storage, transportation, and disposal as the number of systems grows. Experience over the past decade shows that rechargeable batteries are not as useful as once thought, owing to the requirement to recharge them in places where recharging is not available or is difficult. Additionally, rechargeable batteries require actual activity to recharge (someone needs to move the battery to the recharging station, monitor the recharging, and then swap it out) and may present a fire hazard.⁵⁴ In fast-paced operational environments, use of nonrechargeable batteries may be preferred, even though that incurs a resupply cost.

Integrated Operations

In order to synergize operational capabilities, increase the speed at which operations can be conducted, and increase the flexibility of military options, the U.S. military has developed the concept of Joint All-Domain Operations (JADO). The intent of this approach is to harmonize and coordinate activities across all services and weapons platforms. In order to achieve these goals, significant communications and data exchange between services and units, particularly forward-deployed units, is necessary. There will be significant energy needed to support the communications and data exchange. As the planning for JADO is still in its early stages, the amount of energy needed at the tactical edge to support such operations is yet to be discovered.⁵⁵

___________________

⁵³ See G. Torbet, 2019, “How Much Energy Does Your PC Use? (And 8 Ways to Cut It Down),” MUO, December 18, https://www.muo.com/tag/much-energy-pc-use-8-ways-cut/.

⁵⁴ See S. Witman, 2019, “Are Rechargable Batteries Better Than Alkaline? Most of the Time,” New York Times, June 6, https://www.nytimes.com/2019/06/06/smarter-living/wirecutter/are-rechargeablebatteries-better-than-alkaline.html.

⁵⁵ See NASEM, 2020, Energy Challenges and Opportunities for Future Data-Driven Operations in the United States Air Force: Proceedings of a Workshop—in Brief, The National Academies Press, Washington, DC, https://doi.org/10.17226/25872.

Finding 18: The energy needs associated with JADO for tactical-edge data operations will be significant but are as of yet not totally understood.

Effect on Operational Readiness

The Department of Defense doctrinally defines “operational readiness” as “The capability of a unit/formation, ship, weapon system, or equipment to perform the missions or functions for which it is organized or designed.” Each of the services uses varying approaches toward measuring and reporting readiness, but all are required to conform with directives from the Chairman of the Joint Chiefs of Staff. The latest Air Force guidance is in AFI 10-201, 22 December 2020.⁵⁶ There are two distinct measured areas of readiness: resource readiness and capability readiness. These are defined in AFI 10-201 as

• “Resource readiness is a commander’s objective measurement of the unit’s resources and training to execute the mission for which it is organized and designed.”
• “Capability readiness is a commander’s subjective assessment of the unit’s ability to accomplish its designed mission essential tasks (METs) based on the unit’s full spectrum mission and the unit’s ability to perform assigned missions.”

The preceding discussion associated with energy dependencies for data-driven capabilities at the tactical edge implies that there may be readiness effects arising from both energy assurance and data assurance. Without energy, data-driven systems cannot operate. Without data, systems cannot see, hear, or speak.

Compounding these issues, it is currently not a requirement that data dependencies or needs be included in weapon systems specifications.⁵⁷ Because data dependencies and needs are not specified, it follows that the energy required for both direct data dependencies and indirect data dependencies for weapon systems readiness and operational capabilities, particularly when deployed, may not be well understood.

Finding 19: Resource readiness is affected when the unit does not have the resources needed to execute the mission.

___________________

⁵⁶ See Secretary of the Air Force, 2020, Air Force Instruction 10-201, December 22, https://static.e-publishing.af.mil/production/1/af_a3/publication/afi10-201/afi10-201.pdf.

⁵⁷ Conversations with AFRL personnel on October 2020 and Jeff Stanley, June 24, 2020, Appendix B.

Finding 20: Capability readiness is affected when the unit does not have the data needed to support the full-spectrum mission.

Finding 21: In determining the requirements for new systems, be that combat or combat support, the USAF has not directed the need for energy minimization, energy consumption monitoring, or power generation as part of the system requirements.

There are significant research efforts under way in many different technological areas that are focused on more effective and efficient use of energy. These include reducing energy usage for computation, both in software construction and in hardware components, as well as developing more efficient computational approaches to structuring the storage and movement of data within systems.

The challenge of delivering energy to deployed forces is complicated by logistics, which tend to favor simplicity over complexity and large users over small users. At the tactical edge, small users may have a larger role in data collection, analysis, and communication under the JADO concepts, which would make them a dependency for the operational readiness of the larger units.

Finding 22: Delivering energy to small units becomes a critical need for the overall system.

Energy Consumption Reduction Architectures

Perhaps the largest opportunities for energy efficiency gains in data-driven operations will be in the combination of algorithms and data system architecture design choices. There is an increasing trend toward edge computing and disaggregation, which is a data architecture with different requirements from a model of cloud-based or otherwise centralized computing centers.⁵⁸ In parallel with the trend toward edge computing, however, there is also a trend toward optimizing energy efficiency of computing hardware by specializing to computing chip platforms tailored to neural network machine learning (such as Google’s tensor processing units [TPUs]⁵⁹ and

___________________

⁵⁸ See NASEM, 2020, Energy Challenges and Opportunities for Future Data-Driven Operations in the United States Air Force: Proceedings of a Workshop—in Brief, The National Academies Press, Washington, DC, https://doi.org/10.17226/25872, presentations by Mukhopadhyay, M. Linderman, Benigni.

⁵⁹ “Tensor Processing Units (TPUs) are Google’s custom-developed application-specific integrated circuits (ASICs) used to accelerate machine learning workloads.” See Google, 2021, “Cloud Tensor Processing Units (TPUs),” https://cloud.google.com/tpu/docs/tpus.

many emerging neuromorphic⁶⁰ platforms).⁶¹ These combined trends raise these questions:

• How long and under which operational use cases will neural network-based ML remain the dominant paradigm?
• How, in what time scales, and under which preprocessing, compression, and data fusion models do the data required for training and inference in ML algorithms flow across a disaggregated network?
• When is a data architecture constrained by energy costs versus bandwidth limitations to move the data to where they need to be processed?

Industry ML deployments are increasingly using custom computer chips to accelerate model training algorithms. While these chips show performance gains in both processing time and energy requirements for commercial cloud deep-learning applications, there is concern within the Air Force that mission-specific applications will require further processor development to realize similar gains. DoD and intelligence community (IC) use cases, including fast decryption of files and cyber-hardening features for use in contested environments, do not yet have an off-the-shelf computer chip solution from industry. Investigations into synaptic or neuromorphic computing platforms (such as Blue Raven⁶²) or low cost, size, weight, and power high-power computing (C-SWAP HPC; such as Agile Condor⁶³) are showing potential benefits for edge deployment of ML algorithms for sensor data processing. High-power computing (HPC) that can be deployed on edge platforms may have particular advantages in those use cases where, for example, graph- and logic-based algorithms are preferred over neural network-based algorithms.

A key question for the neural network-based algorithm paradigm is whether the training data requirements and retraining requirements during operations are practical to meet in the desired mission applications. Algorithm advances that can

___________________

⁶⁰ “Neuromorphic computing research emulates the neural structure of the human brain.” See Intel, 2021, “Neuromorphic Computing,” https://www.intel.com/content/www/us/en/research/neuromorphic-computing.html.

⁶¹ NASEM, 2020, Energy Challenges and Opportunities for Future Data-Driven Operations in the United States Air Force: Proceedings of a Workshop—in Brief, The National Academies Press, Washington, DC, https://doi.org/10.17226/25872, presentations by Crisman, Carol-Jean Wu, Sherief Reda.

⁶² Blue Raven is the project name for “the world’s largest neuromorphic digital synaptic super computer.” See J. O’Brien, 2018, “AFRL, IBM Unveil World’s Largest Neuromorphic Digital Synaptic Super Computer,” Wright-Patterson AFB, July 24, https://www.wpafb.af.mil/News/Article-Display/Article/1582310/afrl-ibm-unveil-worlds-largest-neuromorphic-digital-synaptic-super-computer/.

⁶³ “The Agile Condor high-performance embedded computing (HPEC) architecture offers sensor-agnostic, on-board, real-time data processing to deliver actionable intelligence to the warfighter.” See SRC, 2021, “Agile Condor High Performance Embedded Computing,” https://www.srcinc.com/products/intel-collection-and-analysis/agile-condor-high-performance-embeded-computing.html.

perform with less training data would have potential advantages both in the number of application areas where they could be deployed and in the potential energy requirements for training the algorithms. While less training data may be advantageous for reducing computing and energy requirements, such reductions could be either confounded or enhanced depending on the suitability of the computing and communications platforms for the chosen algorithms. Without further specification of the data flows, algorithms, and computing platforms within an architecture plan, it is difficult to make assessments about energy needs. The committee has not seen the desired level of detail in any data architecture plan.

Finding 23: There is a trend toward optimizing energy efficiency of computing hardware by specializing to computing chip platforms tailored to neural network machine learning.

Finding 24: Mission-specific applications require further processor development to realize similar gains in reducing processing time and energy requirements.

Finding 25: Algorithm advances that can perform with less training data would have potential advantages both in the number of application areas where they could be deployed and in the potential energy requirements for training the algorithms.

Energy-Efficient Computation

The growing demand for computing capabilities, particularly at the tactical edge, has forced the design community to consider all aspects of a fielded product, including size, weight, environmental factors, and energy usage. Energy, a scarce and expensive resource, is becoming a critical design constraint. Software engineers, hardware manufacturers, and system designers are exploring new techniques to reduce the energy consumption of computing devices. One way to reduce energy consumption of a computing device is through the design of software and algorithms that are energy consumption aware. In a given scenario, the goal is to design energy-efficient computational algorithms that reduce energy consumption without sacrificing performance. Research in this area includes both theoretical inquiry and application-level exploration.

Theoretical work⁶⁴ has shown that algorithms for sleep states, power-down mechanisms, dynamic speed scaling, temperature management, and energy--

___________________

⁶⁴ See E. Demaine et al., 2016, “Energy-Efficient Algorithms,” Proceedings of the 2016 ACM Conference on Innovations in Theorectical Computer Science, pp. 321–332, Association for Computing Machinery (ACM), New York, NY.

minimizing scheduling can have a very significant effect on energy consumption. Power-down mechanisms have been developed⁶⁵ that conserve energy by switching a device into low-power modes. The process of dynamic speed scaling in variable-speed processors can save energy by utilizing the full speed and frequency capabilities of a processor and utilizing low speeds whenever possible.⁶⁶ Experiments to understand energy consumption for algorithms have been performed⁶⁷ for popular vector operations, matrix operations, sorting, and graph algorithms. Energy consumption for a given algorithm depends on the extent to which that algorithm can exhibit memory parallelism for a given data layout in the random access memory (RAM) with variations up to 100 percent for many popular algorithms.⁶⁸

In addition, fundamental work⁶⁹ has explored energy complexity of algorithms based on Landauer’s principle, which gives a lower bound on the amount of energy a system must dissipate if it destroys information. This approach lays a theoretical foundation for a new field of semi-reversible computing and provides a framework for the investigation of algorithms. Using this theoretical framework, energy-aware algorithms for control logic, memory allocation, garbage collection, comparison sort, insertion sort, counting sort, breadth-first search, Bellman-Ford, Floyd-Warshall, matrix all-pairs shortest paths, Adelson-Velsky and Landis (AVL) trees, binary heaps, and dynamic arrays have been developed.⁷⁰

Besides theoretical research on operating system-level algorithms, there is also a substantial body of literature on methods to save energy during computation. Many data-rich applications for computing at the edge exist, and the number of use

___________________

⁶⁵ See S. Albers, 2010, Energy-efficient algorithms, Communications of the Association for Computing Machinery 53(5):86–96.

⁶⁶ See S. Irani et al., 2007, Algorithms for power savings, ACM Transactions on Algorithms 3(4); S. Irani et al., 2003, Online strategies for dynamic power management in systems with multiple power-saving states, ACM Transactions on Embedded Computing Systems 2(3).

⁶⁷ See S. Albers and H. Fujiwara, 2007, Energy-efficient algorithms for flow time minimization, ACM Transactions on Algorithms 3(4):Article 49 (November), http://doi.acm.org/10.1145/1290672.1290686; C. Ambühl, 2005, “An Optimal Bound for the MST Algorithm to Compute Energy Efficient Broadcast Trees in Wireless Networks,” ICALP ’05: Proceedings of the 32nd International Conference on Automata, Languages and Programming, pp. 1139–1150, ACM, New York, NY; M. Li and F.F. Yao, 2005, An efficient algorithm for computing optimal discrete voltage schedules, SIAM Journal on Computing 35(3).

⁶⁸ See S. Albers and H. Fujiwara, 2007, Energy-efficient algorithms for flow time minimization, ACM Transactions on Algorithms 3(4):Article 49 (November), http://doi.acm.org/10.1145/1290672.1290686; C. Ambühl, 2005, “An Optimal Bound for the MST Algorithm to Compute Energy Efficient Broadcast Trees in Wireless Networks,” ICALP ’05: Proceedings of the 32nd International Conference on Automata, Languages and Programming, pp. 1139–1150, ACM, New York, NY; M. Li and F.F. Yao, 2005. An efficient algorithm for computing optimal discrete voltage schedules, SIAM Journal on Computing 35(3).

⁶⁹ See E. Demaine et al., 2016, “Energy-Efficient Algorithms,” Proceedings of the 2016 ACM Conference on Innovations in Theorectical Computer Science, pp. 321–332, ACM, New York, NY.

⁷⁰ See E. Demaine et al., 2016, “Energy-Efficient Algorithms,” Proceedings of the 2016 ACM Conference on Innovations in Theorectical Computer Science, pp. 321–332, ACM, New York, NY.

cases is growing. This includes signal and image processing, deep neural networks, computer vision, and machine learning. These applications have special needs for on-board processing: size, weight, and power (SWaP) constraints and hard real-time constraints.⁷¹

There are three key design principles for energy-efficient embedded processing of data-rich applications: (1) hardware accelerators, (2) approximate design and synthesis, and (3) software/hardware co-design.

Hardware acceleration with custom processing circuits eliminates many loads and stores, and executes all possible concurrent operations simultaneously. Custom accelerators can be used in embedded field programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs). In terms of hardware for data-intensive processing, accelerators provide the best energy efficiency for data-rich applications, followed by graphics processing units (GPUs).⁷²

Many data-rich applications in machine learning, signal processing, computer vision, and cognitive computing have inherent error resiliency: noisy inputs, approximate algorithms, and loose constraint on output. Approximate computation can be leveraged to trade off accuracy for hardware resources (e.g., design area, power, latency). The benefits of approximations in embedded accelerators include a large fraction of energy spent in arithmetic units, customization of the data path down to individual bits, and reduced internal memory and memory bandwidth.⁷³

___________________

⁷¹ See L. Benini et al., 2000, A survey of design techniques for system-level dynamic power management, IEEE Transactions on Very Large Scale Integration (VLSI) Systems 8(3):299–316; S. Irani et al., 2007, Algorithms for power savings, ACM Transactions on Algorithms 3(4); K. Pruhs et al., 2008, Speed scaling of tasks with precedence constraints, Theory of Computing Systems 43(1); S. Roy et al., 2014, “Energy Aware Algorithmic Engineering,” MASCOTS ’14: Proceedings of the 2014 IEEE 22nd International Symposium on Modelling, Analysis & Simulation of Computer and Telecommunication Systems, pp. 321–330, IEEE, Piscataway, NJ.

⁷² See S. Albers and F. Müller, 2007, “Speed Scaling on Parallel Processors,” SPAA ’07: Proceedings of the Nineteenth Annual ACM Symposium on Parallel Algorithms and Architectures, pp. 289–298, ACM, New York, NY; J. Augustine et al. 2004, “Optimal Power-Down Strategies,” 45th Annual IEEE Symposium on Foundations of Computer Science, pp. 530–539, IEEE, Piscataway, NJ; N. Bansal et al., 2007, Speed scaling to manage energy and temperature, Journal of the ACM 54(1):Article 3 (March); K. Pruhs et al., 2008, Speed scaling of tasks with precedence constraints, Theory of Computing Systems 43(1).

⁷³ See S. Albers and H. Fujiwara, 2007, Energy-efficient algorithms for flow time minimization, ACM Transactions on Algorithms 3(4):Article 49 (November), http://doi.acm.org/10.1145/1290672.1290686; E. Demaine et al., 2016, “Energy-Efficient Algorithms,” Proceedings of the 2016 ACM Conference on Innovations in Theorectical Computer Science, pp. 321–332, ACM, New York, NY; M. Flammini et al., 2004, “Improved Approximation Results for the Minimum Energy Broadcasting Problem,” Proceedings of the 2004 Joint Workshop on Foundations of Mobile Computing, pp. 85–91, ACM, New York, NY; S. Roy et al., 2014, “Energy Aware Algorithmic Engineering,” MASCOTS ’14: Proceedings of the 2014 IEEE 22nd International Symposium on Modelling, Analysis & Simulation of Computer and Telecommunication Systems, pp. 321–330, IEEE, Piscataway, NJ.

Software and hardware co-design for embedded data-rich systems involved integrated design of the accelerator, approximation techniques, and computation algorithms employed. Examples show 2–3 × reduction in power consumption of accelerators with 2–8 percent reduction in accuracy.⁷⁴

A substantial amount of research has been performed to create energy consumption-aware algorithms at the operating system level and the application level, and it appears this line of work has great potential for reducing energy needs for computing at the tactical edge, but there is still more research that needs to be performed to yield practical and deployed energy-aware algorithms. Needed research includes theoretical work on the application-level side, particularly applications that will be used in computing at the edge such as neural networks and deep learning. In addition, practical algorithm prototypes need to be tested in comprehensive campaigns, structured around operational concept use cases, for energy analysis and best practices.

Finding 26: Clever algorithm design can yield orders of magnitude of energy savings for computer energy consumption.

Further, this body of knowledge suggests pathways in research that can expand fundamental ideas surrounding energy-aware numerical algorithms and research on practical and prototyped algorithms for energy-aware computations.

Computing Hardware

Air Force systems operating at the tactical edge increasingly use highly capable computing hardware that process distributed sensor system data with sophisticated algorithms.⁷⁵ Using these sophisticated algorithms to analyze large amounts of data requires considerable computational capacity. In some circumstances, most processing takes place in back-end systems, where the data is moved from the edge to off-premise support locations. In other circumstances, time constraints and operational security considerations require that the time-sensitive processing take place in near real time at the tactical edge. These edge computing systems have special

___________________

⁷⁴ See S. Irani et al., 2003, Online strategies for dynamic power management in systems with multiple power-saving states, ACM Transactions on Embedded Computing Systems 2(3); K. Pruhs et al., 2008, Speed scaling of tasks with precedence constraints, Theory of Computing Systems 43(1); S. Roy et al., 2014, “Energy Aware Algorithmic Engineering,” MASCOTS ’14: Proceedings of the 2014 IEEE 22nd International Symposium on Modelling, Analysis & Simulation of Computer and Telecommunication Systems, pp. 321–330, IEEE, Piscataway, NJ.

⁷⁵ See NASEM, 2020, Energy Challenges and Opportunities for Future Data-Driven Operations in the United States Air Force: Proceedings of a Workshop—in Brief, The National Academies Press, Washington, DC, https://doi.org/10.17226/25872.

needs for on-board algorithm execution, which include both hard real-time processing and SWaP constraints. For this reason, when possible, supporting and direct computation for devices and platforms at the edge is done off-premise. However, when data-driven decisions must be made instantaneously, this analysis must be done at the edge using field-deployed data centers and platform processors. This is especially important in situations with limited connectivity or inadequate time to transfer the data back and forth from back-end servers. This has led to the growing trend of deploying advanced machine learning methods to the tactical edge.⁷⁶

There are a variety of hardware systems relevant for edge-deployed computational systems. To optimize within SWaP constraints, the current trend is to specialize computing hardware to specific applications.⁷⁷ Because the algorithms used at the tactical edge are complex and may require large parallel matrix multiplications, heterogenous computer architectures are used for tactical-edge devices. Hardware specialization within heterogeneous computing systems with custom processing circuits results in the elimination of many loads and stores, in addition to executing all possible concurrent operations simultaneously. In terms of hardware for data-intensive processing, heterogeneous computing systems provide the best energy efficiency for data rich applications.⁷⁸Box 2.1 discusses the current state of machine learning hardware in more detail.

Finding 27: Operational constraints, such as communication capability and speed requirements for data-driven decisions, cause situations where advanced data processing must take place at the tactical edge. Deploying these types of capabilities to the tactical edge requires an architectural approach that optimizes design constraints to minimize power usage while delivering very fast processing on-premise.

Software

As the development and availability of high-capacity data storage devices has continued to increase the amount of data that can be collected and stored, the

___________________

⁷⁶ Conversations with USMC Major Aaron Stone on September 2020, Appendix B.

⁷⁷ See M. Iriarte, 2018,“UAS SWaP Constraints Driving RF and Microwave Designs,”Military Embedded Systems, April 23, https://militaryembedded.com/unmanned/isr/uas-driving-and-microwave-designs.

⁷⁸ See M. Hallock et al., 2016. “Evaluation of Emerging Energy-Efficient Heterogeneous Computing Platforms for Biomolecular and Cellular Stimulation Workloads,” University of Illinois, Urbana-Champaign, https://www.ks.uiuc.edu/~johns/files/ipdps2016/papers/hcw2016-08-ArmGPU-final. pdf; and Z. Cheng et al., 2017, “Rethinking Energy-Efficiency of Heterogeneous Computing for CNN-Based Mobile Applications,” 2017 IEEE International Symposium on Parallel and Distributed Processing with Applications and 2017 IEEE International Conference on Ubiquitous Computing and Communications (ISPA/IUCC), pp. 454–458, doi: 10.1109/ISPA/IUCC.2017.00074.

software development community has been challenged to develop approaches to data access and usage that are fast enough to avoid frustrating users and to meet the capabilities of the hardware. There are energy trade-offs in how software accomplishes the various steps that support task execution.⁷⁹

___________________

⁷⁹ See J. Michanan et al., 2014, Understanding the power-performance tradeoff through pareto analysis of live performance data, Journal of the ACM 54(1):Article 3, 1–39, https://doi.org/10.1145/1206035.1206038; S. Akramullah, 2014, “Performance, Power, and Quality Tradeoff Analysis,” Digital Video Concepts, Methods, and Metrics, Apress, Berkeley, CA, https://doi.org/10.1007/978-1-4302-6713-3_8; ScaleFocus, 2019, “Top 13 Reasons to Modernize Your Legacy Systems,” March 6, https://www.scalefocus.com/insights/business/top-13-reasons-to-modernizeyour-legacy-systems; R. Mack, 2020, “How Utilities Can Rewire Customer Engagement by Replacing Legacy Systems,” Veracity, March 11, https://veracityit.com/2020/03/11/utilities-legacy-systems/.

An increased amount of data manipulated for complex analyses implies the need for many computation cycles.⁸⁰ There are better and worse ways to approach computational strategy. Some computational approaches are energy hogs, while others are energy efficient. There is an entire research field associated with identifying energy-efficient computational approaches, which encompass both hardware and software aspects. To date, “researchers have proved mathematically that relatively simple hardware modifications could cut in half the energy consumed in running today’s standard software procedures.”⁸¹ The implications of efficient computing cascade to the energy needed for cooling as well: fewer computational cycles generate less heat.

Finding 28: Modifying how computational systems execute instructions can reduce energy usage of the system.

Finding 29: Reducing the energy used in computation also reduces the amount of heat generated, which not only affects the amount of energy needed for environmental controls but also affects the operational heat signature of deployed units.

For small data users, such as home computer users, the amount of energy consumed by software processes may not be noticeable. However, as individuals come to rely on mobile devices, which are battery powered, software processes that run down batteries quickly become a market differentiator. Legacy software on more modern devices are typical sources of power drain; concomitantly, advanced software installed on older computational systems (if possible at all) can overwhelm system resources (such as RAM) and drain power supplies. Keeping many programs in active memory is also implicated in draining power from devices. Activities that are inherently resource-intensive, such as mining bitcoins, can overwhelm regional power grids and may serve as a warning of what is to come.⁸² Further, the use of device software that relies on cloud-based software processes rely on communication between the device and the cloud, which requires extensive use of communications software. For example, voice-activated capabili-

___________________

⁸⁰ See T. Brown and D. Hernandez, 2020, “AI and Efficiency,” OpenAI Blog, May 5, https://openai.com/blog/ai-and-efficiency/.

⁸¹ See N.W. Stauffer, 2013, “Energy-Efficient Computing: Enabling Smaller, Lighter, Faster computers,” MIT Energy Institute News, June 20, http://energy.mit.edu/news/energy-efficientcomputing/.

⁸² See P. Roberts, 2018, “This Is What Happens When Bitcoin Miners Take Over Your Town: Eastern Washington Had Cheap Power and Tons of Space. Then the Suitcases of Cash Started Arriving,” Politico, March/April, https://www.politico.com/magazine/story/2018/03/09/bitcoin-mining-energyprices-smalltown-feature-217230.

ties typically require back-end cloud-based computational support for interpretation of the voice commands.⁸³ The integration of many small end users’ devices (the IoT phenomenon) magnifies this issue. Very large data users are strongly motivated to implement software that minimizes energy consumption. This desire has spurred a dynamic research area in how to construct software to minimize energy consumption.

Finding 30: There is a relationship between the software use profiles and the hardware architecture of small devices that affects battery life between charges.

There are some straightforward approaches to reducing energy use by software components, such as reducing the amount and frequency of data logged during operation, removing unneeded data from actively searched repositories, and eliminating data format translations during software execution. Every unneeded process consumes energy, so a first step is to reduce the number of operations that are needed. Beyond that, research into reversible energy computing is yielding results that are impacting the approaches to developing software programs. “Existing work shows that the use of parallel and approximate programming, source code analyzers, efficient data structures, coding practices, and specific programming languages can significantly increase energy efficiency. Moreover, the utilization of energy monitoring tools and benchmarks can provide insights for the software practitioners and raise energy-awareness during the development phase.”⁸⁴

Big Data

The plan for “AI to the Edge” means that data sets will be needed at the edge. As noted in the section “Energy Consumption Reduction Architectures” above, the choices of applications and engineering solutions that depend on big data can greatly affect energy requirements for tactical-edge computing support. The next-generation tactical-edge information collection, processing, and usage is conceptually based on what can be referred to as “big data,” which is intended to feed machine learning support to operations. There are both practical and operational considerations for the use of big data machine learning solutions at the tactical edge. The practical aspect includes the problem of training the machine learning application: training takes a great deal of data, time, and energy, which makes it problematic in a fast-paced tactical environment. The operational aspect includes

___________________

⁸³ See V. Sharma, 2017, “How Do Digital Voice Assistants (e.g., Alexa, Siri) Work?” USC Marshall, October 17, https://www.marshall.usc.edu/blog/how-do-digital-voice-assistants-eg-alexa-siri-work.

⁸⁴ See S. Georgiou et al., 2019, Software development lifecycle for energy efficiency: Techniques and tools, ACM Computing Surveys 52(4).

the challenge of performing the machine learning training at the back end and then performing inference based only on that training at the tactical edge: because the inference is based on historical data, an adversary need only change its tactics and processes to reduce the utility of the output.

A conventional definition of big data is that it is based on velocity, volume, variety, veracity, and value.⁸⁵ A critical value proposition in big data analytics is the discovery of subtle patterns and hidden correlations that are not obvious. The ability to use the sheer volume of data to make qualitatively different discoveries is an important part of the big data environment. The increased number and variety of data collection at the tactical edge provides all of these aspects. It is one thing to collect a rich data set; it is a very different thing to access it from the tactical edge. A huge lesson learned from global business operations that “follow the sun” is that you do not want to be pushing entire data sets over networks from time zone to time zone: it would take the entire work cycle to update the baseline.⁸⁶ This creates a challenge: sending the sheer volume of data collected to back-end processors is tactically difficult in that it requires significant bandwidth, but processing the data locally removes the context of prior knowledge that is available at the back end and requires significant processing infrastructure at the tactical edge. A middle-ground solution that deploys advanced IT infrastructure with tactical-edge operators can address this issue, but that also implies advanced energy support for those cutting-edge technologies.

Big data processing consumes a lot of energy.⁸⁷ The energy needs are not just for the computational devices but also for environmental control. Because of the huge energy consumption needs, the details of implementation are important: energy losses over transmission lines can figure quite significantly in the requirements calculations. That is why very large data centers are located as close as possible to power generation facilities.⁸⁸

___________________

⁸⁵ See Big Data, 2020, “The Five V’s of Big Data,” BBVA, May 26, https://www.bbva.com/en/fivevs-big-data/.

⁸⁶ The problem with moving data over networks is so problematic that actual trucks are used to move large amounts of data. See, for example, this very real service: “AWS Snowmobile is an Exabytescale data transfer service used to move extremely large amounts of data to AWS. You can transfer up to 100PB per Snowmobile, a 45-foot long ruggedized shipping container, pulled by a semi-trailer truck. … Transferring data with Snowmobile is more secure, fast and cost effective.” See AWS, 2019, “AWS Snowmobile Overview,” https://aws.amazon.com/snowmobile/.

⁸⁷ See R. Danilak, 2017, “Why Energy Is a Big and Rapidly Growing Problem for Data Centers,” Forbes, December 15, https://www.forbes.com/sites/forbestechcouncil/2017/12/15/why-energy-is-abig-and-rapidly-growing-problem-for-data-centers/?sh=714d0c155a30.

⁸⁸ See F. Boshell and S. Ratka, 2020, “The Nexus Between Data Centers Efficiency and Renewables: A Role Model for the Energy Transition,” Energypost, June 26, https://energypost.eu/the-nexusbetween-data-centres-efficiency-and-renewables-a-role-model-for-the-energy-transition/.

Finding 31: Collecting rich data at the tactical edge means that some capability for big data processing exists at the tactical edge. Big data processing at the tactical edge requires adequate energy.

Finding 32: Dependence on conventional cloud computing solutions is not a viable solution for every tactical situation and therefore cannot be the sole solution to the data-driven operational problems. The development of portable data centers may provide an intermediate solution.

Data Variety

The types of data being collected, stored, and analyzed is increasingly varied in format. The formats include video, audio, textual, and other sensory systems (such as telemetry). Mixing all these types of data into a comprehensive analytical framework requires advanced approaches to data integration and meaning extraction. Research on how to do this quickly and energy efficiently touches on storage approaches, searching algorithms, and display efficiencies (such as background refresh options). Data in forms other than textual (such as video, audio, or analog signals) increases communication bandwidth needs, both locally and externally, which increases energy needed for communication infrastructure.

Concomitantly, fusing data in multiple formats from a variety of sources is not a trivial undertaking. The term “data fusion” refers to combining distinct data sets to obtain unified analytic results. With appropriate methods of data fusion, information from distinct edge sensors may be combined to produce inferences that exploit modality or geographic diversities that are qualitatively improved over any single edge analysis alone. The whole is greater than the sum of its parts. The fusion process must be engineered to allow for the time aspects associated with various data (some data phenomena occur frequently, while others occur rarely or periodically), the precision (or lack thereof) of different data types, and the reliability of various data sources. Poorly engineered fusion approaches use excessive computing cycles, which increases the energy needed.

Finding 33: Variety in data types increases communication bandwidth needs, which increases energy needed for communication.

Finding 34: Data fusion that integrates many different types of data requires careful consideration and engineering to minimize energy used in the fusion process.

INFRASTRUCTURE AND ENTERPRISE NEEDS

A system is an integrated set of capabilities. For one part of a system to operate as designed, any dependencies in other parts of the system must be considered. For tactical-edge advanced computing, the following components will make the overall system work better or worse: architecture design, software and hardware energy efficiencies, and manpower support.

Architecture Design

There is a growing trend toward edge computing and disaggregated data architectures, which have different infrastructure (including energy) requirements from architecture models of centralized cloud or command and control (C2) centers where data is aggregated for processing. Part of the reason for the move to edge computing is that it is common to produce large volumes of data at the tactical edge that cannot be backhauled to a central computing facility within a time frame that meets mission requirements.⁸⁹ Leaving time aside, the communications infrastructure is also often not in place to move large volumes of data that are generated on a regular or continual basis. Data triage is a viable solution to data transfer bottlenecks, but it does not address the stated need for more data-driven operations.⁹⁰ Instead, volumes and rates of data processing at the edge are expected to continue increasing, with associated increases in energy demands. The pressing questions are: (1) What algorithms will be run to meet which mission needs? (2) What data are needed where to support such algorithms? (3) What energy and communications infrastructure requirements does this imply? The answers to these questions are given by a data architecture design.

As noted previously, for most data-centric algorithms it is important to separate the operations of “training” versus “inference.” During training, models are fit to the available data. During inference, fitted models are used to analyze new data. In a recent analysis by the company OpenAI, the computational power requirements to train state-of-the-art AI algorithms have been doubling every 3 to 4 months since 2012—seven times faster than Moore’s law.⁹¹ With such trends in mind, the (often implicit) architecture model for training AI systems is to aggregate training data in a large, centralized computing center, followed by deployment of the trained model to the edge. Such a data architecture may be appropriate when

___________________

⁸⁹ See NASEM, 2020, Energy Challenges and Opportunities for Future Data-Driven Operations in the United States Air Force: Proceedings of a Workshop—in Brief, The National Academies Press, Washington, DC, https://doi.org/10.17226/25872, presentation by Col. Benigni.

⁹⁰ See Conversation with Frank Konieczny, July 2020, Appendix B.

⁹¹ See T. Brown and D. Hernandez, 2020, “AI and Efficiency,” OpenAI Blog, May 5, https://openai.com/blog/ai-and-efficiency/.

the necessary training data can be aggregated offline and when the trained models can be trusted to maintain relevance in the deployed domain. On the other hand, when data must be aggregated from or otherwise shared across edge networks over limited bandwidths or when the models require frequent retraining, other data architecture designs may be required.

Because training AI models is the most demanding part of the computation in terms of energy requirements, the data architecture may become a dominant factor in determining energy needs for data-driven operations. In addition, current activities to optimize the energy efficiency of computing hardware is leading to specialization of computing chips for performing AI inference. These specialized computing platforms are showing promise for reducing SWaP requirements but are not addressing the architecture questions of when (re-)training of such AI systems may be needed and how the training data is made available.⁹²

Finding 35: The data architecture may become a dominant factor in determining energy needs for data-driven operation.

Data fusion as an aspect of data architecture design is worth calling out. When mission applications may be enhanced or even reliant on edge node data fusion capabilities, data processing and communications requirements must be set at the system or architecture level. This is a complicated and difficult problem that is easy to overlook. However, without a system-level requirements process for data exploitation, the delivered architecture may fail to meet the promise of data-driven operations and at the same time become overengineered into isolated edge node or centralized processing models.

Finding 36: A system-level requirements process for data acquisition, communications, and processing is likely required to achieve the desired data-driven operations capabilities.

Manpower

One critical part of the infrastructure that is often overlooked is that of people. In the field of energy, particularly as it applies to energy needed to operate dispersed data centers, the Air Force has a shortage of qualified personnel, which is exacerbated by the reality of competition with the civilian sector for the available

___________________

⁹² See Defense Advanced Research Projects Agency, 2021, “AI Next Campaign,” https://www.darpa.mil/work-with-us/ai-next-campaign.

talent pool.⁹³ In the past, the Air Force has treated energy as a commodity and then outsourced both the planning and execution of this to the contractors.⁹⁴ As energy becomes more critical to mission success, this specialty field needs to become a core field within the Air Force. There are several concerns. First, in order to contract for, and supervise, contractors, the Air Force needs in-house experts. Second, the energy needs for deployed units and weapons systems are different based on mission profiles, operational needs, and component elements, which implies a need for specific expertise on a deployable basis.

Critical skill sets needed are balancing load power, controlling microgrids, battery maintenance and sustainment, maintaining a low signature, and the ability to stand up and take down the power sources quickly across the operational and tactical battle space. Additionally, the Air Force must have the expertise to power not just the data center but also the networks associated with moving the data from place to place as needed. This is, in and of itself, a specialty area where power may need to be turned on and off, in an instant. Last, within the field of research and development, it is likely that the Air Force will have unique requirements for the power consumption of chips, which large commercial providers do not have. Therefore, this scientific skill set, for minimizing chip power consumption, needs to be considered as an investment within the Air Force science and technology (S&T) enterprise.

At a national or strategic level, the number of electrical engineers entering the career field, including the subset who are interested in and capable of attaining security clearances, is not growing as fast as industry or the Air Force needs.⁹⁵ A career in electric power is not viewed as being as lucrative as a career in computer science, programming, or application development. This impacts both the Air Force and industry, and competition for trained and experienced electrical engineers may drive the available pool more toward industry than toward government or military service. Furthermore, as the average age of experienced electrical engineers increases, the awareness of new technologies and new threats diminishes.

___________________

⁹³ See C. Murray, 2000,“Engineer Shortage Called Danger to Military Readiness,” EETimes, September 21, https://www.eetimes.com/engineer-shortage-called-danger-to-military-readiness/#; R. Wilson, 2016, “EuMW: Skills Shortage Is Sector’s Biggest Challenge,” Electronics Weekly, October 4, https://www.electronicsweekly.com/market-sectors/military-aerospace-electronics/eumw-skills-shortage-sectorsbiggest-challenge-2016-10/; L. Frenzel, 2013, “Is There Really a Shortage of Engineers?” Electronic Design, September 16, https://www.electronicdesign.com/community-home/article/21798513/isthere-really-a-shortage-of-engineers; M. Lapedus, 2019, “Engineering Talent Shortage Now Top Risk Factor,” Semiengineering, February 25, https://semiengineering.com/engineering-talent-shortagenow-top-risk-factor/.

⁹⁴ See Air Force Civil Engineer Center, 2017, “Energy Flight Plan 2017–2036,” https://www.afcec.af.mil/Portals/17/documents/Energy/AFEnergyFlightPlan2017.pdf?ver=2019-12-16-105948-090.

⁹⁵ See M. Lapedus, 2019 “Engineering Talent Shortage Now Top Risk Factor,” Semiengineering, February 25, https://semiengineering.com/engineering-talent-shortage-now-top-risk-factor/.

However, there may be unexpected continued resilience in new power facilities owing to the tendency of these engineers to use designs that rely more on physical switches, relays, protective systems, and human control, and less on automation and computer control.

Finding 37: There is a need for deployable personnel to set up, manage, maintain, and optimize the energy support for tactical-edge computational systems.

Finding 38: There is a shortage of trained personnel in electrical engineering and data center support operations.

Finding 39: Owing to the design needs of tactical edge computational systems, it is likely that there will be a need for S&T investment.

The current assumption that data-driven operations will rely on large data centers has falsely led the Air Force to assume that its current methods of providing energy to large operational installations, using contracted power and labor, is sufficient. With the need to perform the compute and store functions dispersed across the tactical, operational, and strategic level of warfare, the manpower skill sets needed for this change in energy patterns is significant and is a barrier to the successful implementation of data-driven operations.

The Air Force must have the organic manpower in order to learn how its needs differ from the commercial sector, to determine what organic intellectual property meets its needs, to manage contractors who will implement solutions, and to make the trade-offs needed between operational needs and energy needs. Simply stated, without the organic manpower that understands the entire spectrum of energy needs, the Air Force may never achieve a solution that strengthens its operational goals and instead will subject itself to substantial tactical, operational, and strategic risk. For the purpose of this discussion, the term “manpower” includes recruiting, educating, training, contractor/military blend, and incentives for education.

Finding 40: The USAF does not have the organic manpower to manage, lead, supervise, or solve the challenges associated with the design, operation, and management of energy consumption associated with data-driven operations.

Research

There is a gap today between current technologies for data collection and exploitation and stated aspirations such as intelligent automated decision support and real-time collaborative autonomy (as in the “kill mesh” described by Air Force Warfighting Integration Capability [AFWIC] and the “mosaic warfare”

concept from Defense Advanced Research Projects Agency Strategic Technology Office [DARPA STO]). The National Security Commission on Artificial Intelligence (NSCAI) in its interim report identified a need for AI-enabled autonomous operations to include predictive analysis, decision support systems, unmanned platforms, robotics, and weapons.⁹⁶ While individual algorithms and platforms exist today for each of these capabilities, an enterprise architecture integrating such capabilities has yet to be demonstrated.

Finding 41: There is a gap between current technologies and planned capabilities for collaborative data-driven decisions.

Innovative Approaches to Energy

There are several important aspects regarding innovation in energy sourcing, storage, and logistics. These include distributing energy resources, utilizing alternative energy sources in innovative ways, and being clever in the storage and usage of energy in order to minimize loss and maximize the use of available resources.

Distributed Energy Resources

Distributed Energy Resources (DER) are an important element in an overall energy architecture. The traditional model of electricity infrastructure is to generate power where it is cheapest or has the smallest impact, move the electricity via transmission lines to substations close to the user, then distribute the electricity from those substations to the users. DER technologies are typically used with solar, wind, or other “renewable” sources and can either be connected to a larger power grid or in a microgrid⁹⁷ configuration.⁹⁸ Renewable energy technologies have matured significantly in the past several decades and have achieved significant market penetration.⁹⁹ They have also emerged as an important technology

___________________

⁹⁶ See National Security Commission on Artificial Intelligence, 2019, Interim Report: November 2019, https://www.nscai.gov/whitepaper/interim-report-november-2019/.

⁹⁷ Microgrids are a key component for improving power reliability and quality, efficiency, and providing grid-independence to users. See M. Smith and D. Ton, 2016, “The U.S. Department of Energy’s Microgrid Initiative,” U.S. Department of Energy, Washington, DC.

⁹⁸ DER is defined as “any resource on the distribution system that produces electricity and is not otherwise included in the formal NERC definition of the Bulk Electric System (BES).” See North American Electric Reliability Corporation, 2017, “Distributed Energy Resources,” February, https://www.nerc.com/comm/other/essntlrlbltysrvcstskfrcdl/distributed_energy_resources_report.pdf.

⁹⁹ See National Renewable Energy Laboratory, 2021, “Renewable Electricity Futures Study,” https://www.nrel.gov/analysis/re-futures.html.

for disaster and emergency relief operations.¹⁰⁰ There are efforts under way to establish the utility of renewable energy-based DER in garrison via microgrids, such as at Scott Air Force Base in Illinois, where solar and other systems are being integrated to increase resiliency.¹⁰¹ Renewable energy for deployed military units is seen as advantageous owing to the historic poor performance of diesel generators, which have an expected probability of performance of 80 percent at initial start and reducing after that with extended operations¹⁰² and with the challenges associated with deploying to areas with extensive lack of infrastructure.¹⁰³

Finding 42: DER based on renewable sources, such as wind and solar power, has become a viable technology for deployed microgrids.

Finding 43: DER at the tactical edge have been in the form of diesel generators, batteries, and other forms of local electric power generation. The use of renewable energy sources such as solar, wind, or thermal has been limited owing to the cost of deployment and the fact that they are generally not “hardened” against physical damage during conflict.

Innovative Storage Concepts and Resiliency

Beyond DER innovations, there is potential for innovative approaches to using electric vehicles for storage and reuse of generated power at the tactical edge. DoD is in the process of electrifying its nontactical fleet of vehicles¹⁰⁴ and the U.S. government recently committed to an all-electric fleet of vehicles,¹⁰⁵ making the

___________________

¹⁰⁰ See Federal Emergency Management Agency, 2020, “Integrated Sustainability into Recovery,” July 22, https://www.fema.gov/emergency-managers/national-preparedness/frameworks/communityrecovery-management-toolkit/recovery-planning/integrating-sustainability.

¹⁰¹ See Office of Energy Efficiency and Renewable Energy, 2020, “2020 Assisting Federal Facilities with Energy Conservation Technologies (AFFECT) Federal Agency Call Funding Recipients,” https://www.energy.gov/eere/femp/2020-assisting-federal-facilities-energy-conservation-technologiesaffect-federal-agency.

¹⁰² See NASEM, 2020, Energy Challenges and Opportunities for Future Data-Driven Operations in the United States Air Force: Proceedings of a Workshop—in Brief, The National Academies Press, Washington, DC, https://doi.org/10.17226/25872.

¹⁰³ See V. Miyagi, 1996, “National Guard in Disaster Relief Operations—Hurricane Iniki (6 September–6 November 1992)—A Case Study,” University of Pittsburgh, March 25, https://apps.dtic.mil/dtic/tr/fulltext/u2/a310669.pdf.

¹⁰⁴ See C. Marnay et al., 2013, “Los Angeles Air Force Base Vehicle to Grid Pilot Project,” Lawrence Berkeley National Laboratory, https://eta-publications.lbl.gov/sites/default/files/lbnl-6154e.pdf.

¹⁰⁵ See R. Meyer, 2021, “The Weekly Planet: Why Biden Is Buying 645,000 New Cars,” The Atlantic, January 26, https://www.theatlantic.com/science/archive/2021/01/weekly-planet-why-biden-isbuying-645000-new-cars/617828/.

concept of vehicle-to-grid (V2G)¹⁰⁶ power sharing potential an interesting option to explore. In the V2G concept, battery-equipped vehicles are used for power storage when not in use, which enables the potential for using those batteries as a source of energy for other equipment. A V2G solution can assist in bulk energy storage (particularly from deployed renewable energy generation capabilities), power frequency and fluctuation regulation, and holding operational reserves. Advances in stable large batteries, such as the Tesla PowerWall, can also be part of this potential solution space.

Resiliency in power supplies at the tactical edge, as well as any other contested or compromised location, is a critical component for continued operational capability. This may include concepts such as isolating microgrids from each other or creating self-sufficient microgrids as required on emergency bases. The reliance on traditional energy sources, such as diesel generators, can imbue vulnerabilities into the system that can be ameliorated with alternative energy sourcing and storage. Having multiple energy sources with versatile storage capabilities can be an advantage.¹⁰⁷

There are several ways to think about increasing the resiliency of energy systems in order to minimize effects from direct or indirect attacks, including those resulting from natural phenomena such as weather. One of the first options is to harden the system, simply to minimize effects. How the energy systems at the tactical edge can be hardened is an interesting challenge, because by definition the systems must be deployable, which implies that they are movable, relatively lightweight, and potentially stand-alone. There has been some research into replacing petroleum-based fuels with alternatives, such as liquefied natural gas (LNG) or batteries, which reduces the size and weight of the energy component while improving reliability of the system.¹⁰⁸

Another way to improve resiliency is to design systems that can be quickly restored following outages or systems that can react to and isolate the effects of attacks, through a self-aware triage capacity. In other words, to design systems

___________________

¹⁰⁶ See D. Steward, 2021, “Critical Elements of Vehicle-to-Grid (V2G) Economics,” National Renewable Energy Laboratory, https://www.nrel.gov/docs/fy17osti/69017.pdf; W. Kempton and J. Tomić, 2004, “Vehicle-to-Grid Power Implementation: From Stabilizing the Grid to Supporting Large Scale Renewable Energy,” University of Delaware, November 12, http://www1.udel.edu/V2G/KempTom-V2G-Implementation05.PDF.

¹⁰⁷ See National Renewable Energy Laboratory, 2018, “Valuing the Resilience Provided by Solar and Battery Energy Storage Systems,” March, https://www.energy.gov/sites/prod/files/2018/03/f49/Valuing-Resilience.pdf.

¹⁰⁸ See Conversations with AFRL on October 2020, Appendix B.

that are self-aware to the point where they can react to insults or attacks by taking themselves off-line in order to protect the rest of the system.¹⁰⁹

Innovation in storage and reuse of energy is another way to improve resiliency. For example, enlisting unused capabilities, such as the batteries in dormant vehicles, as storage devices for generated electricity, can create a significant alternative bank of energy resources. The V2G concept, discussed previously, is an example of such a solution.¹¹⁰ Such innovative approaches make the most out of the components onsite, thereby augmenting conventional capabilities and increasing the resiliency of power systems.

Finding 44: Innovating integration of multiple-source energy generation and storage capability can increase resiliency at the tactical edge.

Finding 45: The increasing use of electric vehicles combined with deployed energy generation systems creates the potential for the innovative system integration of those two elements into a V2G capability.

POTENTIAL THREATS

Throughout the history of competition between adversaries, energy sources have been among the preferred targets: deny energy to your adversary and they lose some or all ability to operate. Therefore, understanding the types and potential of threats to energy sources throughout the supply chain is important.

The Threat Ecosystem

Threats come in many flavors, including those yet to be imagined. When considering the ecosystem of problems that might arise with regard to the availability of energy at the tactical edge to power data-driven operations, one must consider all aspects of how energy is made available. There are two particular dimensions that bear analysis: the supply chain for energy and the system of systems that manage the energy usage at the tactical edge. The supply chain of energy includes the encapsulation of energy into a transportable medium, the transportation of the stored energy, the receipt of the energy, and the integration of the energy into the tactical-edge environment. Figure 2.3 shows a generalized view of the energy

___________________

¹⁰⁹ See B. Carrier, 2019, “Incident Response KPIs: SPEED Is Critical. Here Are Five Reasons Why,” Cyber Triage, April 9, https://www.cybertriage.com/2019/incident-response-kpis-time-is-criticalhere-are-five-reasons-why/; J. James and M. Koopmans, 2013, Automated network triage, Digital Investigation 10(2):129–137.

¹¹⁰ See University of Delaware, 2021, “What Is V2G?” http://www1.udel.edu/V2G/.

supply chain in more detail. The management of energy use at the tactical edge includes the systems for converting stored energy to usable energy, the infrastructure through which energy flows from source to user, the administration of scarce resources, and the communications of energy requirements for both future and current operations.

Transforming a source of energy into a usable form of energy is an important first step in the supply chain. Nonrenewable sources, such as coal or oil, can be extracted from various sources, such as mines or underground reservoirs, but are not readily usable without processing. In this process, impurities are removed, and the product is refined to a usable state for its intended purpose. Energy from renewable sources, such as wind and solar, must be captured and then transformed into storable energy, such as in batteries, stored water, or molten salt. Threats to energy supplies during these transformation processes include both physical and logical attacks on the processes. It is unlikely that nonrenewable sources would ever be processed and transformed at or near the tactical edge, so this is not a primary consideration for those types of energy sources. For example, drilling for oil requires significant infrastructure, so it is unlikely that an oil extraction and refining capability would ever be associated with tactical deployments. A possible exception could be small nuclear generators, given the extraction and refinement of the nuclear energy prior to the deployment of the small nuclear generators. However, for renewable energy, it is quite likely that collection, processing, and transformation could be near or at the tactical edge.

Moving energy, once it has been transformed into storable energy, is necessary in order to get it to the end user. When the energy is in transit, it is a likely target: if energy can be denied to an end user, it cannot be used. Moving nonrenewable energy to end users, the “last mile,” is a recognized challenge. While the security of supply convoys has always been a priority, studies recently have established “a strong relationship between fuel consumption and casualty rates owing to fuel and

water convoy protection.”¹¹¹ It follows, therefore, that reducing the dependency on energy supply convoys or shortening the distance between energy source and energy use may have some benefits in terms of reducing casualty rates. On the other hand, exposing the energy source and processing systems to attack also has drawbacks in that if the collection and processing mechanisms are destroyed, then all energy from that source is denied. Last, what has not been seriously contemplated is the potential scale of the distributed energy need and how to distribute, in a potentially clandestine manner, energy across the battlefield in tiny packets.

Finding 46: The transportation of energy to the tactical edge is a recognized target for adversaries.

Receiving and integrating energy into end user systems can present some challenges as well. Energy sources that are highly volatile require special handling and use. Lithium-ion batteries, for example, can explode under certain storage environments as well as when subjected to shocks. For example, there have been “several incidents where Tesla batteries appeared to spontaneously burst into flames—sometimes while driving, other times while the car was parked.”¹¹² The volatility of hydrocarbon fuel sources is well known and requires special storage.

Attacks on the energy supply and use can stem from physical attack, such as explosives, or from logical attacks, such as compromising control systems. Malicious software is regularly implicated in battery life reduction in handheld devices, such as smart phones, and could in the future be specifically designed to drain energy from such systems.¹¹³ The vulnerability of systems to attack varies at each stage in the system and is limited only by the limitations of the adversary. Another attack vector—a compromise of the technology supply system—was used in conjunction with the SolarWinds incident in late 2020.¹¹⁴ The technology supply chain is different from the energy supply chain discussed above. Rather than disrupting

___________________

¹¹¹ See Army Technology, 2020, “Casualty Costs of Fuel and Water Resupply Convoys in Afghanistan and Iraq,” February 7, https://www.army-technology.com/features/feature77200/.

¹¹² See R. Mitchell, 2019, “Federal Safety Agency Launches Probe of Tesla Battery Fires,” Los Angeles Times, November 1, https://www.latimes.com/business/story/2019-11-01/federal-safety-agencylaunches-investigation-of-tesla-battery-fires.

¹¹³ See I. Morris, 2018, “Android Virus May Be Draining Your Battery and You Should Remove it NOW,” Mirror, December 17, https://www.mirror.co.uk/tech/android-virus-draining-yourbattery-13744925; D. Riley, 2019, “New Android Malware Drains Batteries and Uses Data in Ad Fraud Scam,” Silicon Angle, February 21, https://siliconangle.com/2019/02/21/new-android-malwaredrains-batteries-uses-data-ad-fraud-scam/.

¹¹⁴ See FireEye, 2020, “Highly Evasive Attacker Leverages SolarWinds Supply Chain to Compromise Multiple Global Victims with SUNBURST Backdoor,” December 13, https://www.fireeye.com/blog/threat-research/2020/12/evasive-attacker-leverages-solarwinds-supply-chain-compromises-withsunburst-backdoor.html.

parts of the flow of energy, an attacker that can successfully compromise a technical system used widely by energy managers could potentially disrupt energy flow. In the SolarWinds incident, tens of thousands of businesses were impacted when updates to the Orion system (a popular network and asset management tool) contained hidden malicious code that gave the attackers virtually unrestricted access to the victims’ networks.

Cyberattacks on Energy Grid Operations

Availability of commercial power has not traditionally been a concern for CONUS installations. Utilities are regulated at the state and federal level and must adhere to reliability standards that ensure near 100 percent availability. While outages caused by natural events (weather, seismic activity, forest fires, etc.) are infrequent, they are not rare. Weather and seasonal impacts can often be anticipated and planned for. A different type of outage—mandatory “rolling blackouts” as seen in California in 2020¹¹⁵—will be a factor in planning for future commercial power availability. Other factors such as policies that reduce or eliminate electricity generation by fossil or nuclear fuels may also reduce available commercial power. Renewable energy (wind, solar, etc.) may not provide the steady base load of commercial power needed by military installations.

Fortunately, there have been no cyberattacks on the North American electric grid that resulted in a loss of power or loss of control. All grid operators are required by regulation to report any cyber incident that impacts their operations, and over the past several years there have been very few reports of cyberattacks. However, two incidents are worth noting.

In 2019, a control center in the western part of the United States lost communications with several remote power generation sites. The loss was intermittent, lasting a few minutes before communications were restored, then failing again for a few minutes. An investigation revealed that the utility’s Internet-facing firewalls were rebooting in response to malicious actors attempting to exploit a known vulnerability in that specific type of firewall. Patching the firewalls with updated software corrected the issue. While the generation and transmission of electricity was not disrupted, the utility did not have full control over its assets for about 10 hours. There is no evidence that this incident was a targeted attack, and in fact the attackers may have been scanning the Internet looking for vulnerable firewalls

___________________

¹¹⁵ See J.D. Morris, 2020,“PG&E Error at Power Plant May Help Explain California’s Rolling Blackouts,” San Francisco Chronicle, September 14, https://www.sfchronicle.com/business/article/PG-E-error-atpower-plant-may-help-explain-15567028.php; PG&E, 2021, “Learn About Public Safety Power Shutoff (PSPS) Events,” https://www.pge.com/en_US/residential/outages/public-safety-power-shuttoff/learnabout-psps.page.

and were unaware that they had successfully degraded the operations of an electric utility.

In 2015, Ukraine suffered distribution outages resulting from a successful cyberattack. Three power distribution substations were disabled via unauthorized remote access into a utility control center. An extensive investigation by U.S. and Ukrainian experts revealed that the attackers had successfully obtained usernames and passwords of the utility’s employees 6 months prior to the attack via phishing e-mails. A second similar attack on parts of Ukraine’s electrical grid occurred in 2016. In 2020, the U.S. government publicly identified the attackers as members of the Russian military.

The electric grid functions at three main layers—generation, transmission, and distribution. Disruptions in any of these layers can cause loss of power to military installations and other customers. Historically, the most frequent causes of power disruptions at the generation and transmission layers are related to weather or natural events. At the distribution level, an outage is more likely to be caused by either lightning or animals such as squirrels, birds, or snakes getting in the wires and causing a short, which trips a breaker. Cyberattacks against the grid are a constant threat, but to date the number of successful attacks that resulted in power loss to a customer are exceptionally low.

Finding 47: The reality of cyberattacks against energy infrastructure is that the number and severity of attacks are growing. Even if not aimed directly at the energy infrastructure, the effects of a cyberattack may affect it adversely.