Implications of Artificial Intelligence–Related Data Center Electricity Use and Emissions Proceedings of a Workshop (2025) / Chapter Skim
Currently Skimming:

8 Efficiency Through Technology Advancement: HardwareSoftware Interactions
Pages 72-84

The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.


From page 72...
... , and Dally and other panelists explored the software strategies required to achieve high performance and efficiency on these architectures in an open discussion. KEYNOTE PRESENTATION Dally highlighted how AI developments have accelerated energy demands and described some of the factors that may influence this trajectory in the future.1 While huge efficiency gains have been achieved -- today's AI chips are upward of 1,000 times more efficient than their predecessors -- some experts subscribe to Jevons paradox, that more efficiency will just create more demand.
From page 73...
... In addition, structured sparsity schemes make algorithms much more efficient while preserving accuracy, although more research is needed to build effective sparse matrix graphics processing unit (GPU) hardware.3,4 NVIDIA's current GPU has 20 PetaFLOPS available for inference operations, providing state-of-the-art efficiency and performance.
From page 74...
... improved. Structured sparsity also improves neural networks through pruning and retraining.6 As a result of all of these developments, between 2012 and 2024, it is estimated that single-chip inference performance increased by 5,000-fold, AI hardware efficiency increased by 1,250-fold, and AI software efficiency increased by more than 1,000-fold, Dally said.
From page 75...
... Andrew Chien, University of Chicago, asked how NVIDIA was making GPUs efficient during the AI training phase, especially as size and capacity increases. Dally replied that it is harder to measure training efficiency than inference efficiency, and therefore harder to reduce the precision on training, but NVIDIA's chips have no loss of accuracy for inference.
From page 76...
... The panelists were Miloš Popovic, Ayar Labs; Eilam; Valerie Taylor, Argonne National Laboratory; Vivienne Sze, Massachusetts Institute of Technology; and Dally. Improving Artificial Intelligence Infrastructure Efficiency with Optical Input-Output AI processors have made great gains in density, but these architectures are currently limited by data bandwidth and data movement.
From page 77...
... Finally, optics need very little energy to move data when they are close to and deeply integrated with computing elements. Current processors that use optical interconnection have both efficiencies and limitations.10 To make progress, Ayar Labs scientists have experimented with optically interconnected GPUs and divided workloads and found a 20-fold improvement in energy efficiency, especially when implementing parallel programming across multiple processors.
From page 78...
... Hardware trends include innovating memory access to reduce data movement, using mixed- and low-precision computations, and considering biologically inspired hardware.13 Software trends, such as LLM inference benchmarks, explore different open source LLMs, AI accelerators, and inference frameworks for improved efficiency.14 11 C Murray, S
From page 79...
... A Decade of Efficiency Gains Dally highlighted some of the history he shared in his keynote presentation regarding drivers of increasing efficiency in AI systems over the past decade. GPUs are increasingly energy efficient thanks to innovations in number representation, complex instructions, and sparsity.
From page 80...
... Popovic noted that optical interconnection has been possible for more than a decade, but it was not used or useful until it gained high-volume capabilities, and its manufacturing ecosystem still needs 2–3 years to develop for it to have a large-scale impact. Dally agreed, noting that many new technologies start this way -- several breakthroughs are needed before an innovation reaches a stage where it can be easily built; is energy efficient; and has an obvious use, a ready audience, and no viable alternatives.
From page 81...
... Reid Lifset, Yale University, asked about the potential for reuse of AI hardware. Dally replied that outdated hardware should be retired if the goal is to improve energy efficiency.
From page 82...
... Instead, he suggested using the ratio of total power versus total computing as measured in OPS, not watts. Laura Gonzalez Guerrero, Clean Virginia, asked what data centers can do today to avoid increasing fossil fuel use.
From page 83...
... She identified a need for more investment in hardware–software co-design, especially algorithm innovation in neural network architecture. AI also has the potential to accelerate renewable energy production and use if computing can fully leverage its flexibility.


This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More information on Chapter Skim is available.