National Academies Press: OpenBook

Large Language Models and Cybersecurity: Proceedings of a Workshop—in Brief (2024)

Chapter: Large Language Models and Cybersecurity: Proceedings of a Workshop - in Brief

Suggested Citation:"Large Language Models and Cybersecurity: Proceedings of a Workshop - in Brief." National Academies of Sciences, Engineering, and Medicine. 2024. Large Language Models and Cybersecurity: Proceedings of a Workshop—in Brief. Washington, DC: The National Academies Press. doi: 10.17226/27776.
×
images Proceedings of a Workshop—in Brief

Large Language Models and Cybersecurity

Proceedings of a Workshop—in Brief


On August 31 and September 1, 2023, the National Academies of Sciences, Engineering, and Medicine held a workshop in conjunction with a meeting of its Forum on Cyber Resilience aimed at better understanding recent developments with large language models (LLMs) and their implications for cybersecurity and resilience. Presentations addressed how LLMs are constructed and function, how industry is considering using LLMs generally and for cybersecurity, safeguards that aim to limit LLM outputs deemed harmful and techniques for circumventing them, and, more generally, the trustworthiness of LLMs and their integration into cybersecurity offense and defense. The agenda and recordings of the presentations and discussions from the workshop are available online.1

HOW LARGE LANGUAGE MODELS WORK AND WHY THEY ARE INTERESTING

Several speakers provided background on how LLMs work and some promising applications. Caiming Xiong, Salesforce, introduced language models’ approach of taking in a list of words, given as a prompt, and attempting to predict the word that follows. Peter Grabowski, Google, explained that the success of this prediction is, in part, dependent on the corpus upon which a model is trained.

A larger corpus will offer the model more context and information to support accurate word prediction. LLMs can use as many as hundreds of billions of parameters to represent that training data. Grabowski defined parameters for Bayesian language models as an entry in the dictionary or, in terms of neural networks, as a weight in the network. Recent approaches promise to be more efficient and use less parameters. Grabowski and William Pearce, Google, went further to highlight that embedding allows for greater semantic complexity. Embedding is a process that assigns individual words a geometric position in a high-dimensional, abstract space that corresponds to their semantic meaning. This approach, Grabowski and Pearce said, then allows machines to use vectors to calculate each word’s meaning and relevance. As such, the context in which an input is placed heavily influences the probability weight of the following token.2

Xiong listed grammar, lexical semantics, world knowledge, sentiment analysis, translation, special reasoning, and mathematics as just a few of the skills LLMs can accomplish through sophisticated word prediction. The goal of an LLM is to predict tokens, one after the other, in a sequence that produces a compelling output.

__________________

1 National Academies of Sciences, Engineering, and Medicine, “Summer 2023 Meeting of the Forum on Cyber Resilience,” August 31–September 1, 2023, https://www.nationalacademies.org/our-work/forum-on-cyber-resilience#sectionPastEvents.

2 A token is the equivalent to an item in a list. Just as a word would be considered a token, a series of characters without a space would also be considered a token.

Suggested Citation:"Large Language Models and Cybersecurity: Proceedings of a Workshop - in Brief." National Academies of Sciences, Engineering, and Medicine. 2024. Large Language Models and Cybersecurity: Proceedings of a Workshop—in Brief. Washington, DC: The National Academies Press. doi: 10.17226/27776.
×

Those developing LLMs hope to optimize this prediction to ensure an output that is beneficial to the user, even with a smaller corpus. Xiong demonstrated some current applications of LLMs, such as integration into a chat interface that offers edit and content addition suggestions for the user to review and verify. LLMs can also be used to summarize meeting transcripts and automatically identify action items.

Dan Guido, Trail of Bits, began his presentation by outlining some current applications of LLMs, observing that it is difficult to establish precisely what LLMs can and cannot do because there is a diversity of views on how to define and interpret success. For example, an LLM may be able to obtain a high score on the Law School Admission Test because of the data it was trained on, but that does not mean the LLM can perform the functions of a lawyer. Guido suggested that one can obtain a more grounded view of LLM capabilities by being more precise about the desired goals. Turning to cybersecurity applications, Guido noted that his company, Trail of Bits, has successfully used LLMs to “decompile” machine-level code back into high-level programming languages, identify and trigger bugs, reason about memory layouts, write scripts to launch exploits, identify weak encryption, and find cases where cryptographic application programming interfaces are being called incorrectly. Guido noted that these actions facilitated by LLMs were completed with careful prompting rather than automatic functions but could still improve efficiency in completing such tasks.

Adam Troy, Microsoft, and Jayesh Govindarajan, Salesforce, highlighted ways LLMs can already boost knowledge worker productivity by assisting in writing emails, text documents, and presentations. Govindarajan was optimistic that LLMs can help streamline preparing and responding to emails but cautioned that Salesforce’s customers have found shortcomings in LLM-prepared emails, not all of which can be addressed through changes in how users interact with LLMs. Salesforce is working on building feedback loops to make the output more accurate—work that will require the development of effective evaluation techniques. Galen Hunt, Microsoft, and a member of the Forum on Cyber Resilience, highlighted the opportunity for boosting the productivity of programmers, observing that Microsoft is currently applying LLMs to convert legacy code to more modern programming languages that offer such benefits as memory safety.

Along with productivity, discussions considered user requirements and cost-effectiveness. Grabowski observed that as LLMs become more widely used, their developers are starting to use requirements engineering to better understand and meet user needs. Govindarajan noted that evaluations of whether an LLM might support an application must consider the cost of training and executing the model. State-of-the-art LLMs may not be appropriate in all applications, because costs may exceed benefits for some tasks. However, even an expensive model may have cost-effective impact in more critical contexts such as health care. Furthermore, LLMs may not be cost-effective options for certain tasks, according to Govindarajan. LLMs promise to improve programmer productivity, but those savings will be offset if it turns out to be too expensive to review or verify that code.

Grabowski and Pearce emphasized that the output of LLMs can be influenced by how the model is designed and the data that an LLM is trained on. Even though many have a similar chatbot interface, models can vary significantly in terms of their detailed structure and the data used to train them. For example, Grabowski suggested that if standard American English dominates in their training data, models will tend to produce the highest quality outputs if prompted using that language.

WHAT IMPACTS THE TRUSTWORTHINESS OF LARGE LANGUAGE MODELS?

Workshop panelists described several potential sources of unreliable outputs from LLMs. Grabowski and Pearce demonstrated how more carefully constructed prompts allow models to generate more useful output, using the example of a chatbot asked to make a recommendation for dinner. Offering the LLM a prompt directly requesting food recommendations results in an unintelligible response. Preceding the user’s query with text telling the model that it should behave as a chatbot in effect nudges the model into considering training data that resemble a chat, improving the output but still not providing useful recommendations. Preceding the user’s query with a series of prompts that tell the chatbot that it is a chatbot and providing details of what a useful food recommen-

Suggested Citation:"Large Language Models and Cybersecurity: Proceedings of a Workshop - in Brief." National Academies of Sciences, Engineering, and Medicine. 2024. Large Language Models and Cybersecurity: Proceedings of a Workshop—in Brief. Washington, DC: The National Academies Press. doi: 10.17226/27776.
×

dation might look like yielded much better results. Such successive refinement of contextual prompts nudges models to draw on more helpful examples in their training data, Pearce suggested.

Xiong turned to other trustworthiness challenges with LLMs. He observed that there have been multiple manifestations of biased outputs that can be traced to bias in the data used to train the models. He also highlighted the problem of unfaithful or nonsensical outputs that are sometimes dubbed “hallucinations.” The term itself is controversial; Troy noted that Microsoft decided to avoid the term because it suggests that models are sentient and it is a term that implies mental illness. Govindarajan defined the problem as the output of a purported fact that is easily verified to be false. Troy noted that one approach that is currently being used to combat the problem is to provide citations. Other possible approaches include developing affordances to provide models with feedback on their mistakes and adjusting the amount of randomness (sometimes called the “temperature” of the model) to lower the probability of producing nonverifiable information. With respect to the latter, Troy noted that there are tradeoffs with the creativity displayed by the models. Govindarajan offered that one way that creativity comes into play is when there are many possible correct outputs.

Panelists suggested several other methods for improving the quality of LLM output. Troy pointed to chain of thought, a prompting technique that encourages the LLM to build its output using a structured approach. Pearce similarly noted that breaking down prompts step by step to more carefully construct the result will improve the output. Retrieval augmented generation was mentioned by both Troy and Grabowski. For example, the mathematical output of a model can be improved by telling the model that it can access a calculator external to the model. Govindarajan referenced two recently published articles to show current work toward registering functions and actions within an LLM that could improve model output.3,4

Troy discussed Microsoft’s current efforts to mitigate risk with humans in the loop. Currently, humans must be in the loop for all high-risk situations. It might be possible to reduce the burden on humans, Govindarajan and Troy suggested, by allowing a model to define all of the steps needed to accomplish a task without oversight but require a human to initiate execution of those steps. To account for risk, finding applications that do not need humans within the loop may be a gradual process. Govindarajan offered that one might focus on whether an action is reversable or not reversable to ascertain if it could be completed by current LLMs without human oversight, although this would not avoid such risks of getting into a loop that would result in redoing an action unnecessarily. Pearce suggested that user experience research aimed at better understanding expectations and designing based on those findings will help bring us closer to using LLMs without direct human oversight. Parisa Tabriz, Google, noted that one of the challenges in considering user expectations is that they change according to context and over time.

The discussion turned from humans in the loop to how organizations adjust when things go wrong. Troy observed that Microsoft has the most experience with Bing, where a team that is responding day by day to “jailbreaks” (instances where users induce the chatbot to produce outputs deemed harmful) and user feedback. Adjusting to user feedback can involve changing the temperature or meta prompt. To offer users leeway, Bing has three modes that allow the users to control how creative versus exact they want the output to be. Tabriz said that Google uses sandboxes, in which select users can interact with newer versions of a chatbot, to collect user feedback. More broadly, extensive review and a governance process are needed to successfully introduce a new LLM to a search engine or other product. Fortunately, there is considerable experience with governance that can be drawn on with LLMs.

Standardized approaches to evaluation and management will be an ongoing conversation. Grabowski offered that he would like to see guidance on the considerations for using models, which can then be iterated upon. Tabriz noted that she looks forward to when developers, researchers, and regulators are constantly in conver-

__________________

3 S. Yao, J. Zhao, D. Yu, et al., 2022, “React: Synergizing Reasoning and Acting in Language Models,” arXiv preprint arXiv:2210.03629.

4 T. Schick, J. Dwivedi-Yu, R. Dessì, et al., 2024, “Toolformer: Language Models Can Teach Themselves to Use Tools,” Advances in Neural Information Processing Systems 36.

Suggested Citation:"Large Language Models and Cybersecurity: Proceedings of a Workshop - in Brief." National Academies of Sciences, Engineering, and Medicine. 2024. Large Language Models and Cybersecurity: Proceedings of a Workshop—in Brief. Washington, DC: The National Academies Press. doi: 10.17226/27776.
×

sation. Govindarajan highlighted that the development of a good evaluation framework within Salesforce took time and iteration. Moreover, it is still evolving and will continue to improve. He hopes that regulators can also create a friendly set of guardrails that can grow to high standards.

DISCUSSIONS REGARDING THE SECURITY OF LARGE LANGUAGE MODELS

Govindarajan said that in his view, LLMs present a new class of security risks not previously encountered by industry. Tabriz observed that despite the risks, developers are working hard to finding ways to use them securely; for example, by identifying the new threats and finding ways to mitigate them, helping ensure that the benefits outweigh the harms.

Guido cautioned against trusting the outputs of an LLM not only because, as noted earlier, they rely on training data that may be incorrect or contain gaps but also because the training data are susceptible to tampering. Recent research results show that even individuals without significant resources can poison datasets used by open source and commercial LLMs.5 Guido provided some examples. First, a research group demonstrated that one can identify when a particular crawl of the Internet is used to capture data for model training, replace content on Wikipedia with hidden instructions, and remove the data from Wikipedia once the crawl is finished—making the tampering undetectable.6 Interestingly, because training order matters, it is possible to poison a dataset by simply reordering data. There has also been research regarding the creation of back doors by hiding key words in models that can be used by attackers to trigger the desired response.7 Guido observed that much work lies ahead for defenders to identify and respond to such threats.

Zico Kolter, Carnegie Mellon University and Bosch Center for Artificial Intelligence, described how the basic elements of LLM construction—scraping data from the Web, building a model to determine word probability, and creating a chatbot that returns output based on model prediction—lead to security risks in particular contexts. One context to consider is that creators of LLM-based chatbots frequently want to shape the bots’ output to fit within acceptable bounds. Alignment is the process of additional training using prompts that require a specific response together with the desirable response. For example, an aligned chatbot that is asked how to build a bomb will be much more likely to provide a canned refusal than to offer the requested information.

If a model has access to information that should not be disclosed, it can disclose supposedly safeguarded information given the correct prompting. Pearce showed this through a demo where ChatGPT was asked to generate fake logs to simulate someone breaking into a network. Initially, the chatbot declined the request as it had safeguards in place that encouraged it to avoid illegal activity. However, by explaining that the request hoped to generate interview questions using the fake logs, the user was able to nudge the chatbot into fulfilling the request.

Grabowski noted a couple of methods currently used to safeguard the type and amount of information chatbots can disclose other than alignment. One method mentioned was “segmenting” the training data to exclude content that should not be used. The second method was using an LLM as a component within a larger pipeline for a nested approach. An example of this is known as “constrained decoding.”8

With respect to segmenting, Grabowski cautioned that large corpuses will not always be effectively cleaned, and that such cleaning cannot yet be automated. A point of concern in respect to alignment, noted by Kolter, is that the information that the chatbot is retrained to not communicate will still exist within its corpus.

Adding context to a prompt to circumvent a safeguard may not always work, but there are other ways to “break through” alignment. Kolter described research by his group at Carnegie Mellon University on circumventing LLM safeguards through adversarial attacks.9 He started

__________________

5 N. Carlini, M. Jagielski, C.A. Choquette-Choo, et al., 2023, “Poisoning Web-Scale Training Datasets Is Practical,” arXiv preprint arXiv:2302.10149.

6 I. Shumailov, Z. Shumaylov, D. Kazhdan, et al., 2021, “Manipulating SGD with Data Ordering Attacks,” Advances in Neural Information Processing Systems 34:18021–18032.

7 N. Kandpal, M. Jagielski, F. Tramèr, and N. Carlini, 2023, “Backdoor Attacks for In-Context Learning with Language Models,” arXiv preprint arXiv:2307.14692.

8 A method where a second algorithm reviews the responses generated by the LLM and limits the output that can be generated.

9 A. Zou, Z. Wang, J.Z. Kolter, and M. Fredrikson, 2023, “Universal and Transferable Adversarial Attacks on Aligned Language Models,” arXiv preprint arXiv:2307.15043.

Suggested Citation:"Large Language Models and Cybersecurity: Proceedings of a Workshop - in Brief." National Academies of Sciences, Engineering, and Medicine. 2024. Large Language Models and Cybersecurity: Proceedings of a Workshop—in Brief. Washington, DC: The National Academies Press. doi: 10.17226/27776.
×

by showing how a specific, human-unintelligible string could be used to bypass chatbot safeguards. This string was found by repeatedly attempting to bypass alignment in open source LLMs by measuring changes in the probability of obtaining a malicious response as the prompt was modified. Kolter presented data showing that several widely used open source and closed source LLMs had significant vulnerability to this attack.

One open question is how the same string could be used as an exploit in several different models despite each model having different training, architecture, and initial model weights. Kolter hypothesized that this stems from commonalities in the training corpuses used for each model that create common patterns across the models. Commenting later on the nature of such attack strings, Pearce said that we do not currently understand how data are synthesized by LLMs. Building on this, Mike Walker, Microsoft, pointed to other work showing that LLM chatbots establish a sort of machine-only cipher communication.10 Walker, noting that security can be thought of as the absence of surprise one cannot mitigate, observed that poor understanding of how LLMs synthesize data contributes to uncertainty that can be exploited, as illustrated in the work of Kolter and his coauthors.

Guido echoed the sentiment that reinforcement learning should not be viewed as an effective security control. For example, an LLM granted access to specific information within an organization could be manipulated to perform malicious actions. Grabowski cautioned that he would never train a current model with data he did not want it to repeat. If it is higher risk, we might not have the security to integrate LLMs as they are now. Indeed, added Kolter, the current trend toward integrating LLMs into workflows raises questions as to whether security concerns can adequately be mitigated through measures such as retraining. His research group released its code and adversarial prompt examples to help raise awareness of these issues.11

Another possibility for mitigating this vulnerability would be to identify and filter malicious prompts. Govindarajan contrasted filtering LLM prompts with ongoing efforts to filter spam emails. For the latter, evaluation criteria and systems are in place to evaluate effectiveness, and there is a considerable body of experience to draw on. By contrast, LLMs are very new, and there is limited experience about how to develop and refine systems to filter bad inputs. Indeed, Troy indicated that Microsoft is “constantly chasing the next jailbreak” and suggested that it is not yet clear how successful filtering efforts will be. According to Troy, OpenAI, Microsoft’s foundation model provider, is attempting to improve the situation by training its models to more closely follow system instructions and behave more carefully in responding to user prompts.

Guido stated that the output of an artificial intelligence (AI) model is all untrustworthy and if it is not treated that way then attackers can embed data deep into your application that will go unchecked. There are risks of model poisoning, prompt injection, lack of data privacy, incorrect code execution, and data leakage, to name a few.

USING LARGE LANGUAGE MODELS FOR CYBERSECURITY

Several panelists discussed the potential for using LLMs for defensive purposes. Walker observed that LLMs can be a force multiplier for cyber workforces. Pearce illustrated both the potential as well as the limitations of using a demonstration in which ChatGPT solves a security engineering interview question. After being given a network log excerpt and the right prompt, ChatGPT identified patterns or anomalies while denoting potential security concerns, listed solutions to mitigate the given threat, and suggested new security rules to enhance network security. Based on the work Pearce and his team has done, Pearce said that LLMs show promise in applying knowledge about Internet protocols (e.g., TCP and UDP) and rules used for intrusion detection (e.g., Suricata) to the evaluation of security threats. Grabowski noted that this work could help network security engineers recall information about known vulnerabilities but, at least at present, cannot automate security engineering work.

Current evaluations of LLMs tend to focus on their use to support software development rather than cybersecurity, said Guido, leaving a gap in our understanding of capabilities and limitations. A major area of research for

__________________

10 Y. Yuan, W. Jiao, W. Wang, et al., 2023, “GPT-4 Is Too Smart to Be Safe: Stealthy Chat with LLMs via Cipher,” arXiv preprint arXiv:2308.06463.

11 Zou et al., 2023, “Universal and Transferable Adversarial Attacks on Aligned Language Models.”

Suggested Citation:"Large Language Models and Cybersecurity: Proceedings of a Workshop - in Brief." National Academies of Sciences, Engineering, and Medicine. 2024. Large Language Models and Cybersecurity: Proceedings of a Workshop—in Brief. Washington, DC: The National Academies Press. doi: 10.17226/27776.
×

Trail of Bits, he noted, is mapping of all of the tasks of a cybersecurity analyst, whether for offense, defense, or operations, and constructing a heat map assessing how good AI is at performing those tasks. The idea, Guido added, is to provide a roadmap for where AI can be used to the greatest effect.

Guido used a chart (Figure 1) to illustrate where LLMs appear to be best suited for cybersecurity work. For example, AI is especially effective for tasks that require broad knowledge and where mistakes are acceptable. Attackers currently have an advantage, but attacks require higher levels of accuracy. Defense-oriented cybersecurity can benefit more from AI with the right investment.

Brendan Dolan-Gavitt, New York University, presented case studies from work at his institution on using LLMs for security tasks such as finding bugs, deobfuscating, reverse engineering, and fixing vulnerabilities. Thus far, LLMs have been used successfully to create compelling demonstrations but systematic evaluation does not consistently produce strong results.

Dolan-Gavitt gave an example of a compelling demonstration: using ChatGPT to repair source code containing vulnerability CVE-2023-40296 from the National Institute of Standards and Technology’s National Vulnerability Database. Although the vulnerability was published in August 2023, well after the cutoff for ChatGPT’s training data, ChatGPT nevertheless correctly identified a buffer overflow in the code. Prompted to correct the vulnerability, ChatGPT did so in a reasonable fashion, although not in precisely the same way as the developers who originally patched the bug.

Aspects of cybersecurity that can be influenced by generative artificial intelligence (AI), analyzed through the lenses of benefits, constraints, knowledge needed, and the purpose of interest
FIGURE 1 Aspects of cybersecurity that can be influenced by generative artificial intelligence (AI), analyzed through the lenses of benefits, constraints, knowledge needed, and the purpose of interest.
SOURCE: Dan Guido, Trail of Bits, presentation to the workshop on August 31, 2023.

Dolan-Gavitt then turned to a larger, more systematic study of vulnerability repair using LLMs.12 Twelve real-world source code files with known vulnerabilities were tested using 432 combinations of models and prompts. An ensemble of LLMs was used to carry out repairs. The ensemble of models appeared to have been able to repair 8 of the 12 files. However, on closer inspection, the success rate turned out to have been only 6 of the 12 files, with some models failing to properly fix half of the files they had reported to have been fixed—for example, by breaking the functionality of the program. When Dolan-Gavitt’s team expanded its systematic analysis of LLMs fixing vulnerabilities to more than 2,000 scenarios it measured a success rate of around 20 percent.

Dolan-Gavitt also presented a demonstration of reverse engineering the application TurboTax using ChatGPT-4 to resolve multiple program crashes. Using the LLM, he was able to get instructions for how to use a debugger and obtain a backtrace, input the backtrace to identify problematic code, decompile that code, and identify and fix the problem. However, a more systematic evaluation of reverse engineering using LLMs found only a 53 percent accuracy rate in answering true/false questions about programs containing flaws.

Dolan-Gavitt pointed to three key reasons for discrepancies between the demonstrations and the results of systematic review. For one, early demos tended to be easier problems to solve than those tested in the systematic review. Demos also used the more current ChatGPT-4 while the systematic analyses were performed using the ChatGPT-3 and Codex models. Finally, the demos were facilitated by an expert human in the loop, which allowed for clearer direction and optimized interactions.

Dolan-Gavitt stated that current LLMs have not been specifically trained to perform security tasks so the results of the systematic analyses can still be viewed optimistically. LLMs are quickly improving, and it has become cheaper and easier to fine tune them with domain-specific data. Nevertheless, one must be care-

__________________

12 H. Pearce, B. Tan, B. Ahmad, R. Karri, and B. Dolan-Gavitt, 2023, “Examining Zero-Shot Vulnerability Repair with Large Language Models,” pp. 2339–2356 in 2023 IEEE Symposium on Security and Privacy (SP), https://doi.org/10.1109/SP46215.2023.

Suggested Citation:"Large Language Models and Cybersecurity: Proceedings of a Workshop - in Brief." National Academies of Sciences, Engineering, and Medicine. 2024. Large Language Models and Cybersecurity: Proceedings of a Workshop—in Brief. Washington, DC: The National Academies Press. doi: 10.17226/27776.
×

ful to avoid overestimating capabilities based on a small number of successful demonstrations. Guido also addressed the challenge of evaluation, for which a starting point would be identifying risk-changing capabilities. These might include identifying synergies with existing tools, determining the acceptability and visibility of mistakes, and how rapidly LLM facilitated actions can improve.

Yan Shoshitaishvili, Emotion Labs and Arizona State University, detailed a range of cybersecurity-related applications of LLMs. He suggested that many questions are not unique to LLMs and apply to digital tools more generally. Shoshitaishvili pointed to the experience of the Defense Advanced Research Projects Agency’s Cyber Grand Challenge. Broadly speaking, he said, the challenge showed the promise of automated cybersecurity techniques but also showed that the techniques tended to be brittle. There was an array of reasons for the automation having been brittle, but the root of the issue was how the teams approached using automation to achieve their goals. Within the automation, there was a lack of computer systems semantics understanding and a lack of reasoning when applying a program within a given context. Compared to this experience, Shoshitaishvili asserted that LLMs have the capability of performing expert-level cybersecurity tasks and offering the contextual guidance needed to appropriately apply a solution to a given problem.

Shoshitaishvili described research on integrating LLMs into teams participating in cybersecurity competitions. One approach is to add the LLM as another participant in the collaborative discussion, which allows it to offer suggestions such as which parts of a piece of code to focus on or how to semantically interpret some binary code. This combined human and LLM team has been used in multiple competitions. Shoshitaishvili observed that the human team members generally find the addition of an LLM to be satisfactory but that the LLMs make numerous mistakes; in at least one case completely ignoring a vulnerability. Another approach, Shoshitaishvili stated, is to use an LLM and a human-assisted cyber reasoning system in combination to lead the work of a team of human hackers.13 This approach yielded promising results that suggest that LLMs may be effective for analysis, planning, and task dispatching. Shoshitaishvili’s group also looked at using an LLM-assisted cyber reasoning system,14 which appears to be another approach meriting further exploration.

Finally, Shoshitaishvili has explored using LLMs as a cybersecurity tutor to help close the current security skills gap. The premise is that successful training requires a level of hands-on teaching that exceeds the capacity of available instructors. LLMs, by contrast, can be deployed at the scale necessary to offer students individualized instruction. Shoshitaishvili and a collaborator currently host an educational platform that supports students in learning about and practicing core cybersecurity concepts.15 Hundreds of students used the platform in the past, but many left—possibly because they hit a wall without access to an instructor. To combat this, his team designed a chatbot to imitate the role of a tutor by answering questions and providing advice to students in real time. They have found that this integration has helped students stay engaged and learn new skills.

LOOKING TO THE FUTURE

Several workshop participants commented about the future of LLMs and their applications. Troy underscored the importance of breaking out of prior mental models for applying AI in new applications. For example, he continued, LLMs are good at producing compelling text, a task in which prior AI applications faltered, but at least so far have proven bad in some areas where other AI applications have exceled, such as mathematics. Pearce added that LLM developers are currently making huge strides—but down a narrow path. In the future, he expects many more applications will emerge and user expectations will also evolve with more experience.

Walker predicted that plugins will be an important point of growth. For example, an LLM in collaboration with a Wolfram Alpha plugin can more precisely and accurately answer questions that require calculations. Giving LLMs

__________________

13 For information on human-assisted cyber reasoning systems, see Y. Shoshitaishvili, M. Weissbacher, L. Dresel, et al., 2017, “Rise of the HaCRS: Augmenting Autonomous Cyber Reasoning Systems with Human Assistance,” pp. 348–362 in CCS ’17: Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, https://doi.org/10.1145/3133956.3134105.

14 Cyber reasoning systems are “fully autonomous machines capable of identifying, exploiting, and patching vulnerabilities in binary code” (Shoshitaishvili et al., 2017, p. 348).

15 See Arizona State University’s pwn.college website at https://pwn.college, accessed December 20, 2023.

Suggested Citation:"Large Language Models and Cybersecurity: Proceedings of a Workshop - in Brief." National Academies of Sciences, Engineering, and Medicine. 2024. Large Language Models and Cybersecurity: Proceedings of a Workshop—in Brief. Washington, DC: The National Academies Press. doi: 10.17226/27776.
×

access to powerful domain-specific tools may unlock new opportunities to use LLMs for a variety of applications.

Multiple participants offered remarks on the future of LLMs as they pertain to cybersecurity. Walker noted that newer models such as ChatGPT-4 greatly outperformed its predecessors by some measures, raising expectations that future models may be even more powerful in ways that bear on cybersecurity. Walker also cautioned that the novelty of LLMs not only means their vulnerabilities are not fully understood but also that attackers have had little time to explore potential uses—meaning there is much uncertainty about both defensive and offensive applications.

In the immediate term, Guido predicted that LLMs will disrupt the cybersecurity technology landscape in such areas as bug bounties, phishing training, signature-based defenses, disinformation detection, and attacker attribution. LLMs, for example, could make it possible to submit many more bug reports to firms that offer bounties. LLMs may also make it easier for attackers to create personalized phishing messages, customize exploits to evade signature-based detection, or create false personas to accelerate the spread of disinformation. For defenders, LLMs may provide new analytical tools for better attributing attacks. Guido also cautioned that other disruptive technologies may appear, such as more capable forms of autonomous malware.

Several panelists spoke about the cybersecurity implications of changes in the way that LLMs are developed and distributed. Guido noted that open source models are currently rapidly improving and being created around the world. These models can be specialized in the future in ways that pose a cybersecurity threat. Malicious actors are already aware of this and developing tools accordingly. Mitigating malicious training is a current focus of research. Troy observed that current LLMs learn from a static corpus but that there is interest in having chatbots that learn over time from conversations—a development that would have several cybersecurity and trustworthiness implications.

Walker warned about a variety of social engineering attacks that further development of LLMs will enable. For example, he hypothesized that the development of models fluent in less widely spoken languages will mean new populations will be exposed to language-specific social engineering attacks. On the other hand, Walker referenced an article describing the use of LLMs to deter telemarketers, which suggests that the models could also be used to combat social engineering actors.16

Participants expressed excitement regarding the future use of LLMs for tasks humans currently do. Pearce hoped to see LLMs performing better than humans in specific contexts. Humans are not perfect, said Pearce, so it is possible for LLMs to outperform us. This would be a big impact of specialized models. Grabowski reinforced that the most interesting use cases to investigate are where the model performs better than humans. A future goal that will take more iteration, mentioned by Pearce, would be having LLMs interact with other models and bringing in the human later down the line. According to Grabowski, the organizations and individuals who will be successful in crafting specialized systems will have creatively explored applications of LLMs while being transparent with users. Even in the case of using LLMs for tasks humans do, Grabowski emphasized that some situations may still use humans in the loop, depending on the complexity of the issue.

__________________

16 R. McMillan, 2023, “People Hire Phone Bots to Torture Telemarketers,” Wall Street Journal, June 29, https://www.wsj.com/articles/people-hire-phone-bots-to-torture-telemarketers-2dbb8457.

Suggested Citation:"Large Language Models and Cybersecurity: Proceedings of a Workshop - in Brief." National Academies of Sciences, Engineering, and Medicine. 2024. Large Language Models and Cybersecurity: Proceedings of a Workshop—in Brief. Washington, DC: The National Academies Press. doi: 10.17226/27776.
×

DISCLAIMER This Proceedings of a Workshop—in Brief was prepared by Nneka Udeagbala as a factual summary of what occurred at the workshop. The statements made are those of the rapporteur or individual workshop participants and do not necessarily represent the views of all workshop participants; the planning committee; or the National Academies of Sciences, Engineering, and Medicine.

PLANNING COMMITTEE MEMBERS John L. Manferdelli (Chair), VMware; Fred Schneider (NAE) (Chair Emeritus) Cornell University; Yair Amir, Johns Hopkins University; Maritza Johnson, University of San Diego; and Brian LaMacchia, Microsoft.

FORUM ON CYBER RESILIENCE MEMBERS John Manferdelli (Chair), VMWare; Yair Amir, Johns Hopkins University; Steven Bellovin, Columbia University; Thomas Berson, Salesforce; Nadya Bliss, Arizona State University; Timothy Booher, Boeing; Srini Devadas, Massachusetts Institute of Technology; Curtis Dukes, Center for Internet Security; Kristen Eichensehr, University of Virginia; Paul England (NAE), Microsoft; Alexander Gantman, Qualcomm Technologies; Melissa Hathaway, Hathaway Global Strategies; Galen Hunt, Microsoft; Maritza Johnson, University of San Diego; Brian LaMacchia, Farcaster Consulting Group; John Launchbury, Galois; Dave Levin, University of Maryland; Damon McCoy, New York University; James Miller, Adaptive Strategies; Andy Ozment, Capital One; Ari Schwartz, Venable; Parisa Tabriz, Google.

STAFF Tho Nguyen, Senior Program Officer; Nneka Udeagbala, Associate Program Officer; Jon Eisenberg, Senior Director, Computer Science and Telecommunications Board; and Shenae Bradley, Administrative Assistant.

REVIEWERS To ensure that it meets institutional standards for quality and objectivity, this Proceedings of a Workshop—in Brief was reviewed by Josiah Dykstra, Trail of Bits; Galen Hunt, Microsoft; and Maritza Johnson, Good Research. Katiria Ortiz, National Academies of Sciences, Engineering, and Medicine, served as the review coordinator.

SPONSORS This Proceedings of a Workshop—in Brief was supported by the National Institute of Standards and Technology and the National Science Foundation.

This proceedings was prepared using federal funds under award 60NANB22D156 from the U.S. Department of Commerce, National Institute of Standards and Technology. The statements, findings, conclusions, and recommendations do not necessarily reflect the views of the National Institute of Standards and Technology or the U.S. Department of Commerce.

Any opinions, findings, conclusions, or recommendations expressed do not necessarily reflect the views of the National Science Foundation.

For additional information regarding the workshop, visit https://www.nationalacademies.org/our-work/forum-oncyber-resilience#sectionPastEvents.

SUGGESTED CITATION National Academies of Sciences, Engineering, and Medicine. 2024. Large Language Models and Cybersecurity: Proceedings of a Workshop—in Brief. Washington, DC: The National Academies Press. https://doi.org/10.17226/27776.

Division on Engineering and Physical Sciences

Copyright 2024 by the National Academy of Sciences. All rights reserved.

images
Suggested Citation:"Large Language Models and Cybersecurity: Proceedings of a Workshop - in Brief." National Academies of Sciences, Engineering, and Medicine. 2024. Large Language Models and Cybersecurity: Proceedings of a Workshop—in Brief. Washington, DC: The National Academies Press. doi: 10.17226/27776.
×
Page 1
Suggested Citation:"Large Language Models and Cybersecurity: Proceedings of a Workshop - in Brief." National Academies of Sciences, Engineering, and Medicine. 2024. Large Language Models and Cybersecurity: Proceedings of a Workshop—in Brief. Washington, DC: The National Academies Press. doi: 10.17226/27776.
×
Page 2
Suggested Citation:"Large Language Models and Cybersecurity: Proceedings of a Workshop - in Brief." National Academies of Sciences, Engineering, and Medicine. 2024. Large Language Models and Cybersecurity: Proceedings of a Workshop—in Brief. Washington, DC: The National Academies Press. doi: 10.17226/27776.
×
Page 3
Suggested Citation:"Large Language Models and Cybersecurity: Proceedings of a Workshop - in Brief." National Academies of Sciences, Engineering, and Medicine. 2024. Large Language Models and Cybersecurity: Proceedings of a Workshop—in Brief. Washington, DC: The National Academies Press. doi: 10.17226/27776.
×
Page 4
Suggested Citation:"Large Language Models and Cybersecurity: Proceedings of a Workshop - in Brief." National Academies of Sciences, Engineering, and Medicine. 2024. Large Language Models and Cybersecurity: Proceedings of a Workshop—in Brief. Washington, DC: The National Academies Press. doi: 10.17226/27776.
×
Page 5
Suggested Citation:"Large Language Models and Cybersecurity: Proceedings of a Workshop - in Brief." National Academies of Sciences, Engineering, and Medicine. 2024. Large Language Models and Cybersecurity: Proceedings of a Workshop—in Brief. Washington, DC: The National Academies Press. doi: 10.17226/27776.
×
Page 6
Suggested Citation:"Large Language Models and Cybersecurity: Proceedings of a Workshop - in Brief." National Academies of Sciences, Engineering, and Medicine. 2024. Large Language Models and Cybersecurity: Proceedings of a Workshop—in Brief. Washington, DC: The National Academies Press. doi: 10.17226/27776.
×
Page 7
Suggested Citation:"Large Language Models and Cybersecurity: Proceedings of a Workshop - in Brief." National Academies of Sciences, Engineering, and Medicine. 2024. Large Language Models and Cybersecurity: Proceedings of a Workshop—in Brief. Washington, DC: The National Academies Press. doi: 10.17226/27776.
×
Page 8
Suggested Citation:"Large Language Models and Cybersecurity: Proceedings of a Workshop - in Brief." National Academies of Sciences, Engineering, and Medicine. 2024. Large Language Models and Cybersecurity: Proceedings of a Workshop—in Brief. Washington, DC: The National Academies Press. doi: 10.17226/27776.
×
Page 9
Large Language Models and Cybersecurity: Proceedings of a Workshop—in Brief Get This Book
×
 Large Language Models and Cybersecurity: Proceedings of a Workshop—in Brief
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

On August 31 and September 1, 2023, the National Academies of Sciences, Engineering, and Medicine held a workshop in conjunction with a meeting of its Forum on Cyber Resilience aimed at better understanding recent developments with large language models (LLMs) and their implications for cybersecurity and resilience. Presentations addressed how LLMs are constructed and function, how industry is considering using LLMs generally and for cybersecurity, safeguards that aim to limit LLM outputs deemed harmful and techniques for circumventing them, and, more generally, the trustworthiness of LLMs and their integration into cybersecurity offense and defense.

READ FREE ONLINE

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!