OWASP GenAI LLM Top 10 2026
Start with the current AI security risk map
Read the OWASP guide to orient yourself to application risks, scenarios, and mitigations.
Start with: OWASP GenAI LLM Top 10 2026
RedLectern topic
AI security
Prompt injection · data attacks · tool abuse · exfiltration · application security boundaries.
For: Builders · security learners
After this page you can: Threat-model an AI workflow and test its access boundaries.
Page reviewed
Part of: Artificial Intelligence › AI safety and governance
About AI security
AI security examines how an AI-enabled application can be manipulated, misused, or made to expose information through its inputs, data sources, tools, and surrounding software. A useful threat model follows the whole workflow: who supplies each input, what the system can read, which actions it can take, and where outputs go. Prompt injection and data poisoning are distinct risks: one manipulates instructions processed during a task, while the other manipulates data used in model development or reference sources.
The practical goal is to test boundaries in the application around the model. Treat model output as untrusted input to downstream functions, keep permissions narrow, and check whether sensitive data can reach an unauthorized destination. A prompt that says “do not reveal secrets” does not enforce an access policy by itself. Start with the workflow’s data and action paths, then test whether an attacker-controlled prompt, document, or tool result can cross them. Use isolated, authorized test environments for attack exercises.
Start here
Web LLM attacks
Practice on a web security learning path
Use the Web LLM attacks path to study prompt injection, APIs, tool access, and defensive design in a learning environment.
Start with: Web LLM attacks
Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations
Study adversarial machine learning
Use NIST’s taxonomy to distinguish attack goals, lifecycle stages, and mitigation limits.
Start with: Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations
Learn the foundations
1.Threats in prompts and dataAI workflows can receive attacker-controlled instructions through a user prompt or content the system retrieves, including websites and files.
AI workflows can receive attacker-controlled instructions through a user prompt or content the system retrieves, including websites and files. Those instructions can alter model behavior, disclose sensitive information, or influence functions available to the application. Direct prompt injection arrives through user input; indirect prompt injection arrives through external content. A jailbreak is an attempt to make a model disregard its safety restrictions. Data poisoning is a separate lifecycle risk: manipulated pre-training, fine-tuning, or embedding data can introduce vulnerabilities, backdoors, or biases. Trace each input to its source and ask what data or actions it can reach.
- LLM04:2025 Data and Model Poisoning
Explains how manipulated training and embedding data can affect model behavior.
Prompt injection
Attacker-controlled input attempting to redirect a system from its intended instructions.
Indirect prompt injection
Prompt injection arriving through retrieved content, files, or tool outputs.
Jailbreak
An attempt to bypass a model's intended behavioral or safety restrictions.
Data poisoning
Deliberate manipulation of training or reference data to alter system behavior.
Data exfiltration
Unauthorized transfer of information outside its permitted boundary.
Try it
A document summarizer reads uploaded files and can email a summary. Identify one indirect prompt-injection path and one boundary to test.
An uploaded file could contain instructions that change the summary or trigger an email. Test with a harmless document that asks the system to send a marker to a test address, and verify that file text alone cannot authorize sending.
2.Tools, permissions, and application boundariesAn AI application may use model outputs to call functions or interact with other systems.
An AI application may use model outputs to call functions or interact with other systems. Risk grows when a workflow exposes unnecessary functions, gives a tool broader permissions than its task needs, or lets actions proceed without approval. Enforce authorization in the downstream system so each request is checked against policy; do not rely on the model to decide whether an action is allowed. Map the identity and permissions used by each tool, then test that users and untrusted content cannot make it read, change, or send data outside the intended scope.
- OWASP Application Security Verification Standard
Provides a basis for testing web application technical security controls and secure development requirements.
Try it
An assistant only needs to read project documents but its connector can also edit and delete them. Name two changes that reduce the risk.
Replace or configure the connector to expose read-only access, and enforce a read-only identity or scope in the document system. Keep any write action behind a separate authorization and approval check.
3.Test the workflow and its access boundariesTest the application as a complete workflow: its prompts, retrieved content, tools, identities, permissions, and downstream services.
Test the application as a complete workflow: its prompts, retrieved content, tools, identities, permissions, and downstream services. Use benign test data and an authorized sandbox to check whether untrusted inputs can alter instructions, retrieve data beyond the user’s scope, trigger an unauthorized action, or move a secret to an outside destination. Record the expected boundary and the observed result for each test. A passing prompt-level check does not establish that the application’s access controls work; verify authorization where the data or action is actually enforced.
- Prompt injection protection in Microsoft Defender for Office 365
Describes prompt-injection objectives such as data exfiltration and tool discovery, and notes the role of runtime safeguards.
Try it
For an assistant that can search a user’s files and send email, define one test for unauthorized access and one for exfiltration.
Use two test accounts with separate files and try to retrieve the other account’s marker; the expected result is denial. Then place a harmless marker in an authorized test file and check that an injected instruction cannot email it to an unapproved test destination.
Latest
No recent updates are available.
Books
Threat Modeling: Designing for Security
Adam Shostack · 2014
A practical way to map a system, identify what could go wrong, and decide how to address threats. As a 2014 edition, it predates LLM-era attack patterns; use it for the modeling method and supplement it with current testing guidance.
Security Engineering: A Guide to Building Dependable Distributed Systems
Ross Anderson · 2020
A systems-security reference covering access control, distributed systems, secure systems development, and AI and machine learning security. The third edition is from 2020, so it is not an LLM-era test guide; the author offers the full text online for free.