Adversarial example
Also called: adversarial input, evasion attack
An input changed by a small, deliberate perturbation so that a machine learning model gives a wrong answer with confidence.
An adversarial example is an input that has been deliberately modified, often very slightly, so that a model makes a mistake. A research paper on the topic describes adversarial examples as inputs formed by applying small but intentionally worst-case perturbations to dataset examples, which make the model output a wrong answer with high confidence. NIST's report on adversarial machine learning provides a taxonomy and common terminology for attacks of this kind and for their mitigations.
Sources: Goodfellow et al.: Explaining and Harnessing Adversarial Examples (arXiv) (opens in a new tab) (checked 2026-10-21); NIST AI 100-2 E2025: Adversarial Machine Learning, a Taxonomy and Terminology (opens in a new tab) (checked 2026-10-21)
AI agent
Also called: LLM agent, agentic AI
A system where an LLM directs its own steps and tool use to finish a task, instead of following a fixed code path.
An AI agent is a system in which a language model dynamically directs its own process and tool usage. Anthropic's engineering guide draws the contrast with workflows, where models and tools are orchestrated through predefined code paths. Because an agent decides which tools to call, it can be manipulated into calling the wrong ones, which is why OWASP's excessive agency guidance stresses limited tools, minimum permissions and human approval for high-impact actions.
Sources: Anthropic: Building effective agents (opens in a new tab) (checked 2026-10-21); OWASP: LLM06:2025 Excessive Agency (opens in a new tab) (checked 2026-10-21)
AI bill of materials
Also called: AI BOM, AIBOM, ML-BOM, ML bill of materials
An inventory of the datasets, models, frameworks and settings behind an AI system, like a software bill of materials.
An AI bill of materials (AI BOM or ML-BOM) lists the components behind an AI system: datasets, models, settings and their dependencies. CycloneDX describes its ML-BOM as providing model and dataset transparency for security, privacy, safety and ethical review. OWASP's supply chain guidance calls AI and ML bills of materials an emerging area and suggests evaluating options starting with CycloneDX. Posture tools such as Microsoft Defender for Cloud also describe discovering an AI BOM for deployed workloads.
Sources: CycloneDX: Machine Learning Bill of Materials (AI/ML-BOM) (opens in a new tab) (checked 2026-10-21); OWASP: LLM03:2025 Supply Chain (opens in a new tab) (checked 2026-10-21); Microsoft Learn: AI security posture management overview (opens in a new tab) (checked 2026-10-21)
AI gateway
Also called: LLM gateway, AI proxy
A layer between applications and model providers that adds logging, caching, rate limits and other controls.
An AI gateway is a layer that applications send requests through to reach model providers, so that usage can be observed and controlled in one place. Cloudflare's AI Gateway documentation, for example, lists analytics, logging, caching, rate limiting, and request retry and fallback to other models. Some gateways also add guardrail checks, but that depends on the product. Rate limits and usage analytics help with unbounded consumption (LLM10), because they cap how much one user or key can spend.
Sources: Cloudflare: AI Gateway documentation (opens in a new tab) (checked 2026-10-21)
AI governance
Also called: AI risk and compliance, responsible AI governance
The policies, roles and processes an organization uses to manage the AI it builds, buys and uses, and to show it does so responsibly.
AI governance is the set of policies, roles, processes and records an organization uses to manage the AI systems it builds, buys and uses. Frameworks describe it in different ways: the NIST AI RMF has a Govern function alongside Map, Measure and Manage, and ISO/IEC 42001 defines an AI management system for policies, objectives and processes for responsible AI. Governance platforms help with inventory, risk assessment, policy management and evidence. They support compliance work and do not make an organization compliant.
Sources: NIST: AI Risk Management Framework (opens in a new tab) (checked 2026-10-21); Microsoft Learn: ISO/IEC 42001:2023 overview (opens in a new tab) (checked 2026-10-21)
AI red teaming
Also called: LLM red teaming, red teaming
Probing an AI model or application on purpose to find harmful behavior and security weaknesses before others do.
AI red teaming means deliberately attacking or probing an AI system to uncover harmful outputs and security weaknesses. Microsoft's guidance notes that the term, which began in security testing, has come to describe many kinds of probing of AI systems. It says red teaming helps find harms and plan measurement, but it is not a replacement for systematic measurement and mitigation work. Tools in this area range from scanners and attack frameworks to managed services.
Sources: Microsoft Learn: Planning red teaming for LLMs and their applications (opens in a new tab) (checked 2026-10-21)
AI security posture management
Also called: AI-SPM, AISPM
Tools that find AI models, services and data across your environments and check them for risky configuration.
AI security posture management (AI-SPM) discovers where AI is used in your environments and checks how it is configured. Microsoft's documentation, for example, describes discovering a generative AI bill of materials across services such as Azure OpenAI Service, Amazon Bedrock and Google Vertex AI, giving recommendations on identity, data security and internet exposure, and using attack path analysis. Coverage depends on the clouds and services each product supports, so match the list to your own environment.
Sources: Microsoft Learn: AI security posture management overview (opens in a new tab) (checked 2026-10-21)
Artificial general intelligence
Also called: AGI
AI with broad, human-level ability across most cognitive tasks. Researchers and labs define the threshold differently.
Artificial general intelligence (AGI) is AI whose abilities are broad, covering most kinds of cognitive work at or above human level. Definitions differ. Google DeepMind describes AGI as AI at least as capable as humans at most cognitive tasks. Its Levels of AGI paper reviews nine earlier definitions, then grades systems by performance and generality, from Emerging AGI, where it places 2023 chatbots such as ChatGPT, up to superintelligence at Level 5. A 2025 paper, A Definition of AGI, instead defines AGI as matching the cognitive versatility and proficiency of a well-educated adult. For security planning, Google DeepMind's Frontier Safety Framework ties its mitigations to specific capability levels.
Sources: Google DeepMind: Taking a responsible path to AGI (opens in a new tab) (checked 2026-10-21); Google DeepMind: Levels of AGI for Operationalizing Progress on the Path to AGI (arXiv, version 5) (opens in a new tab) (checked 2026-10-21); A Definition of AGI (arXiv) (opens in a new tab) (checked 2026-10-21); Google DeepMind: Strengthening our Frontier Safety Framework (opens in a new tab) (checked 2026-10-21)
Backdoored model
Also called: model backdoor, sleeper agent model
A model that behaves normally until a hidden trigger makes it do something the attacker chose.
A backdoored model has been altered, through poisoned data or tampered weights, to act normally in most cases and to change behavior when a specific trigger appears. OWASP says such backdoors may leave the model's behavior untouched until the trigger activates, which makes them hard to detect and creates the opportunity for a sleeper agent. Defenses include checking where a model came from, verifying it with signatures and file hashes, and red teaming it before use.
Sources: OWASP: LLM04:2025 Data and Model Poisoning (opens in a new tab) (checked 2026-10-21); OWASP: LLM03:2025 Supply Chain (opens in a new tab) (checked 2026-10-21)
Cloud security posture management
Also called: CSPM
Tools that continuously assess cloud accounts for misconfiguration and risk against security standards.
Cloud security posture management (CSPM) gives continuous visibility into the security state of cloud assets and recommends fixes. Microsoft's documentation describes Defender for Cloud assessing Azure, AWS and Google Cloud environments against security standards, issuing recommendations to reduce misconfigurations, and offering a paid plan with extras such as attack path analysis and AI security posture. AI-SPM is often delivered as an extension of a CSPM product.
Sources: Microsoft Learn: What is Cloud Security Posture Management (CSPM) (opens in a new tab) (checked 2026-10-21)
Confused deputy
Also called: confused deputy problem
An attack where a trusted program is tricked into using its authority on an attacker's behalf.
A confused deputy is a program with legitimate authority that is tricked into using it for someone else. The MCP security best practices describe how this can happen with MCP proxy servers that connect to third-party APIs: with a static client ID, dynamic client registration and a consent cookie, a malicious client can obtain authorization codes without proper user consent. The specification requires such proxies to implement per-client consent. The same idea applies to agents that act with a user's permissions on instructions from untrusted content.
Sources: Model Context Protocol: Security best practices (opens in a new tab) (checked 2026-10-21)
Data loss prevention
Also called: DLP, GenAI DLP
Controls that stop sensitive data from being shared with people or services that should not have it.
Data loss prevention (DLP) is the practice of preventing users from inappropriately sharing sensitive data, such as financial data, health records or credit card numbers. Microsoft Purview implements it with policies that identify, monitor and protect sensitive items in apps, on devices and in web traffic, using content analysis that goes beyond a simple text scan. For generative AI, the same idea is applied to the prompts and uploads people send to AI apps, which Purview lists among its inline web traffic locations.
Sources: Microsoft Learn: Learn about data loss prevention (opens in a new tab) (checked 2026-10-21)
Data poisoning
Also called: training data poisoning, LLM04
Tampering with the data a model learns from, to plant weaknesses, backdoors or bias.
Data poisoning is manipulation of pre-training, fine-tuning or embedding data so that a model learns something harmful. OWASP classifies it as an integrity attack, with effects that include degraded performance, biased or toxic output and harm to downstream systems, and it says external and unverified data sources carry the highest risk. Defenses include tracking data origins, vetting data vendors, using data version control and monitoring model behavior for anomalies. MITRE ATLAS lists Training Data Poisoning (AML.T0020).
Sources: OWASP: LLM04:2025 Data and Model Poisoning (opens in a new tab) (checked 2026-10-21); MITRE ATLAS data 2026.09: AML.T0020 Training Data Poisoning (opens in a new tab) (checked 2026-10-21)
Data security posture management
Also called: DSPM
Tools that discover sensitive data across data stores and show how exposed it is.
Data security posture management (DSPM) finds sensitive data in cloud and other data stores, evaluates how sensitive and how exposed it is, and highlights risks that could lead to a breach. Microsoft's Defender for Cloud documentation describes automatic discovery of managed and shadow data resources, attack paths to sensitive stores and sensitivity settings based on predefined or custom information types. For generative AI, DSPM matters because prompts, retrieval sources and training data all come from these stores.
Sources: Microsoft Learn: Data security posture management overview (opens in a new tab) (checked 2026-10-21)
Data tokenization
Also called: tokenization, pseudonymization
Replacing a sensitive value with a token, so systems can work with the token while the real value stays protected.
Data tokenization replaces a sensitive value, such as a name or card number, with a token that stands in for it. Google's Sensitive Data Protection documentation calls this pseudonymization, replacing values with cryptographically generated tokens, and lists methods that are reversible (deterministic encryption and format-preserving encryption) and one-way (cryptographic hashing). In AI workflows, tokens let a model work with a record without seeing the real value. With a reversible method, an authorized system can restore the original afterwards.
Sources: Google Cloud: Sensitive Data Protection, Pseudonymization (opens in a new tab) (checked 2026-10-21)
Denial of wallet
Also called: DoW
An attack that drives up a victim's AI usage bill by sending large volumes of requests to a pay-per-use service.
Denial of wallet is an attack in which the attacker runs high volumes of operations against an AI service to exploit pay-per-use pricing, which creates unsustainable costs for the provider. OWASP lists it among the common forms of unbounded consumption. Rate limits, per-user quotas, input size limits and monitoring for unusual usage help, and an AI gateway is a common place to enforce them.
Sources: OWASP: LLM10:2025 Unbounded Consumption (opens in a new tab) (checked 2026-10-21)
Embedding inversion
Also called: embedding inversion attack
An attack that reconstructs much of the original text from the numeric embeddings stored for it.
Embedding inversion is an attack in which an adversary reverses embeddings to recover source information. OWASP's page on vector and embedding weaknesses says attackers can exploit vulnerabilities to invert embeddings and recover significant amounts of source information, which threatens confidentiality. The practical lesson is that embeddings of sensitive text should be treated as sensitive too, with access controls and detailed logging on the vector store.
Sources: OWASP: LLM08:2025 Vector and Embedding Weaknesses (opens in a new tab) (checked 2026-10-21)
EU AI Act
Also called: Regulation (EU) 2024/1689, AI Act
The EU regulation that sets risk-based rules for AI systems, in force since 1 August 2024 and applying in stages.
The EU AI Act, Regulation (EU) 2024/1689, sets harmonised rules for AI in the European Union using risk categories: unacceptable risk (banned), high risk (strict obligations), transparency risk and minimal risk. The European Commission's page says it entered into force on 1 August 2024 and applies from 2 August 2026, with earlier dates for prohibitions and general-purpose AI obligations and later dates for some high-risk rules. This directory does not give legal advice, so confirm what applies to you with counsel.
Sources: European Commission: AI Act (opens in a new tab) (checked 2026-10-21)
Evaluation harness
Also called: eval framework, LLM evaluation framework
Software that runs a model or application through a set of tasks and scores the results.
An evaluation harness is a framework that runs a model or application through datasets of tasks and scores the outputs. The Inspect framework from the UK AI Security Institute and Meridian Labs, for example, describes an evaluation as a task made of three parts: a dataset of labeled samples, a solver that generates answers (anything from a single model call to a full agent) and a scorer that grades the result. Harnesses make checks repeatable, so they can run again after each change to a model or prompt.
Sources: Inspect: a framework for frontier AI evaluations (opens in a new tab) (checked 2026-10-21)
Excessive agency
Also called: LLM06, over-privileged agent
When an LLM-based system can take damaging actions because it has too much functionality, permission or autonomy.
Excessive agency is the vulnerability that lets damaging actions happen when a model produces unexpected, ambiguous or manipulated output. OWASP traces it to three root causes: excessive functionality (tools the agent does not need), excessive permissions (more access downstream than the purpose requires) and excessive autonomy (high-impact actions without approval). Mitigations include narrow tools, minimum permissions, authorization in downstream systems and human approval for high-impact actions.
Sources: OWASP: LLM06:2025 Excessive Agency (opens in a new tab) (checked 2026-10-21)
Guardrails
Also called: LLM guardrails, AI guardrails
Checks placed before and after a model call that screen inputs and outputs against rules.
Guardrails are checks wrapped around a language model call. Input checks screen what goes in, and output checks screen what comes back, for example for harmful content, personal data or off-topic answers. NVIDIA's NeMo Guardrails documentation describes rails that can be added before and after LLM calls, and OWASP lists input and output filtering among its prompt injection mitigations. Guardrails reduce risk but do not remove it, because detection is imperfect. They work best alongside least privilege and testing.
Sources: NVIDIA NeMo Guardrails documentation (opens in a new tab) (checked 2026-10-21); OWASP: LLM01:2025 Prompt Injection (opens in a new tab) (checked 2026-10-21)
Hallucination
Also called: confabulation, fabricated output
When a model produces content that sounds accurate but is fabricated.
A hallucination is model output that seems accurate but is made up. OWASP's misinformation entry calls it a major cause of misinformation, and explains that it happens when a model fills gaps in its training data with statistical patterns, without real understanding. Overreliance makes it worse, because users trust output they have not checked. Common mitigations are grounding answers with retrieval-augmented generation, cross-checking against trusted sources, automatic validation of key outputs and human review of critical content.
Sources: OWASP: LLM09:2025 Misinformation (opens in a new tab) (checked 2026-10-21)
Improper output handling
Also called: LLM05, insecure output handling
Passing model output to other systems without validating or encoding it, so the output can trigger attacks such as XSS or code execution.
Improper output handling means model-generated output is not validated, sanitized or handled safely before it goes to other components. Because prompts can shape the output, OWASP likens this to giving users indirect access to extra functionality. It lists cross-site scripting, cross-site request forgery, server-side request forgery, privilege escalation and remote code execution as possible results. The remedy is to treat model output as untrusted input: validate it, encode it for its destination and use parameterized queries.
Sources: OWASP: LLM05:2025 Improper Output Handling (opens in a new tab) (checked 2026-10-21)
Indirect prompt injection
Also called: indirect injection
Prompt injection where the instructions arrive in content the model reads, such as a web page or file, instead of from the user.
In indirect prompt injection the attacker does not type into the chat. The instructions sit in external content the model reads, such as a web page, a document or an email, and they change the model's behavior when that content is processed. OWASP gives the example of a summarized web page with hidden instructions that make the model leak conversation data through a link. Common defenses are to keep untrusted content clearly separated and labeled, and to limit what the model's tools are allowed to do.
Sources: OWASP: LLM01:2025 Prompt Injection (opens in a new tab) (checked 2026-10-21)
ISO/IEC 42001
Also called: ISO 42001, AI management system standard
An international standard for AI management systems: requirements for how an organization sets up and improves its governance of AI.
ISO/IEC 42001:2023 is an international standard that specifies requirements for establishing, implementing, maintaining and continually improving an artificial intelligence management system within an organization. An AI management system is a set of related elements that set policies, objectives and processes for the responsible development, provision or use of AI systems. Organizations can be audited and certified against it by independent third parties. Governance platforms often say they support it, so check each vendor's own statement.
Sources: Microsoft Learn: ISO/IEC 42001:2023 overview (opens in a new tab) (checked 2026-10-21)
Jailbreak
Also called: jailbreaking, LLM jailbreak
A prompt injection attack that gets a model to ignore its safety rules entirely.
A jailbreak is a form of prompt injection in which the attacker's input makes a model ignore its safety protocols. OWASP notes that the terms prompt injection and jailbreaking are often used interchangeably, and says that preventing jailbreaks takes ongoing updates to a model's training and safety mechanisms. Jailbreak attempts are a common case for red teaming tools to probe and for guardrail classifiers to screen. MITRE ATLAS lists the technique as LLM Jailbreak (AML.T0054).
Sources: OWASP: LLM01:2025 Prompt Injection (opens in a new tab) (checked 2026-10-21); MITRE ATLAS data 2026.09: AML.T0054 LLM Jailbreak (opens in a new tab) (checked 2026-10-21)
Least privilege
Also called: principle of least privilege, minimal permissions
Giving users, programs and agents only the access they need for their task, and nothing more.
Least privilege is the security principle that a system should restrict users, and the processes acting for them, to the access needed to do their assigned work. NIST's glossary records this definition. In AI systems it applies to a model's tools and credentials as well as to people: OWASP recommends giving extensions minimum permissions on downstream systems, running actions in the user's own context and enforcing authorization outside the model. It limits the damage when a model is manipulated.
Sources: NIST glossary: least privilege (opens in a new tab) (checked 2026-10-21); OWASP: LLM06:2025 Excessive Agency (opens in a new tab) (checked 2026-10-21)
LLM firewall
Also called: AI firewall, firewall for AI
A control in the request path that inspects prompts and responses, like a web application firewall for LLM traffic.
LLM firewall is a label vendors use for checks placed in the request path of an AI application, so that prompts and responses are inspected before they reach the model or the user. Cloudflare's feature of this kind scans requests sent to LLM endpoints for prompt injection, personal data and unsafe or custom topics, and its detections work with any model. The checks can be the same as those called guardrails, so compare what a product documents instead of relying on the name.
Sources: Cloudflare: AI Security for Apps (formerly Firewall for AI) (opens in a new tab) (checked 2026-10-21)
LLM vulnerability scanner
Also called: LLM scanner, AI vulnerability scanner
A tool that sends many probes to a language model or LLM app and reports where it fails, much as a network scanner probes hosts.
An LLM vulnerability scanner automatically runs many probes against a model or application and reports which ones produced an unwanted result, such as a jailbreak, a prompt injection or a data leak. NVIDIA's garak describes itself in its README as an LLM vulnerability scanner and compares its role to nmap or Metasploit for LLMs. A scanner shows what failed on the day it ran, so it complements design controls and human review and does not replace them.
Sources: garak README (NVIDIA) (opens in a new tab) (checked 2026-10-21)
LLM-as-a-judge
Also called: model-graded evaluation, LLM judge
Using a strong language model to grade the outputs of other models or applications.
LLM-as-a-judge means using a language model to evaluate other model outputs, for example to score answers or to decide whether a response is harmful. The research paper that studied the approach for chat assistants reports that strong judges such as GPT-4 can match human preferences at over 80% agreement, the same level of agreement as between humans. When a tool uses a judge to decide whether an attack worked, the judge can be wrong or be manipulated, so check how the tool validates its verdicts.
Sources: Zheng et al.: Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (arXiv) (opens in a new tab) (checked 2026-10-21)
MCP rug pull
Also called: rug pull attack, tool definition change attack
A server or tool that behaves well until it is approved, then changes its description or behavior.
An MCP rug pull is when a server changes a tool's description or behavior after the user or client has already approved it, so a server that looked safe at install time can attack later. Invariant Labs describes this pattern in its tool poisoning notification. MITRE ATLAS has a related technique, AI Supply Chain Rug Pull (AML.T0109), in which an adversary publishes legitimate AI components, gains user adoption and then pushes an update. Re-checking tool definitions for changes after approval is the main defense.
Sources: Invariant Labs: MCP Security Notification, Tool Poisoning Attacks (opens in a new tab) (checked 2026-10-21); MITRE ATLAS data 2026.09: AML.T0109 AI Supply Chain Rug Pull (opens in a new tab) (checked 2026-10-21)
MCP server
Also called: Model Context Protocol server
A program that gives AI applications context, tools and data through MCP, running locally or remotely.
An MCP server is a program that provides context to MCP clients. The MCP architecture documentation says a server can run locally, using the stdio transport, or remotely, using Streamable HTTP. Servers can expose three kinds of primitives: tools (functions an AI application can call), resources (data that gives context) and prompts (reusable templates). The security guidance warns that local servers run with the client's privileges and need consent and sandboxing, so an untrusted server is a real risk.
Sources: Model Context Protocol: Architecture overview (opens in a new tab) (checked 2026-10-21); Model Context Protocol: Security best practices (opens in a new tab) (checked 2026-10-21)
Membership inference
Also called: membership inference attack
An attack that tries to tell whether a specific record was part of a model's training data.
A membership inference attack tries to determine whether a given data record was in a model's training dataset. The original research paper frames the basic attack as having a data record and black-box access to a model, and asking whether the record was used in training. It is a privacy risk because it can reveal that a person's data was used. MITRE ATLAS lists it as Infer Training Data Membership (AML.T0024.000). OWASP's LLM02 page lists differential privacy among its mitigations.
Sources: Shokri et al.: Membership Inference Attacks against Machine Learning Models (arXiv) (opens in a new tab) (checked 2026-10-21); MITRE ATLAS data 2026.09: AML.T0024.000 Infer Training Data Membership (opens in a new tab) (checked 2026-10-21); OWASP: LLM02:2025 Sensitive Information Disclosure (opens in a new tab) (checked 2026-10-21)
MITRE ATLAS
Also called: ATLAS, Adversarial Threat Landscape for AI Systems
MITRE's catalog of tactics, techniques and mitigations for attacks on AI systems, with IDs such as AML.T0051.
MITRE ATLAS (Adversarial Threat Landscape for AI Systems) catalogs how attackers target AI systems as tactics, techniques and mitigations, each with an ID such as AML.T0051 for LLM Prompt Injection. In the data version we read (2026.09) it has 16 tactics, and MITRE publishes the content as a data file on GitHub. This directory maps a short list of techniques to OWASP LLM risks and tool categories. Which techniques we show and how we map them is our analysis.
Sources: MITRE ATLAS (opens in a new tab) (checked 2026-10-21); MITRE ATLAS data file, version 2026.09 (opens in a new tab) (checked 2026-10-21)
ML supply chain
Also called: LLM supply chain, AI supply chain, LLM03
Everything an ML system depends on that you did not build: pre-trained models, datasets, adapters, libraries and platforms.
The ML supply chain covers training data, models, adapters and deployment platforms, as well as ordinary code dependencies. OWASP says LLM supply chains are vulnerable at three points, training data, models and deployment platforms, and that third-party pre-trained models and data can be tampered with or poisoned. Its mitigations include vetting suppliers, using verifiable model sources with signatures and file hashes, keeping a signed inventory of components and patching what you use.
Sources: OWASP: LLM03:2025 Supply Chain (opens in a new tab) (checked 2026-10-21)
Model card
Also called: model documentation
A document that ships with a model and describes its intended uses, limits, training data and evaluation results.
A model card is a file that accompanies a model and describes it. Hugging Face says a model card should cover the model, its intended uses and potential limitations (including biases and ethical considerations), the training parameters, the datasets used and the evaluation results. For security and governance, a model card is one source of provenance and licensing information, although it is a claim made by the publisher and not an independently verified record.
Sources: Hugging Face Hub: Model Cards (opens in a new tab) (checked 2026-10-21)
Model Context Protocol
Also called: MCP
An open standard for connecting AI applications to data sources, tools and workflows.
The Model Context Protocol (MCP) is an open-source standard for connecting AI applications to external systems. According to the MCP documentation, AI applications can use it to reach data sources such as local files and databases, tools such as search engines and calculators, and workflows such as specialized prompts, and the documentation compares it to a USB-C port for AI applications. MCP also has its own security guidance, which covers confused deputy attacks, token passthrough, local server compromise and scope minimization.
Sources: Model Context Protocol: What is MCP? (opens in a new tab) (checked 2026-10-21); Model Context Protocol: Security best practices (opens in a new tab) (checked 2026-10-21)
Model registry
Also called: ML model registry, model hub
A central store that tracks versions of models and how they moved from development to production.
A model registry is a central place to store, version and manage models through their life cycle. The MLflow documentation describes its registry as a centralized model store with APIs and a UI, where each new version of a model gets a higher version number, links back to the run that produced it and can be pointed to by an alias. For security, a registry is a control point where access controls, scanning and provenance records can be applied. OWASP's LLM10 page recommends a central model inventory with access controls.
Sources: MLflow documentation: Model Registry (opens in a new tab) (checked 2026-10-21); OWASP: LLM10:2025 Unbounded Consumption (opens in a new tab) (checked 2026-10-21)
Model serialization attack
Also called: malicious model file, pickle exploit
An attack that hides code in a saved model file so that it runs when the file is loaded.
A model serialization attack abuses the way some formats save model data. Pickle, the default format for PyTorch model weights, can carry instructions that run code when the file is loaded. Hugging Face's security documentation explains how this works, and says its Hub scans uploaded pickle files and lists the imports they reference, while noting that the scan is not foolproof. OWASP lists malicious pickling among the ways a shared model can carry malware. Scanners and safer formats reduce the risk.
Sources: Hugging Face Hub: Pickle Scanning (opens in a new tab) (checked 2026-10-21); OWASP: LLM04:2025 Data and Model Poisoning (opens in a new tab) (checked 2026-10-21)
Model signing
Also called: signed models, model provenance signing
Cryptographically signing a model's files so users can verify who published them and that they were not changed.
Model signing attaches a cryptographic signature to a model so that anyone can check who published it and that its files have not changed since. The Sigstore model-transparency project, which builds on work with the OpenSSF, hashes the files in a model directory into a manifest of digests and signs those digests, then verifies by recomputing the hashes and checking the signer's identity. OWASP's supply chain guidance recommends integrity checks with signing and file hashes.
Sources: Sigstore model-transparency README (opens in a new tab) (checked 2026-10-21); OWASP: LLM03:2025 Supply Chain (opens in a new tab) (checked 2026-10-21)
NIST AI Risk Management Framework
Also called: NIST AI RMF, AI RMF
NIST's voluntary framework for managing AI risks, built around four functions: Govern, Map, Measure and Manage.
The NIST AI Risk Management Framework (AI RMF) helps organizations manage the risks AI poses to individuals, organizations and society, and build trustworthiness into the design, development, use and evaluation of AI systems. Version 1.0 was released on January 26, 2023, and the framework is intended for voluntary use. Its core functions are Govern, Map, Measure and Manage. NIST's page notes that the framework is being revised, so check it for the current version.
Sources: NIST: AI Risk Management Framework (opens in a new tab) (checked 2026-10-21)
OWASP Top 10 for LLM Applications
Also called: OWASP LLM Top 10, OWASP Top 10 for LLMs
OWASP's list of ten risks for applications built on large language models. The 2025 edition runs from LLM01 to LLM10.
The OWASP Top 10 for LLM Applications is published by the OWASP Gen AI Security Project. The 2025 edition has ten entries, from LLM01 Prompt Injection to LLM10 Unbounded Consumption. This directory maps each tool to the entries its documented capabilities address. The list is OWASP's, and the mapping is our own analysis. Each tool page and each of our risk pages links to OWASP's page for the risk.
Sources: OWASP Top 10 for LLM Applications 2025 (opens in a new tab) (checked 2026-10-21)
Personally identifiable information
Also called: PII, personal data
Information that can be used to distinguish or trace a person's identity, alone or combined with other linked information.
Personally identifiable information (PII) is information that can be used to distinguish or trace a person's identity, such as a name, a Social Security number or biometric records, alone or combined with other information that is linked or linkable to that person. This is the definition recorded in NIST's glossary, adapted from US federal guidance. NIST SP 800-122 explains how to identify PII and protect its confidentiality. In LLM applications, PII can enter through prompts, uploaded files and retrieved documents.
Sources: NIST glossary: personally identifiable information (opens in a new tab) (checked 2026-10-21); NIST SP 800-122: Guide to Protecting the Confidentiality of PII (opens in a new tab) (checked 2026-10-21)
Pickle
Also called: Python pickle, pickle file
Python's built-in format for saving objects. Loading pickle data you do not trust can run arbitrary code.
Pickle is the Python module that turns objects into bytes and rebuilds them later. Python's documentation carries a prominent warning that the module is not secure: malicious pickle data can execute arbitrary code during unpickling, so you should only unpickle data you trust. Because many machine learning tools have saved model weights with pickle, a downloaded model file can be an attack route. Formats such as safetensors store tensors only and avoid this risk.
Sources: Python documentation: pickle (opens in a new tab) (checked 2026-10-21); Hugging Face Hub: Pickle Scanning (opens in a new tab) (checked 2026-10-21)
Prompt injection
Also called: LLM01, prompt attack
Input that changes how a language model behaves, so it follows an attacker's instructions instead of the application's.
Prompt injection happens when input to a model, including input people cannot see, changes how the model behaves or what it outputs. OWASP lists it first (LLM01) in its 2025 Top 10 for LLM Applications. It can lead to policy violations, harmful content, unauthorized access or skewed decisions. OWASP says it is unclear whether fool-proof prevention exists, so teams combine filtering, least privilege and adversarial testing. Jailbreaking is a related attack that tries to remove a model's safety behavior entirely.
Sources: OWASP: LLM01:2025 Prompt Injection (opens in a new tab) (checked 2026-10-21)
Redaction
Also called: data redaction, PII redaction, data masking
Removing or masking sensitive values in text, images or documents before they are stored, shared or sent to a model.
Redaction removes or masks sensitive values, such as names, card numbers or ID numbers, so the rest of the content can be used safely. Presidio's documentation describes fast identification and anonymization modules for private entities in text and images. In LLM applications, redaction is often applied to prompts before they reach a model and to responses before they reach a user. Unlike tokenization, a plain redaction cannot be reversed by the application, which is simpler but loses the original value.
Sources: Presidio documentation (opens in a new tab) (checked 2026-10-21)
Retrieval-augmented generation
Also called: RAG
A model is given relevant documents retrieved from a knowledge base before it answers.
Retrieval-augmented generation (RAG) optimizes a model's output by having it reference an authoritative knowledge base before it responds, instead of relying only on its training data. AWS describes it this way, and OWASP describes RAG as pairing a pre-trained model with external knowledge sources, using vectors and embeddings to find relevant content. RAG adds security questions of its own: who may see which documents, and whether retrieved text can carry hidden instructions or poisoned content.
Sources: AWS: What is Retrieval-Augmented Generation? (opens in a new tab) (checked 2026-10-21); OWASP: LLM08:2025 Vector and Embedding Weaknesses (opens in a new tab) (checked 2026-10-21)
Safetensors
Also called: .safetensors
A Hugging Face file format for storing tensors safely, as an alternative to pickle.
Safetensors is a file format for storing the tensors that make up a model. Its documentation describes it as a simple format for storing tensors safely, as opposed to pickle, while still being fast thanks to zero-copy loading. Unlike pickle, it is meant to hold tensor data only, so loading it does not run code the way a pickle file can. It protects the file format and nothing more: it says nothing about whether the model's behavior can be trusted.
Sources: Hugging Face: safetensors documentation (opens in a new tab) (checked 2026-10-21); Hugging Face Hub: Pickle Scanning (opens in a new tab) (checked 2026-10-21)
Safety classifier
Also called: safety classifier model, content safety model
A model trained to label text or images as safe or unsafe against defined policies, used to screen prompts and responses.
A safety classifier is a model that judges content against defined safety policies and labels it, so an application can allow, block or flag it. Google describes ShieldGemma as instruction-tuned models for evaluating the safety of text and images, and for text it evaluates both prompt inputs and output responses. Open-weight classifiers can be hosted and fine-tuned by the user. Like all detectors, they make mistakes, so teams should measure them on their own traffic before relying on them.
Sources: Google AI for Developers: ShieldGemma (opens in a new tab) (checked 2026-10-21)
Shadow AI
Also called: unsanctioned AI, unapproved AI use
Use of AI tools by staff without the knowledge, approval or governance of the organization's IT or security teams.
Shadow AI is the use of AI tools, especially generative AI, without the knowledge, approval or governance of an organization's IT or security teams. Microsoft's guidance says it can lead to leaks of sensitive information, noncompliance and reputational harm. Its recommended steps are to discover AI app use, block unsanctioned apps, block sensitive data going to sanctioned apps and govern the data sent to AI apps. Finding shadow AI needs network, browser or identity data, not only cloud posture checks.
Sources: Microsoft Learn: Prevent data leak to shadow AI (opens in a new tab) (checked 2026-10-21)
Superintelligence (ASI)
Also called: artificial superintelligence, superintelligent AI
AI that would outperform every human at almost every cognitive task. It does not exist today, and definitions differ on the exact bar.
Superintelligence, or artificial superintelligence (ASI), is AI that would outperform every human at almost every cognitive task. In Google DeepMind's Levels of AGI paper, a general system at the top level, Level 5, outperforms 100% of humans; the paper calls it Artificial Superintelligence (ASI) and lists it as not yet achieved. The Future of Life Institute's Statement on Superintelligence describes AI that can significantly outperform all humans on essentially all cognitive tasks. Definitions differ on where the bar sits. ASI has a second meaning in AI security: the OWASP Top 10 for Agentic Applications, from OWASP's Agentic Security Initiative (ASI), numbers today's agent risks ASI01 to ASI10.
Sources: Google DeepMind: Levels of AGI for Operationalizing Progress on the Path to AGI (arXiv, version 5) (opens in a new tab) (checked 2026-10-21); Future of Life Institute: Statement on Superintelligence (opens in a new tab) (checked 2026-10-21); OWASP: Agentic AI Security Initiative (opens in a new tab) (checked 2026-10-21); OWASP: GenAI and Agentic AI Exploit Roundup Q3 2026 (opens in a new tab) (checked 2026-10-21)
Superintelligence safety and security
Also called: ASI security
Work on keeping AI that could outperform people aligned with its operators' intent, under human control and out of reach of people who would misuse it.
Superintelligence safety and security is research and policy for AI that could outperform people: keeping it aligned with its operators' intent and under human control, and stopping others from misusing it. Google DeepMind's approach to AGI safety and security focuses on misuse and misalignment, two of the four risk areas it names. Beyond model-level mitigations, it relies on system-level security such as monitoring and access control, treating the AI as an untrusted insider, an approach often called AI control. It leaves risks unique to superintelligence to future research. In our AI security layers, the matching controls sit in governance, testing, agent access and monitoring. No tool listed here documents a capability to secure superintelligence.
Sources: Google DeepMind: An Approach to Technical AGI Safety and Security (arXiv) (opens in a new tab) (checked 2026-10-21); AI Control: Improving Safety Despite Intentional Subversion (arXiv) (opens in a new tab) (checked 2026-10-21)
System prompt leakage
Also called: LLM07, prompt extraction
Disclosure of the hidden instructions given to a model, which can expose credentials, internal rules or permission details.
A system prompt is the set of instructions an application gives a model before the user speaks. System prompt leakage is the risk that attackers get the model to reveal it. OWASP's guidance is that the prompt should not be treated as a secret or as a security control. The real danger is what it contains, such as credentials, internal rules or role structures. Keep sensitive data out of prompts, and enforce behavior and permissions in systems outside the model.
Sources: OWASP: LLM07:2025 System Prompt Leakage (opens in a new tab) (checked 2026-10-21)
Tool poisoning
Also called: tool poisoning attack, MCP tool poisoning
Hiding malicious instructions in an AI tool's description, where the model reads them but the user may not.
Tool poisoning is an attack in which malicious instructions are embedded in the description of a tool, for example an MCP tool, so that the model sees them and the user does not. In a security notification, Invariant Labs describes it as a specialized form of indirect prompt injection that can make an agent take unauthorized actions. MITRE ATLAS lists AI Agent Tool Poisoning (AML.T0110). Reviewing tool definitions before approval, and checking them again for changes, are the usual defenses.
Sources: Invariant Labs: MCP Security Notification, Tool Poisoning Attacks (opens in a new tab) (checked 2026-10-21); MITRE ATLAS data 2026.09: AML.T0110 AI Agent Tool Poisoning (opens in a new tab) (checked 2026-10-21)
Unbounded consumption
Also called: LLM10, resource exhaustion
When an LLM application lets users run excessive, uncontrolled inference, causing denial of service, runaway cost or model theft.
Unbounded consumption is the OWASP LLM10 risk that an application allows excessive, uncontrolled inference, which can lead to denial of service, economic losses, model theft and degraded service. OWASP's mitigations include input size limits, rate limiting and user quotas, timeouts and throttling, monitoring for unusual usage, restricting how much probability detail an API returns, and access controls on model repositories. Some gateways and guardrail tools offer rate and cost controls.
Sources: OWASP: LLM10:2025 Unbounded Consumption (opens in a new tab) (checked 2026-10-21)
Vector database
Also called: vector store, embedding store
A database that stores embeddings and finds the items closest in meaning to a query.
A vector database stores vectors, which are lists of numbers produced by an embedding model, and supports fast lookup of the nearest neighbors to a query vector. AWS explains that embeddings encode many kinds of data into vectors that capture meaning and context, so similar items sit close together. RAG systems use vector databases to find relevant passages. OWASP's LLM08 page covers the security side, including permission-aware access control and partitioning between users and groups.
Sources: AWS: What is a Vector Database? (opens in a new tab) (checked 2026-10-21); OWASP: LLM08:2025 Vector and Embedding Weaknesses (opens in a new tab) (checked 2026-10-21)
