The Words, Defined

Adversarial testing of an AI system to establish what it can be made to do that it should not, by an attacker who controls some of the text it reads or the tools it can call. For LLM applications it covers prompt injection, jailbreaks, tool-call abuse and data exfiltration, and its useful output is a list of reachable consequences ordered by damage rather than a score.

An attack in which instructions are placed in text a language model processes, so that the model follows them as if they came from its operator. It works because a model receives one stream of tokens and has no reliable way to separate content it was asked to read from an instruction addressed to it. Defences at the prompt layer reduce the rate; they do not bound the consequence.

Prompt injection where the malicious instruction is planted in content the model retrieves or is given - a web page, an email, a shared document, a support ticket, a database row - rather than typed by the user. It is the form that matters for agents, because the attacker needs no access to the application and the legitimate user may never know.

A user deliberately getting a model to bypass its own safety training or operator instructions. The person talking to the model is the attacker, which distinguishes it from prompt injection, where the attacker and the user are different people.

Everything an AI system is technically able to reach or change if it is compromised or simply wrong: the tools and APIs it can call, the data its credentials can read, and which of those actions are irreversible. Blast radius is set by permissions and architecture, not by the model, and it is the figure that turns an unbounded AI risk into an ordinary security question.

An LLM application holding more capability, permission or autonomy than the task requires - a service account reused from a pilot, a tool that can delete when it only needs to read, an action taken with no human in front of it. It appears in the OWASP Top 10 for LLM Applications and is the most common root cause behind serious findings in an agent security review.

Causing an agent to invoke one of its tools - an API, a database query, an email send, a file write - in a way its operator did not intend, typically through injected instructions. It is the mechanism by which prompt injection turns from a wrong answer into an unauthorised action.

An injected instruction causing a model to take sensitive content from its context window and deliver it somewhere an attacker can read - as a query string on a URL the agent fetches, a rendered Markdown image, a message sent through a tool. Any capability to make an outbound request is a capability to exfiltrate, which is why outbound destinations are allowlisted rather than blocklisted.

A community-maintained list of the most significant security risks specific to applications built on large language models - prompt injection, sensitive information disclosure, excessive agency and improper output handling among them. It is a useful checklist for making findings legible to a security team; it is not a substitute for scoping tests by what an application's agents can actually reach.

A collection of real inputs, each paired with an output a knowledgeable person has verified as correct, used to measure an AI system and to detect regressions when anything about it changes. Its value depends almost entirely on label quality and on how well it covers the awkward cases; a hundred carefully verified cases outperform a thousand careless ones.

Whether each claim in an AI system's answer actually follows from the source text it was given, as distinct from whether the answer is fluent or true in general. Measuring grounding separately from retrieval splits the complaint "it hallucinates" into two different bugs - the right document was not found, or it was found and not followed - which are fixed in different places.

A model producing a confident claim that is not supported by its inputs or by fact. In document-answering systems most of what gets called hallucination is a grounding failure - the model answered from a document that did not contain the answer, or cited a source it did not read - and is measurable as such.

Using a language model to score another model's outputs against an eval set, because scoring by hand does not scale. It works well enough to be worth doing on narrow, checkable criteria, with three caveats: judges are biased towards length, their own phrasing and position order; vague criteria produce meaningless numbers; and the judge itself must be validated against hand-scored cases before its numbers are trusted.

An automated check in the build pipeline that runs the eval set on every change to a prompt, model version, retrieval configuration, tool schema or routing rule, and fails the build when a metric drops past a threshold agreed in advance. The threshold being agreed before the first red build is what makes it a gate rather than a negotiation.

The practice of producing evidence that an AI system does what it is meant to do, cannot do what it must not do, and can be shown afterwards to have done either. It is distinct from governance (the rules) and from certification (a third party's statement that a standard was met): assurance is the evidence both of those rest on.

Giving an agent the narrowest credential that lets it do its specific job, under its own identity rather than a shared one, read-only wherever reading is enough. The principle is the same one that applies to any other process; it is applied to agents less often because their permissions tend to be inherited casually from a pilot and never narrowed.

A control that stops an agent before an irreversible action - moving money, sending external communication, deleting, changing permissions - until a person has seen what is about to happen and approved it. To be a control rather than a formality it must be enforced somewhere the model cannot reach, show the approver enough to judge in seconds, and fire rarely enough that it is actually read.

A maintained census of every AI system an organisation has in use - including ones that arrived through a SaaS feature, a default-on copilot or an unregistered script - recording its owner, the data it touches, what it is permitted to do and its purpose. It is the prerequisite for every other governance control, since controls are applied per system, and it is required by ISO/IEC 42001.

The international management-system standard for artificial intelligence, published in 2023, which sets out how an organisation should govern its AI systems: an inventory, risk assessment and treatment, defined roles, controls, and records showing all of it operates. It is certifiable by accredited bodies; a firm that prepares an organisation for it should not also be the one certifying it.

The European Union's regulation of AI systems, in force since 2024 with obligations phasing in through 2027, which classifies systems by risk and imposes logging, documentation, human-oversight and transparency requirements on the regulated categories. It applies by where a system is used, not where its provider is based, so it can reach companies outside the EU whose systems or outputs are used within it.

A voluntary framework from the US National Institute of Standards and Technology for managing risk in AI systems, organised around four functions - Govern, Map, Measure and Manage. It is not certifiable and is widely used as a structure for an AI risk programme and as common vocabulary in customer security reviews.

India's 2023 data protection law, which governs how personal data of individuals in India is collected, processed and retained, and which reaches AI systems wherever personal data enters a prompt, a retrieval index, a trace or a provider's logs. Its practical questions for an AI deployment are what personal data reaches which processor, under what consent, and for how long it is kept.

A record from which any run of an AI system can be reconstructed: the raw input, every version in play (prompt, model, retrieval index, tool schema, release), the retrieved context, each tool call with its arguments and result, the authorising identity, and the output - all linked by one trace id. A log of the request and response alone is not an audit trail, because nothing in an AI system is rerunnable afterwards.

Instrumentation that lets an operator see what an AI system did and why, across model calls, retrieval and tool calls, from live production traffic. It is usually built on OpenTelemetry traces and spans - a run is a trace, each call is a span - so that the AI feature is inspected in the same system as the rest of the software rather than in a separate console.

A change in an AI system's behaviour that nobody deployed: users start asking different things, the document corpus changes, or the provider updates the model underneath a fixed prompt. Offline evaluation catches regressions a team caused; drift is caught only by measuring in production and comparing against what the eval set says normal looks like.

Sending each request to the smallest or cheapest model that still passes the evaluation set for that kind of task, rather than sending everything to the largest available model. It is a behaviour change, so it is only safe after measurement; routed before an eval set exists, it is how a team downgrades the one call that needed the larger model.

A provider-side optimisation that reuses the processed form of a prompt prefix that is byte-identical across requests - typically the system prompt and any fixed context - so that it is not re-billed and re-computed each time. It is one of the cheapest efficiency levers available and is frequently found switched off on prompts that would benefit most.

The total model and infrastructure cost of one completed unit of work - a resolved ticket, a processed document, a qualified lead - including retries and every step of a chain, as opposed to a per-token rate or a monthly invoice. It is the only figure that lets a cheaper model needing two attempts be compared honestly with a dearer one needing one.