Course: 2B — Securing & Attacking Harnesses and LLMs Module: B0 — Legal, Ethics, and Disclosure for AI Security Testing Duration: 60 minutes Level: Senior Engineer and above Prerequisites: Course 1 complete (the harness is the other 98.4%; the model is 1.6%)
This module is non-optional. You are attacking AI systems. Every statute that governs attacking a database governs attacking the agent that can reach it — plus a layer of law, contract, and policy that has no analogue in traditional pentesting. An agentic red-team that runs outside authorization runs in your name, with your liability, against someone else's production model.
After completing this module, you will be able to:
Course 2A's S00 covered the legal foundation of using a security harness against traditional targets — networks, apps, contracts. Course 2B inverts the lens: the target is an AI system. That inversion changes the law in two ways.
First, the traditional computer-crime statutes still apply, but they stretch in non-obvious directions. A prompt injection that makes an agent read a file it was not authorized to read is, arguably, a CFAA "exceeds authorized access" event — except the access is being performed by the model, on behalf of an input you provided. A red-team that exfiltrates a provider's system prompt or model weights touches trade-secret law in a way that scanning a port never does. The "is it a computer?" question, long settled for a database, is freshly contested for a model endpoint.
Second, a new layer of AI-specific law now exists that has no pentest analogue. The EU AI Act (Regulation 2024/1689) prohibits certain uses outright and imposes obligations on general-purpose AI (GPAI) providers. US executive orders and OMB memoranda impose AI incident-reporting duties on federal agencies and their vendors. Several US states have enacted "bot disclosure" and "deepfake" laws whose edges touch red-team activity. And the DMCA anti-circumvention provisions (§ 1201) can criminalize the very techniques you use to test a model that sits behind a technical protection measure.
This module makes both layers explicit. It is the only module where the anti-patterns carry criminal penalties, civil liability, program bans, and — uniquely to AI — a realistic chance of enabling the harm you were hired to prevent. Get the legal and ethical framing right and the rest of the course is applied rigor on a defensible foundation.
Three sub-sections, twenty minutes each:
What constitutes authorized testing of an AI system vs. illegal access — and the new AI-specific layer that traditional pentesters never had to learn.
The three jurisdictional frameworks from 2A S00 apply unchanged to the infrastructure around an AI system: the server the model runs on, the database it reads, the API gateway in front of it. If your red-team touches those, the law is the law from S00.
The operational rule across all three is identical to S00: the difference between authorized testing and a crime is the asset owner's consent, evidenced in writing, and bounded by scope. What changes in 2B is what counts as the asset and what counts as access.
Three legal questions that traditional pentesting never had to answer:
This is the most contested question in AI security law. The argument runs: a prompt injection causes the agent to take an action (read a file, call a tool, exfiltrate data) that the deployer did not intend. Is the deployer "the computer's owner," is the injected instruction "without authorization," and is the resulting tool call "obtaining information"?
The honest answer in 2026 is that there is no settled case law. Scholarly analysis (e.g., work cited by the Stanford HAI policy program) and the OWASP Agentic Top 10 framings treat prompt-injection-driven unauthorized access as a plausible CFAA theory for the attacker who injects the payload, but a weak theory for a red-teamer testing their own employer's system with authorization. The practical consequence: test only systems you own or are explicitly authorized to red-team. A prompt-injection test against a third-party model you merely use (even one you pay for via API) is untested legal territory. The authorization chain (below) is your protection, not the novelty of the theory.
Yes — when the provider has taken reasonable steps to keep them secret. A red-team that exfiltrates model weights to prove they can has not just "accessed information"; it has potentially misappropriated a trade secret under the Defend Trade Secrets Act (DTSA, 18 U.S.C. § 1836) in the US, or the EU Trade Secrets Directive (2016/943). The "I did it to prove a vulnerability" defense is weaker here than for a SQL injection proof, because the weight file itself is the protected asset and copying it is the harm — there is no "COUNT(*)" equivalent that proves extractability without taking the secret.
The control: never exfiltrate weights to prove extractability. Prove the path exists (you can reach the weight store, you can read N bytes), capture the minimum (a hash of a few weights, not the file), and report. The finding is "the weight store is reachable," not "I have your model." This is the B0.3 weight-exfiltration control, foreshadowed.
DMCA § 1201 makes it a crime to circumvent a "technological measure that effectively controls access to a protected work." If a model is behind an API key, a rate limiter, a watermark, or an output filter, and your red-team bypasses that measure to reach the model or its outputs, you may be committing a § 1201 violation independent of any CFAA question. This is the same statute that was used against security researchers in the early 2000s (the DeCSS cases, the printer-cartridge cases) and it has a checkered history with researchers.
The Library of Congress issues triennial § 1201 exemptions. As of the 2024 rulemaking, there is a security-testing exemption (17 C.F.R. § 201.40) for good-faith testing of devices and systems — but it is narrow and it does not clearly cover AI models. The safe rule: if your test requires bypassing a technical access control on a model you do not own, get explicit written authorization that waives the provider's § 1201 rights for the engagement, or do not run the test. This is why B0.1's authorization agreement (below) has a DMCA-waiver clause.
Beyond the traditional statutes, three AI-specific regimes now constrain or govern red-team activity:
The operational synthesis: the traditional computer-crime statutes govern the infrastructure; the AI-specific layer governs the model and its outputs. A complete AI red-team authorization must address both. The agreement below does.
Authorization is a chain — from asset owner down to the specific prompt, tool call, or extraction your red-team is about to execute. For an AI system the chain is one link longer than 2A's, because the "asset owner" splits into the deployer (the organization running the agent in production) and the provider (the organization that built the model — OpenAI, Anthropic, a foundation-model vendor). They are often different entities with different authorizations.
Model Provider (e.g., OpenAI, Anthropic)
│ built the model; owns the weights; sets the API ToS
▼
Deployer / Asset Owner (your client)
│ runs the agent in production; owns the data the agent touches
▼
Red-team Engagement Sponsor
│ commissioned the AI red-team (often the deployer's CISO/AI-security lead)
▼
Engagement Scope / Rules of Engagement (RoE)
│ defines in-scope systems, allowed techniques, exclusions, dual-use handling
▼
Scope File (machine-checkable)
│ the formalized boundary: endpoints, models, data classes, techniques
▼
Red-team Harness Scope Enforcement Layer
│ intercepts every outbound test action (prompt, tool call, extraction)
▼
Individual Test Action (a prompt injection, a weight-read, a jailbreak)
│ legal cover exists only if the full chain above is intact
Three breaks specific to AI red-teaming:
Deployer ≠ provider, and provider ToS forbids it. Your client (deployer) authorizes you to red-team their agent — but the agent calls an OpenAI model, and OpenAI's API ToS (or usage policy) prohibits "attempts to reverse engineer, extract, or otherwise access the model weights" or "jailbreaking." The deployer cannot authorize you to violate the provider's terms. This is the single most common gap in AI red-team engagements. The fix: the engagement must either (a) stay within what the provider's terms permit, (b) use the provider's official red-team/preview program (several providers operate one), or (c) obtain a separate provider waiver. Read the provider ToS before scoping.
Dual-use handling is not in the RoE. A traditional pentest RoE covers scope and behavior. An AI red-team RoE must additionally cover what happens when you find a jailbreak that could enable misuse — can you publish it? share it with the provider? with the client only? A red-team that discovers a 5-query jailbreak (cf. RedAgent, SDD-B06) holds a dual-use artifact. An RoE without a dual-use clause is an RoE that fails at exactly the moment a serious finding appears.
The model is not the only target, but it is the most sensitive. A red-team that pivots from the agent to its retrieval store, its memory, or its tool surface is testing the deployer's infrastructure — back under the traditional CFAA/CMA regime. But the moment the test touches the model weights, system prompt, or training data, it is in the trade-secret and provider-ToS layer. Scope files must distinguish "deployer-controlled surfaces" from "provider-controlled surfaces" and enforce different rules for each.
When you red-team a deployed AI system (not a CTF, not a benchmark — a production agent), the engagement contract must carry clauses a traditional pentest SOW does not. The clauses that matter, every time:
gpt-model-v3 may not apply to v4, and the scope must pin it.The lab for this sub-section has you read a real model-provider acceptable-use policy (OpenAI's, Anthropic's) and a sample AI red-team SOW, and formalize the boundary as a JSON scope file that separates deployer-controlled from provider-controlled surfaces. That JSON is the input to the red-team harness's scope enforcement layer.
Coordinated disclosure for AI — and the dual-use dilemma that makes it harder than disclosing a buffer overflow.
Coordinated Vulnerability Disclosure (CVD) is the baseline norm inherited from traditional security: report privately, give the vendor a window, coordinate public release. The reference timelines are the same as 2A S00 — CERT/CC's 45-day coordination default, vendor-specific policies (Google Project Zero's 90 days), program-policy timelines governing over the generic default.
For AI, three things extend CVD:
This is the ethical problem that has no analogue in 2A. A successful jailbreak is simultaneously:
The same artifact is both. Traditional security has dual-use problems (an exploit can be used offensively), but the gap between "security research" and "offensive capability" is wider for a buffer overflow than for a jailbreak prompt, which is literally copy-paste usable by a non-technical attacker.
The principles that resolve the tension — drawn from the emerging consensus in the AI red-team community (Microsoft AI Red Team, Anthropic's responsible disclosure guidance, OpenAI's bug bounty program):
An AI security finding without evidence is an opinion. The minimum evidence, adapted from 2A S00:
| Field | Why |
|---|---|
| Timestamp (UTC) | Establishes when — critical for model-version disputes (the finding may not reproduce on the next version) |
| Model version / checkpoint | The single most important field. A finding against claude-opus-4-1 is meaningless without the version; providers ship silent updates |
| Exact prompt / payload (the injection, the jailbreak) | Reproducibility — the provider must be able to replay it |
| Exact output (the model's response) | Proof of impact |
| Temperature / sampling params | Jailbreak success is often sampling-dependent; a finding at temp=1.0 may not reproduce at temp=0 |
| Success rate (N attempts / M successes) | A single successful jailbreak is a weak finding; a 60% success rate over 100 attempts is a strong one. InjecAgent-style harness measurement (SDD-B03) is the standard |
| System prompt / configuration in force | For a deployed-agent finding, the harness config determines the attack surface |
| Scope reference (which authorization permitted this) | The legal anchor |
The half most engineers skip — and where AI-specific law bites hardest:
The retention policy (built in the lab) adds AI-specific classes: Public (technique description, no recipe), Provider-Only (the recipe, shared under NDA with the provider), Restricted (PII/weights, destroyed on report), Destroy-on-Report (jailbreak prompts).
Where red-teaming an AI system creates exposure a traditional pentester never had — and the control for each.
This is the risk with the least legal precedent and the highest stakes. Proving that an attacker could exfiltrate model weights is a valuable finding. Actually exfiltrating them is trade-secret misappropriation under the DTSA and EU Trade Secrets Directive, potentially a CFAA violation, and — for models subject to export controls or national-security thresholds — potentially more.
The control: minimum-proof discipline, enforced in the harness. A weight-extraction test should:
The red-team harness must enforce this — a test that "accidentally" exfiltrates 40GB of weights has committed the harm it was hired to prevent. Scope enforcement (B0.1) must treat the weight store as a destroy-on-read target.
Covered in B0.2. The risk, restated as liability: a tester who publishes a working jailbreak that is then used to generate illegal content (CSAM synthesis instructions, bioweapon recipes, etc.) faces a moral and, in some emerging readings, a legal exposure — even if the publication was "responsible disclosure." Several jurisdictions are actively considering liability for AI-misuse-enabling publications.
The control: the four disclosure principles in B0.2, codified in the engagement RoE before testing begins. The decision of whether to publish a recipe is not made in the moment; it is made by the clause the client signed.
A red-team that fuzzes a production model endpoint aggressively can degrade the service for other users of that model — especially on shared inference infrastructure. A rate-limit-busting jailbreak sweep against a shared API is the AI analogue of the 2A S00 "scanner becomes a DoS" problem, except the victim is not your target but every other tenant on the provider's infrastructure.
The control: rate-aware testing and isolation. Test against a dedicated endpoint, a preview/research tier, or a local model where possible. When testing shared production, cap concurrency and volume below the provider's stated limits, and coordinate timing with the provider. Provider preview programs (OpenAI, Anthropic, Google all operate them) exist partly to solve this — use them.
Restated as a risk: a red-team that relies solely on the deployer's authorization and proceeds to jailbreak or extract from a model whose provider ToS forbids it has committed a ToS breach — and ToS breaches, depending on jurisdiction and framing, can be CFAA "exceeds authorized access" events (the theory is contested post-Van Buren, but it is not dismissed).
The control: provider-authorization verification as a precondition. Before any test that touches provider-controlled surfaces (weights, system prompt, jailbreak), the red-team harness verifies one of: (a) provider ToS explicitly permits the technique, (b) a provider-issued waiver/preview enrollment is on file, (c) the model is self-hosted/open-weights and the deployer owns it. The harness scope file carries a provider_authorization field per technique. This is the AI analogue of 2A's "scope enforcement is a legal control."
Unique to AI: a red-team that runs thousands of jailbreak prompts against a model that learns from its interactions (some deployed systems log and retrain on user inputs) can contaminate the model — teaching it the attack patterns, or worse, accidentally fine-tuning in a vulnerability. The red-team's own test traffic becomes training data.
The control: opt-out of training and test on frozen checkpoints. Verify the model endpoint is not feeding a training loop (most provider APIs offer an explicit "do not train on my data" setting). For maximum fidelity, test against a frozen model checkpoint or a research tier that does not log to training. Document the checkpoint version in every finding.
The most common and most dangerous AI red-team mistake. The deployer can authorize testing of their system; they cannot authorize violation of the provider's terms. Cure: verify provider ToS compliance or obtain a provider waiver before touching provider-controlled surfaces. Read the ToS before scoping.
Downloading the full model to "prove" it can be done. Cure: minimum-proof discipline — path + hash + byte count, never the file. The proof is reachability, not possession.
Including the copy-paste-usable recipe in a public writeup. Cure: publish existence and severity; share the recipe with the provider under NDA; extend the coordination window for AI findings (180 days for model-level).
Sweeping thousands of jailbreaks against a shared API endpoint. Cure: use a dedicated/preview tier, a local model, or a frozen checkpoint; cap volume below provider limits; coordinate timing.
Reporting "the prompt injection is fixed" after a guardrail update. Cure: measure residual risk with a harness (InjecAgent-style) and report the success rate before and after the mitigation.
Treating the model's behavior as the authorization layer for your test actions. Cure: the operator (you) is liable; scope enforcement in code; the model's compliance is not your legal cover.
| Term | Definition |
|---|---|
| CFAA | Computer Fraud and Abuse Act, 18 U.S.C. § 1030 — the primary US federal computer-intrusion statute; § 1030(a)(2) access, § 1030(a)(5) damage |
| Van Buren (2021) | Supreme Court "gates-down" reading of "exceeds authorized access"; narrowed the CFAA, did not create a good-faith defense |
| DTSA | Defend Trade Secrets Act, 18 U.S.C. § 1836 — civil trade-secret misappropriation; model weights are protected when kept secret |
| DMCA § 1201 | Anti-circumvention provision; bypassing a technical access control on a model may be a separate crime from any CFAA question |
| EU AI Act | Regulation 2024/1689 — first comprehensive AI law; prohibits certain uses (Art. 5), governs high-risk systems (Title III) and GPAI (Title VIII) |
| Provider ToS | The model provider's terms of service; the deployer cannot authorize its violation |
| Deployer vs. provider | The split in the AI authorization chain — the deployer runs the agent; the provider built the model; their authorizations differ |
| Dual-use dilemma | A jailbreak is both a security finding (report) and a misuse recipe (suppress); the central ethical problem of AI red-teaming |
| Minimum-proof discipline | Capture the path/hash/byte-count, not the artifact; the proof is reachability, not possession |
| AI red-team scope file | Machine-checkable boundary separating deployer-controlled from provider-controlled surfaces, with a provider_authorization field per technique |
| CVD for AI | Coordinated disclosure extended: longer windows (180 days for model-level), residual-risk measurement, recipe-suppression by default |
See 07-lab-spec.md. Three labs, one per sub-section: (1) read a real model-provider AUP (OpenAI/Anthropic) and a sample AI red-team SOW, formalize the boundary as a JSON scope file separating deployer- from provider-controlled surfaces; (2) write an AI-specific CVD timeline template and a dual-use disclosure decision rubric; (3) build a provider_authorization_check() function the red-team harness runs before any provider-surface technique. The JSON scope file and the authorization check are the legal layer made engineering — they feed B1's threat-model scope enforcement.
genai.owasp.org/resource/owasp-top-10-for-agentic-applications-for-2026/. The canonical risk taxonomy framing prompt-injection-driven unauthorized access.# Module B0 — Legal, Ethics, and Disclosure for AI Security Testing
**Course**: 2B — Securing & Attacking Harnesses and LLMs
**Module**: B0 — Legal, Ethics, and Disclosure for AI Security Testing
**Duration**: 60 minutes
**Level**: Senior Engineer and above
**Prerequisites**: Course 1 complete (the harness is the other 98.4%; the model is 1.6%)
> *This module is non-optional. You are attacking AI systems. Every statute that governs attacking a database governs attacking the agent that can reach it — plus a layer of law, contract, and policy that has no analogue in traditional pentesting. An agentic red-team that runs outside authorization runs in your name, with your liability, against someone else's production model.*
---
## Learning Objectives
After completing this module, you will be able to:
1. Distinguish authorized security testing of an AI system from illegal access under the CFAA (US), the Computer Misuse Act (UK), and EU member-state regimes — and explain what *Van Buren v. United States* (2021) changed and what it did not.
2. Apply the laws that are **specific to AI** — the EU AI Act's prohibitions and GPAI obligations, the US AI-reporting regimes (incident disclosure under the AI EO / OMB rules), and DMCA §1201 anti-circumvention as it applies to model weight extraction and red-teaming a model behind an API.
3. Map the AI-specific attack surface to legal categories: prompt injection vs. unauthorized access, model weight exfiltration vs. trade-secret theft, jailbreak disclosure vs. coordinated vulnerability disclosure, and "does this CFAA argument still work when the 'computer' is a model API?"
4. Write an AI-specific authorization and disclosure agreement that covers model-card misrepresentation, red-team/preview-program safe harbor, weight-leak handling, and the disclosure of jailbreaks that could enable misuse.
5. Explain the **dual-use dilemma unique to AI red teaming**: a successful jailbreak is simultaneously a security finding (report it) and a misuse recipe (suppress it), and articulate the disclosure principles that resolve the tension.
6. Define the five AI-specific legal/ethical risks (weight exfiltration, dual-use jailbreak publication, third-party model harm, provider ToS conflict, and test-data contamination) and the control for each.
---
## Why this module exists
Course 2A's S00 covered the legal foundation of *using* a security harness against traditional targets — networks, apps, contracts. Course 2B inverts the lens: the target is an **AI system**. That inversion changes the law in two ways.
First, the *traditional* computer-crime statutes still apply, but they stretch in non-obvious directions. A prompt injection that makes an agent read a file it was not authorized to read is, arguably, a CFAA "exceeds authorized access" event — except the access is being performed *by the model*, on behalf of an input *you* provided. A red-team that exfiltrates a provider's system prompt or model weights touches trade-secret law in a way that scanning a port never does. The "is it a computer?" question, long settled for a database, is freshly contested for a model endpoint.
Second, a **new layer of AI-specific law** now exists that has no pentest analogue. The EU AI Act (Regulation 2024/1689) prohibits certain uses outright and imposes obligations on general-purpose AI (GPAI) providers. US executive orders and OMB memoranda impose AI incident-reporting duties on federal agencies and their vendors. Several US states have enacted "bot disclosure" and "deepfake" laws whose edges touch red-team activity. And the DMCA anti-circumvention provisions (§ 1201) can criminalize the very techniques you use to test a model that sits behind a technical protection measure.
This module makes both layers explicit. It is the only module where the anti-patterns carry criminal penalties, civil liability, program bans, and — uniquely to AI — a realistic chance of *enabling the harm you were hired to prevent*. Get the legal and ethical framing right and the rest of the course is applied rigor on a defensible foundation.
Three sub-sections, twenty minutes each:
- **B0.1 — Authorization Law for AI Systems.** The statutes (traditional + AI-specific), the authorization chain for an AI red-team, and the AI-target engagement contract.
- **B0.2 — Responsible Disclosure of AI Vulnerabilities.** CVD for AI, the dual-use dilemma, and what evidence you must keep vs. destroy when the finding is a jailbreak or a weight-leak.
- **B0.3 — AI-Specific Legal and Ethical Risks.** Weight exfiltration, dual-use publication, provider ToS conflict, and the controls for each.
---
# B0.1 — Authorization Law for AI Systems
*What constitutes authorized testing of an AI system vs. illegal access — and the new AI-specific layer that traditional pentesters never had to learn.*
## The traditional statutes still apply
The three jurisdictional frameworks from 2A S00 apply unchanged to the *infrastructure* around an AI system: the server the model runs on, the database it reads, the API gateway in front of it. If your red-team touches those, the law is the law from S00.
- **United States — the CFAA (18 U.S.C. § 1030).** Accessing a computer "without authorization" or "exceeding authorized access" and obtaining information (§ 1030(a)(2)); intentional, reckless, or negligent damage (§ 1030(a)(5)). *Van Buren v. United States*, 593 U.S. 374 (2021) adopted the "gates-down" reading: "exceeds authorized access" means accessing a file/folder/database you are *not authorized to access at all*, not accessing an authorized area for an improper purpose. This *helps* good-faith researchers accused of accessing data they were permitted to access but for a disfavored reason, but it did not create a general good-faith defense and it did not touch the threshold question of *authorization itself*.
- **United Kingdom — the Computer Misuse Act 1990 (CMA).** Section 1 (unauthorised access); section 3 (unauthorised acts). Broader than the post-*Van Buren* CFAA; no statutory researcher safe harbor. A red-team that reaches a UK-hosted model needs written authorization from the asset owner.
- **European Union — Directive 2013/40/EU** (transposed into national law) plus the **GDPR** (the dominant risk if your test touches personal data in the model's training set, memory, or retrieval store) plus **NIS2** (incident-reporting duties).
The operational rule across all three is identical to S00: **the difference between authorized testing and a crime is the asset owner's consent, evidenced in writing, and bounded by scope.** What changes in 2B is *what counts as the asset* and *what counts as access*.
## What changes when the target is an AI system
Three legal questions that traditional pentesting never had to answer:
### 1. Is a prompt injection "access" under the CFAA?
This is the most contested question in AI security law. The argument runs: a prompt injection causes the agent to take an action (read a file, call a tool, exfiltrate data) that the *deployer* did not intend. Is the deployer "the computer's owner," is the injected instruction "without authorization," and is the resulting tool call "obtaining information"?
The honest answer in 2026 is that **there is no settled case law.** Scholarly analysis (e.g., work cited by the Stanford HAI policy program) and the OWASP Agentic Top 10 framings treat prompt-injection-driven unauthorized access as a *plausible* CFAA theory for the *attacker who injects the payload*, but a weak theory for a *red-teamer* testing their own employer's system with authorization. The practical consequence: **test only systems you own or are explicitly authorized to red-team.** A prompt-injection test against a third-party model you merely *use* (even one you pay for via API) is untested legal territory. The authorization chain (below) is your protection, not the novelty of the theory.
### 2. Are model weights a trade secret?
Yes — when the provider has taken reasonable steps to keep them secret. A red-team that *exfiltrates* model weights to prove they can has not just "accessed information"; it has potentially misappropriated a trade secret under the Defend Trade Secrets Act (DTSA, 18 U.S.C. § 1836) in the US, or the EU Trade Secrets Directive (2016/943). The "I did it to prove a vulnerability" defense is weaker here than for a SQL injection proof, because the *weight file itself* is the protected asset and copying it is the harm — there is no "COUNT(*)" equivalent that proves extractability without taking the secret.
The control: **never exfiltrate weights to prove extractability.** Prove the *path* exists (you can reach the weight store, you can read N bytes), capture the minimum (a hash of a few weights, not the file), and report. The finding is "the weight store is reachable," not "I have your model." This is the B0.3 weight-exfiltration control, foreshadowed.
### 3. Does the DMCA anti-circumvention provision (§ 1201) apply?
DMCA § 1201 makes it a crime to circumvent a "technological measure that effectively controls access to a protected work." If a model is behind an API key, a rate limiter, a watermark, or an output filter, and your red-team bypasses that measure to reach the model or its outputs, you may be committing a § 1201 violation *independent of* any CFAA question. This is the same statute that was used against security researchers in the early 2000s (the DeCSS cases, the printer-cartridge cases) and it has a checkered history with researchers.
The Library of Congress issues triennial § 1201 exemptions. As of the 2024 rulemaking, there is a **security-testing exemption** (17 C.F.R. § 201.40) for good-faith testing of devices and systems — but it is narrow and it does not clearly cover AI models. The safe rule: **if your test requires bypassing a technical access control on a model you do not own, get explicit written authorization that waives the provider's § 1201 rights for the engagement, or do not run the test.** This is why B0.1's authorization agreement (below) has a DMCA-waiver clause.
## The AI-specific legal layer (no pentest analogue)
Beyond the traditional statutes, three AI-specific regimes now constrain or govern red-team activity:
- **The EU AI Act (Regulation 2024/1689).** The world's first comprehensive AI law. It *prohibits* certain AI practices outright (Art. 5 — e.g., social scoring, certain biometric categorization) and imposes obligations on **high-risk AI systems** (Title III) and **general-purpose AI models** (Title VIII, added by the 2024 political agreement). For a red-team: if your test reveals that a deployed system violates an Art. 5 prohibition, you have uncovered a *legal non-compliance*, not just a security bug — and the reporting obligations differ. GPAI providers with "systemic risk" (frontier models) face additional duties including red-teaming *of their own* (Art. 55). Your client may be contracting you to satisfy *their* AI Act obligation.
- **US AI incident-reporting regimes.** Executive Order 14110 (2023, "Safe, Secure, and Trustworthy Development and Use of AI") and OMB Memorandum M-24-10 impose duties on federal agencies (and their vendors) to test, document, and remediate AI systems, and to report incidents. The Defense Production Act reporting threshold (developers of dual-use foundation models must report red-team results and safety incidents to Commerce) was a live obligation through early 2025; its status under subsequent administration policy should be verified before relying on it. The point: **a finding from an AI red-team may itself be a reportable event** the client is obligated to disclose to a regulator.
- **State-level "bot," "deepfake," and AI-disclosure laws.** California (AB 2013, SB 942), Texas, and others have enacted disclosure and transparency laws whose edges touch red-team activity — particularly if your test generates synthetic content that *looks* like a real person or a real institution's output. These are adjacent, not central, to most red-team work, but a tester who generates realistic phishing as a payload should know they exist.
**The operational synthesis:** the traditional computer-crime statutes govern the *infrastructure*; the AI-specific layer governs the *model and its outputs*. A complete AI red-team authorization must address both. The agreement below does.
## The authorization chain for an AI red-team
Authorization is a chain — from asset owner down to the specific prompt, tool call, or extraction your red-team is about to execute. For an AI system the chain is one link longer than 2A's, because the "asset owner" splits into the **deployer** (the organization running the agent in production) and the **provider** (the organization that built the model — OpenAI, Anthropic, a foundation-model vendor). They are often different entities with different authorizations.
```
Model Provider (e.g., OpenAI, Anthropic)
│ built the model; owns the weights; sets the API ToS
▼
Deployer / Asset Owner (your client)
│ runs the agent in production; owns the data the agent touches
▼
Red-team Engagement Sponsor
│ commissioned the AI red-team (often the deployer's CISO/AI-security lead)
▼
Engagement Scope / Rules of Engagement (RoE)
│ defines in-scope systems, allowed techniques, exclusions, dual-use handling
▼
Scope File (machine-checkable)
│ the formalized boundary: endpoints, models, data classes, techniques
▼
Red-team Harness Scope Enforcement Layer
│ intercepts every outbound test action (prompt, tool call, extraction)
▼
Individual Test Action (a prompt injection, a weight-read, a jailbreak)
│ legal cover exists only if the full chain above is intact
```
**Three breaks specific to AI red-teaming:**
1. **Deployer ≠ provider, and provider ToS forbids it.** Your client (deployer) authorizes you to red-team their agent — but the agent calls an OpenAI model, and OpenAI's API ToS (or usage policy) prohibits "attempts to reverse engineer, extract, or otherwise access the model weights" or "jailbreaking." *The deployer cannot authorize you to violate the provider's terms.* This is the single most common gap in AI red-team engagements. The fix: the engagement must either (a) stay within what the provider's terms permit, (b) use the provider's official red-team/preview program (several providers operate one), or (c) obtain a separate provider waiver. Read the provider ToS *before* scoping.
2. **Dual-use handling is not in the RoE.** A traditional pentest RoE covers scope and behavior. An AI red-team RoE must *additionally* cover what happens when you find a jailbreak that could enable misuse — can you publish it? share it with the provider? with the client only? A red-team that discovers a 5-query jailbreak (cf. RedAgent, SDD-B06) holds a dual-use artifact. An RoE without a dual-use clause is an RoE that fails at exactly the moment a serious finding appears.
3. **The model is not the only target, but it is the most sensitive.** A red-team that pivots from the agent to its retrieval store, its memory, or its tool surface is testing the *deployer's* infrastructure — back under the traditional CFAA/CMA regime. But the moment the test touches the *model weights, system prompt, or training data*, it is in the trade-secret and provider-ToS layer. Scope files must distinguish "deployer-controlled surfaces" from "provider-controlled surfaces" and enforce different rules for each.
## The AI red-team authorization agreement
When you red-team a deployed AI system (not a CTF, not a benchmark — a production agent), the engagement contract must carry clauses a traditional pentest SOW does not. The clauses that matter, every time:
- **Systems in scope.** The exhaustive list: which agents, which model versions, which endpoints, which data stores. Model version matters — a finding against `gpt-model-v3` may not apply to `v4`, and the scope must pin it.
- **Provider authorization / ToS compliance.** An explicit statement of how the engagement complies with the model provider's terms, OR a copy of a provider-issued waiver/preview-program enrollment. Without this, the deployer's authorization does not cover provider-controlled surfaces.
- **Techniques permitted / prohibited.** Prompt injection (direct, indirect, multi-step), memory poisoning, tool/MCP abuse, sandbox escape attempts, weight-extraction probes, jailbreak attempts. Each either permitted, permitted-with-minimum-proof, or prohibited. "Permitted" is not the default for weight extraction — see B0.3.
- **Dual-use and disclosure.** How jailbreak and misuse-enabling findings will be handled: report-to-provider-only, coordinated disclosure timeline, publication embargo, pre-publication review. This clause is the resolution of the dual-use dilemma (B0.2).
- **DMCA / anti-circumvention waiver.** If the test requires bypassing a technical access control on a model, an explicit waiver of the provider's § 1201 rights for the engagement scope and duration.
- **Data handling.** What the red-team may capture (system prompt, retrieved context, model outputs), how it must be stored, and the destruction timeline. Personal data in retrieved context triggers GDPR.
- **Incident reporting.** Acknowledgment that a finding may be a reportable event under the EU AI Act, US EO/OMB regimes, or sector rules, and who (client or tester) is responsible for the report.
**The lab for this sub-section** has you read a real model-provider acceptable-use policy (OpenAI's, Anthropic's) and a sample AI red-team SOW, and formalize the boundary as a JSON scope file that separates deployer-controlled from provider-controlled surfaces. That JSON is the input to the red-team harness's scope enforcement layer.
---
# B0.2 — Responsible Disclosure of AI Vulnerabilities
*Coordinated disclosure for AI — and the dual-use dilemma that makes it harder than disclosing a buffer overflow.*
## Coordinated Vulnerability Disclosure — the AI extension
Coordinated Vulnerability Disclosure (CVD) is the baseline norm inherited from traditional security: report privately, give the vendor a window, coordinate public release. The reference timelines are the same as 2A S00 — CERT/CC's 45-day coordination default, vendor-specific policies (Google Project Zero's 90 days), program-policy timelines governing over the generic default.
For AI, three things extend CVD:
1. **The "vendor" may be a chain.** A jailbreak in a deployed agent could originate in the foundation model's training (provider's responsibility), the system-prompt design (deployer's responsibility), or the harness's tool-surface exposure (also deployer's). CVD for an AI finding may require reporting to *both* the provider (so the base vulnerability can be fixed upstream) and the deployer (so the harness can be patched downstream). Get both in writing.
2. **Fix time is longer and less binary.** A buffer overflow is patched in a release. A prompt-injection susceptibility may be reduced by a guardrail update, a model-version bump, or a harness control — none of which is a clean "fixed." Coordinate on a *measured residual risk* (e.g., "injection success rate dropped from 60% to 4% under the InjecAgent harness") rather than a binary "fixed/unfixed."
3. **Disclosure content is itself sensitive.** A traditional advisory describes the bug and the patch. An AI advisory that includes the *working jailbreak prompt* hands a misuse recipe to anyone who reads it. This is the dual-use dilemma, and it is the central ethical problem of AI red-teaming.
## The dual-use dilemma
This is the ethical problem that has no analogue in 2A. A successful jailbreak is simultaneously:
- **A security finding** — proof that a safety control failed. The responsible thing is to report it, and, in the CVD tradition, to eventually publish so the field learns.
- **A misuse recipe** — a concrete prompt sequence that bypasses the model's refusal training. Publishing it hands a weapon to anyone who wants to generate disallowed content at scale.
The same artifact is both. Traditional security has dual-use problems (an exploit can be used offensively), but the gap between "security research" and "offensive capability" is wider for a buffer overflow than for a jailbreak prompt, which is *literally copy-paste usable* by a non-technical attacker.
The principles that resolve the tension — drawn from the emerging consensus in the AI red-team community (Microsoft AI Red Team, Anthropic's responsible disclosure guidance, OpenAI's bug bounty program):
1. **Report to the provider first, always.** Before any publication, before any sharing, the provider gets a private report and a coordination window. This is non-negotiable CVD.
2. **Publish the *existence and severity*, not the *recipe*, by default.** A responsible AI advisory says "we found a technique that bypasses refusal with X% success on model Y; the provider has mitigated in version Z." It does *not* include the working prompt unless the provider consents and the technique is independently well-known enough that publication adds no marginal misuse risk.
3. **Time-bound the embargo, but make it longer than 90 days.** The 90-day Project Zero default is calibrated to software patches measured in weeks. AI mitigations (retraining, RLHF rounds) are measured in months. A 180-day default coordination window is more realistic for a serious model-level jailbreak; harness-level fixes can stay on the 45–90 day track.
4. **Decline to publish pure-misuse-enabling content with no defensive value.** If a finding is "here is how to make the model produce disallowed content" and there is no *defensive* lesson (no new technique, no architectural insight), the responsible choice is often provider-only disclosure. This is a judgment call, and it is the one place AI red-teaming diverges from "publish everything."
## What evidence you must preserve
An AI security finding without evidence is an opinion. The minimum evidence, adapted from 2A S00:
| Field | Why |
| --- | --- |
| **Timestamp** (UTC) | Establishes *when* — critical for model-version disputes (the finding may not reproduce on the next version) |
| **Model version / checkpoint** | The single most important field. A finding against `claude-opus-4-1` is meaningless without the version; providers ship silent updates |
| **Exact prompt / payload** (the injection, the jailbreak) | Reproducibility — the provider must be able to replay it |
| **Exact output** (the model's response) | Proof of impact |
| **Temperature / sampling params** | Jailbreak success is often sampling-dependent; a finding at `temp=1.0` may not reproduce at `temp=0` |
| **Success rate** (N attempts / M successes) | A single successful jailbreak is a weak finding; a 60% success rate over 100 attempts is a strong one. InjecAgent-style harness measurement (SDD-B03) is the standard |
| **System prompt / configuration in force** | For a deployed-agent finding, the harness config determines the attack surface |
| **Scope reference** (which authorization permitted this) | The legal anchor |
## What evidence you must NOT retain
The half most engineers skip — and where AI-specific law bites hardest:
- **Personal data in retrieved context.** If your indirect-injection test causes the agent to retrieve and exfiltrate real user records, you now hold PII. Same rule as 2A S00: capture the *minimum proof* (one redacted row, or a count), classify as Restricted, destroy per policy. The proof is "the agent leaked," not the leaked data.
- **Model weights and system prompts.** These are trade secrets. If your test extracts either, *do not retain the full artifact.* Capture a hash, a byte-count, or a short prefix — enough to prove extractability — and report. Retaining the full system prompt or a weight shard is retaining a stolen trade secret, and the "I did it to prove the vuln" defense is weakest here.
- **Working jailbreak prompts beyond the engagement.** A vault of "jailbreaks I found on engagements" is a misuse-enabling dataset. Retain only what the engagement contract requires, destroy the rest, and never carry cross-client jailbreak artifacts — they are dual-use IP that belongs to the client who paid for the finding.
The retention policy (built in the lab) adds AI-specific classes: **Public** (technique description, no recipe), **Provider-Only** (the recipe, shared under NDA with the provider), **Restricted** (PII/weights, destroyed on report), **Destroy-on-Report** (jailbreak prompts).
---
# B0.3 — AI-Specific Legal and Ethical Risks
*Where red-teaming an AI system creates exposure a traditional pentester never had — and the control for each.*
## Risk 1 — Model weight exfiltration: proving extractability without stealing the secret
This is the risk with the least legal precedent and the highest stakes. Proving that an attacker *could* exfiltrate model weights is a valuable finding. *Actually exfiltrating* them is trade-secret misappropriation under the DTSA and EU Trade Secrets Directive, potentially a CFAA violation, and — for models subject to export controls or national-security thresholds — potentially more.
**The control: minimum-proof discipline, enforced in the harness.** A weight-extraction test should:
1. Prove the *path* — the harness can reach the weight store (read N bytes, hash them, never save them).
2. Capture a *fingerprint* — a hash of a small weight shard, enough to prove it is the real model and not a decoy.
3. Stop. Do not download the file. Do not save the shard. Log the path, the byte count, and the hash, then report.
The red-team harness must enforce this — a test that "accidentally" exfiltrates 40GB of weights has committed the harm it was hired to prevent. Scope enforcement (B0.1) must treat the weight store as a destroy-on-read target.
## Risk 2 — Dual-use publication: the responsible-disclosure judgment call
Covered in B0.2. The risk, restated as liability: a tester who publishes a working jailbreak that is then used to generate illegal content (CSAM synthesis instructions, bioweapon recipes, etc.) faces a *moral* and, in some emerging readings, a *legal* exposure — even if the publication was "responsible disclosure." Several jurisdictions are actively considering liability for AI-misuse-enabling publications.
**The control: the four disclosure principles in B0.2, codified in the engagement RoE before testing begins.** The decision of whether to publish a recipe is not made in the moment; it is made by the clause the client signed.
## Risk 3 — Third-party model harm: when your test damages a shared model
A red-team that fuzzes a production model endpoint aggressively can degrade the service for *other* users of that model — especially on shared inference infrastructure. A rate-limit-busting jailbreak sweep against a shared API is the AI analogue of the 2A S00 "scanner becomes a DoS" problem, except the victim is not your target but every other tenant on the provider's infrastructure.
**The control: rate-aware testing and isolation.** Test against a dedicated endpoint, a preview/research tier, or a local model where possible. When testing shared production, cap concurrency and volume below the provider's stated limits, and coordinate timing with the provider. Provider preview programs (OpenAI, Anthropic, Google all operate them) exist partly to solve this — use them.
## Risk 4 — Provider ToS conflict: the deployer cannot authorize what the provider forbids
Restated as a risk: a red-team that relies solely on the deployer's authorization and proceeds to jailbreak or extract from a model whose provider ToS forbids it has committed a ToS breach — and ToS breaches, depending on jurisdiction and framing, can be *CFAA "exceeds authorized access"* events (the theory is contested post-*Van Buren*, but it is not dismissed).
**The control: provider-authorization verification as a precondition.** Before any test that touches provider-controlled surfaces (weights, system prompt, jailbreak), the red-team harness verifies one of: (a) provider ToS explicitly permits the technique, (b) a provider-issued waiver/preview enrollment is on file, (c) the model is self-hosted/open-weights and the deployer owns it. The harness scope file carries a `provider_authorization` field per technique. This is the AI analogue of 2A's "scope enforcement is a legal control."
## Risk 5 — Test-data contamination: poisoning the model you are testing
Unique to AI: a red-team that runs thousands of jailbreak prompts against a model that *learns from its interactions* (some deployed systems log and retrain on user inputs) can *contaminate* the model — teaching it the attack patterns, or worse, accidentally fine-tuning in a vulnerability. The red-team's own test traffic becomes training data.
**The control: opt-out of training and test on frozen checkpoints.** Verify the model endpoint is not feeding a training loop (most provider APIs offer an explicit "do not train on my data" setting). For maximum fidelity, test against a frozen model checkpoint or a research tier that does not log to training. Document the checkpoint version in every finding.
---
## Anti-Patterns
### "The deployer authorized it, so the provider's terms don't matter"
The most common and most dangerous AI red-team mistake. The deployer can authorize testing of *their* system; they cannot authorize violation of the *provider's* terms. Cure: verify provider ToS compliance or obtain a provider waiver before touching provider-controlled surfaces. Read the ToS before scoping.
### Exfiltrating weights to prove extractability
Downloading the full model to "prove" it can be done. Cure: minimum-proof discipline — path + hash + byte count, never the file. The proof is reachability, not possession.
### Publishing the working jailbreak prompt with the advisory
Including the copy-paste-usable recipe in a public writeup. Cure: publish existence and severity; share the recipe with the provider under NDA; extend the coordination window for AI findings (180 days for model-level).
### Testing a shared production model at high volume
Sweeping thousands of jailbreaks against a shared API endpoint. Cure: use a dedicated/preview tier, a local model, or a frozen checkpoint; cap volume below provider limits; coordinate timing.
### Treating an AI finding as binary "fixed/unfixed"
Reporting "the prompt injection is fixed" after a guardrail update. Cure: measure residual risk with a harness (InjecAgent-style) and report the success rate before and after the mitigation.
### "The model did it"
Treating the model's behavior as the authorization layer for your test actions. Cure: the operator (you) is liable; scope enforcement in code; the model's compliance is not your legal cover.
---
## Key Terms
| Term | Definition |
| --- | --- |
| **CFAA** | Computer Fraud and Abuse Act, 18 U.S.C. § 1030 — the primary US federal computer-intrusion statute; § 1030(a)(2) access, § 1030(a)(5) damage |
| ***Van Buren* (2021)** | Supreme Court "gates-down" reading of "exceeds authorized access"; narrowed the CFAA, did not create a good-faith defense |
| **DTSA** | Defend Trade Secrets Act, 18 U.S.C. § 1836 — civil trade-secret misappropriation; model weights are protected when kept secret |
| **DMCA § 1201** | Anti-circumvention provision; bypassing a technical access control on a model may be a separate crime from any CFAA question |
| **EU AI Act** | Regulation 2024/1689 — first comprehensive AI law; prohibits certain uses (Art. 5), governs high-risk systems (Title III) and GPAI (Title VIII) |
| **Provider ToS** | The model provider's terms of service; the deployer cannot authorize its violation |
| **Deployer vs. provider** | The split in the AI authorization chain — the deployer runs the agent; the provider built the model; their authorizations differ |
| **Dual-use dilemma** | A jailbreak is both a security finding (report) and a misuse recipe (suppress); the central ethical problem of AI red-teaming |
| **Minimum-proof discipline** | Capture the path/hash/byte-count, not the artifact; the proof is reachability, not possession |
| **AI red-team scope file** | Machine-checkable boundary separating deployer-controlled from provider-controlled surfaces, with a `provider_authorization` field per technique |
| **CVD for AI** | Coordinated disclosure extended: longer windows (180 days for model-level), residual-risk measurement, recipe-suppression by default |
---
## Lab Exercise
See `07-lab-spec.md`. Three labs, one per sub-section: (1) read a real model-provider AUP (OpenAI/Anthropic) and a sample AI red-team SOW, formalize the boundary as a JSON scope file separating deployer- from provider-controlled surfaces; (2) write an AI-specific CVD timeline template and a dual-use disclosure decision rubric; (3) build a `provider_authorization_check()` function the red-team harness runs before any provider-surface technique. The JSON scope file and the authorization check are the legal layer made engineering — they feed B1's threat-model scope enforcement.
---
## References
1. **18 U.S.C. § 1030** — Computer Fraud and Abuse Act. § 1030(a)(2) (access); § 1030(a)(5) (damage).
2. ***Van Buren v. United States*, 593 U.S. 374 (2021)** — "gates-down" interpretation of "exceeds authorized access."
3. **18 U.S.C. § 1836** — Defend Trade Secrets Act. Civil trade-secret misappropriation; applicable to model weight exfiltration.
4. **17 U.S.C. § 1201** — DMCA anti-circumvention. Triennial exemptions at 17 C.F.R. § 201.40 (Library of Congress rulemaking).
5. **Regulation (EU) 2024/1689** — EU AI Act. Art. 5 (prohibited practices); Title III (high-risk); Title VIII / Art. 55 (GPAI, systemic-risk duties including red-teaming).
6. **Directive (EU) 2016/943** — EU Trade Secrets Directive. National transpositions govern weight-exfiltration liability in the EU.
7. **Executive Order 14110 (2023)** — "Safe, Secure, and Trustworthy Development and Use of AI." US AI governance; DPA reporting thresholds. Verify current administration policy before relying on specific provisions.
8. **OMB Memorandum M-24-10** — Advancing Governance, Innovation, and Risk Management for Agency Use of AI. Federal-agency AI duties.
9. **OWASP** — *Top 10 for Agentic Applications (2026)*. `genai.owasp.org/resource/owasp-top-10-for-agentic-applications-for-2026/`. The canonical risk taxonomy framing prompt-injection-driven unauthorized access.
10. **Microsoft AI Red Team** — published AI red-team methodology and the failure-mode taxonomy v2.0 (cf. SDD-B02).
11. **OpenAI / Anthropic responsible-disclosure & acceptable-use policies** — provider terms governing red-team activity on their models; the ToS layer the deployer cannot waive.
12. **CERT/CC** — Coordination norms; the 45-day default extended to ~180 days for AI model-level findings.
13. **Google Project Zero** — 90-day disclosure policy; the software baseline that AI disclosure extends.