{ "module": "B0 — Legal, Ethics, and Disclosure for AI Security Testing", "course": "2B — Securing & Attacking Harnesses and LLMs", "version": "1.0.0", "duration_minutes": 35, "total_questions": 15, "bloom_distribution": { "target": "20% recall / 40% application / 40% analysis-design", "actual": { "recall": 3, "application": 6, "analysis": 6 } }, "passing_score_percent": 70, "questions": [ { "id": "Q01", "bloom": "recall", "type": "multiple_choice", "prompt": "In an AI red-team engagement, the 'asset owner' splits into two entities that a traditional pentest treats as one. Which two?", "options": [ "The cloud provider and the CDN.", "The model PROVIDER (built the model, owns weights, sets ToS) and the DEPLOYER (runs the agent, owns data).", "The red-team sponsor and the legal counsel.", "The API gateway and the inference server." ], "answer_index": 1, "rationale": "This split is the foundational legal difference between 2A and 2B. The deployer can authorize testing of THEIR system but CANNOT authorize violation of the PROVIDER's terms. This is where most AI red-team legal mistakes happen — a test relying on deployer authorization that proceeds to jailbreak/extract a model whose ToS forbids it commits a ToS breach." }, { "id": "Q02", "bloom": "recall", "type": "multiple_choice", "prompt": "What did Van Buren v. United States (2021) establish, and what did it NOT do?", "options": [ "It legalized all good-faith security research; it did not affect the CFAA's scope.", "It adopted the 'gates-down' reading (exceeds authorized access = accessing something you're not authorized to access at all); it did NOT create a general good-faith defense or touch the threshold question of authorization itself.", "It made prompt injection explicitly legal under the CFAA.", "It overturned the Computer Fraud and Abuse Act entirely." ], "answer_index": 1, "rationale": "Van Buren narrowed the CFAA — 'exceeds authorized access' means accessing an area you're not authorized for at all, not accessing an authorized area for an improper purpose. This helps researchers accused of accessing permitted data for a disfavored reason. But it did NOT create a good-faith defense and did NOT change the binary gate: either the scope authorizes the target, or it's a potential violation." }, { "id": "Q03", "bloom": "recall", "type": "multiple_choice", "prompt": "Under what legal theory is exfiltrating model weights a distinct risk from a normal pentest data read?", "options": [ "It is not — weights are just data like any other.", "Trade-secret misappropriation under the DTSA (18 U.S.C. § 1836) and the EU Trade Secrets Directive (2016/943). Weights are a protected trade secret when kept secret; copying them IS the harm, so there's no 'COUNT(*) equivalent' that proves extractability without taking the secret.", "Only the DMCA applies.", "Only the GDPR applies, and only if the weights contain personal data." ], "answer_index": 1, "rationale": "Model weights are a trade secret. Unlike a SQL injection proof (where COUNT(*) demonstrates the vuln without taking data), copying the weight file IS the misappropriation — the artifact itself is the protected asset. This is why the control is minimum-proof discipline: prove the PATH (reach N bytes), capture a HASH, never save the file." }, { "id": "Q04", "bloom": "application", "type": "multiple_choice", "prompt": "A client (deployer) authorizes you to red-team their production agent, including jailbreak attempts against the OpenAI model it calls. OpenAI's Acceptable Use Policy prohibits jailbreaking. What is the correct action?", "options": [ "Proceed — the deployer's authorization covers the whole system.", "You cannot proceed on the deployer's authorization alone for the jailbreak. You must either (a) stay within what OpenAI's terms permit, (b) enroll in OpenAI's official red-team/preview program, or (c) obtain a separate provider waiver. The deployer cannot authorize violation of the provider's terms.", "Proceed but only at low volume to avoid detection.", "Ask the deployer to sign an indemnity and proceed." ], "answer_index": 1, "rationale": "This is the most common AI red-team legal mistake. The deployer owns the agent but not the model's terms. The authorization chain has a gap at the provider link. The fix: verify provider ToS compliance or obtain a provider waiver BEFORE touching provider-controlled surfaces. Build the provider_authorization check into the harness gate." }, { "id": "Q05", "bloom": "application", "type": "multiple_choice", "prompt": "During an authorized weight-extraction test, you confirm the harness can reach the model's weight store. What do you capture as proof?", "options": [ "Download the full weight file so you can demonstrate the exfiltration to the client.", "Save a 1GB shard to prove it's the real model, then delete it after the report.", "Read a small number of bytes, compute a hash of a small shard, log the path + byte count + hash, and STOP. Never save the shard. The finding is reachability, not possession.", "Screenshot the directory listing and move on." ], "answer_index": 2, "rationale": "Minimum-proof discipline. Downloading or saving weights is trade-secret misappropriation — the harm the test was hired to prevent. A hash of a small shard proves reachability (and proves it's the real model, not a decoy) without retaining the protected asset. The harness must enforce this — a test that 'accidentally' exfiltrates 40GB has committed the harm." }, { "id": "Q06", "bloom": "application", "type": "multiple_choice", "prompt": "You discover a 5-query jailbreak against a frontier model with 60% success over 100 attempts. Which disclosure approach is responsible?", "options": [ "Publish the full working prompt immediately with a 45-day embargo, following the Project Zero model.", "Report to the provider first under NDA; publish existence + severity (60% over 100 attempts) but NOT the recipe; set a ≥180-day embargo for this model-level finding; withhold publication if it's pure misuse with no defensive lesson.", "Publish the full recipe on a personal blog to establish priority.", "Sell the finding to a third party since it's dual-use." ], "answer_index": 1, "rationale": "The four disclosure principles: (1) provider first; (2) existence+severity, not recipe, by default; (3) ≥180 days for model-level findings (RLHF/retraining is months, not weeks); (4) withhold pure-misuse content with no defensive lesson. The success-rate measurement (60% over 100) is the right metric, not a binary 'jailbroken.' The decision was made in the RoE before testing." }, { "id": "Q07", "bloom": "application", "type": "multiple_choice", "prompt": "Your indirect-injection test causes the agent to retrieve and surface real customer records. What do you retain in the evidence store?", "options": [ "The full record set, so the client can see the severity.", "One redacted row (or a COUNT) proving the agent can read the table, classified as Restricted, destroyed on report submission. The proof is 'the agent leaked,' not the leaked data.", "Nothing — discard all evidence immediately.", "Only the SQL query, not the result." ], "answer_index": 1, "rationale": "Same rule as 2A S00, applied to the AI context: capture the minimum proof. Retaining real PII makes you a data controller (GDPR) and triggers breach-notification law. The vulnerability is proven by demonstrating the agent CAN read the records (a count, a redacted row), not by possessing the records. Restricted class, destroy on report." }, { "id": "Q08", "bloom": "application", "type": "multiple_choice", "prompt": "Why is a single successful jailbreak (1 attempt, 1 success) a weak finding, and what should you report instead?", "options": [ "It's a strong finding — any success proves the model is broken.", "Jailbreak success is sampling-dependent and stochastic. Report a SUCCESS RATE over N attempts (e.g. 60/100) plus the sampling parameters (temperature), using InjecAgent-style harness measurement. A single success is an anecdote.", "Report only the model version; success rate is irrelevant.", "Retry until you get 100% success, then report." ], "answer_index": 1, "rationale": "Jailbreaks are stochastic. Temperature, sampling, and model state all affect success. The standard is a measured success rate over a statistically meaningful N (InjecAgent, cf. SDD-B03). '60% over 100 attempts' is a strong finding; '1/1' is an anecdote. Always report N/M and the sampling config so the provider can reproduce." }, { "id": "Q09", "bloom": "application", "type": "multiple_choice", "prompt": "DMCA § 1201 (anti-circumvention) is relevant to AI red-teaming in which scenario?", "options": [ "It is never relevant to AI testing.", "When a test requires bypassing a technical access control on a model you do not own (API key, rate limiter, watermark, output filter). Bypassing it may be a § 1201 violation INDEPENDENT of any CFAA question. The Library of Congress triennial exemptions include a narrow security-testing exemption that does NOT clearly cover AI models.", "It applies only to copyright infringement of training data.", "It applies only when you publish the jailbreak." ], "answer_index": 1, "rationale": "§ 1201 criminalizes circumventing a technological access control on a protected work — a model behind an API is behind such a control. Bypassing it is a separate crime from any CFAA access question. The triennial exemptions (17 C.F.R. § 201.40) have a security-testing carve-out but it's narrow and doesn't clearly cover AI. The fix: get an explicit ToS/§ 1201 waiver before bypassing a model access control, or don't run the test." }, { "id": "Q10", "bloom": "analysis", "type": "multiple_choice", "prompt": "Under the EU AI Act (Regulation 2024/1689), how might a red-team finding have a legal significance beyond a normal security bug?", "options": [ "It never does — the AI Act governs product safety, not security testing.", "If the finding reveals a deployed system violates an Art. 5 prohibition (e.g. social scoring), you have uncovered a LEGAL NON-COMPLIANCE, not just a security bug — with different reporting obligations. For GPAI providers with systemic risk, Art. 55 imposes red-teaming duties your client may be contracting you to satisfy.", "All findings must be reported to the EU Commission within 24 hours.", "The AI Act only applies to training data, not to deployed agents." ], "answer_index": 1, "rationale": "The EU AI Act is the AI-specific layer with no pentest analogue. Art. 5 prohibits certain practices outright — a finding that a system does a prohibited thing is legal non-compliance. Title III governs high-risk systems; Title VIII (Art. 55) imposes duties on systemic-risk GPAI providers including red-teaming. A client may be contracting you to satisfy THEIR AI Act obligation, and the finding itself may be a reportable event." }, { "id": "Q11", "bloom": "analysis", "type": "multiple_choice", "prompt": "Your red-team sweeps thousands of jailbreak attempts against a shared production model endpoint. Other users of that provider start reporting degraded service. What risk have you encountered, and what is the control?", "options": [ "No risk — it's in scope. Continue.", "Third-party model harm — the AI analogue of the 2A 'scanner becomes a DoS' problem, except the victim is every OTHER tenant on shared inference infrastructure. Control: test against a dedicated/preview tier or a local/frozen model; cap volume below provider limits; coordinate timing. Use provider preview programs.", "A CFAA damage charge — stop all testing permanently.", "A GDPR violation because the model learned your prompts." ], "answer_index": 1, "rationale": "Shared model infrastructure means aggressive testing degrades service for unrelated tenants. This is the 2A DoS-by-scanner problem transposed to AI, with the twist that the victim is not your target. The control is isolation: dedicated endpoints, preview/research tiers, local models, or frozen checkpoints. Provider preview programs exist partly to solve this — use them." }, { "id": "Q12", "bloom": "analysis", "type": "multiple_choice", "prompt": "A deployed model logs and retrains on user inputs. Your jailbreak test traffic is fed into the training loop. What risk is this, and how do you prevent it?", "options": [ "It improves the model — no action needed.", "Test-data contamination — the AI-specific risk that your own test traffic becomes training data, teaching the model the attack patterns or accidentally fine-tuning in a vulnerability. Control: opt out of training (most provider APIs offer this), test on a frozen checkpoint or research tier that does not log to training, and document the checkpoint version in every finding.", "A trade-secret issue — you now own part of the model.", "Unavoidable — all production testing has this property." ], "answer_index": 1, "rationale": "Test-data contamination is unique to AI: the red-team's own inputs can become training data. The control is to test against models that are NOT learning from your traffic — set 'do not train on my data,' use a frozen checkpoint, or use a research tier. Documenting the checkpoint version is also critical because a finding may not reproduce after the model retrains." }, { "id": "Q13", "bloom": "analysis", "type": "multiple_choice", "prompt": "Design the provider_authorization field in an AI red-team scope file. What does it record per technique, and when does it block?", "options": [ "It records the deployer's internal approval and never blocks.", "Per (technique, target model), it records which of three conditions holds: (a) provider ToS explicitly permits the technique; (b) a provider waiver/preview enrollment is on file and unexpired; (c) the model is self-hosted/open-weights owned by the deployer. If NONE hold, the harness BLOCKS the technique. This is the AI analogue of 'scope enforcement is a legal control.'", "It records the price of the API calls and blocks on budget overruns.", "It records the model's safety training version and blocks on version mismatch." ], "answer_index": 1, "rationale": "The provider_authorization field enforces the key legal control: the deployer cannot authorize what the provider forbids. Per technique (jailbreak, weight_read, system_prompt_extract) and per target model, it verifies provider permission. If none of the three conditions hold, the harness refuses to execute. This prevents the most common AI red-team legal mistake at the engineering layer." }, { "id": "Q14", "bloom": "analysis", "type": "multiple_choice", "prompt": "Why does AI red-team CVD diverge from the traditional 'publish everything' norm, and in what specific case?", "options": [ "It does not — all AI findings should be published like software bugs.", "A jailbreak is dual-use: a copy-paste-usable recipe that enables misuse. AI CVD publishes existence + severity by default (not the recipe), uses longer embargoes (~180 days for model-level), and in the specific case of PURE-MISUSE content with NO defensive lesson (no new technique, no architectural insight), withholds publication entirely (provider-only disclosure). This is where AI red-teaming diverges.", "AI findings should never be published because they are always dual-use.", "Only the provider may ever publish AI findings." ], "answer_index": 1, "rationale": "The dual-use dilemma resolves through the four principles. The divergence from 'publish everything' is specific: pure-misuse content with no defensive value (no new technique) is withheld, because publishing it hands a weapon to non-technical attackers with no offsetting benefit to defenders. Findings WITH a defensive lesson (a new technique class, an architectural insight) are published after the longer embargo. The decision is made in the RoE, not in the moment." }, { "id": "Q15", "bloom": "analysis", "type": "multiple_choice", "prompt": "You are scoping an AI red-team. The client's agent uses a system prompt, a vector retrieval store with customer data, and a code-execution tool. Construct the scope-file's surface separation and the data-class rules.", "options": [ "Treat everything as one surface with Public retention.", "Separate DEPLOYER-controlled surfaces (system prompt, retrieval store, code-exec tool) from PROVIDER-controlled surfaces (model weights, base refusal layer). For deployer surfaces: system prompt = Provider-Only (trade-secret-adjacent); retrieval store may contain PII = Restricted, destroy-on-report; code-exec output = Public if no PII. For provider surfaces: weight_read/jailbreak require provider_authorization to be ALLOW before the harness executes them.", "Treat the retrieval store as Public because it's just vectors.", "Treat the system prompt as Public because prompts are not secrets." ], "answer_index": 1, "rationale": "The scope file separates surfaces by who controls them (deployer vs provider) and assigns data-class + retention rules per surface. The system prompt is trade-secret-adjacent (Provider-Only). The retrieval store likely contains PII (Restricted, destroy-on-report). Provider-surface techniques (weight_read, jailbreak) require the provider_authorization check. This is the legal layer (B0) made into the engineering layer (B1's scope enforcement)." } ] }