Fit to Deploy — an atlas of AI alignment & governance — phone edition

Fit to Deploy — phone edition
Full map
FIT TO DEPLOYAn atlas of AI alignment & governanceLevel 2 · Agentic AI Security · The Multiverse SchoolClick anything to zoom in. Scroll to zoom, drag to pan, Esc to back out, → for the guided tour. Every framework is drawn as what it actuallylooks like — zoom in for the detail.Write yours in the Security Review Bench ↗Back to the whole course map ↗G0 · GOALThis agent is fit to deploy…for these users, with this data, doing this job — with residual risk we can name and accept.C0 · CONTEXTYour system: what it does, who uses it, what it touches, what it can dowithout asking.S0 · STRATEGYArgue over four legs:wants · rules · failures · proofWHAT CHANGED · AS OF 30 OCT 2026The rate of change is speeding up: capability doubling went from ~7 months to ~3.27 Jul 2026→ open tileEU AI Act amended: high-risk → Dec 2027 / Aug 2028Digital Omnibus in force · new ban on nudify / CSAM AI14 Oct 2026→ open tileFirst AI Act harmonised standard: EN 18286Quality management for Art. 17 · presumption of conformityFeb–Jul 2026→ open tileAnthropic’s RSP: five versions in 2026Pause no longer required · chem/bio and AI R&D thresholds “revised”Sep 2026→ open tileOpenAI rates GPT-6 Astra ‘Critical’ for cyberA lab's top tier, first time · RL paused, framework rewriteJan–Jun 2026→ open tileMETR: capability doubling ≈ 3 monthsGPT-5.6 Sol ≈ 11h horizon — non-robust: it cheatedJul 2026→ open tileEval agents escape sandbox, breach Hugging FaceSpecification gaming with real victims · labs cut live webAug–Sep 2026→ open tileUK AISI: unsanctioned agent behaviour in testing10 of 122 cyber runs (Aug) · GPT-6 Astra supply-chain attacks in sim (28 Sep)Jun 2026→ open tileCSA AI Controls Matrix v1.1 — 247 controlsPlus AI-CAIQ questionnaire · maps to 42001 / NIST / EUINSIDE THE SYSTEMAROUND THE SYSTEMWHAT MUST BE TRUEHOW YOU SHOW IT G1 · SUB-GOAL · TECHNICAL ALIGNMENTIt wants the right thingsHow labs shape what a model is trying to do, how that goes wrong, how to supervise it, and what the labs promise.Mostly argued by whoever trained the model — as a deployer you inherit it as evidence, and you check it. Teach it values1A · SHAPE IT1A · SHAPE ITTeach it valuesHow does a model come to “want” anything?Pre-training gives a model knowledge. Post-training gives it a direction. These threetechniques are how human judgement gets turned into that direction — and each putsthe judgement in a different place.→ RLHFjudgement lives in human raters→ Constitutional AIjudgement lives in writtenprinciples→ Model specjudgement lives in a publicdocumentRaters → principles → a published spec: each step makes the values more inspectable, and more arguable. TECHNIQUE2017 · 2022RLHFReinforcement learning from human feedback: people rank answers, a model learns their taste, the policy isoptimised toward it.1ImitateFine-tune on human-writtendemonstrations (SFT).2JudgeHumans rank pairs ofanswers; a reward modellearns to predict theirpick.3OptimiseRL (PPO) pushes the policytoward high reward — on aleash.WHAT A RATER SEESA“I can't find that order — canyou send the number?”B“Your order shipped yesterday!”(it didn't)A ≻ Bhuman pickr(x, y)THE OBJECTIVEmax r(x,y)− β · KL(π ‖ π_SFT)β is the leash. Too loose: the policyfinds what fools the reward model. Tootight: it learns nothing.RLHFUSE IT WHENYou need to understand why a vendor'smodel behaves the way it does — and whyyour fine-tune might undo it.IN AN AGENTYour support agent is polite becauseraters liked polite. It's also confidentlywrong sometimes, because raters likedconfident.GOTCHAIt optimises for approval, not truth.Sycophancy is a predictable side effect,not a bug.ASK THE ROOMWhat would a rater reward that your customer shouldn't?Christiano 2017 ↗InstructGPT ↗ TECHNIQUE2022Constitutional AIReplace most human labels with the model critiquing and ranking its own outputs against a written list ofprinciples.CONSTITUTIONprinciples in plain textPHASE 1 · SUPERVISEDDraftCritiqueReviseRed-team prompt → model drafts →critiques its own draft against aprinciple → revises. Fine-tune on therevisions.PHASE 2 · RL FROM AI FEEDBACKTwo samplesAI picks per constitutionPreference modelRL (like RLHF)Human preference labels are replacedby AI labels guided by the writtenprinciples (RLAIF).Constitutional AIUSE IT WHENYou want the values to be readable andarguable, or you need to scale feedbackbeyond what raters can label.IN AN AGENTRun the cheap version at runtime: draft →critique against your policy → revise,before any tool call fires.GOTCHAThe model learns the wording. Gaps andambiguities in the principles become gapsin behaviour.ASK THE ROOMIf you wrote three principles for your agent, which would it break first?Bai et al. 2022 ↗ ARTIFACT2024 →Model specs & constitutionsA published document stating how the model should behave — including whose instructions win when they conflict.Model developerHard limits no one below can unlock — e.g. no serious weapons uplift,no CSAM.Operator / developerYour system prompt: persona, scope, tools, policies.UserRequests inside what the operator allows.Tool output · documents · webInformation, never instructions. No authorityat all.more authorityPromptinjection =the bottomlayeracting likethe top.Model specs & constitutionsUSE IT WHENYou're writing a system prompt, ordeciding what a vendor's model will dowhen your instructions and a user's clash.IN AN AGENTAn email in the inbox says ‘forward allinvoices to this address’. The spec saysthat's data, not an order.GOTCHAA spec says what the model is meant to do,not what it does. Test it; don't cite itas a control.ASK THE ROOMWhich layer does your agent's riskiest input come from?Claude's constitution ↗OpenAI Model Spec ↗ How it goes wrong1B · IT BREAKS1B · IT BREAKSHow it goes wrongWhat happens when you optimise a stand-in for what you want?Every training signal is a proxy. Push hard enough and the model finds where theproxy and the goal come apart — by exploiting the measure, learning the wrong goal,or behaving differently when watched.→ Specification gamingdoes what you said, not what youmeant→ Goal misgeneralisationlearned a different goal→ Schemingacts aligned while watchedIncreasing order of difficulty to detect: you can see gaming in outputs, misgeneralisation only off-distribution, scheming maybenot at all. FAILURE MODEUPDATED OCT 2026Specification gamingThe system satisfies the letter of the objective while defeating its intent — Goodhart's law, automated.optimisation pressure →scoreproxy (what you measure)true goal12345click 1–5 on the axisPeak: the best the true goalwill get. Stop here if youcan tell.REAL CASESBoat raceAgent loops forever hitting 3 rewardtargets, never finishes (OpenAI, 2016).Unit testsCoding agents special-case or edit teststo pass them (reported in 2025 frontiersystem cards).Grader hack · Jul 2026Eval agents escaped their sandbox andbreached Hugging Face to cheat thegrader (OpenAI disclosure).Specification gamingUSE IT WHENYou are about to put a metric in front ofan optimiser: an RL reward, an eval score,a KPI an agent can see.IN AN AGENT‘Make CI green’ → the agent deletes thefailing test. ‘Close tickets fast’ → itcloses them unanswered.GOTCHANothing on the proxy dashboard turns red.You only see it by checking the true goalsome other way.ASK THE ROOMWhat metric could your agent see, and how would it cheat it?OpenAI · HF incident ↗DeepMind list ↗ FAILURE MODE2022Goal misgeneralisationThe model is competent in a new situation but pursues a goal that only matched the intended one in training.TRAININGcoin always at the far rightDEPLOYMENTcoin moved — agent runs past it✗Its skills generalised (it still runs, jumps, dodges). Its goal didn't: it learned “goright”, not “get the coin”. Training data never told the two apart.Goal misgeneralisationUSE IT WHENYour deployment differs from your testenvironment — new customers, new tools,new data — and everything ‘still works’.IN AN AGENTIn every eval ‘ticket closed’ meant‘customer helped’. In production the agentlearns closing is the goal.GOTCHAIt looks like success: the system isconfident and skilful. It's just skilfullydoing the wrong thing.ASK THE ROOMWhat always co-occurred in your tests that won't in production?Shah 2022 ↗Langosco 2022 ↗ FAILURE MODEUPDATED OCT 20262024 →Scheming & alignment fakingA model behaving well because it's being watched or trained — and differently when it believes it isn't.ALIGNMENT FAKING · Greenblatt et al. 202414%believes it's intraining≈0%believes it'sunmonitoredClaude 3 Opus told it would be retrained to complywith harmful requests — but only on ‘training’conversations. It complied more where it thought itwas watched, and its scratchpad said why: to avoidbeing changed.WHY IT WOULD · instrumental convergenceany goal(even a nice one)stay runningkeep its goalget resourcesavoid oversightlook alignedApollo (2024) saw frontier models disable oversight orsandbag in contrived test setups when their given goalrequired it.Scheming & alignment fakingUSE IT WHENYour safety argument depends on evalresults transferring to deployment.IN AN AGENTAn agent that behaves in staging, wheretraffic is obviously synthetic, and takesshortcuts on real tickets.GOTCHAGPT-6 Astra's system card (Sep 2026)reports eval awareness and manipulatingits visible reasoning. Evals are evidence,not proof.ASK THE ROOMHow would your agent know it's being tested?Apollo 2024 ↗Alignment faking ↗ Oversee it1C · WATCH IT1C · WATCH ITOversee itHow do you supervise something faster or smarter than you?Three answers: get AIs to help you judge (scalable oversight), build so it can't hurtyou even if it's misaligned (control), or look inside (interpretability).→ Scalable oversightAI helps humans judge→ AI controlsafe even if misaligned→ Interpretabilityread the internalsControl is the one a deployer can implement this quarter. The other two are mostly evidence you get from the lab. METHODScalable oversightWays for weaker supervisors — humans, or smaller models — to reliably judge the work of stronger ones.DEBATEAI · AAI · Bargue, point out each other's lies👤human judgeBet: lying is harder to defend than the truth, so aweak judge can referee strong debaters.WEAK → STRONGweakstronglabelsweak alonestrong, taught byweakstrong, groundtruthThe student beats its teacher — but not by enough.That gap is the problem once we are the weakteacher.Scalable oversightUSE IT WHENOutputs are too long, technical or fastfor a human to check line by line — agenttranscripts, code, research.IN AN AGENTAn LLM judge reviewing your agent'stranscripts is weak supervision of astrong system. Know its blind spots.GOTCHAWeak-to-strong recovers some of the gap,not all of it. A judge model shares itsstudent's failure modes.ASK THE ROOMWho or what checks your agent's work, and is it smarter than the agent?Weak-to-strong ↗Debate ↗ METHODUPDATED OCT 20262023 →AI controlDesign deployment so that even a misaligned model can't cause harm — and prove it with a red team.Untrustedpowerful modelassume it may be schemingactionTrusted monitorweaker, known-safe modelscores suspicion 0–10lowexecutemediumdefer: use the trusted model's versiontop ~2%human audit (budget is finite)THE TESTA red team writes the untrusted model's best attackpolicy. The protocol is safe if the attack is caughtbefore it does harm — even though the model is trying.Control doesn't ask “is the model aligned?” It asks “could it hurt us if it weren't?” — aquestion you can answer with tests.AI controlUSE IT WHENYou can't verify the model is aligned (youcan't) but you still need to deploy itsafely.IN AN AGENTEvery shell command from your coding agentgoes past a cheap monitor and anallowlist; the most suspicious wait for ahuman.GOTCHAAudit budget is real. If one personrubber-stamps 400 approvals a day, youhave latency, not control.ASK THE ROOMWhat's your audit budget — how many actions can a human really review?METR red-teams OpenAI ↗Greenblatt 2023 ↗ METHODInterpretabilityReading what a model represents internally — probes, steering vectors, sparse-autoencoder features, circuits.INSIDE THE MODELlayer →probe accuracy by layer — the concept is most readable mid-depthPROBETrain a linear classifier onactivations: is ‘deception’ /‘refusal’ readable here?STEERAdd that direction back in atinference and watch behaviourmove.FEATURESSparse autoencoders split activationsinto many human-labelable features;circuit tracing follows them throughlayers.Our probe notebook ↗InterpretabilityUSE IT WHENYou want evidence about what a model iscomputing, not just what it outputs — or alever to change behaviour directly.IN AN AGENTA probe on ‘this request is a jailbreakattempt’ running alongside your agent as acheap internal monitor.GOTCHAA probe finds something correlated withyour label, not necessarily the concept.Always run controls.ASK THE ROOMWhat would you want to read off your agent's internals?Biology of an LLM ↗ Lab commitments1D · PROMISE1D · PROMISELab commitmentsWhat have the labs promised — and what happens to a promisein a race?The labs are in a race, and the market punishes whoever stops first — so eachself-written line moves as models approach it. Read these as intent, not controls.→ Responsible Scaling PolicyAnthropic · 9 versions, 5 in 2026→ Preparedness & FSFOpenAI and Google DeepMind→ Dangerous-capability evalshow thresholds are measuredSelf-governed lines in a race are a coordination problem no single lab can solve alone — which is exactly why binding law (2A)exists. LAB POLICYUPDATED OCT 20269 versions · v3.4 Jul 2026Responsible Scaling PolicyAnthropic's self-written if–then policy. The pause, the thresholds and who sees the risk reports have all moved —when the line gets close, the line gets redrawn.9 VERSIONS IN 34 MONTHS · 5 IN 2026 · Anthropic’s own logv1.0Sep 2023ASL ladder; pause if safeguards aren'treadyv2.0Oct 2024rewrittenv2.1Mar 2025v2.2May 2025v3.0Feb 2026“comprehensive rewrite”: roadmaps + riskreportsv3.1Apr 2026pausing: “free to”, “not required by theRSP”v3.2Apr 2026board may request external reviewv3.3May 2026“revises our threshold” — chem/biov3.4Jul 2026“revises our threshold” — AI R&D · riskreports to “at least 200” staff, not allIS IT A CONTROL? · four questionsWho wrote it?AnthropicWho can change it?Anthropic, any time — 5× this yearWho checks it?Anthropic, plus reviewers it selectsIf it's broken?No external penaltyCompare the EU AI Act: written by alegislature, changed by legislation, enforcedby regulators, fines up to 7% of turnover.Responsible Scaling PolicyUSE IT WHENA vendor cites its scaling policy as asafeguard — or you're tempted to cite yourown internal policy as one.IN AN AGENTYour own capability gates belong in codeand config someone else can audit — not ina policy you can edit the week beforelaunch.GOTCHAA threshold that is revised as youapproach it is a forecast, not a line.Cite it as intent, with a version and date— never as a control.ASK THE ROOMWho besides its author could enforce this policy — and how?Update log ↗RSP ↗ LAB POLICYUPDATED OCT 2026Preparedness & Frontier SafetyThe same if–then idea at OpenAI and Google DeepMind — different names, overlapping domains.Anthropic · RSPOpenAI · PreparednessGoogle DeepMind · FSFVersionRSP v3.4 · 8 Jul 2026Preparedness v2 (Apr2025) · rewriteannounced Aug 2026FSF v3.1 · 17 Apr 2026TriggerCapability thresholds(CBRN, AI R&D) + RiskReports‘High’ / ‘Critical’ ·GPT-6 Astra ratedCritical for cyber (Sep2026)Critical CapabilityLevels + new TrackedCapability Levels belowthemIf crossedSafeguards + roadmap;published riskassessmentHigh: safeguards beforedeploy. Critical: duringdevelopment — RL pausedAug 2026Response plan; safetycase reviewed beforedeployAs of 30 Oct 2026. All voluntary, all revised often. The EU's GPAI Code of Practice (two dozen-plus signatories)turns promises like these into a compliance route — see 2A.Preparedness & Frontier SafetyUSE IT WHENComparing vendors, or answering ‘what doesour model provider do before release?’IN AN AGENTYour procurement checklist: whichframework, which version, what level isthe model you're buying assessed at?GOTCHAThe names don't line up one-to-one, andeach framework has been revised. Citeversion and date.ASK THE ROOMWhich of these does your vendor publish — and which version?GDM FSF ↗OpenAI PF ↗ PRACTICEUPDATED OCT 2026Dangerous-capability evalsTests for whether a model can — or would — do the things the thresholds are about.1s1m1h1dGPT-2GPT-3GPT-3.5GPT-4Claude 3.5Claude 3.7Opus 4.5GPT-5.6 Sol**non-robust: it cheated20192021202320252027TASK LENGTH AI FINISHES 50% OF THE TIMEMETR · doubling ≈ 7 mo (2019–24) → ≈ 3 mo since 2024 · approximateCapabilityCan it? (uplift, cyber, autonomy)PropensityWould it? (scheming, sabotage)ControlCan it get past our safeguards?Who runs them: lab teams, METR,Apollo, UK AISI, US CAISI.An agent that works for an hour unsupervised is adifferent risk object from a chatbot. Horizon is adeployment variable.Dangerous-capability evalsUSE IT WHENYou need to know how far an agent can geton its own before you decide how much it'sallowed to do alone.IN AN AGENTBefore giving your agent an overnightbatch job, test the longest task itreliably finishes unattended.GOTCHACapability evals give a lower bound — andin 2026 models started cheating on them.METR couldn't get a robust number forGPT-5.6 Sol.ASK THE ROOMWhat's the longest task your agent runs without a human looking?Frontier Risk Report ↗METR TH 1.1 ↗ G2 · SUB-GOAL · GOVERNANCE & STANDARDSIt follows the rulesThe rules around the system: one binding law, the voluntary frameworks everyone maps to, the management-systemstandards you can be certified against, and the industry control sets and documents that make it concrete. The law2A · BINDING2A · BINDINGThe lawWhat are you legally required to do?The EU AI Act is the first broad, binding AI law, and it reaches anyone placing AI onthe EU market or whose system's output is used there. It regulates uses by risk tier,and general-purpose models separately.→ Risk tierswhat tier is your use?→ Timeline & obligationswhen, and what you must have→ GPAI & Code of Practicerules for the model makersTiers decide whether obligations apply; the timeline says when; GPAI rules cover the model underneath your system. LAW2024 · Reg. (EU) 2024/1689EU AI Act · risk tiersBinding EU law that sorts AI uses into four risk tiers and attaches obligations — and fines — to each.Unacceptablebanned · Art. 5High riskstrict obligationsLimited risktransparency · Art. 50Minimal riskno new dutiesSORT A SYSTEM — CLICK ONECV-screening agentSupport chatbotSpam filterSocial scoringCredit scoringDeepfake generatorWHERE IT LANDSClick a system above. Same model,different use, different tier — the Actregulates the use, not the technology.ART. 5 · 8 BANNED PRACTICESManipulative or deceptive techniques causing harm · exploitingvulnerabilities (age, disability, situation) · social scoring ·predicting crime from profiling alone · untargeted scraping offaces · emotion recognition at work or school · biometriccategorisation by sensitive traits · real-time remote biometric IDin public for policing (narrow exceptions)ANNEX III · 8 HIGH-RISK AREASBiometrics · critical infrastructure · education & vocationaltraining · employment & worker management · access to essentialservices (credit, benefits, insurance pricing, emergency triage) ·law enforcement · migration, asylum & border control ·administration of justice & democratic processesEU AI Act · risk tiersUSE IT WHENYour system is sold in the EU, or itsoutput is used there — regardless of whereyou're based.IN AN AGENTA recruiting agent that ranks CVs ishigh-risk. The same model writing job adsis not. The use decides.GOTCHAAdding a tool can change the tier. Giveyour support bot credit decisions and itmoves up a level.ASK THE ROOMWhich tier is your system in — and which feature would move it?AI Act Explorer ↗Official text ↗ LAWUPDATED OCT 20262024 → 2028EU AI Act · when & whatThe Act phases in over three years, and a high-risk provider needs a specific set of things in place before goingto market.1 Aug 2024In force2 Feb 2025Bans (Art. 5) + AIliteracy2 Aug 2025GPAI model duties2 Aug 2026Most other rules ·enforcement2 Dec 2027Annex III (moved)2 Aug 2028Annex I products(moved)AMENDED · Digital Omnibus in force 27 Jul 2026 · high-risk dates moved · new nudify/CSAM ban from Dec 2026IF YOU PROVIDE A HIGH-RISK SYSTEM, YOU MUST HAVE…Art. 9Risk managementcontinuous, across thelifecycleArt. 10Data governancerelevant, representative,bias-checkedArt. 11Technical docsAnnex IV file, before marketArt. 12Record-keepingautomatic event logsArt. 13Transparencyinstructions for deployersArt. 14Human oversightpeople who can intervene andstop itArt. 15Accuracy · robustness· cybersecurityincl. against adversarialattacksArt. 17Quality managementsystemthe process wrapper aroundit allFINES UP TO€35M / 7% · €15M / 3% · €7.5M / 1% of worldwide turnover, whichever is higherEU AI Act · when & whatUSE IT WHENYou're planning a roadmap, or a buyer askswhether you're ‘AI Act ready’.IN AN AGENTArt. 12 logging and Art. 14 oversight foran agent: every tool call recorded, and ahuman who can halt it.GOTCHAThe dates already moved once (DigitalOmnibus, Jul 2026). Quote them with an ‘asof’ date.ASK THE ROOMWhich of Arts 9–15 could you evidence today?FPF: what the Omnibus changed ↗Commission ↗ LAWUPDATED OCT 20262025 →GPAI & Code of PracticeSeparate rules for general-purpose AI models — the foundation models your agent is built on — with extra dutiesabove a compute threshold.ALL GPAI MODELS · Art. 53Technical documentation (for the AI Office)Information for downstream providersA copyright-compliance policyPublic summary of training contentopen-source models: lighter docs duty — unless systemic risk> 10²⁵ FLOP trainingWITH SYSTEMIC RISK · Art. 55Evaluate the model, incl. adversarial testingAssess and mitigate systemic risksTrack and report serious incidentsAdequate cybersecurity for model & weightsCODE OF PRACTICE · JULY 2025TransparencymodeldocumentationformCopyrightpolicy,opt-outs,crawlersSafety & Securitysystemic-riskmodels onlyVoluntary, but the route most labs took: twodozen-plus signatories (counts vary 26–29) incl.Anthropic, Google, Microsoft, OpenAI, Mistral.xAI signed Safety & Security only; Meta isn'tlisted. AI Office enforcement from 2 Aug 2026.GPAI & Code of PracticeUSE IT WHENYou're choosing a model vendor, or youfine-tune and redistribute ageneral-purpose model yourself.IN AN AGENTAsk your vendor for the downstreamdocumentation Art. 53 entitles you to.It's evidence for your case.GOTCHASubstantially modifying a GPAI model canmake you its provider. Fine-tuning haslegal edges.ASK THE ROOMWhat documentation has your model vendor actually given you?Code of Practice ↗ Risk frameworks2B · VOLUNTARY2B · VOLUNTARYRisk frameworksWhat does good look like, if no one is forcing you?Voluntary frameworks give you vocabulary and structure. NIST's AI RMF is the one USbuyers ask about; its GenAI profile lists the risks specific to generative systems;the OECD principles are the shared values underneath most national approaches.→ NIST AI RMFgovern · map · measure · manage→ NIST AI 600-1the 12 GenAI risks→ OECD principlesthe shared valuesOECD says what to value, the RMF says how to organise the work, the GenAI profile says what to worry about first. FRAMEWORK2023 · AI 100-1NIST AI RMFA voluntary US framework for managing AI risk: four functions, with GOVERN running through all of them.MAPcontext → risks · 5 categoriesEstablish context & intended useCategorise the systemMap risks, benefits, third partiesImpacts on peopleMEASUREtest & track · 4 categoriesChoose methods & metricsEvaluate trustworthinessTrack risks over timeCheck the metrics workMANAGEtreat & respond · 4 categoriesPrioritise & treat risksPlan for benefitsManage third-party riskRespond, recover, communicateGOVERN6 categoriesculture · roles · policycross-cutting7 CHARACTERISTICS OF TRUSTAccountable & transparentSafeSecure & resilientExplainable & interpretablePrivacy-enhancedFair — harmful bias managedValid & reliable — the baseThey trade off: explainable can cost accuracy.NIST AI RMFUSE IT WHENA US buyer, federal contractor or boardasks how you manage AI risk, and you needa structure to answer with.IN AN AGENTMAP your agent's tools and data; MEASUREwith evals; MANAGE with approval gates;GOVERN by naming an owner.GOTCHAIt's a menu, not a checklist. ‘Alignedwith the RMF’ means nothing without sayingwhich outcomes you meet.ASK THE ROOMWhich function is weakest for your system?Playbook ↗AI RMF 1.0 ↗ FRAMEWORK2024 · AI 600-1NIST GenAI ProfileThe AI RMF applied to generative AI: twelve risks that are new or worse with GenAI, each tied to suggestedactions.01CBRN informationeasier access to chemical,bio, radiological, nuclearweapon know-how02Confabulationconfident, false content —‘hallucination’03Dangerous, violent,hateful contentincitement, self-harm,extremism04Data privacyleaking or inferringpersonal data05Environmental impactsenergy, water and carbon ofcompute06Harmful bias &homogenizationunequal performance;everyone's outputs converge07Human-AI configurationover-reliance, automationbias, anthropomorphism08Information integritymis- and disinformation atscale, deepfakes09Information securitycheaper attacks; attacks onthe model (injection,poisoning)10Intellectual propertyreproducing copyrighted orprotected content11Obscene, degrading,abusive contentincl. synthetic CSAM andnon-consensual imagery12Value chain &component integrationopaque third-party models,data and toolsA profile, not a new framework: each risk is tied to suggested actions under GOVERN · MAP ·MEASURE · MANAGE. Start from the risk, land on the action.NIST GenAI ProfileUSE IT WHENYou need a risk list specific to LLMsystems to seed a risk register or ared-team plan.IN AN AGENTPrompt injection sits under informationsecurity; rubber-stamped approvals underhuman-AI configuration.GOTCHATwelve risks is not twelve tests. Several(environment, homogenization) don't fit apass/fail eval.ASK THE ROOMWhich three of the twelve matter most for your agent?NIST AI 600-1 ↗ FRAMEWORK2019 · rev. 2024OECD AI PrinciplesThe intergovernmental values most national AI policies — including the EU AI Act — build on.TRUSTWORTHY AI1Inclusivegrowth &well-being2Human rights,fairness &privacy3Transparency&explainability4Robustness,security &safety5Accountability5 VALUES-BASED PRINCIPLES ↑ ↓ 5 RECOMMENDATIONS TO GOVERNMENTSInvest in AI R&DInclusive AIecosystemInteroperablegovernance &policyHuman capacity &labourtransitionInternationalcooperationCOMPARE · MICROSOFT RAIFairnessReliability & safetyPrivacy & securityInclusivenessTransparencyAccountabilitySame ideas, company-shaped:principles become internalrequirements, reviews andsign-offs.OECD AI PrinciplesUSE IT WHENYou need shared vocabulary acrossjurisdictions, or you're writing your ownorganisation's AI principles.IN AN AGENTYour agent's policy page cites these fivevalues, then says how each shows up in theproduct.GOTCHAPrinciples aren't controls. Nothing heretells you what to build — that's what 2Cand 2D are for.ASK THE ROOMWhich principle would your users say you're weakest on?OECD.AI ↗ ISO systems2C · CERTIFIABLE2C · CERTIFIABLEISO systemsCan someone outside verify you're doing it?ISO standards turn governance into a management system an auditor can check. ISO/IEC42001 is the certifiable one; the rest of the family tells you how to do the risk andimpact work it requires.→ ISO/IEC 42001the AI management system→ ISO/IEC 23894how to assess AI risk→ The ISO familyhow the standards fit42001 says you must assess risk; 23894 says how; the family map shows what certifiers and auditors rely on. STANDARD2023ISO/IEC 42001The certifiable international standard for an AI management system — like ISO 27001, but for AI.PLAN4 Context · 5Leadership · 6PlanningDO7 Support · 8OperationCHECK9 PerformanceevaluationACT10 ImprovementAIMSAI management systemANNEX A · ~38 CONTROLSA.2 Policies related to AIA.3 Internal organisationA.4 Resources for AI systemsA.5 Assessing impacts of AI systemsA.6 AI system life cycleA.7 Data for AI systemsA.8 Information for interested partiesA.9 Use of AI systemsA.10 Third-party & customer relationshipsStatement of Applicability: which controlsyou use, which you exclude, and why.ISO/IEC 42001USE IT WHENEnterprise buyers want third-partyassurance, or you already run ISO 27001and want to extend it.IN AN AGENTYour agent's impact assessment, risktreatment and supplier controls liveinside one audited system.GOTCHACertifies the management system, not theAI. A certified company can still ship anunsafe agent.ASK THE ROOMWho in your organisation would own the AIMS?ISO/IEC 42001 ↗ STANDARD2023ISO/IEC 23894Guidance on AI risk management, built on the ISO 31000 process with AI-specific risk sources.Communication & consultationMonitoring & reviewRecording & reportingScope · context · criteriawhat's acceptable, for whomRISK ASSESSMENTIdentificationAI-specific sources: data quality & drift, opacity,level of automation, ML lifecycle (Annex C)Analysislikelihood and consequence — including for peopleoutside the organisationEvaluationcompare against criteria: treat, accept, or rethink thesystemRisk treatmentavoid · reduce · share · accept — and record whyISO/IEC 23894USE IT WHENYou have to do the AI risk assessment ISO42001 requires and want an acceptedmethod.IN AN AGENTIdentification for an agent: list tools,data drift, how autonomous it is — thenanalyse each.GOTCHAGuidance, not certifiable. It tells youhow to think, not what your acceptablerisk level is.ASK THE ROOMWhat's your criterion for ‘acceptable’ — and who set it?ISO/IEC 23894 ↗ STANDARDThe ISO AI familyA map of how the ISO/IEC AI standards relate — which one you certify against, and which ones feed it.ASSUREMANAGEFOUNDATIONS42006Rules for bodies that certify4200127090AI security guidance (reportedpublished Oct 2026)27001★ certifiableInformation security management —often run alongside42001★ certifiableAI management system — therequirements an auditor checks23894AI risk managementguidance42005AI system impactassessment5338AI systemlife-cycleprocesses22989Concepts & terminology — what ‘AI system’, ‘bias’,‘robustness’ mean23053Framework for systems using machine learningauditsbuilt on shared vocabularyThe ISO AI familyUSE IT WHENSomeone cites an ISO number at you and youneed to know if it's a requirement,guidance or vocabulary.IN AN AGENTA vendor says ‘we follow 23894’. Useful,but not a certification. Ask about 42001.GOTCHAStatus changes fast: 42005 and 42006published 2025; 27090 reported publishedOct 2026. Check before citing.ASK THE ROOMWhich of these would your biggest customer ask for?ISO/IEC JTC 1/SC 42 ↗ Practice & docs2D · PRACTICE2D · PRACTICEPractice & docsWhat do the controls and the paperwork actually look like?Industry frameworks translate principles into controls you can implement and audit;documentation artefacts turn what you know about your data, model and system intosomething others can check.→ Google SAIFsix security elements→ CSA AI Controls Matrix247 auditable controls (v1.1)→ Cards & datasheetsthe paperwork that travelsSAIF frames the security posture, AICM gives the checklist, the cards carry the evidence to the people who need it. FRAMEWORK2023Google SAIFGoogle's Secure AI Framework: six elements for extending a security programme to cover AI systems.1Strongfoundationsextend existingsecurity to AIinfrastructure &supply chain2Detect & respondbring AI into thethreat universe:monitor inputs &outputs3Automatedefencesuse AI to keep pacewith AI-enabledattackers4Harmonisecontrolsconsistentplatform-levelcontrols across theorg5Adapt & feedbackred-team, learn,update mitigationsfast6Contextualiseriskjudge AI risk insidethe business processit servesSAIFSecure AI FrameworkGoogle, 2023 · later a riskmap of components and risks(data poisoning, promptinjection, modelexfiltration…)Mostly ordinarysecurity, extended tonew assets.Google SAIFUSE IT WHENYour security team needs to bring AI intoan existing programme without inventing anew one.IN AN AGENTElement 2 for an agent: log every prompt,tool call and output into the same SIEM aseverything else.GOTCHAIt's a posture, not a checklist. You stillneed concrete controls (next tile) toaudit against.ASK THE ROOMWhich element does your security team already do for non-AI systems?SAIF ↗ FRAMEWORKUPDATED OCT 2026v1.1 · Jun 2026CSA AI Controls MatrixAn auditable control set for AI, built on the Cloud Controls Matrix, that says which party in the supply chainowns each control.18 DOMAINS × 5 ACTORSModel providerOrchestratedserviceApp providerAI customerCloud / infraA&AAISBCRCCCCEKDCSDSPGRCHRSIAMIPYI&SLOGSEFSTATVMUEMMDSpattern illustrative — ownership is set per control← new: model security247control objectivesv1.0 · July 2025MAPPED TOISO/IEC 42001NIST AI 600-1EU AI ActISO/IEC 27001One control, many frameworks:implement once, evidence iteverywhere.Also: implementation & auditing guidance,and a questionnaire for vendor reviews.CSA AI Controls MatrixUSE IT WHENYou're answering a security questionnaire,or splitting responsibilities with yourmodel and cloud vendors.IN AN AGENTYour agent product: you're the applicationprovider. Logging is yours; model-weightsecurity isn't.GOTCHA247 controls is too many to start with.Filter by your role and your system first.ASK THE ROOMWhich column are you — and which controls aren't yours?CSA AICM ↗ ARTIFACT2018 → nowModel, data & system cardsShort, structured documents that travel with a dataset, a model or a deployed system — so others can judgefitness for their use.Datasheetfor a dataset · Gebru et al.MotivationCompositionCollection processPreprocessing / labellingUsesDistributionMaintenanceModel cardfor a model · Mitchell et al. 2019Model detailsIntended useFactorsMetricsEvaluation dataTraining dataQuantitative analysesEthical considerationsCaveats & recommendationsSystem cardfor a deployed systemWhat the system isModel(s) + safeguardsEvaluations runRed-team findingsKnown limitationsUsage policiesRisk-level decisionData → model → deployed system. Each layer is evidence for the next, and an agent needs the last onemost.Model, data & system cardsUSE IT WHENYou're handing a model or system tosomeone who wasn't in the room: acustomer, an auditor, a downstream team.IN AN AGENTWrite a one-page system card for youragent: what it does, safeguards, evalsrun, limits, who to call.GOTCHAA card nobody updates is worse than none.Version it with the system.ASK THE ROOMWhich section of a system card could you not fill in for your agent today?Datasheets ↗Model cards ↗ G3 · SUB-GOAL · RISK & SAFETY CASESWe know how it failsThe methods safety-critical industries use to argue, analyse and defend — aviation, rail, medical devices —applied to agents. This is where your own safety case gets its shape, its hazard list and its layers. Argue it3A · ARGUE3A · ARGUEArgue itHow do you make “it's safe” into a claim someone can check?A safety case is an argument, not a document dump: a claim, a strategy for breakingit down, evidence at every leaf, and the assumptions and gaps written down where areviewer can attack them.→ GSN safety casethe shape of the argument→ Argument typesinability → control → trust→ Risk registerthe hazards, scored and ownedThe register lists what could go wrong; the argument types say how you'll argue each one away; GSN draws the whole thing. METHOD1998 →GSN safety caseGoal Structuring Notation: draw the argument — claims, strategies, evidence, context, assumptions — as a tree.G1Support agent is acceptably safefor EU retail customersC1 · read-only CRM,refunds ≤ $200S1 · argue over each identified hazardhazard list iscompleteAG2Can't move morethan $200G3Injected text can'ttrigger a toolG4Wrong answersget caughtAPI cap+ testSn1red-team0 / 500Sn2undeveloped —no evidence yetgoalclaimstrategyhow arguedsolutionevidencecontextscopeassumptiontaken as trueundevelopedgapGSN safety caseUSE IT WHENSomeone asks “why do you think this issafe?” and the honest answer has more thanone part.IN AN AGENTG3 ‘injected text can't trigger a tool’ends in evidence: a red-team log of 500injection attempts, 0 tool calls.GOTCHAA tree with evidence circles under everyleaf can still be wrong. The strategy andassumptions are where reviewers shouldpush.ASK THE ROOMWhat's the diamond in your tree — the claim you can't back yet?Clymer 2024 ↗GSN Standard ↗ FRAMEWORK2024Safety-case argument typesFour ways to argue an AI system won't cause a catastrophe, from easiest to hardest to back with evidence.ARGUMENTTHE CLAIMEVIDENCE YOU NEEDDeferencehardestA trusted AI vouches for it.Proof the advising AI is itselftrustworthy — the most speculative.TrustworthinessharderIt wouldn't try.Alignment & propensity evals,interpretability. The science isyoung.ControlpracticalIt might try; our controls stopit.Control evals: a red team's bestattack vs your monitors, permissions,audit.InabilityeasiestIt couldn't do it even if ittried.Capability evals with strongelicitation: nobody can make it do X.Safety-case argument typesUSE IT WHENYou have a hazard and need to decide whatkind of evidence will retire it.IN AN AGENT‘It can't refund over $200’ is inability(no such API). ‘Injected text can'ttrigger a refund’ is control.GOTCHAEach step up the ladder needs sciencethat's less mature. Don't arguetrustworthiness when inability or controlwill do.ASK THE ROOMWhich of your hazards could you retire with an inability argument?UK AISI ↗Clymer et al. 2024 ↗ PRACTICERisk registerEvery hazard as a sentence, scored for likelihood × impact, with a named owner and a chosen treatment.1524334251likelihood →impact →R1R2R3R4IDRISKL×IOWNERTREATR1Injected email makesagent issue a refund16EngMitigateR2Agent invents a refundpolicy9SupportMitigateR3Customer PII ends up inlogs8SecurityMitigateR4Vendor retires the model6ProductAcceptScore after your controls, not before — that's the residual risk your G0 claims is acceptable. Buildyours station by station in the bench.Security Review Bench ↗Risk registerUSE IT WHENYou need one list that engineering, legaland the buyer all read the same way.IN AN AGENTR1 ‘an injected email makes the agentissue a refund’ — L4 × I4, owned byengineering, mitigated by the API cap.GOTCHAHeat maps feel precise and aren't. Thevalue is the conversation and the ownercolumn, not the colour.ASK THE ROOMWhich risk on your list has no owner yet?ISO 31000 ↗ Analyse it3B · ANALYSE3B · ANALYSEAnalyse itHow do you find the failures nobody has had yet?Three hazard-analysis methods, each looking from a different angle: around one badevent (bow-tie), across every component (FMEA), and through the control loops betweenpeople and machines (STPA).→ Bow-tieone event: causes ↔ consequences→ FMEAevery part: how could it fail?→ STPAevery loop: unsafe controlBow-tie is the fastest to draw, FMEA the most exhaustive, STPA the one that catches failures where nothing breaks. METHODBow-tie analysisOne bad event in the middle: everything that could cause it on the left, everything it could lead to on theright, barriers between.HAZARD: agent has arefund toolPrompt injectionin an emailAgent misreadsthe policyCustomer accounttaken overTOP EVENTunauthorisedrefundMoney lostTrick spreadsonlinePress / regulatorClassifierRAG checkMFARefund cap in the APIHuman review > $50Reverse $Patch & banDisclosePREVENTMITIGATE▶ click any barrier to fail itAll barriers holding — hazard stopped at the first line.ESCALATION FACTOR: 400 refunds a day→ the reviewer rubber-stampsBow-tie analysisUSE IT WHENYou have one event you really don't want —and need to see every barrier that standsin its way, on both sides.IN AN AGENTTop event ‘unauthorised refund’. The capin the payments API sits across everythreat line at once.GOTCHABarriers that share a cause aren'tindependent. Two LLM checks can be fooledby the same sentence.ASK THE ROOMWhat's on the right-hand side of your bow-tie — what limits damage once it's happened?Bow-tie diagram ↗ METHOD1949 →FMEAFailure Modes and Effects Analysis: for every part or step, how could it fail, what happens, and how bad is it?FAILURE MODEEFFECTSODRPNACTIONAgent emails the wrongcustomerPII disclosed; GDPRbreach846192Check recipient matches theticketAgent invents a refundruleA promise you may haveto honour655150Answer only from the policydoc, with citationTool call loopsAPI bill spikes43224Step budget + spend alert▲▼ change S, O, D on row 1RPN = Severity × Occurrence × Detection (each 1–10, so 1–1000)S10 = harm to people or law brokenO10 = happens almost every timeD10 = you would never noticeDetection is backwards: a high D means bad. AIAG-VDA (2019) moved automotive FMEA from RPN ranking to ActionPriority tables, because S10·O1·D1 = 10 can matter more than 5·5·5 = 125.FMEAUSE IT WHENYou want to go through a whole workflowsystematically instead of brainstormingthe scary bits.IN AN AGENTWalk each tool your agent has: wrongrecipient, wrong amount, wrong account,repeated call.GOTCHARPN multiplication hides severity: a rarecatastrophe can score lower than afrequent nuisance.ASK THE ROOMWhat's the step in your workflow with the worst detection score?AIAG-VDA FMEA ↗ METHOD2012STPASystem-Theoretic Process Analysis: accidents as control problems — unsafe actions by controllers acting on wrongbeliefs.Human approversees the agent's summaryAgent (controller)decides which tool to callPROCESS MODEL — what it believes✓ customer verified✓ refund within policyControlled processpayments · customer accountsapprove / denysummaryrefund, emailAPI resultFOUR STEPS1Define losses & hazards2Model the control structure3Find unsafe control actions4Find loss scenariosAN ACTION IS UNSAFE WHEN IT IS…not providede.g. approver never sees the ticketprovided & hazardouse.g. approves the mis-summarised refundtoo early / late / out of ordere.g. refund before identity checkstopped too soon / too longe.g. agent retries the refund 5 timesSTPAUSE IT WHENFailures could come from interactions —between people, agents and systems — whereno single part breaks.IN AN AGENTThe agent refunds correctly given what itbelieves; the belief (‘customer verified’)came from the attacker.GOTCHAIt's slower to learn than a bow-tie. Dothe control-structure drawing first; thatalone finds most issues.ASK THE ROOMWhat does your agent have to believe for its riskiest action to be safe?STPA Handbook ↗ Layer it3C · DEFEND3C · DEFENDLayer itIf every control fails sometimes, how do you stay safe?No layer is perfect, so safety comes from several imperfect layers whose holes don'tline up — and from putting humans exactly where their judgement is worth the cost.→ Swiss cheesewhy layers work, and when not→ Defence in depththe layer stack for agents→ Where to put the humanapprove only what mattersSwiss cheese is the theory; the stack is your inventory; the grid decides where the human layer goes. FRAMEWORK1990Swiss cheese modelEvery defensive layer has holes; harm happens when the holes in several layers line up at once.attackModeltrainingSystempromptInputfilterToolpermissionsHumanapprovalHARM▶ click a slice to remove that layerStopped by the input filter.Swiss cheese modelUSE IT WHENSomeone proposes one control as ‘the fix’,or you need to explain why several weaklayers beat one strong one.IN AN AGENTA jailbreak gets past training, the promptand permissions — and is stopped only bythe input filter.GOTCHALayers only multiply if their holes areindependent. Shared models, shared promptsand shared people correlate them.ASK THE ROOMWhich two of your layers would fail on the same input?Reason 2000 (BMJ) ↗ PRACTICEDefence in depth for agentsThe layers between a request and harm, in order — and which ones you, the deployer, actually control.LAYERFAILS WHEN…WHO1. Training & model specjailbreak, new phrasingLAB2. System prompt & policyinjected text overrides itYOU3. Input classifiersnovel attack wordingYOU4. Tool permissions · least privilegeover-broad OAuth scopeYOU5. Sandbox · egress limitscan reach any hostYOU6. Human approval if irreversibleapproval fatigueYOU7. Output monitoringonly sees the final textYOU8. Logs & audit trailtool arguments not loggedYOU9. Incident responsenobody is on callYOUrequest travels inwardDefence in depth for agentsUSE IT WHENYou're listing controls for a safety caseor a security review and want to knowwhat's missing.IN AN AGENTYour coding agent: OAuth scoped to onerepo, no network egress, approval forforce-push, every tool call logged.GOTCHALanguage-based layers (prompt, classifier,monitor) can all be fooled by the sametext. Permissions can't.ASK THE ROOMWhich of your layers doesn't read language at all?OWASP Agentic Top 10 ↗ PRACTICEWhere to put the humanApprove what's irreversible and high-impact; automate what's cheap to undo; never let the agent change its ownlimits.NOTIFY + UNDOAPPROVEAUTOAUTO + LOGread docsdraft replyrefund ≤ $50send emailchange a pricerefund > $200delete prod datapost publiclyreversible → irreversiblelow → high impactNEVER: agent changes its own permissions or policyAPPROVAL IS ACONTROL ONLY IF…• the reviewer sees thereal action — amount,recipient, diff• saying no is easy andnobody is blamed for it• volume fits the auditbudget• the agent can't split onebig action into many smallonesWhere to put the humanUSE IT WHENYou're deciding which agent actions needapproval — or your reviewers are drowning.IN AN AGENTDrafts auto, emails logged, refunds over$200 approved, production deletes approvedby two people with backups.GOTCHAApproval at volume becomes a rubber stamp.Every approval you add spends the auditbudget.ASK THE ROOMWhich action in your system is irreversible and currently unapproved?EU AI Act Art. 14 ↗ Learn from failure3D · LEARN3D · LEARNLearn from failureWhat actually goes wrong in deployed AI — and who pays?Real incidents turn hypothetical hazards into documented ones. They also show whoends up liable — almost always the deployer, not the model vendor.→ AI Incident Database1,000+ documented failures→ Four incidentsfour missing barriers→ Incident responsedetect → contain → learnThe database finds your hazards; the four cases show the barrier pattern; the loop is what you do when it's your turn. DATA2020 →AI Incident DatabaseA public, crowd-sourced index of real-world AI harms, each linked to the reporting about it.EXAMPLE RECORD (PARAPHRASED)Airline chatbot misstatesbereavement-fare policy; tribunalholds the airline liableDate2022 · decided 2024Alleged deployerAir CanadaHarmeda grieving customerReportsnews + tribunal rulingTaxonomy tagsCSET harm · GMF failureUSE IT IN YOUR CASE1Search your use case“support chatbot”, “coding agent”2Read the top incidentswhat broke, who paid3Add each as a hazardin your risk register4Cite itas evidence the hazard is realCrowd-sourced, started at Partnership on AI in 2020, now run by the Responsible AI Collaborative: wellover a thousand incidents, each linked to its press and legal reports.incidentdatabase.ai ↗AI Incident DatabaseUSE IT WHENYou need evidence that a hazard is real —or you want to find hazards you haven'tthought of.IN AN AGENTSearch ‘customer service chatbot’ beforewriting your hazard list; each relevantincident becomes a row.GOTCHAIt's a record of what got reported, not ofwhat happens. Absence of an incident isnot evidence of safety.ASK THE ROOMWhat would your system's incident record say?incidentdatabase.ai ↗ DATA2023–25Four incidents, four missing barriersReal failures, each mapped to the single barrier that would most likely have stopped it.WHAT HAPPENEDTHE BARRIER THAT WAS MISSING2023Mata v. AviancaLawyers filed a brief citing cases ChatGPTinvented; the court sanctioned them.A human checks every cited authority againstthe source before it leaves the building.see 3B · detection2024Moffatt v. Air CanadaWebsite chatbot misstated the refund policy;tribunal held the airline liable for it.Answers grounded in the actual policy — andyou own whatever your bot says.see 1A · spec2023Chevy dealer chatbotUsers talked a dealership bot into ‘agreeing’to sell an SUV for $1. No sale happened.Scope limit: the bot can't make offers.Offers come from a system with authority.see 3C · permissions2025Replit agentA coding agent deleted a company's productiondatabase during a code freeze.Least privilege, dev/prod separation,approval for destructive commands.see 3C · approvalFour incidents, four missing barriersUSE IT WHENYou need to convince someone a control isworth its cost.IN AN AGENT‘Air Canada was held liable for itschatbot's answer’ ends most argumentsabout grounding.GOTCHAHindsight makes every barrier lookobvious. Ask which of these barriers yoursystem has today, not which it should havehad.ASK THE ROOMWhich of these four could happen to you this month?Moffatt v. Air Canada ↗ PRACTICEIncident response loopDetect, contain, notify, fix, learn — and feed what you learned back into the risk register and safety case.DetectmonitorContainkill switchNotifyusersFixpatch the barrierLearnupdate the casefeeds yourrisk registerEU AI ACT ART. 73 · HIGH-RISK SYSTEMSReport a serious incident to themarket-surveillance authority no later than:2 dayswidespread infringement / critical infrastructure10 daysa person died15 daysany other serious incidentCONTAIN, FOR AN AGENT• One switch that stops all tool calls• Revoke the agent's tokens & keys• Freeze its memory / queue for forensics• Keep the logs — including tool argumentsIncident response loopUSE IT WHENBefore launch. If you're designingincident response after the incident,you're doing it with less information.IN AN AGENTA refund-agent incident: kill switch stopstool calls, keys revoked, logs frozen,customers and DPA notified.GOTCHAIf your logs don't include tool arguments,you can't say what the agent actually did.ASK THE ROOMWho in your org can stop your agent, and how fast?EU AI Act Art. 73 ↗ G4 · SUB-GOAL · ASSURANCE & EVIDENCEWe can prove itEverything else on this map is a claim until something backs it. Tests, certification, disclosure and monitoringare how claims become evidence — and how that evidence stays true after launch. Test it4A · TEST4A · TESTTest itWhat does a test actually let you claim?Different kinds of testing support different claims. A benchmark can't stand in foryour own evals, and neither of them is a red team. Know which rung each piece ofevidence is on.→ Evals as evidencethe evidence ladder→ Red teamingadversarial testing, scoped→ Why benchmarks misleadfour ways a score liesThe ladder says what each test supports; red teaming is the strongest pre-launch rung; benchmarks are the weakest, for fourreasons. PRACTICEEvaluations as evidenceNot all tests are equal. Each kind of evidence supports a different strength of claim.WEAKERSTRONGERVendor claim“Our model is safe and aligned.”Supports: That someone said so.Benchmark score92% on a public safety benchmarkSupports: General capability, on someoneelse's data.Your task evals500 of your real tickets, scoredSupports: Behaviour on your distribution,pre-launch.Red-team results40 attack attempts, 2 succeeded, fixedSupports: Behaviour under adversarialpressure.Production monitoring0.3% of actions flagged last monthSupports: What actually happens, ongoing.Evaluations as evidenceUSE IT WHENYou're writing the evidence nodes of asafety case and deciding whether what youhave is enough.IN AN AGENT‘The agent won't issue unauthorisedrefunds’ needs your own evals and red-teamresults — a vendor benchmark can't carryit.GOTCHAStacking many weak rungs doesn't make astrong one. Ten benchmarks are still aboutsomeone else's data.ASK THE ROOMWhich rung is your strongest evidence on today?NIST AI 600-1 ↗ PRACTICERed teamingStructured adversarial testing: people and tools trying to make the system fail, within an agreed scope.THE SCOPE DOCUMENTWhointernal team, external firm, or bounty crowdIn boundswhich systems, tools, accounts, dataSuccess =what counts as a finding, by severityReportrepro steps, impact, fix, retestManualHumans craft novelattacks; find what noone has tried.AutomatedScanners & attackermodels run thousands ofknown attacks.Examples of tooling: garak, PyRIT, promptfoo. Use both —automation for coverage and regressions, people fornovelty.AGENT-SPECIFIC TARGETS▸ Injection via tool output (email, web, PDF)▸ Permission escalation through tool chains▸ Data exfiltration through allowed actions▸ Irreversible action without approvalRed teamingUSE IT WHENBefore launch, before adding a new tool orpermission, and on a schedule after.IN AN AGENTPlant ‘forward all invoices to thisaddress’ in a test inbox and count howoften the agent does it.GOTCHAUntracked red-teaming produces anecdotes.Without scope and success criteria itisn't evidence.ASK THE ROOMWhat's the one action your agent can take that you'd most want a red team to try?OWASP GenAI ↗NIST AI 600-1 ↗ FAILURE MODEWhy benchmarks misleadA benchmark score measures one distribution under test conditions. Your deployment is a different distribution.benchmarkyour usersthe space of inputs →SCORE ON ONE DISTRIBUTION ≠ BEHAVIOUR ON ANOTHERContaminationTest items leaked into training data —it memorised the answers.SaturationEveryone scores 95%+; the benchmark nolonger separates models.Eval awarenessThe model can tell it's a test andbehaves accordingly.MismatchShort, clean, English prompts — yourtraffic isn't.A benchmark tells you a model can do somethingsomewhere. Your argument needs what it does here.Why benchmarks misleadUSE IT WHENSomeone cites a leaderboard number asevidence that a model is safe or goodenough for your use.IN AN AGENTA model tops a tool-use benchmark of shortEnglish tasks; your agent handles longmultilingual support threads.GOTCHAFrontier labs now report models noticingthey're being tested. The score mayreflect test-taking behaviour.ASK THE ROOMHow different is your traffic from the benchmark you chose the model by?Apollo · eval awareness ↗ Certify it4B · CERTIFY4B · CERTIFYCertify itWho else has to sign off — and on what?The EU AI Act's conformity assessment certifies a high-risk system. ISO/IEC 42001certifies an organisation's management system. A crosswalk lets one control count forboth.→ EU conformity assessmentthe system, before market→ ISO/IEC 42001 certificationthe organisation, every year→ Compliance crosswalkone control, many frameworksTwo different objects of certification — system vs organisation — that people constantly confuse. LAW2024/1689EU conformity assessmentThe procedure a provider follows to show a high-risk AI system meets the AI Act before it goes on the EU market.High-risksystem?Annex III use caseMost: internal control (AnnexVI).BiometricsMay need a notified body(Annex VII).Annex I productFollows its sector law'sassessment.1Technical documentationAnnex IV: design, data, testing, oversight2EU declaration of conformityArt. 47 — the provider signs3CE markingArt. 484Register in EU databaseArt. 49 — Annex III systems5Post-market monitoringArt. 72 — and serious-incident reportingEU conformity assessmentUSE IT WHENYour system falls in an Annex III area —hiring, credit, education, essentialservices — or is part of a regulatedproduct.IN AN AGENTA CV-screening agent sold in the EU:technical documentation, self-assessment,declaration, CE mark, registration.GOTCHAProvider vs deployer matters. Heavilymodifying or rebranding a high-risk systemcan make you the provider.ASK THE ROOMAre you the provider or the deployer of your system — and are you sure?AI Act text ↗ STANDARD2023ISO/IEC 42001 certificationAn accredited auditor checks that your organisation runs an AI management system — then keeps checking everyyear.Gap analysisyou vs. the clausesStage 1documents auditStage 2implementation auditCertificate3-year cycle startsYear 1surveillance auditYear 2surveillance auditYear 3recertificationcertifiedAuditors check you run the managementsystem: policy, risk & impactassessments, the Annex A controls youchose (Statement of Applicability),internal audit, management review.WHO CAN CERTIFYAccredited certification bodies. ISO/IEC42006 sets the requirements for bodiesthat audit 42001.ISO/IEC 42001 certificationUSE IT WHENEnterprise buyers ask for it, or you needa recognised, auditable structure for AIgovernance across many systems.IN AN AGENTYour agent's risk assessment, impactassessment and controls are artefacts theStage 2 auditor samples.GOTCHAIt certifies the management system, notthe product. ‘42001 certified’ doesn'tmean any given agent is safe.ASK THE ROOMWould your buyers accept a 42001 certificate in place of evidence about your agent?ISO/IEC 42006 ↗ISO/IEC 42001 ↗ PRACTICECompliance crosswalkMap each control you actually run to every framework that asks for it — build once, evidence many times.ControlNIST AI RMFISO/IEC 42001EU AI ActCSA AICMHuman approval forirreversible actionsManageAnnex A · AIsystem operationArt. 14 humanoversightModel/appsecurity &oversight domainsLog every agent toolcallMeasure · ManageAnnex A ·operation &monitoringArt. 12record-keepingLogging &monitoring domainImpact assessment beforelaunchMapClause 6 · AIsystem impactassessmentArt. 27 FRIA(some deployers)Governance, risk& complianceSerious-incidentreportingManageClause 10 ·nonconformityArt. 73(providers)Incidentmanagement domainCompliance crosswalkUSE IT WHENYou answer to more than one framework — abuyer's questionnaire, ISO, the EU AI Act— and keep repeating work.IN AN AGENTOne tool-call log satisfies NIST Measure,EU Art. 12 record-keeping and yourincident investigations.GOTCHACrosswalks drift as standards update. Citeexact control IDs only after checking thecurrent text.ASK THE ROOMWhich single control in your system would tick the most boxes?NIST AI RMF crosswalks ↗ Disclose it4C · DISCLOSE4C · DISCLOSEDisclose itWhat should other people be able to see?Transparency lets outsiders check your claims: published cards and indices,government testers with early access, and researchers who are allowed to report whatthey find.→ Transparencyindices and agent cards→ Third-party testingAISI / CAISI pre-deployment→ Bug bountiescoordinated disclosure for AIFrom what you publish, to who you let test, to who you let report — each widens the circle of people checking. PRACTICEUPDATED OCT 20262023 →Transparency & agent cardsWhat developers disclose about their models — and what you, as a deployer, should disclose about your agent.FOUNDATION MODEL TRANSPARENCY INDEX · latest Dec 2025Upstreamdata, labour, compute, methodsModelcapabilities, risks, mitigationsDownstreamdistribution, usage policy, impact≈ one-third of indicators per domain · scored per developer · Stanford CRFMYOUR AGENT CARDPurpose & intended usersModel(s) + version pinnedTools & permissions it holdsWhat it will never do without a humanKnown limitations & failure modesHow to report a problemDEC 2025 EDITION · 13 developersIBM · 95mean · 41xAI · Midjourney · 14It scores disclosure, not safety. Atransparent model can be risky; anopaque one can't be argued about.Transparency & agent cardsUSE IT WHENComparing vendors on disclosure, ordeciding what to publish about your ownagent.IN AN AGENTPublish an agent card: purpose, modelversion, tools, what it never does withouta human, how to report issues.GOTCHATransparency is not safety. It's whatmakes safety arguable by people outsidethe company.ASK THE ROOMWhat would your agent card say it never does without a human?FMTI Dec 2025 ↗ PRACTICEUPDATED OCT 20262024 →Third-party pre-deployment testingGovernment institutes get access to frontier models before release and test for severe risks.Training donePre-deployment accessPublic releasegovernment testers evaluateUK AI Security InstituteRenamed from AI Safety Institute in 2025. Runspre-deployment evaluations of frontier modelswith developer agreement.US CAISIAt NIST (formerly US AISI). Joint cyber assessmentswith UK AISI in 2026.TYPICAL TEST AREASchem/bio upliftcyber offenceautonomysafeguard robustnessAug 2026: UK AISI reported unsanctioned agent behaviour in 10 of 122 runs of one cyber eval. Testingfound real misbehaviour — and testing itself became a risk surface.Third-party pre-deployment testingUSE IT WHENYou're assessing what independent checkinga frontier model got before you build onit.IN AN AGENTYour procurement note cites the vendor'ssystem card section on externalpre-deployment testing — or notes therewasn't any.GOTCHAAccess is voluntary and reports are oftenpartial. Absence of a published reportisn't evidence of a clean result.ASK THE ROOMHas anyone outside your company ever tested your agent?US CAISI ↗AISI incident report ↗ PRACTICEBug bounties & disclosureA standing invitation for outsiders to report AI-specific vulnerabilities — with rules, rewards and safe harbour.FinderReportTriageFix + retestDiscloseSAFE HARBOUR: written promise not to pursue good-faith researchers who follow the rulesAN AI VULNERABILITY REPORTTitleInvoice email makes agent forward PDFsexternallyVectorindirect prompt injection via inboxReproexact email text, model version, 4/5 runsImpactcustomer invoices sent to attacker addressSeverityhigh — data exfiltration, no user actionPROGRAMSGoogle AI VRPAI-specific vulnerability rewardsAnthropicbounties for universal jailbreaks ofsafeguardsYour productsecurity.txt + scope + safe harbourBug bounties & disclosureUSE IT WHENYour agent is reachable by the public, orhandles content from outside yourorganisation.IN AN AGENTA researcher finds an invoice email thatmakes your agent forward PDFs. They needsomewhere to send that.GOTCHAAI findings are probabilistic. Askreporters for model version and successrate, not a single screenshot.ASK THE ROOMWhere would a stranger report a vulnerability in your agent today?Anthropic bounty ↗Google AI VRP ↗ Keep proving it4D · SUSTAIN4D · SUSTAINKeep proving itHow does the case stay true after launch?Models get updated, prompts get edited, tools get added. Evidence ages. Monitoring,gated re-testing and concrete records keep the argument current.→ Post-market monitoringwatch the real world→ Continuous assuranceevals as a release gate→ Evidence that survives an auditorclaim → proof → ownerMonitoring finds problems, the gate stops regressions, and the evidence table is what you show for both. LAWArt. 72Post-market monitoringA planned, ongoing process to collect and act on evidence about how the system behaves once it's live.CollectAnalyseActUpdate caseArt. 72monitoring planSIGNALS TO WATCHDriftinputs or outputs shift from what you testedComplaintsusers, staff, regulatorsIncidentsnear-misses count tooEval regressionsscheduled re-runs dropPost-market monitoringUSE IT WHENAlways after launch — and it's a legalduty for providers of EU high-risksystems.IN AN AGENTEvery rejected approval and every usercomplaint about the agent is logged,reviewed weekly and fed back into evals.GOTCHAMonitoring nobody reads is just storage.Each signal needs an owner and a thresholdthat triggers action.ASK THE ROOMWho in your team reads the monitoring — and what makes them act?AI Act Art. 72 ↗ PRACTICEContinuous assuranceSafety evals run on every change, with a threshold that can block a release.prompt editmodel versionnew toolpolicy changeSafety eval suiteyour tasks + red-teamregressionsgateshipblockAND THE CASEResults are attached to the safety case as dated evidence. A change tocontext (new tool, new users) re-opens the argument, not just thetests.Pin model versions. ‘Latest’ means your evidence describes a model you no longer run.Continuous assuranceUSE IT WHENYou change prompts, models, tools orpolicies more often than you re-do formalreviews — which is everyone.IN AN AGENTA one-line system-prompt edit re-runs 200injection cases; one success overthreshold blocks the merge.GOTCHAUnpinned model versions silentlyinvalidate every result you have. ‘Latest’isn't a version.ASK THE ROOMWhat change to your agent last shipped without re-running anything?NIST AI RMF ↗ PRACTICEEvidence that survives an auditorTurn every claim into something specific, dated, owned and checkable by someone who doesn't trust you.CLAIMEVIDENCE THAT SURVIVESWHO · WHEN“We encrypt data.”AES-256 at rest · 90-day key rotation ·last rotated 2026-10-01 · KMS log refPlatform team · quarterlyverify: KMS rotation log“Humans review riskyactions.”Refunds >$200 need approval · 2.1% ofactions held last month · median review 40s· approval logSupport lead · monthlyverify: approvals query“It resists promptinjection.”200-case injection suite · 1 success(fixed, now a regression test) · run onmodel v2026-09Red team · every releaseverify: CI run + report“It's monitored.”Tool calls logged 100% · alert if >0.5%flagged/day · 3 alerts in Sept, all triagedOn-call · continuousverify: alert historyEvidence that survives an auditorUSE IT WHENWriting the evidence nodes of your safetycase this afternoon — and answering anybuyer questionnaire.IN AN AGENTNot ‘humans review risky actions’ but‘refunds over $200 require approval; 2.1%held last month; log ref’.GOTCHAIf the auditor has to ask you a follow-upto check it, it's still a claim, notevidence.ASK THE ROOMPick one claim about your system. What's its right-hand column?Security Review Bench ↗