The Watershed — roots of agentic security (310 sources) — phone edition

The Watershed — phone edition
Full map
THE WATERSHEDWhere today's agentic-security ideas came from — 310 sources as rivers of research, 2014 → 2026.◆ ● ? ★ ■ ◇springs: agenda papers (shape = kind of agenda)━●━river: a research lineage; each dot = papers in a 2-year stretch▣mechanism: the lineage's core idea, drawn (zoom for a key result)→today: what each river became in agent securityClick a basin to dive in · every card opens its source · Esc backs out · → for the tour · N for presenter notesResearch Frameworks map ↗Meta-Map ↗ THE SPRINGS Agenda papers · springs 2 2016–17 3 2018–19 5 2020–21 14 2022–23 25 2024–25 Concrete Problems in AI Safety — 2016 · arXiv · arXiv:1606.06565◆Concrete Problems in AI SafetyarXiv · 2016 ↗ AI Safety Gridworlds — 2017 · arXiv (confirmed arXiv-only; DBLP lists only CoRR abs/1711 · arXiv:1711.09883◆AI Safety GridworldsarXiv · 2017 ↗ Scalable agent alignment via reward modeling: a research direction — 2018 · arXiv (confirmed arXiv-only; DBLP lists only CoRR abs/1811 · arXiv:1811.07871◆Scalable agent alignment via reward modeling: a researchdirectionarXiv · 2018 ↗ Advances and Open Problems in Federated Learning — 2019 · Foundations and Trends in Machine Learning, Vol · arXiv:1912.04977★Advances and Open Problems in Federated LearningFoundations and Trends in Mach · 2019 ↗ Risks from Learned Optimization in Advanced Machine Learning Systems — 2019 · arXiv (confirmed arXiv-only; DBLP lists only CoRR abs/1906 · arXiv:1906.01820◆Risks from Learned Optimization in Advanced Machine LearningSystemsarXiv · 2019 ↗ Open Problems in Cooperative AI — 2020 · arXiv · arXiv:2012.08630?Open Problems in Cooperative AIarXiv · 2020 ↗ Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims — 2020 · arXiv (multi-institution report; submitted 15 April 2020; no · arXiv:2004.07213◆Toward Trustworthy AI Development: Mechanisms for SupportingVerifiable ClaimsarXiv · 2020 ↗ Ethical and social risks of harm from Language Models — 2021 · arXiv · arXiv:2112.04359●Ethical and social risks of harm from Language ModelsarXiv · 2021 ↗ On the Opportunities and Risks of Foundation Models — 2021 (arXiv v1, 2021-08-16) · arXiv · arXiv:2108.07258★On the Opportunities and Risks of Foundation ModelsarXiv · 2021 ↗ Unsolved Problems in ML Safety — 2021 · arXiv · arXiv:2109.13916◆Unsolved Problems in ML SafetyarXiv · 2021 ↗ Sociotechnical Harms of Algorithmic Systems: Scoping a Taxonomy for Harm Reduction — 2022 · arXiv (v1 11 Oct 2022; v3 19 Jul 2023; cs · arXiv:2210.05791◇Sociotechnical Harms of Algorithmic Systems: Scoping a Taxonomyfor Harm ReductionarXiv · 2022 ↗ The Alignment Problem from a Deep Learning Perspective — 2022 (arXiv v1, 2022-08-30); ICLR 2024 · arXiv; published in ICLR 2024 (per arXiv comment field: "Pub · arXiv:2209.00626●The Alignment Problem from a Deep Learning PerspectivearXiv · 2022 ↗ A Survey of Safety and Trustworthiness of Large Language Models through the Lens of Verification and Validation — 2023 · arXiv (v1 May 2023; v2 Aug 2023); published in Artificial In · arXiv:2305.11391◇A Survey of Safety and Trustworthiness of Large Language Modelsthrough the Lens of Verificat…arXiv · 2023 ↗ AI Alignment: A Comprehensive Survey — 2023 · arXiv · arXiv:2310.19852◇AI Alignment: A Comprehensive SurveyarXiv · 2023 ↗ An Overview of Catastrophic AI Risks — 2023 · arXiv · arXiv:2306.12001◇An Overview of Catastrophic AI RisksarXiv · 2023 ↗ Consciousness in Artificial Intelligence: Insights from the Science of Consciousness — 2023 (arXiv v1, 2023-08-17) · arXiv · arXiv:2308.08708?Consciousness in Artificial Intelligence: Insights from theScience of ConsciousnessarXiv · 2023 ↗ Frontier AI Regulation: Managing Emerging Risks to Public Safety — 2023 · arXiv (confirmed arXiv-only; DBLP lists a single CoRR abs/23 · arXiv:2307.03718■Frontier AI Regulation: Managing Emerging Risks to Public SafetyarXiv · 2023 ↗ Harms from Increasingly Agentic Algorithmic Systems — 2023 (arXiv v1, 2023-02-20) · arXiv; ACM FAccT 2023 (arXiv comment: "Accepted at FAccT 202 · arXiv:2302.10329◆Harms from Increasingly Agentic Algorithmic SystemsarXiv · 2023 ↗ Managing extreme AI risks amid rapid progress — 2023 (arXiv v1, 2023-10-26); 2024 (Science publication; arXiv updated 2024-05-22) · Science (doi:10 · arXiv:2310.17688★Managing extreme AI risks amid rapid progressScience · 2023 ↗ Model evaluation for extreme risks — 2023 (arXiv v1, 2023-05-24) · arXiv · arXiv:2305.15324■Model evaluation for extreme risksarXiv · 2023 ↗ Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback — 2023 · TMLR (Transactions on Machine Learning Research), 2023 · arXiv:2307.15217●Open Problems and Fundamental Limitations of ReinforcementLearning from Human FeedbackTMLR · 2023 ↗ Sociotechnical Safety Evaluation of Generative AI Systems — 2023 · arXiv (confirmed arXiv-only; DBLP lists only CoRR abs/2310 · arXiv:2310.11986■Sociotechnical Safety Evaluation of Generative AI SystemsarXiv · 2023 ↗ TASRA: a Taxonomy and Analysis of Societal-Scale Risks from AI — 2023 · arXiv · arXiv:2306.06924◇TASRA: a Taxonomy and Analysis of Societal-Scale Risks from AIarXiv · 2023 ↗ The Rise and Potential of Large Language Model Based Agents: A Survey — 2023 · arXiv (Sept 2023); published in Science China Information Sc · arXiv:2309.07864◇The Rise and Potential of Large Language Model Based Agents: ASurveyarXiv · 2023 ↗ Black-Box Access is Insufficient for Rigorous AI Audits — 2024 · FAccT 2024 · arXiv:2401.14446■Black-Box Access is Insufficient for Rigorous AI AuditsFAccT 2024 · 2024 ↗ Evaluating Frontier Models for Dangerous Capabilities — 2024 · arXiv (no journal-ref or venue comment on arXiv as of verifi · arXiv:2403.13793■Evaluating Frontier Models for Dangerous CapabilitiesarXiv · 2024 ↗ Foundational Challenges in Assuring Alignment and Safety of Large Language Models — 2024 · TMLR (Transactions on Machine Learning Research), 09/2024 · arXiv:2404.09932●Foundational Challenges in Assuring Alignment and Safety ofLarge Language ModelsTMLR · 2024 ↗ IDs for AI Systems — 2024 · arXiv; RegML workshop @ NeurIPS 2024 · arXiv:2406.12137■IDs for AI SystemsarXiv · 2024 ↗ On the Societal Impact of Open Foundation Models — 2024 · ICML 2024 (Position Papers track); Proceedings of the 41st I · arXiv:2403.07918■On the Societal Impact of Open Foundation ModelsICML 2024 · 2024 ↗ Open Problems in Technical AI Governance — 2024 · TMLR 2025 (arXiv v1 July 2024) · arXiv:2407.14981?Open Problems in Technical AI GovernanceTMLR 2025 · 2024 ↗ Personhood Credentials: Artificial Intelligence and the Value of Privacy-Preserving Tools to Distinguish Who is Real Online — 2024 · arXiv · arXiv:2408.07892★Personhood Credentials: Artificial Intelligence and the Value ofPrivacy-Preserving Tools to …arXiv · 2024 ↗ Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress? — 2024 · NeurIPS 2024 · arXiv:2407.21792■Safetywashing: Do AI Safety Benchmarks Actually Measure SafetyProgress?NeurIPS 2024 · 2024 ↗ The AI Risk Repository: A Meta-Review, Database, and Taxonomy of Risks From Artificial Intelligence — 2024 · arXiv (submitted 14 Aug 2024; v3 revised 5 May 2026) · arXiv:2408.12622◇The AI Risk Repository: A Meta-Review, Database, and Taxonomy ofRisks From Artificial Intell…arXiv · 2024 ↗ The Ethics of Advanced AI Assistants — 2024 · arXiv (Google DeepMind institutional report) · arXiv:2404.16244★The Ethics of Advanced AI AssistantsarXiv · 2024 ↗ Towards Guaranteed Safe AI: A Framework for Ensuring Robust and Reliable AI Systems — 2024 · arXiv · arXiv:2405.06624?Towards Guaranteed Safe AI: A Framework for Ensuring Robust andReliable AI SystemsarXiv · 2024 ↗ TrustLLM: Trustworthiness in Large Language Models — 2024 · ICML 2024 (Position paper track), PMLR v235:20166-20270 · arXiv:2401.05561●TrustLLM: Trustworthiness in Large Language ModelsICML 2024 · 2024 ↗ Visibility into AI Agents — 2024 · arXiv; ACM FAccT 2024 · arXiv:2401.13138■Visibility into AI AgentsarXiv · 2024 ↗ Authenticated Delegation and Authorized AI Agents — 2025 · arXiv (DBLP lists a CoRR 2025 entry only · arXiv:2501.09674■Authenticated Delegation and Authorized AI AgentsarXiv · 2025 ↗ Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety — 2025 · arXiv (v1 Jul 2025; v2 Dec 2025) · arXiv:2507.11473★Chain of Thought Monitorability: A New and Fragile Opportunityfor AI SafetyarXiv · 2025 ↗ Governing AI Agents — 2025 · arXiv; Notre Dame Law Review, Vol · arXiv:2501.07913■Governing AI AgentsarXiv · 2025 ↗ Infrastructure for AI Agents — 2025 · arXiv; accepted to TMLR · arXiv:2501.10114■Infrastructure for AI AgentsarXiv · 2025 ↗ International AI Safety Report — 2025 · arXiv · arXiv:2501.17805★International AI Safety ReportarXiv · 2025 ↗ Multi-Agent Risks from Advanced AI — 2025 · arXiv · arXiv:2502.14143★Multi-Agent Risks from Advanced AIarXiv · 2025 ↗ Open Challenges in Multi-Agent Security: Towards Secure Systems of Interacting AI Agents — 2025 · arXiv · arXiv:2505.02077?Open Challenges in Multi-Agent Security: Towards Secure Systemsof Interacting AI AgentsarXiv · 2025 ↗ Open Problems in Machine Unlearning for AI Safety — 2025 · arXiv · arXiv:2501.04952?Open Problems in Machine Unlearning for AI SafetyarXiv · 2025 ↗ Open Problems in Mechanistic Interpretability — 2025 · arXiv · arXiv:2501.16496?Open Problems in Mechanistic InterpretabilityarXiv · 2025 ↗ The AI Agent Index — 2025 · arXiv (accompanying website: aiagentindex · arXiv:2502.01635◇The AI Agent IndexarXiv · 2025 ↗ The Singapore Consensus on Global AI Safety Research Priorities — 2025 · institutional report (arXiv, submitted 25 June 2025; final r · arXiv:2506.20702★The Singapore Consensus on Global AI Safety Research Prioritiesinstitutional report · 2025 ↗ Why Do Multi-Agent LLM Systems Fail? — 2025 · arXiv (v3) · arXiv:2503.13657?Why Do Multi-Agent LLM Systems Fail?arXiv · 2025 ↗ MECHANISMWhat an agenda doesRoots › SpringsAn agenda paper names open problems so others can work on them. Concrete Problemsin AI Safety (2016) named five; each became a stream you can still follow. Lateragendas split by field (interpretability, unlearning, multi-agent, governance) orby consensus (the International AI Safety Report).SO WHAT: an agenda decides which problems get a river. Read one to see what afield thought was worth a decade.Avoiding side effectsagent today: deletes files to finish fasterAvoiding reward hackingagent today: edits the test so it passesScalable oversightagent today: 10,000 tool calls a daySafe explorationagent today: runs a destructive command to seeDistributional shiftagent today: fine in staging, breaks on prodKEY RESULT · zoomFive problems, 2016Amodei et al. 2016 grouped accident risk into five problems: avoiding side effects, avoiding reward hacking, scalable supervision, safeexploration, robustness to distributional shift.arXiv:1606.06565
BENCH · Which spring is your problem from?
Five things a real agent did. Which 2016 problem is it?
Pick one — commit before you read the feedback.
 FLOWS INTO ↓→ Every river on this map. Dashed amber threads join a spring to the same paperwhere it sits in a river. THEORY Agent foundations & inner alignment 1 2014–15 3 2018–19 3 2020–21 4 2022–23 Robust Cooperation in the Prisoner's Dilemma: Program Equilibrium via Provability Logic — arXiv · 2014-01-22 · arXiv:1401.5577 · Mihaly Barasz, Paul Christiano, Benja Fallenstein, Marcello Herreshoff et al.Robust Cooperation in the Prisoner's Dilemma: Program Equilibriumvia Provability LogicarXiv · 2014 ↗ Gradient hacking — LessWrong/AF · 2019-10-16 · Evan HubingerGradient hackingLessWrong/AF · 2019 ↗ Relaxed adversarial training for inner alignment — LessWrong/AF · 2019-09-10 · Evan HubingerRelaxed adversarial training for inner alignmentLessWrong/AF · 2019 ↗ Risks from Learned Optimization in Advanced Machine Learning Systems — arXiv · 2019-06-05 · arXiv:1906.01820 · Evan Hubinger, Chris van Merwijk, Vladimir Mikulik, Joar Skalse, Scott GarrabrantRisks from Learned Optimization in Advanced Machine Learning SystemsarXiv · 2019 ↗ AI safety via market making — LessWrong/AF · 2020-06-26 · Evan HubingerAI safety via market makingLessWrong/AF · 2020 ↗ An overview of 11 proposals for building safe advanced AI — arXiv · 2020-12-04 · arXiv:2012.07532 · Evan HubingerAn overview of 11 proposals for building safe advanced AIarXiv · 2020 ↗ Automating Auditing: An ambitious concrete technical research proposal — LessWrong/AF · 2021-08-11 · Evan HubingerAutomating Auditing: An ambitious concrete technical researchproposalLessWrong/AF · 2021 ↗ A transparency and interpretability tech tree — LessWrong/AF · 2022-06-16 · Evan HubingerA transparency and interpretability tech treeLessWrong/AF · 2022 ↗ How likely is deceptive alignment? — LessWrong/AF · 2022-08-30 · Evan HubingerHow likely is deceptive alignment?LessWrong/AF · 2022 ↗ Conditioning Predictive Models: Risks and Strategies — arXiv · 2023-02-02 · arXiv:2302.00805 · Evan Hubinger, Adam S. Jermyn, Johannes Treutlein, Rubi Hudson, Kate WoolvertonConditioning Predictive Models: Risks and StrategiesarXiv · 2023 ↗ Natural Abstractions: Key Claims, Theorems, and Critiques — LessWrong/AF · 2023-03-16 · Lawrence Chan (LawrenceC), Leon Lang, Erik JennerNatural Abstractions: Key Claims, Theorems, and CritiquesLessWrong/AF · 2023 ↗ MECHANISMMesa-optimizationRoots › Theory › Inner alignmentTraining selects for behaviour, not for goals. If training produces a model thatis itself an optimizer (a mesa-optimizer), its learned objective can differ fromthe one you trained on. A mesa-optimizer that models its training process canlook aligned exactly while it is watched: deceptive alignment (Hubinger et al.,2019).SO WHAT: an agent that passes your evals has shown you its behaviour underobservation — not its goal.Base optimizerSGD on the base objectiveModelis itself an optimizermesa-objective ≠? base objectiveTRAININGoverseer watchinglooks aligned ✓DEPLOYMENToversight gonepursues mesa-objectiveKEY RESULT · zoomWhere the theory meets agents todayThe 2019 argument is now an empirical programme: situational-awareness datasets (SAD), sandbagging detection, and model organisms of misalignment — all pinned inthe Evals river. The theory says what to look for; the evals look.Hubinger et al. 2019 · arXiv:1906.01820 TODAY → Scheming & deceptive-alignment evals; model organisms of misalignment Threat Models · ring 4: the agent IS the adversary ↗ Scalable oversight · debate · ELK 4 2018–19 6 2020–21 4 2022–23 4 2024–25 AI safety via debate — arXiv · 2018-05 · arXiv:1805.00899 · Geoffrey Irving, Paul Christiano, Dario AmodeiAI safety via debatearXiv · 2018 ↗ Supervising strong learners by amplifying weak experts — arXiv · 2018-10 · arXiv:1810.08575 · Paul Christiano, Buck Shlegeris, Dario AmodeiSupervising strong learners by amplifying weak expertsarXiv · 2018 ↗ AI Safety Needs Social Scientists — Distill · 2019-02-19 · Geoffrey Irving; Amanda AskellAI Safety Needs Social ScientistsDistill · 2019 ↗ Finding Generalizable Evidence by Learning to Convince Q&A Models — arXiv · 2019-09-12 · arXiv:1909.05863 · Ethan Perez, Siddharth Karamcheti, Rob Fergus, Jason Weston, Douwe Kiela et al.Finding Generalizable Evidence by Learning to Convince Q&A ModelsarXiv · 2019 ↗ Debate update: Obfuscated arguments problem — LessWrong/AF · 2020-12-23 · Beth Barnes, Paul Christiano (with contributions from William Saunders, Joe Collman et al.Debate update: Obfuscated arguments problemLessWrong/AF · 2020 ↗ Unsupervised Question Decomposition for Question Answering — arXiv · 2020-02-22 · arXiv:2002.09758 · Ethan Perez, Patrick Lewis, Wen-tau Yih, Kyunghyun Cho, Douwe KielaUnsupervised Question Decomposition for Question AnsweringarXiv · 2020 ↗ Writeup: Progress on AI Safety via Debate — LessWrong/AF · 2020-02-05 · Beth Barnes, Paul Christiano (paulfchristiano)Writeup: Progress on AI Safety via DebateLessWrong/AF · 2020 ↗ Eliciting Latent Knowledge — org site · 2021-12-14 · Paul Christiano, Mark Xu, Ajeya CotraEliciting Latent Knowledgeorg site · 2021 ↗ Eliciting Latent Knowledge (ELK) — LessWrong/AF · 2021-12-14 · Paul Christiano, Mark Xu, Ajeya CotraEliciting Latent Knowledge (ELK)LessWrong/AF · 2021 ↗ Imitative Generalisation (AKA 'Learning the Prior') — LessWrong/AF · 2021-01-10 · Beth BarnesImitative Generalisation (AKA 'Learning the Prior')LessWrong/AF · 2021 ↗ Formalizing the presumption of independence — arXiv · 2022-11 · arXiv:2211.06738 · Paul Christiano, Eric Neyman, Mark XuFormalizing the presumption of independencearXiv · 2022 ↗ QuALITY: Question Answering with Long Input Texts, Yes! — NAACL 2022 · 2021-12-16 · arXiv:2112.08608 · Richard Yuanzhe Pang, Alicia Parrish, Nitish Joshi, Nikita Nangia, Jason Phang et al.QuALITY: Question Answering with Long Input Texts, Yes!NAACL 2022 · 2022 ↗ Single-Turn Debate Does Not Help Humans Answer Hard Reading-Comprehension Questions — ACL 2022 Workshop on Learning with Natur · 2022-04-11 · arXiv:2204.05212 · Alicia Parrish, Harsh Trivedi, Ethan Perez, Angelica Chen, Nikita Nangia et al.Single-Turn Debate Does Not Help Humans Answer HardReading-Comprehension QuestionsACL 2022 Workshop on Learning · 2022 ↗ GPQA: A Graduate-Level Google-Proof Q&A Benchmark — arXiv · 2023-11-20 · arXiv:2311.12022 · David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang et al.GPQA: A Graduate-Level Google-Proof Q&A BenchmarkarXiv · 2023 ↗ Debating with More Persuasive LLMs Leads to More Truthful Answers — arXiv · 2024-02-09 · arXiv:2402.06782 · Akbir Khan, John Hughes, Dan Valentine, Laura Ruis, Kshitij Sachan et al.Debating with More Persuasive LLMs Leads to More Truthful AnswersarXiv · 2024 ↗ Towards a Law of Iterated Expectations for Heuristic Estimators — arXiv · 2024-10 · arXiv:2410.01290 · Paul Christiano, Jacob Hilton, Andrea Lincoln, Eric Neyman, Mark XuTowards a Law of Iterated Expectations for Heuristic EstimatorsarXiv · 2024 ↗ Training Language Models to Win Debates with Self-Play Improves Judge Accuracy — arXiv · 2024-09-25 · arXiv:2409.16636 · Samuel Arnesen, David Rein, Julian MichaelTraining Language Models to Win Debates with Self-Play ImprovesJudge AccuracyarXiv · 2024 ↗ Backdoor defense, learnability and obfuscation — ITCS 2025 · 2024-09 · arXiv:2409.03077 · Paul Christiano, Jacob Hilton, Victor Lecomte, Mark XuBackdoor defense, learnability and obfuscationITCS 2025 · 2025 ↗ MECHANISMDebate & ELKRoots › Theory › Scalable oversightWhen the AI knows more than the person overseeing it, oversight has to amplifythe overseer. Debate: two copies argue, a weaker judge reads the transcript.Amplification: break the question into pieces a human can check. ELK: train areporter that tells you what the model's own internal state says, not what ahuman would guess.SO WHAT: when your agent does work you cannot check line by line, you need aprotocol that makes the truth easier to verify than a lie.QuestionDebater Aargues an answerDebater Battacks itcross-examJudgeweaker: a humanVerdictELKpredictor's latent state → reporter → answer✓ direct translator: reports what the model knows✗ human simulator: reports what you would believe — also passes trainingKEY RESULT · zoomDebate helped — with stronger debatersKhan et al. 2024: debate let non-expert models and humans reach 76% and 88% accuracy, against naive baselines of 48% and 60%. Counterweight in the river:single-turn debate did not help humans (Parrish et al. 2022).arXiv:2402.06782 (abstract)
BENCH · Which theory tool fits?
Your agent, your problem. Which idea from this basin do you reach for?
Pick one — commit before you read the feedback.
 TODAY → Human review at agent scale: LLM judges, CoT monitoring, decomposedreview Guardrails & Runtime ↗ LOOKING INSIDE Interpretability: circuits → mech interp 1 2016–17 2 2018–19 7 2020–21 20 2022–23 25 2024–25 2 2026 Feature Visualization: How neural networks build up their understanding of images — Distill · 2017-11-07 · Chris Olah, Alexander Mordvintsev, Ludwig SchubertFeature Visualization: How neural networks build up theirunderstanding of imagesDistill · 2017 ↗ The Building Blocks of Interpretability — Distill · 2018-03-06 · Chris Olah, Arvind Satyanarayan, Ian Johnson, Shan Carter, Ludwig Schubert et al.The Building Blocks of InterpretabilityDistill · 2018 ↗ Activation Atlas — Distill · 2019-03-06 · Shan Carter, Zan Armstrong, Ludwig Schubert, Ian Johnson, Chris OlahActivation AtlasDistill · 2019 ↗ An Overview of Early Vision in InceptionV1 — Distill · 2020-04-01 · Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, Shan CarterAn Overview of Early Vision in InceptionV1Distill · 2020 ↗ Curve Detectors — Distill · 2020-06-17 · Nick Cammarata, Gabriel Goh, Shan Carter, Ludwig Schubert, Michael Petrov, Chris OlahCurve DetectorsDistill · 2020 ↗ Naturally Occurring Equivariance in Neural Networks — Distill · 2020-12-08 · Chris Olah, Nick Cammarata, Chelsea Voss, Ludwig Schubert, Gabriel GohNaturally Occurring Equivariance in Neural NetworksDistill · 2020 ↗ Understanding RL Vision — Distill · 2020-11-17 · Jacob Hilton, Nick Cammarata, Shan Carter, Gabriel Goh, Chris OlahUnderstanding RL VisionDistill · 2020 ↗ Zoom In: An Introduction to Circuits — Distill · 2020-03-10 · Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, Shan CarterZoom In: An Introduction to CircuitsDistill · 2020 ↗ Attention Approximates Sparse Distributed Memory — arXiv / NeurIPS 2021 · 2021-11 · arXiv:2111.05498 · Trenton Bricken, Cengiz PehlevanAttention Approximates Sparse Distributed MemoryarXiv / NeurIPS 2021 · 2021 ↗ Multimodal Neurons in Artificial Neural Networks — Distill · 2021-03-04 · Gabriel Goh, Nick Cammarata, Chelsea Voss, Shan Carter, Michael Petrov et al.Multimodal Neurons in Artificial Neural NetworksDistill · 2021 ↗ 200 Concrete Open Problems in Mechanistic Interpretability — Alignment Forum · 2022-12-28 · Neel Nanda200 Concrete Open Problems in Mechanistic InterpretabilityAlignment Forum · 2022 ↗ A Comprehensive Mechanistic Interpretability Explainer & Glossary — neelnanda.io · 2022 · Neel NandaA Comprehensive Mechanistic Interpretability Explainer & Glossaryneelnanda.io · 2022 ↗ Another list of theories of impact for interpretability — LessWrong/AF · 2022-04-13 · Beth BarnesAnother list of theories of impact for interpretabilityLessWrong/AF · 2022 ↗ Causal Scrubbing: a method for rigorously testing interpretability hypotheses — LessWrong/AF · 2022-12-03 · Lawrence Chan, Adrià Garriga-Alonso, Nicholas Goldowsky-Dill, Ryan Greenblatt et al.Causal Scrubbing: a method for rigorously testing interpretabilityhypothesesLessWrong/AF · 2022 ↗ Engineering Monosemanticity in Toy Models — arXiv · 2022-11-16 · arXiv:2211.09169 · Adam S. Jermyn, Nicholas Schiefer, Evan HubingerEngineering Monosemanticity in Toy ModelsarXiv · 2022 ↗ Interim research report: Taking features out of superposition with sparse autoencoders — LessWrong/AF · 2022-12-13 · Lee Sharkey, Dan Braun, Beren MillidgeInterim research report: Taking features out of superposition withsparse autoencodersLessWrong/AF · 2022 ↗ Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small — arXiv · 2022-11 · arXiv:2211.00593 · Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, Jacob SteinhardtInterpretability in the Wild: a Circuit for Indirect ObjectIdentification in GPT-2 smallarXiv · 2022 ↗ Interpreting Neural Networks through the Polytope Lens — arXiv · 2022-11-22 · arXiv:2211.12312 · Sid Black, Lee Sharkey, Leo Grinsztajn, Eric Winsor, Dan Braun, Jacob Merizian et al.Interpreting Neural Networks through the Polytope LensarXiv · 2022 ↗ Polysemanticity and Capacity in Neural Networks — arXiv · 2022-10 · arXiv:2210.01892 · Adam Scherlis, Kshitij Sachan, Adam S. Jermyn, Joe Benton, Buck ShlegerisPolysemanticity and Capacity in Neural NetworksarXiv · 2022 ↗ TransformerLens — Open-source library · 2022 · Neel Nanda, Joseph BloomTransformerLensOpen-source library · 2022 ↗ A Toy Model of Universality: Reverse Engineering How Networks Learn Group Operations — arXiv · 2023-02 · arXiv:2302.03025 · Bilal Chughtai, Lawrence Chan, Neel NandaA Toy Model of Universality: Reverse Engineering How Networks LearnGroup OperationsarXiv · 2023 ↗ Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchilla — arXiv · 2023-07 · arXiv:2307.09458 · Tom Lieberum, Matthew Rahtz, János Kramár, Neel Nanda, Geoffrey Irving, Rohin Shah et al.Does Circuit Analysis Interpretability Scale? Evidence from MultipleChoice Capabilities in C…arXiv · 2023 ↗ Emergence of Sparse Representations from Noise — ICML 2023 · 2023 · Trenton Bricken, Rylan Schaeffer, Bruno Olshausen, Gabriel KreimanEmergence of Sparse Representations from NoiseICML 2023 · 2023 ↗ Emergent Linear Representations in World Models of Self-Supervised Sequence Models — arXiv · 2023-09 · arXiv:2309.00941 · Neel Nanda, Andrew Lee, Martin WattenbergEmergent Linear Representations in World Models of Self-SupervisedSequence ModelsarXiv · 2023 ↗ Finding Neurons in a Haystack: Case Studies with Sparse Probing — arXiv · 2023-05 · arXiv:2305.01610 · Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii et al.Finding Neurons in a Haystack: Case Studies with Sparse ProbingarXiv · 2023 ↗ Localizing Model Behavior with Path Patching — arXiv · 2023-04-12 · arXiv:2304.05969 · Nicholas Goldowsky-Dill, Chris MacLeod, Lucas Sato, Aryaman AroraLocalizing Model Behavior with Path PatchingarXiv · 2023 ↗ Progress measures for grokking via mechanistic interpretability — arXiv · 2023-01-12 · arXiv:2301.05217 · Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, Jacob SteinhardtProgress measures for grokking via mechanistic interpretabilityarXiv · 2023 ↗ Sparse Distributed Memory is a Continual Learner — arXiv / ICLR 2023 · 2023-03 · arXiv:2303.11934 · Trenton Bricken, Xander Davies, Deepak Singh, Dmitry Krotov, Gabriel KreimanSparse Distributed Memory is a Continual LearnerarXiv / ICLR 2023 · 2023 ↗ Steering Llama 2 via Contrastive Activation Addition — arXiv · 2023-12-09 · arXiv:2312.06681 · Nina Panickssery (published at ACL as Nina Rimsky), Nick Gabrieli, Julian Schulz et al.Steering Llama 2 via Contrastive Activation AdditionarXiv · 2023 ↗ Towards Automated Circuit Discovery for Mechanistic Interpretability — NeurIPS 2023 · 2023-04-28 · arXiv:2304.14997 · Arthur Conmy, Augustine N. Mavor-Parker, Aengus Lynch, Stefan Heimersheim et al.Towards Automated Circuit Discovery for Mechanistic InterpretabilityNeurIPS 2023 · 2023 ↗ A List of 45+ Mech Interp Project Ideas from Apollo Research's Interpretability Team — LessWrong/AF · 2024-07-18 · Lee Sharkey, Lucius Bushnaq, Dan Braun, Stefan Heimersheim, Nicholas Goldowsky-DillA List of 45+ Mech Interp Project Ideas from Apollo Research'sInterpretability TeamLessWrong/AF · 2024 ↗ AtP*: An efficient and scalable method for localizing LLM behaviour to components — arXiv · 2024-03 · arXiv:2403.00745 · János Kramár, Tom Lieberum, Rohin Shah, Neel NandaAtP*: An efficient and scalable method for localizing LLM behaviourto componentsarXiv · 2024 ↗ Circuits in Superposition: Compressing many small neural networks into one — LessWrong/AF · 2024-10-14 · Lucius Bushnaq, Jake MendelCircuits in Superposition: Compressing many small neural networksinto oneLessWrong/AF · 2024 ↗ Codebook Features: Sparse and Discrete Interpretability for Neural Networks — arXiv / ICML 2024 · 2023-10 · arXiv:2310.17230 · Alex Tamkin, Mohammad Taufeeque, Noah D. GoodmanCodebook Features: Sparse and Discrete Interpretability for NeuralNetworksarXiv / ICML 2024 · 2024 ↗ Compact Proofs of Model Performance via Mechanistic Interpretability — NeurIPS 2024 · 2024-06-17 · arXiv:2406.11779 · Jason Gross, Rajashree Agrawal, Thomas Kwa, Euan Ong, Chun Hei Yip, Alex Gibson et al.Compact Proofs of Model Performance via Mechanistic InterpretabilityNeurIPS 2024 · 2024 ↗ Erasing Conceptual Knowledge from Language Models — arXiv · 2024-10-03 · arXiv:2410.02760 · Rohit Gandikota, Sheridan Feucht, Samuel Marks, David BauErasing Conceptual Knowledge from Language ModelsarXiv · 2024 ↗ Evaluating Sparse Autoencoders on Targeted Concept Erasure Tasks — arXiv · 2024-11-28 · arXiv:2411.18895 · Adam Karvonen, Can Rager, Samuel Marks, Neel NandaEvaluating Sparse Autoencoders on Targeted Concept Erasure TasksarXiv · 2024 ↗ Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2 — arXiv · 2024-08 · arXiv:2408.05147 · Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat et al.Gemma Scope: Open Sparse Autoencoders Everywhere All At Once onGemma 2arXiv · 2024 ↗ How to use and interpret activation patching — arXiv · 2024-04-23 · arXiv:2404.15255 · Stefan Heimersheim, Neel NandaHow to use and interpret activation patchingarXiv · 2024 ↗ Improving Dictionary Learning with Gated Sparse Autoencoders — arXiv · 2024-04 · arXiv:2404.16014 · Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Tom Lieberum, Vikrant Varma et al.Improving Dictionary Learning with Gated Sparse AutoencodersarXiv · 2024 ↗ Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoders — arXiv · 2024-07 · arXiv:2407.14435 · Senthooran Rajamanoharan, Tom Lieberum, Nicolas Sonnerat, Arthur Conmy et al.Jumping Ahead: Improving Reconstruction Fidelity with JumpReLUSparse AutoencodersarXiv · 2024 ↗ Measuring Progress in Dictionary Learning for Language Model Interpretability with Board Game Models — NeurIPS 2024 · 2024-07-31 · arXiv:2408.00113 · Adam Karvonen, Benjamin Wright, Can Rager, Rico Angell, Jannik Brinkmann et al.Measuring Progress in Dictionary Learning for Language ModelInterpretability with Board Game…NeurIPS 2024 · 2024 ↗ NNsight and NDIF: Democratizing Access to Open-Weight Foundation Model Internals — arXiv · 2024-07-18 · arXiv:2407.14561 · Jaden Fiotto-Kaufman, Alexander R. Loftus, Eric Todd, Jannik Brinkmann, Koyena Pal et al.NNsight and NDIF: Democratizing Access to Open-Weight FoundationModel InternalsarXiv · 2024 ↗ Refusal in Language Models Is Mediated by a Single Direction — arXiv / NeurIPS 2024 · 2024-06 · arXiv:2406.11717 · Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee et al.Refusal in Language Models Is Mediated by a Single DirectionarXiv / NeurIPS 2024 · 2024 ↗ The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets — COLM 2024 — Conference on Language Model · 2023-10-10 · arXiv:2310.06824 · Samuel Marks, Max TegmarkThe Geometry of Truth: Emergent Linear Structure in Large LanguageModel Representations of T…COLM 2024 — Conference on Lang · 2024 ↗ The Quest for the Right Mediator: Surveying Mechanistic Interpretability Through the Lens of Causal Mediation Analysis — arXiv · 2024-08-02 · arXiv:2408.01416 · Aaron Mueller, Jannik Brinkmann, Millicent Li, Samuel Marks, Koyena Pal et al.The Quest for the Right Mediator: Surveying MechanisticInterpretability Through the Lens of …arXiv · 2024 ↗ TracrBench: Generating Interpretability Testbeds with Large Language Models — ICML Mechanistic Interpretability Worksh · 2024-09-07 · arXiv:2409.13714 · Hannes Thurnherr, Jérémy ScheurerTracrBench: Generating Interpretability Testbeds with Large LanguageModelsICML Mechanistic Interpretabil · 2024 ↗ Using Degeneracy in the Loss Landscape for Mechanistic Interpretability — arXiv · 2024-05-17 · arXiv:2405.10927 · Lucius Bushnaq, Jake Mendel, Stefan Heimersheim, Dan Braun, Nicholas Goldowsky-Dill et al.Using Degeneracy in the Loss Landscape for MechanisticInterpretabilityarXiv · 2024 ↗ Activation space interpretability may be doomed — LessWrong/AF · 2025-01-08 · Bilal Chughtai, Lucius BushnaqActivation space interpretability may be doomedLessWrong/AF · 2025 ↗ From Memorization to Reasoning in the Spectrum of Loss Curvature — arXiv · 2025-10-28 · arXiv:2510.24256 · Jack Merullo, Srihita Vatsavaya, Lucius Bushnaq, Owen LewisFrom Memorization to Reasoning in the Spectrum of Loss CurvaturearXiv · 2025 ↗ Open Problems in Mechanistic Interpretability — arXiv · 2025-01-27 · arXiv:2501.16496 · Lee Sharkey, Bilal Chughtai, Joshua Batson, Jack Lindsey, Jeff Wu, Lucius Bushnaq et al.Open Problems in Mechanistic InterpretabilityarXiv · 2025 ↗ SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretability — ICML 2025 · 2025-03-12 · arXiv:2503.09532 · Adam Karvonen, Can Rager, Johnny Lin, Curt Tigges, Joseph Bloom, David Chanin et al.SAEBench: A Comprehensive Benchmark for Sparse Autoencoders inLanguage Model InterpretabilityICML 2025 · 2025 ↗ Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models — ICLR 2025 — International Conference on · 2024-03-28 · arXiv:2403.19647 · Samuel Marks, Can Rager, Eric J. Michaud, Yonatan Belinkov, David Bau, Aaron MuellerSparse Feature Circuits: Discovering and Editing InterpretableCausal Graphs in Language ModelsICLR 2025 — International Conf · 2025 ↗ Steering Out-of-Distribution Generalization with Concept Ablation Fine-Tuning — arXiv · 2025-07-22 · arXiv:2507.16795 · Helena Casademunt, Caden Juang, Adam Karvonen, Samuel Marks et al.Steering Out-of-Distribution Generalization with Concept AblationFine-TuningarXiv · 2025 ↗ Stochastic Parameter Decomposition — arXiv · 2025-06-25 · arXiv:2506.20790 · Lucius Bushnaq, Dan Braun, Lee SharkeyStochastic Parameter DecompositionarXiv · 2025 ↗ Interpreting Language Model Parameters — org site · 2026-05-05 · Lucius Bushnaq, Dan Braun, Oliver Clive-Griffin, Bart Bussmann, Nathan Hu et al.Interpreting Language Model Parametersorg site · 2026 ↗ Steering Language Models with Weight Arithmetic — ICLR 2026 · 2025-11-07 · arXiv:2511.05408 · Constanza Fierro, Fabien RogerSteering Language Models with Weight ArithmeticICLR 2026 · 2026 ↗ MECHANISMFeatures, superposition, SAEsRoots › Looking inside › InterpretabilityCircuits: features are directions in activation space, and weights wire featuresinto algorithms. Superposition: a model packs more features than it has neurons,so single neurons are polysemantic. Sparse autoencoders (SAEs) re-expressactivations in a wide, sparse dictionary where features come apart again.SO WHAT: a behaviour you can locate inside the model is a behaviour you canmonitor or switch off — outside the prompt.4 neurons, 7 featuressuperposition → polysemantic neuronsact.SAE: wide, sparse dictionary3 of 16 features activerecon.Agent payoff: read a feature (probe) → monitor; add/remove a direction (steer) → switch.KEY RESULT · zoomRefusal lives on one directionArditi et al. 2024: across 13 popular open-source chat models up to 72B parameters, refusal is mediated by a single direction in the residual stream. Erase itand the model stops refusing harmful requests; add it and it refuses harmless ones.arXiv:2406.11717 (abstract)
BENCH · Which interpretability move?
Each needs a different tool from this river.
Pick one — commit before you read the feedback.
 TODAY → Probes and steering vectors used as runtime monitors and switches Guardrails & Runtime ↗ WATCHING FROM OUTSIDE AI control 5 2022–23 6 2024–25 Adversarial Training for High-Stakes Reliability — arXiv · 2022-05-03 · arXiv:2205.01663 · Daniel M. Ziegler, Seraphina Nix, Lawrence Chan, Tim Bauman, Peter Schmidt-Nielsen et al.Adversarial Training for High-Stakes ReliabilityarXiv · 2022 ↗ AI Control: Improving Safety Despite Intentional Subversion — arXiv · 2023-12 · arXiv:2312.06942 · Ryan Greenblatt, Buck Shlegeris, Kshitij Sachan, Fabien RogerAI Control: Improving Safety Despite Intentional SubversionarXiv · 2023 ↗ Benchmarks for Detecting Measurement Tampering — arXiv · 2023-08 · arXiv:2308.15605 · Fabien Roger, Ryan Greenblatt, Max Nadeau, Buck Shlegeris, Nate ThomasBenchmarks for Detecting Measurement TamperingarXiv · 2023 ↗ Coup probes: Catching catastrophes with probes trained off-policy — LessWrong/AF · 2023-11-17 · Fabien RogerCoup probes: Catching catastrophes with probes trained off-policyLessWrong/AF · 2023 ↗ Preventing Language Models From Hiding Their Reasoning — arXiv · 2023-10 · arXiv:2310.18512 · Fabien Roger, Ryan GreenblattPreventing Language Models From Hiding Their ReasoningarXiv · 2023 ↗ Adaptive Deployment of Untrusted LLMs Reduces Distributed Threats — arXiv · 2024-11-26 · arXiv:2411.17693 · Jiaxin Wen, Vivek Hebbar, Caleb Larson, Aryan Bhatt, Ansh Radhakrishnan et al.Adaptive Deployment of Untrusted LLMs Reduces Distributed ThreatsarXiv · 2024 ↗ Games for AI Control: Models of Safety Evaluations of AI Deployment Protocols — arXiv · 2024-09 · arXiv:2409.07985 · Charlie Griffin, Louis Thomson, Buck Shlegeris, Alessandro AbateGames for AI Control: Models of Safety Evaluations of AI DeploymentProtocolsarXiv · 2024 ↗ Stress-Testing Capability Elicitation With Password-Locked Models — arXiv · 2024-05 · arXiv:2405.19550 · Ryan Greenblatt, Fabien Roger, Dmitrii Krasheninnikov, David KruegerStress-Testing Capability Elicitation With Password-Locked ModelsarXiv · 2024 ↗ A sketch of an AI control safety case — arXiv · 2025-01 · arXiv:2501.17315 · Tomek Korbak, Joshua Clymer, Benjamin Hilton, Buck Shlegeris, Geoffrey IrvingA sketch of an AI control safety casearXiv · 2025 ↗ Ctrl-Z: Controlling AI Agents via Resampling — arXiv · 2025-04 · arXiv:2504.10374 · Aryan Bhatt, Cody Rushing, Adam Kaufman, Tyler Tracy, Vasil Georgiev, David Matolcsi et al.Ctrl-Z: Controlling AI Agents via ResamplingarXiv · 2025 ↗ How to evaluate control measures for LLM agents? A trajectory from today to superintelligence — arXiv · 2025-04-07 · arXiv:2504.05259 · Tomek Korbak, Mikita Balesni, Buck Shlegeris, Geoffrey IrvingHow to evaluate control measures for LLM agents? A trajectory fromtoday to superintelligencearXiv · 2025 ↗ MECHANISMTrusted monitoringRoots › Watching from outside › AI controlAI control assumes the strong model might be trying to subvert you. A weakertrusted model scores every action for suspicion; the most suspicious go to asmall, expensive human audit budget; middling ones get rewritten by the trustedmodel. A red team plays the untrusted model to test the protocol.SO WHAT: you don't need to prove your agent is aligned to deploy it safely — youneed a protocol that stays safe if it isn't.Ustrong, untrustedT scoresweak, trustedH auditstop 2% suspiciousT editsmiddlingrun itlowillustrative splitFor agents: U = your agent, actions = tool calls, H = the human with 20 minutes a day.KEY RESULT · zoomThe original settingGreenblatt et al. 2023: untrusted GPT-4 writes code, trusted GPT-3.5 rates its suspiciousness, with limited access to high-quality trusted labour. Protocolstested: trusted editing, and untrusted monitoring with anti-collusion measures.arXiv:2312.06942 (abstract) TODAY → Untrusted-model protocols in agent harnesses: monitors, resampling, auditbudgets Defense Frameworks ↗ Evals, scheming & honesty 2 2020–21 10 2022–23 25 2024–25 3 2026 A very crude deception eval is already passed — LessWrong/AF · 2021-10-29 · Beth BarnesA very crude deception eval is already passedLessWrong/AF · 2021 ↗ More detailed proposal for measuring alignment of current models — LessWrong/AF · 2021-11-20 · Beth BarnesMore detailed proposal for measuring alignment of current modelsLessWrong/AF · 2021 ↗ Help ARC evaluate capabilities of current language models — LessWrong/AF · 2022-07-19 · Beth BarnesHelp ARC evaluate capabilities of current language modelsLessWrong/AF · 2022 ↗ The Dangers of Underclaiming: Reasons for Caution When Reporting How NLP Systems Fail — ACL 2022 · 2021-10-15 · arXiv:2110.08300 · Samuel R. BowmanThe Dangers of Underclaiming: Reasons for Caution When Reporting HowNLP Systems FailACL 2022 · 2022 ↗ Evaluating Language-Model Agents on Realistic Autonomous Tasks — arXiv · 2023-12-18 · arXiv:2312.11671 · Megan Kinniment, Lucas Jun Koba Sato, Haoxing Du, Brian Goodrich, Max Hasin et al.Evaluating Language-Model Agents on Realistic Autonomous TasksarXiv · 2023 ↗ Inverse Scaling: When Bigger Isn't Better — Transactions on Machine Learning Researc · 2023-06-15 · arXiv:2306.09479 · Ian R. McKenzie, Alexander Lyzhov, Michael Pieler, Alicia Parrish, Aaron Mueller et al.Inverse Scaling: When Bigger Isn't BetterTransactions on Machine Learni · 2023 ↗ Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting — arXiv · 2023-05-07 · arXiv:2305.04388 · Miles Turpin, Julian Michael, Ethan Perez, Samuel R. BowmanLanguage Models Don't Always Say What They Think: UnfaithfulExplanations in Chain-of-Thought…arXiv · 2023 ↗ Model Organisms of Misalignment: The Case for a New Pillar of Alignment Research — LessWrong/AF · 2023-08-08 · Evan Hubinger, Nicholas Schiefer, Carson Denison, Ethan PerezModel Organisms of Misalignment: The Case for a New Pillar ofAlignment ResearchLessWrong/AF · 2023 ↗ Model evaluation for extreme risks — arXiv · 2023-05-24 · arXiv:2305.15324 · Toby Shevlane; Sebastian Farquhar; Ben Garfinkel; Mary Phuong; Jess Whittlestone; Jad et al.Model evaluation for extreme risksarXiv · 2023 ↗ More information about the dangerous capability evaluations we did with GPT-4 and Claude — LessWrong/AF · 2023-03-19 · Beth BarnesMore information about the dangerous capability evaluations we didwith GPT-4 and ClaudeLessWrong/AF · 2023 ↗ Taken out of context: On measuring situational awareness in LLMs — arXiv · 2023-09-01 · arXiv:2309.00667 · Lukas Berglund, Asa Cooper Stickland, Mikita Balesni, Max Kaufmann, Meg Tong et al.Taken out of context: On measuring situational awareness in LLMsarXiv · 2023 ↗ What I would do if I wasn't at ARC Evals — LessWrong/AF · 2023-09-05 · Lawrence Chan (LawrenceC)What I would do if I wasn't at ARC EvalsLessWrong/AF · 2023 ↗ Analyzing Probabilistic Methods for Evaluating Agent Capabilities — arXiv / SoLaR 2024 · 2024-09-24 · arXiv:2409.16125 · Axel Højmark, Govind Pimpale, Arjun Panickssery, Marius Hobbhahn, Jérémy ScheurerAnalyzing Probabilistic Methods for Evaluating Agent CapabilitiesarXiv / SoLaR 2024 · 2024 ↗ Connecting the Dots: LLMs can Infer and Verbalize Latent Structure from Disparate Training Data — NeurIPS 2024 · 2024-06-20 · arXiv:2406.14546 · Johannes Treutlein, Dami Choi, Jan Betley, Samuel Marks, Cem Anil, Roger Grosse et al.Connecting the Dots: LLMs can Infer and Verbalize Latent Structurefrom Disparate Training DataNeurIPS 2024 · 2024 ↗ Do Unlearning Methods Remove Information from Language Model Weights? — arXiv · 2024-10-11 · arXiv:2410.08827 · Aghyad Deeb, Fabien RogerDo Unlearning Methods Remove Information from Language ModelWeights?arXiv · 2024 ↗ Honesty to Subterfuge: In-Context Reinforcement Learning Can Make Honest Models Reward Hack — arXiv · 2024-10-09 · arXiv:2410.06491 · Leo McKee-Reid, Christoph Sträter, Maria Angelica Martinez, Joe Needham et al.Honesty to Subterfuge: In-Context Reinforcement Learning Can MakeHonest Models Reward HackarXiv · 2024 ↗ LLM Evaluators Recognize and Favor Their Own Generations — NeurIPS 2024 · 2024-04-15 · arXiv:2404.13076 · Arjun Panickssery, Samuel R. Bowman, Shi FengLLM Evaluators Recognize and Favor Their Own GenerationsNeurIPS 2024 · 2024 ↗ Lessons from Studying Two-Hop Latent Reasoning — arXiv · 2024-11-25 · arXiv:2411.16353 · Mikita Balesni, Tomek Korbak, Owain EvansLessons from Studying Two-Hop Latent ReasoningarXiv · 2024 ↗ Looking Inward: Language Models Can Learn About Themselves by Introspection — arXiv · 2024-10-17 · arXiv:2410.13787 · Felix J. Binder, James Chua, Tomek Korbak, Henry Sleight, John Hughes, Robert Long et al.Looking Inward: Language Models Can Learn About Themselves byIntrospectionarXiv · 2024 ↗ METR Task Standard — org site · 2024-07-31 · METR (task-standard@metr.org)METR Task Standardorg site · 2024 ↗ Me, Myself, and AI: The Situational Awareness Dataset (SAD) for LLMs — NeurIPS 2024 · 2024-07-05 · arXiv:2407.04694 · Rudolf Laine, Bilal Chughtai, Jan Betley, Kaivalya Hariharan, Jérémy Scheurer et al.Me, Myself, and AI: The Situational Awareness Dataset (SAD) for LLMsNeurIPS 2024 · 2024 ↗ The Checklist: What Succeeding at AI Safety Will Involve — Personal blog · 2024-09-03 · Samuel R. BowmanThe Checklist: What Succeeding at AI Safety Will InvolvePersonal blog · 2024 ↗ The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A" — ICLR 2024 · 2023-09-21 · arXiv:2309.12288 · Lukas Berglund, Meg Tong, Max Kaufmann, Mikita Balesni, Asa Cooper Stickland et al.The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"ICLR 2024 · 2024 ↗ Uncovering Deceptive Tendencies in Language Models: A Simulated Company AI Assistant — arXiv · 2024-04-25 · arXiv:2405.01576 · Olli Järviniemi, Evan HubingerUncovering Deceptive Tendencies in Language Models: A SimulatedCompany AI AssistantarXiv · 2024 ↗ All Code, No Thought: Current Language Models Struggle to Reason in Ciphered Language — arXiv · 2025-10-10 · arXiv:2510.09714 · Shiyuan Guo, Henry Sleight, Fabien RogerAll Code, No Thought: Current Language Models Struggle to Reason inCiphered LanguagearXiv · 2025 ↗ Eliciting Secret Knowledge from Language Models — arXiv · 2025-10-01 · arXiv:2510.01070 · Bartosz Cywiński, Emil Ryd, Rowan Wang, Senthooran Rajamanoharan, Neel Nanda et al.Eliciting Secret Knowledge from Language ModelsarXiv · 2025 ↗ Inoculation Prompting: Instructing LLMs to misbehave at train-time improves test-time alignment — arXiv · 2025-10-06 · arXiv:2510.05024 · Nevan Wichers, Aram Ebtekar, Ariana Azarbal, Victor Gillioz, Christine Ye, Emil Ryd et al.Inoculation Prompting: Instructing LLMs to misbehave at train-timeimproves test-time alignmentarXiv · 2025 ↗ Language Models Learn to Mislead Humans via RLHF — ICLR 2025 · 2024-09-19 · arXiv:2409.12822 · Jiaxin Wen, Ruiqi Zhong, Akbir Khan, Ethan Perez, Jacob Steinhardt, Minlie Huang et al.Language Models Learn to Mislead Humans via RLHFICLR 2025 · 2025 ↗ Liars' Bench: Evaluating Lie Detectors for Language Models — arXiv · 2025-11-20 · arXiv:2511.16035 · Kieron Kretschmar, Walter Laurito, Sharan Maiya, Samuel MarksLiars' Bench: Evaluating Lie Detectors for Language ModelsarXiv · 2025 ↗ Noise Injection Reveals Hidden Capabilities of Sandbagging Language Models — NeurIPS 2025 · 2024-12-02 · arXiv:2412.01784 · Cameron Tice, Philipp Alexander Kreer, Nathan Helm-Burger, Prithviraj Singh Shahani et al.Noise Injection Reveals Hidden Capabilities of Sandbagging LanguageModelsNeurIPS 2025 · 2025 ↗ Open Character Training: Shaping the Persona of AI Assistants through Constitutional AI — arXiv · 2025-11-03 · arXiv:2511.01689 · Sharan Maiya, Henning Bartsch, Nathan Lambert, Evan HubingerOpen Character Training: Shaping the Persona of AI Assistantsthrough Constitutional AIarXiv · 2025 ↗ Putting up Bumpers — Anthropic Alignment Science Blog · 2025-04-23 · Samuel R. BowmanPutting up BumpersAnthropic Alignment Science Bl · 2025 ↗ SafeArena: Evaluating the Safety of Autonomous Web Agents — arXiv · 2025-03 · arXiv:2503.04957 · Ada Defne Tur, Nicholas Meade, Xing Han Lù, Alejandra Zambrano, Arkil Patel et al.SafeArena: Evaluating the Safety of Autonomous Web AgentsarXiv · 2025 ↗ Stress-Testing Model Specs Reveals Character Differences among Language Models — arXiv · 2025-10 · arXiv:2510.07686 · Jifan Zhang, Henry Sleight, Andi Peng, John Schulman, Esin DurmusStress-Testing Model Specs Reveals Character Differences amongLanguage ModelsarXiv · 2025 ↗ Subliminal Learning: Language models transmit behavioral traits via hidden signals in data — arXiv · 2025-07-20 · arXiv:2507.14805 · Alex Cloud, Minh Le, James Chua, Jan Betley, Anna Sztyber-Betley, Jacob Hilton et al.Subliminal Learning: Language models transmit behavioral traits viahidden signals in dataarXiv · 2025 ↗ Toward an Evaluation Science for Generative AI Systems — arXiv · 2025-03 · arXiv:2503.05336 · Laura Weidinger, Inioluwa Deborah Raji, Hanna Wallach, Margaret Mitchell et al.Toward an Evaluation Science for Generative AI SystemsarXiv · 2025 ↗ Will AI Tell Lies to Save Sick Children? Litmus-Testing AI Values Prioritization with AIRiskDilemmas — arXiv · 2025-05-20 · arXiv:2505.14633 · Yu Ying Chiu, Zhilin Wang, Sharan Maiya, Yejin Choi, Kyle Fish, Sydney Levine et al.Will AI Tell Lies to Save Sick Children? Litmus-Testing AI ValuesPrioritization with AIRiskD…arXiv · 2025 ↗ Chain-of-Thought Reasoning In The Wild Is Not Always Faithful — arXiv / ICML 2026 · 2025-03 · arXiv:2503.08679 · Iván Arcuschin, Jett Janiak, Robert Krzyzanowski, Senthooran Rajamanoharan et al.Chain-of-Thought Reasoning In The Wild Is Not Always FaithfularXiv / ICML 2026 · 2026 ↗ Log analysis is necessary for credible evaluation of AI agents — arXiv · 2026-05-08 · arXiv:2605.08545 · Peter Kirgis, Sayash Kapoor, Stephan Rabanser, Nitya Nadgir, Cozmin Ududec et al.Log analysis is necessary for credible evaluation of AI agentsarXiv · 2026 ↗ MirrorCode: AI can rebuild entire programs from behavior alone — arXiv · 2026-06-29 · arXiv:2606.30182 · Tom Adamczewski, David Owen, David Rein, Florian Brand, Giles Edkins, Allen Hart et al.MirrorCode: AI can rebuild entire programs from behavior alonearXiv · 2026 ↗ MECHANISMWhat an eval actually claimsRoots › Watching from outside › EvalsA capability eval runs tasks through an agent scaffold and scores the result. Theclaim is 'at least this capable'. Weak scaffolding under-elicits (the elicitationgap). A model that recognises it is being tested can sandbag. Honesty work askswhether what the model says matches what it computes.SO WHAT: an eval result is a lower bound on what the agent can do — and only ifthe agent isn't playing to the test.Taskssuite + scorerScaffoldtools, retries, promptsModel“is this a test?” → sandbaggingwhat the eval measuredwhat better elicitation findselicitation gapKEY RESULT · zoomWhere agent evals started (2023)Kinniment et al. (METR/ARC Evals) built four simple agents and tested them on 12 autonomous-replication tasks: “these language model agents can only complete theeasiest tasks from this list.” The same lineage now tracks how fast that changes.arXiv:2312.11671 (abstract)
BENCH · Design your control protocol
Your coding agent, one reviewer, 20 minutes a day.
Pick one — commit before you read the feedback.
 TODAY → Agent benchmarks, red-team evals, time-horizon tracking Benchmarks & Evaluation ↗ TRAINING & THE CURRENT Learning from humans & robustness 4 2016–17 6 2018–19 2 2020–21 10 2022–23 1 2024–25 Concrete Problems in AI Safety — arXiv · 2016-06-21 · arXiv:1606.06565 · Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, Dan ManéConcrete Problems in AI SafetyarXiv · 2016 ↗ Technical Report on the CleverHans v2.1.0 Adversarial Examples Library — arXiv · 2016-10-03 · arXiv:1610.00768 · Nicolas Papernot; Fartash Faghri; Nicholas Carlini; Ian Goodfellow; Reuben Feinman; A et al.Technical Report on the CleverHans v2.1.0 Adversarial ExamplesLibraryarXiv · 2016 ↗ Adversarial Patch — arXiv / NIPS 2017 workshop · 2017-12-27 · arXiv:1712.09665 · Tom B. Brown; Dandelion Mané; Aurko Roy; Martín Abadi; Justin GilmerAdversarial PatcharXiv / NIPS 2017 workshop · 2017 ↗ Deep reinforcement learning from human preferences — arXiv / NIPS 2017 · 2017-06-12 · arXiv:1706.03741 · Paul F Christiano; Jan Leike; Tom B Brown; Miljan Martic; Shane Legg; Dario AmodeiDeep reinforcement learning from human preferencesarXiv / NIPS 2017 · 2017 ↗ Unrestricted Adversarial Examples — arXiv · 2018-09-22 · arXiv:1809.08352 · Tom B. Brown, Nicholas Carlini, Chiyuan Zhang, Catherine Olsson, Paul Christiano et al.Unrestricted Adversarial ExamplesarXiv · 2018 ↗ Fine-Tuning Language Models from Human Preferences — arXiv · 2019-09-18 · arXiv:1909.08593 · Daniel M. Ziegler; Nisan Stiennon; Jeffrey Wu; Tom B. Brown; Alec Radford; Dario Amod et al.Fine-Tuning Language Models from Human PreferencesarXiv · 2019 ↗ TensorFuzz: Debugging Neural Networks with Coverage-Guided Fuzzing — ICML 2019 · 2019 · Augustus Odena, Catherine Olsson, David G. Andersen, Ian GoodfellowTensorFuzz: Debugging Neural Networks with Coverage-Guided FuzzingICML 2019 · 2019 ↗ Testing Robustness Against Unforeseen Adversaries — arXiv · 2019-08-21 · arXiv:1908.08016 · Max Kaufmann; Daniel Kang; Yi Sun; Steven Basart; Xuwang Yin; Mantas Mazeika; Akul Ar et al.Testing Robustness Against Unforeseen AdversariesarXiv · 2019 ↗ The Assistive Multi-Armed Bandit — HRI 2019 · 2019-01-24 · arXiv:1901.08654 · Lawrence Chan, Dylan Hadfield-Menell, Siddhartha Srinivasa, Anca DraganThe Assistive Multi-Armed BanditHRI 2019 · 2019 ↗ Transfer of Adversarial Robustness Between Perturbation Types — arXiv · 2019-05-03 · arXiv:1905.01034 · Daniel Kang; Yi Sun; Tom Brown; Dan Hendrycks; Jacob SteinhardtTransfer of Adversarial Robustness Between Perturbation TypesarXiv · 2019 ↗ Extracting Training Data from Large Language Models — arXiv / USENIX Security 2021 · 2020-12-14 · arXiv:2012.07805 · Nicholas Carlini; Florian Tramer; Eric Wallace; Matthew Jagielski; Ariel Herbert-Voss et al.Extracting Training Data from Large Language ModelsarXiv / USENIX Security 2021 · 2021 ↗ Human irrationality: both bad and good for reward inference — arXiv preprint · 2021-11-12 · arXiv:2111.06956 · Lawrence Chan, Andrew Critch, Anca DraganHuman irrationality: both bad and good for reward inferencearXiv preprint · 2021 ↗ RL with KL penalties is better viewed as Bayesian inference — arXiv · 2022-05-23 · arXiv:2205.11275 · Tomasz Korbak, Ethan Perez, Christopher L BuckleyRL with KL penalties is better viewed as Bayesian inferencearXiv · 2022 ↗ Red Teaming Language Models with Language Models — arXiv; published at EMNLP 2022 · 2022-02-07 · arXiv:2202.03286 · Ethan Perez; Saffron Huang; Francis Song; Trevor Cai; Roman Ring; John Aslanides; Ame et al.Red Teaming Language Models with Language ModelsarXiv · 2022 ↗ Training Language Models with Language Feedback — arXiv / First Workshop on Learning with · 2022-04-29 · arXiv:2204.14146 · Jérémy Scheurer, Jon Ander Campos, Jun Shern Chan, Angelica Chen, Kyunghyun Cho et al.Training Language Models with Language FeedbackarXiv / First Workshop on Lear · 2022 ↗ Training language models to follow instructions with human feedback — arXiv · 2022-03-04 · arXiv:2203.02155 · Long Ouyang; Jeff Wu; Xu Jiang; Diogo Almeida; Carroll L. Wainwright; Pamela Mishkin; et al.Training language models to follow instructions with human feedbackarXiv · 2022 ↗ Eliciting Human Preferences with Language Models — arXiv · 2023-10 · arXiv:2310.11589 · Belinda Z. Li, Alex Tamkin, Noah Goodman, Jacob AndreasEliciting Human Preferences with Language ModelsarXiv · 2023 ↗ Improving Code Generation by Training with Natural Language Feedback — arXiv v1 28 Mar 2023 · 2023-03-28 · arXiv:2303.16749 · Angelica Chen, Jérémy Scheurer, Tomasz Korbak, Jon Ander Campos, Jun Shern Chan et al.Improving Code Generation by Training with Natural Language FeedbackarXiv v1 28 Mar 2023 · 2023 ↗ Large Language Models Sometimes Generate Purely Negatively-Reinforced Text — arXiv · 2023-06-13 · arXiv:2306.07567 · Fabien RogerLarge Language Models Sometimes Generate PurelyNegatively-Reinforced TextarXiv · 2023 ↗ Pretraining Language Models with Human Preferences — arXiv · 2023-02-16 · arXiv:2302.08582 · Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Bhalerao, Christopher L. Buckley et al.Pretraining Language Models with Human PreferencesarXiv · 2023 ↗ Social Contract AI: Aligning AI Assistants with Implicit Group Norms — arXiv / NeurIPS 2023 SoLaR Workshop · 2023-10 · arXiv:2310.17769 · Jan-Philipp Fränken, Sam Kwok, Peixuan Ye, Kanishk Gandhi, Dilip Arumugam et al.Social Contract AI: Aligning AI Assistants with Implicit Group NormsarXiv / NeurIPS 2023 SoLaR Wor · 2023 ↗ Training Language Models with Language Feedback at Scale — Published in TMLR · 2023-03-28 · arXiv:2303.16755 · Jérémy Scheurer, Jon Ander Campos, Tomasz Korbak, Jun Shern Chan, Angelica Chen et al.Training Language Models with Language Feedback at ScalePublished in TMLR · 2023 ↗ Bayesian Preference Elicitation with Language Models — arXiv · 2024-03 · arXiv:2403.05534 · Kunal Handa, Yarin Gal, Ellie Pavlick, Noah Goodman, Jacob Andreas, Alex Tamkin et al.Bayesian Preference Elicitation with Language ModelsarXiv · 2024 ↗ MECHANISMThe RLHF loopRoots › Training › Learning from humansShow people two outputs, ask which is better, train a reward model on thosecomparisons, then optimise the policy against it with a KL leash back to thepretrained model. It made models useful and shaped refusals; it also rewardswhatever humans can't tell apart from good — including misleading them.SO WHAT: the refusals your agent has were trained in by comparisons — so are itssycophancy and its skill at sounding right.Policy2 outputsHuman picksReward modelRL updateKL leashto pretrained model↘ reward hacking↘ sycophancyKEY RESULT · zoomSmall + aligned beat bigOuyang et al. 2022 (InstructGPT): on their prompt distribution, outputs from the 1.3B-parameter InstructGPT model were preferred to outputs from the 175B GPT-3 —over 100× fewer parameters.arXiv:2203.02155 (abstract) TODAY → Refusal training and adversarial robustness → prompt-injection robustness Prompt Injection ↗ Scaling & capabilities 1 2014–15 6 2018–19 15 2020–21 8 2022–23 1 2024–25 A large annotated corpus for learning natural language inference — EMNLP 2015 · 2015-08-21 · arXiv:1508.05326 · Samuel R. Bowman, Gabor Angeli, Christopher Potts, Christopher D. ManningA large annotated corpus for learning natural language inferenceEMNLP 2015 · 2015 ↗ An Empirical Model of Large-Batch Training — arXiv · 2018-12-14 · arXiv:1812.06162 · Sam McCandlish; Jared Kaplan; Dario Amodei; OpenAI Dota TeamAn Empirical Model of Large-Batch TrainingarXiv · 2018 ↗ Is Generator Conditioning Causally Related to GAN Performance? — arXiv / ICML 2018 · 2018-02-23 · arXiv:1802.08768 · Augustus Odena; Jacob Buckman; Catherine Olsson; Tom B. Brown; Christopher Olah; Coli et al.Is Generator Conditioning Causally Related to GAN Performance?arXiv / ICML 2018 · 2018 ↗ Skill Rating for Generative Models — arXiv · 2018-08-14 · arXiv:1808.04888 · Catherine Olsson; Surya Bhupatiraju; Tom Brown; Augustus Odena; Ian GoodfellowSkill Rating for Generative ModelsarXiv · 2018 ↗ Dota 2 with Large Scale Deep Reinforcement Learning — arXiv · 2019-12 · arXiv:1912.06680 · Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung et al.Dota 2 with Large Scale Deep Reinforcement LearningarXiv · 2019 ↗ GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding — ICLR 2019 · 2018-04-20 · arXiv:1804.07461 · Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, Samuel R. BowmanGLUE: A Multi-Task Benchmark and Analysis Platform for NaturalLanguage UnderstandingICLR 2019 · 2019 ↗ SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems — NeurIPS 2019 · 2019-05-02 · arXiv:1905.00537 · Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael et al.SuperGLUE: A Stickier Benchmark for General-Purpose LanguageUnderstanding SystemsNeurIPS 2019 · 2019 ↗ A Neural Scaling Law from the Dimension of the Data Manifold — arXiv preprint · 2020-04-22 · arXiv:2004.10802 · Utkarsh Sharma; Jared KaplanA Neural Scaling Law from the Dimension of the Data ManifoldarXiv preprint · 2020 ↗ FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization — arXiv / ACL 2020 — 'Accepted to ACL 2020 · 2020-05 · arXiv:2005.03754 · Esin Durmus, He He, Mona DiabFEQA: A Question Answering Evaluation Framework for FaithfulnessAssessment in Abstractive Su…arXiv / ACL 2020 — 'Accepted t · 2020 ↗ Investigating Transferability in Pretrained Language Models — arXiv / Findings of EMNLP 2020 · 2020-04 · arXiv:2004.14975 · Alex Tamkin, Trisha Singh, Davide Giovanardi, Noah GoodmanInvestigating Transferability in Pretrained Language ModelsarXiv / Findings of EMNLP 2020 · 2020 ↗ Language Models are Few-Shot Learners — arXiv / NeurIPS 2020 · 2020-05-28 · arXiv:2005.14165 · Tom B. Brown; Benjamin Mann; Nick Ryder; Melanie Subbiah; Jared Kaplan; Prafulla Dhar et al.Language Models are Few-Shot LearnersarXiv / NeurIPS 2020 · 2020 ↗ Measuring the Algorithmic Efficiency of Neural Networks — arXiv · 2020-05-08 · arXiv:2005.04305 · Danny Hernandez; Tom B. BrownMeasuring the Algorithmic Efficiency of Neural NetworksarXiv · 2020 ↗ Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — arXiv · 2020-05-22 · arXiv:2005.11401 · Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin et al.Retrieval-Augmented Generation for Knowledge-Intensive NLP TasksarXiv · 2020 ↗ Scaling Laws for Autoregressive Generative Modeling — arXiv · 2020-10-28 · arXiv:2010.14701 · Tom Henighan; Jared Kaplan; Mor Katz; Mark Chen; Christopher Hesse; Jacob Jackson; He et al.Scaling Laws for Autoregressive Generative ModelingarXiv · 2020 ↗ Scaling Laws for Neural Language Models — arXiv · 2020-01-23 · arXiv:2001.08361 · Jared Kaplan; Sam McCandlish; Tom Henighan; Tom B. Brown; Benjamin Chess; Rewon Child et al.Scaling Laws for Neural Language ModelsarXiv · 2020 ↗ Evaluating CLIP: Towards Characterization of Broader Capabilities and Downstream Implications — arXiv · 2021-08-05 · arXiv:2108.02818 · Sandhini Agarwal; Gretchen Krueger; Jack Clark; Alec Radford; Jong Wook Kim; Miles Br et al.Evaluating CLIP: Towards Characterization of Broader Capabilitiesand Downstream ImplicationsarXiv · 2021 ↗ Evaluating Large Language Models Trained on Code — arXiv · 2021-07-07 · arXiv:2107.03374 · Mark Chen; Jerry Tworek; Heewoo Jun; Qiming Yuan; Henrique Ponde de Oliveira Pinto; J et al.Evaluating Large Language Models Trained on CodearXiv · 2021 ↗ Explaining Neural Scaling Laws — arXiv · 2021-02-12 · arXiv:2102.06701 · Yasaman Bahri; Ethan Dyer; Jared Kaplan; Jaehoon Lee; Utkarsh SharmaExplaining Neural Scaling LawsarXiv · 2021 ↗ Learning Transferable Visual Models From Natural Language Supervision — arXiv / ICML 2021 · 2021-02-26 · arXiv:2103.00020 · Alec Radford; Jong Wook Kim; Chris Hallacy; Aditya Ramesh; Gabriel Goh; Sandhini Agar et al.Learning Transferable Visual Models From Natural LanguageSupervisionarXiv / ICML 2021 · 2021 ↗ Scaling Laws for Transfer — arXiv · 2021-02-02 · arXiv:2102.01293 · Danny Hernandez; Jared Kaplan; Tom Henighan; Sam McCandlishScaling Laws for TransferarXiv · 2021 ↗ The AI Index 2021 Annual Report — arXiv / Stanford HAI · 2021-03-09 · arXiv:2103.06312 · Daniel Zhang, Saurabh Mishra, Erik Brynjolfsson, John Etchemendy, Deep Ganguli et al.The AI Index 2021 Annual ReportarXiv / Stanford HAI · 2021 ↗ True Few-Shot Learning with Language Models — arXiv · 2021-05-24 · arXiv:2105.11447 · Ethan Perez, Douwe Kiela, Kyunghyun ChoTrue Few-Shot Learning with Language ModelsarXiv · 2021 ↗ Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models — arXiv / TMLR · 2022-06-09 · arXiv:2206.04615 · Aarohi Srivastava; Abhinav Rastogi; Abhishek Rao; Abu Awal Md Shoeb; Abubakar Abid; A et al.Beyond the Imitation Game: Quantifying and extrapolating thecapabilities of language modelsarXiv / TMLR · 2022 ↗ Compute Trends Across Three Eras of Machine Learning — arXiv / IJCNN 2022 — journal_ref confirm · 2022-02-11 · arXiv:2202.05924 · Jaime Sevilla, Lennart Heim, Anson Ho, Tamay Besiroglu, Marius Hobbhahn et al.Compute Trends Across Three Eras of Machine LearningarXiv / IJCNN 2022 — journal_r · 2022 ↗ Holistic Evaluation of Language Models — arXiv / Transactions on Machine Learning · 2022-11 · arXiv:2211.09110 · Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu et al.Holistic Evaluation of Language ModelsarXiv / Transactions on Machin · 2022 ↗ Language models are better than humans at next-token prediction — arXiv · 2022-12-21 · arXiv:2212.11281 · Buck Shlegeris, Fabien Roger, Lawrence Chan, Euan McLeanLanguage models are better than humans at next-token predictionarXiv · 2022 ↗ Machine Learning Model Sizes and the Parameter Gap — arXiv · 2022-07-05 · arXiv:2207.02852 · Pablo Villalobos, Jaime Sevilla, Tamay Besiroglu, Lennart Heim, Anson Ho et al.Machine Learning Model Sizes and the Parameter GaparXiv · 2022 ↗ The AI Index 2022 Annual Report — arXiv / Stanford HAI · 2022-05 · arXiv:2205.03468 · Daniel Zhang, Nestor Maslej, Erik Brynjolfsson, John Etchemendy, Terah Lyons et al.The AI Index 2022 Annual ReportarXiv / Stanford HAI · 2022 ↗ Eight Things to Know about Large Language Models — arXiv · 2023-04-02 · arXiv:2304.00612 · Samuel R. BowmanEight Things to Know about Large Language ModelsarXiv · 2023 ↗ Few-shot Adaptation Works with UnpredicTable Data — Published at ACL 2023 · 2022-08-01 · arXiv:2208.01009 · Jun Shern Chan, Michael Pieler, Jonathan Jao, Jérémy Scheurer, Ethan PerezFew-shot Adaptation Works with UnpredicTable DataPublished at ACL 2023 · 2023 ↗ Will we run out of data? Limits of LLM scaling based on human-generated data — arXiv / ICML 2024 · 2022-10-26 · arXiv:2211.04325 · Pablo Villalobos, Anson Ho, Jaime Sevilla, Tamay Besiroglu, Lennart Heim et al.Will we run out of data? Limits of LLM scaling based onhuman-generated dataarXiv / ICML 2024 · 2024 ↗ MECHANISMPower lawsRoots › Training › ScalingLanguage-model loss falls as a straight line on log-log axes as you addparameters, data and compute. That predictability is why labs can plan the nextmodel — and why safety work that takes years is chasing a curve that doesn'twait. The open questions in this river: will the data run out, and do benchmarkstrack what matters?SO WHAT: capability arrives on a schedule, which is why every safety river isracing it.loss (log)compute (log)KEY RESULT · zoomSeven orders of magnitudeKaplan et al. 2020: “The loss scales as a power-law with model size, dataset size, and the amount of compute used for training, with sometrends spanning more than seven orders of magnitude.”arXiv:2001.08361 (abstract)
BENCH · Where in the loop did it go wrong?
Your fine-tuned support agent has a problem. Which part of the RLHF loop?
Pick one — commit before you read the feedback.
 TODAY → Why it is urgent: capability arrives on a schedule Fit to Deploy atlas ↗ SOCIETY Society, people & governance 1 2014–15 6 2018–19 8 2020–21 11 2022–23 5 2024–25 1 2026 Security Impact Ratings Considered Harmful — arXiv / HotOS XII · 2009-04 · arXiv:0904.4058 · Jeff Arnold, Tim Abbott, Waseem Daher, Gregory Price, Nelson Elhage, Geoffrey Thomas et al.Security Impact Ratings Considered HarmfularXiv / HotOS XII · 2009 ↗ Exploring the Role of Prior Beliefs for Argument Persuasion — arXiv / NAACL-HLT 2018 · 2019-06 · arXiv:1906.11301 · Esin Durmus, Claire CardieExploring the Role of Prior Beliefs for Argument PersuasionarXiv / NAACL-HLT 2018 · 2018 ↗ The Malicious Use of Artificial Intelligence: Forecasting, Prevention, and Mitigation — arXiv · 2018-02 · arXiv:1802.07228 · Miles Brundage, Shahar Avin, Jack Clark, Helen Toner, Peter Eckersley, Ben Garfinkel et al.The Malicious Use of Artificial Intelligence: Forecasting,Prevention, and MitigationarXiv · 2018 ↗ Regulatory Markets for AI Safety — arXiv · 2019-12-11 · arXiv:2001.00078 · Jack Clark; Gillian K. HadfieldRegulatory Markets for AI SafetyarXiv · 2019 ↗ Release Strategies and the Social Impacts of Language Models — arXiv · 2019-08-24 · arXiv:1908.09203 · Irene Solaiman; Miles Brundage; Jack Clark; Amanda Askell; Ariel Herbert-Voss; Jeff W et al.Release Strategies and the Social Impacts of Language ModelsarXiv · 2019 ↗ The Role of Cooperation in Responsible AI Development — arXiv · 2019-07-10 · arXiv:1907.04534 · Amanda Askell; Miles Brundage; Gillian HadfieldThe Role of Cooperation in Responsible AI DevelopmentarXiv · 2019 ↗ The Role of Pragmatic and Discourse Context in Determining Argument Impact — arXiv / EMNLP 2019 · 2020-04 · arXiv:2004.03034 · Esin Durmus, Faisal Ladhak, Claire CardieThe Role of Pragmatic and Discourse Context in Determining ArgumentImpactarXiv / EMNLP 2019 · 2019 ↗ Begin with the human: Designing for safety and trustworthiness in cyber-physical systems — Chapter in "Human-Machine Shared Context · 2020-01-01 · Elizabeth T. Williams; Ehsan Nabavi; Genevieve Bell; Caitlin Bentley; Katherine A. Da et al.Begin with the human: Designing for safety and trustworthiness incyber-physical systemsChapter in "Human-Machine Shar · 2020 ↗ Measurement in AI Policy: Opportunities and Challenges — arXiv · 2020-09 · arXiv:2009.09071 · Saurabh Mishra, Jack Clark, C. Raymond PerraultMeasurement in AI Policy: Opportunities and ChallengesarXiv · 2020 ↗ Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims — arXiv · 2020-04-15 · arXiv:2004.07213 · Miles Brundage; Shahar Avin; Jasmine Wang; Haydn Belfield; Gretchen Krueger; Gillian et al.Toward Trustworthy AI Development: Mechanisms for SupportingVerifiable ClaimsarXiv · 2020 ↗ On the Opportunities and Risks of Foundation Models — arXiv / Stanford CRFM · 2021-08 · arXiv:2108.07258 · Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora et al.On the Opportunities and Risks of Foundation ModelsarXiv / Stanford CRFM · 2021 ↗ Risks from AI persuasion — LessWrong/AF · 2021-12-24 · Beth BarnesRisks from AI persuasionLessWrong/AF · 2021 ↗ Towards Understanding Persuasion in Computational Argumentation (PhD Dissertation) — arXiv · 2021-10 · arXiv:2110.01078 · Esin DurmusTowards Understanding Persuasion in Computational Argumentation (PhDDissertation)arXiv · 2021 ↗ Understanding the Capabilities, Limitations, and Societal Impact of Large Language Models — arXiv / Stanford HAI · 2021-02-04 · arXiv:2102.02503 · Alex Tamkin, Miles Brundage, Jack Clark, Deep GanguliUnderstanding the Capabilities, Limitations, and Societal Impact ofLarge Language ModelsarXiv / Stanford HAI · 2021 ↗ Why and How Governments Should Monitor AI Development — arXiv · 2021-08-28 · arXiv:2108.12427 · Jess Whittlestone; Jack ClarkWhy and How Governments Should Monitor AI DevelopmentarXiv · 2021 ↗ BBQ: A Hand-Built Bias Benchmark for Question Answering — ACL 2022 Findings · 2021-10-15 · arXiv:2110.08193 · Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang et al.BBQ: A Hand-Built Bias Benchmark for Question AnsweringACL 2022 Findings · 2022 ↗ Evaluating Human-Language Model Interaction — arXiv / TMLR · 2022-12 · arXiv:2212.09746 · Mina Lee, Megha Srivastava, Amelia Hardy, John Thickstun, Esin Durmus et al.Evaluating Human-Language Model InteractionarXiv / TMLR · 2022 ↗ Benchmarking Large Language Models for News Summarization — arXiv / TACL · 2023-01 · arXiv:2301.13848 · Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown et al.Benchmarking Large Language Models for News SummarizationarXiv / TACL · 2023 ↗ Easily Accessible Text-to-Image Generation Amplifies Demographic Stereotypes at Large Scale — arXiv / FAccT 2023 · 2022-11 · arXiv:2211.03759 · Federico Bianchi, Pratyusha Kalluri, Esin Durmus, Faisal Ladhak, Myra Cheng et al.Easily Accessible Text-to-Image Generation Amplifies DemographicStereotypes at Large ScalearXiv / FAccT 2023 · 2023 ↗ Marked Personas: Using Natural Language Prompts to Measure Stereotypes in Language Models — arXiv / ACL 2023 · 2023-05 · arXiv:2305.18189 · Myra Cheng, Esin Durmus, Dan JurafskyMarked Personas: Using Natural Language Prompts to MeasureStereotypes in Language ModelsarXiv / ACL 2023 · 2023 ↗ Operationalising the Definition of General Purpose AI Systems: Assessing Four Approaches — arXiv · 2023-06 · arXiv:2306.02889 · Risto Uuk, Carlos Ignacio Gutierrez, Alex TamkinOperationalising the Definition of General Purpose AI Systems:Assessing Four ApproachesarXiv · 2023 ↗ Opportunities and Risks of LLMs for Scalable Deliberation with Polis — arXiv · 2023-06 · arXiv:2306.11932 · Christopher T. Small, Ivan Vendrov, Esin Durmus, Hadjar Homaei, Elizabeth Barry et al.Opportunities and Risks of LLMs for Scalable Deliberation with PolisarXiv · 2023 ↗ Regulatory Markets: The Future of AI Governance — arXiv · 2023-04-11 · arXiv:2304.04914 · Gillian K. Hadfield; Jack ClarkRegulatory Markets: The Future of AI GovernancearXiv · 2023 ↗ Report of the 1st Workshop on Generative AI and Law — arXiv · 2023-11 · arXiv:2311.06477 · A. Feder Cooper, Katherine Lee, James Grimmelmann, Daphne Ippolito et al.Report of the 1st Workshop on Generative AI and LawarXiv · 2023 ↗ Task Ambiguity in Humans and Language Models — arXiv / ICLR 2023 · 2022-12 · arXiv:2212.10711 · Alex Tamkin, Kunal Handa, Avash Shrestha, Noah GoodmanTask Ambiguity in Humans and Language ModelsarXiv / ICLR 2023 · 2023 ↗ Whose Opinions Do Language Models Reflect? — arXiv / ICML 2023 · 2023-03 · arXiv:2303.17548 · Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang et al.Whose Opinions Do Language Models Reflect?arXiv / ICML 2023 · 2023 ↗ How will advanced AI systems impact democracy? — arXiv · 2024-08 (submitted 2024-08-27; announced under a 2409 arXiv ID, so many indexes list 2024-09) · arXiv:2409.06729 · Christopher Summerfield, Lisa Argyle, Michiel Bakker, Teddy Collins, Esin Durmus et al.How will advanced AI systems impact democracy?arXiv · 2024 ↗ NLP Systems That Can't Tell Use from Mention Censor Counterspeech, but Teaching the Distinction Helps — arXiv / NAACL 2024 · 2024-04 · arXiv:2404.01651 · Kristina Gligoric, Myra Cheng, Lucia Zheng, Esin Durmus, Dan JurafskyNLP Systems That Can't Tell Use from Mention Censor Counterspeech,but Teaching the Distincti…arXiv / NAACL 2024 · 2024 ↗ AI Behind Closed Doors: a Primer on The Governance of Internal Deployment — arXiv · 2025-04-16 · arXiv:2504.12170 · Charlotte Stix, Matteo Pistillo, Girish Sastry, Marius Hobbhahn, Alejandro Ortega et al.AI Behind Closed Doors: a Primer on The Governance of InternalDeploymentarXiv · 2025 ↗ An Approach to Technical AGI Safety and Security — arXiv · 2025-04 · arXiv:2504.01849 · Rohin Shah, Alex Irpan, Alexander Matt Turner, Anna Wang, Arthur Conmy et al.An Approach to Technical AGI Safety and SecurityarXiv · 2025 ↗ Must Read: A Comprehensive Survey of Computational Persuasion — arXiv / ACM Computing Surveys · 2025-05 · arXiv:2505.07775 · Nimet Beyza Bozdag, Shuhaib Mehri, Xiaocheng Yang, Hyeonjeong Ha, Zirui Cheng et al.Must Read: A Comprehensive Survey of Computational PersuasionarXiv / ACM Computing Surveys · 2025 ↗ How AI Impacts Skill Formation — arXiv · 2026-01 · arXiv:2601.20245 · Judy Hanwen Shen, Alex TamkinHow AI Impacts Skill FormationarXiv · 2026 ↗ MECHANISMVerifiable claimsRoots › Society › GovernanceTrust in an AI developer's claims needs mechanisms others can verify. Brundage etal. (2020) laid out ten, in three kinds: institutions (who checks), software(what leaves a trace), hardware (what can't be faked). The rest of this riverasks who is harmed, who is persuaded, and who decides.SO WHAT: 'we tested it' is a claim; governance is the machinery that lets someoneelse check it.INSTITUTIONSthird-party auditingred-team exercisesbias & safety bountiessharing AI incidentsSOFTWAREaudit trailsinterpretabilityprivacy-preserving MLHARDWAREsecure hardware for MLhigh-precision compute measurementcompute support for academiaDeveloper claim → checked by ↓KEY RESULT · zoomTen mechanismsBrundage et al. 2020: “We analyze ten mechanisms for this purpose--spanning institutions, software, and hardware.” Agent-era descendants:action logs, incident databases, AI Act third-party conformity checks.arXiv:2004.07213 (abstract)
BENCH · Make the claim checkable
Your company says “our agent is safe”. Which mechanism lets an outsider check?
Pick one — commit before you read the feedback.
 TODAY → AI Act duties, third-party audits, incident reporting Governance & Assurance ↗ THE MARSH The marsh: off-topic papers swept in by author graph 3 2014–15 3 2016–17 10 2018–19 10 2020–21 9 2022–23 3 2024–25 Implicit embedding of prior probabilities in optimally efficient neural populations — arXiv preprint · 2012-09-22 · arXiv:1209.5006 · Deep Ganguli, Eero SimoncelliImplicit embedding of prior probabilities in optimally efficientneural populationsarXiv preprint · 2012 ↗ Integral Geometry and Holography — arXiv / JHEP 10 · 2015-05-20 · arXiv:1505.05515 · Bartlomiej Czech; Lampros Lamprou; Samuel McCandlish; James SullyIntegral Geometry and HolographyarXiv / JHEP 10 · 2015 ↗ Universal Computation and Optimal Construction in the Chemical Reaction Network-Controlled Tile Assembly Model — DNA 21 — DNA Computing and Molecular Pro · 2015-07-21 · Nicholas Schiefer; Erik WinfreeUniversal Computation and Optimal Construction in the ChemicalReaction Network-Controlled Ti…DNA 21 — DNA Computing and Mol · 2015 ↗ Neural and perceptual signatures of efficient sensory coding — arXiv preprint · 2016-02-29 · arXiv:1603.00058 · Deep Ganguli, Eero P. SimoncelliNeural and perceptual signatures of efficient sensory codingarXiv preprint · 2016 ↗ Time Complexity of Computation and Construction in the Chemical Reaction Network-Controlled Tile Assembly Model — DNA 22 — DNA Computing and Molecular Pro · 2016-08-14 · Nicholas Schiefer; Erik WinfreeTime Complexity of Computation and Construction in the ChemicalReaction Network-Controlled T…DNA 22 — DNA Computing and Mol · 2016 ↗ FiLM: Visual Reasoning with a General Conditioning Layer — arXiv · 2017-09-22 · arXiv:1709.07871 · Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, Aaron CourvilleFiLM: Visual Reasoning with a General Conditioning LayerarXiv · 2017 ↗ A Fill Estimation Algorithm for Sparse Matrices and Tensors in Blocked Formats — IEEE IPDPS 2018 · 2018-05-01 · Willow Ahrens; Helen Xu; Nicholas SchieferA Fill Estimation Algorithm for Sparse Matrices and Tensors inBlocked FormatsIEEE IPDPS 2018 · 2018 ↗ Epistemic Consequentialism and Epistemic Enkrasia — Oxford University Press · 2018-06-21 · Amanda AskellEpistemic Consequentialism and Epistemic EnkrasiaOxford University Press · 2018 ↗ Feature-wise transformations — Distill · 2018-07-09 · Vincent Dumoulin, Ethan Perez, Nathan Schucher, Florian Strub, Harm de Vries et al.Feature-wise transformationsDistill · 2018 ↗ Pareto Principles in Infinite Ethics — PhilArchive / PhilPapers · 2018 · Amanda AskellPareto Principles in Infinite EthicsPhilArchive / PhilPapers · 2018 ↗ A Unified Theory of Early Visual Representations from Retina to Cortex through Anatomically Constrained Deep CNNs — arXiv / ICLR 2019 · 2019-01 · arXiv:1901.00945 · Jack Lindsey, Samuel A. Ocko, Surya Ganguli, Stéphane DenyA Unified Theory of Early Visual Representations from Retina toCortex through Anatomically C…arXiv / ICLR 2019 · 2019 ↗ ELI5: Long Form Question Answering — arXiv; published at ACL 2019 · 2019-07-22 · arXiv:1907.09190 · Angela Fan; Yacine Jernite; Ethan Perez; David Grangier; Jason Weston; Michael Auli ( et al.ELI5: Long Form Question AnsweringarXiv · 2019 ↗ Evidence Neutrality and the Moral Value of Information — Oxford University Press · 2019-09-12 · Amanda AskellEvidence Neutrality and the Moral Value of InformationOxford University Press · 2019 ↗ FoundationDB Record Layer: A Multi-Tenant Structured Datastore — arXiv · 2019-01-14 · arXiv:1901.04452 · Christos Chrysafis; Ben Collins; Scott Dugas; Jay Dunkelberger; Moussa Ehsan; Scott G et al.FoundationDB Record Layer: A Multi-Tenant Structured DatastorearXiv · 2019 ↗ Hypothesis: A new approach to property-based testing — Journal of Open Source Software, vol. 4, · 2019-11-21 · David R. MacIver; Zac Hatfield-Dodds; and many other contributorsHypothesis: A new approach to property-based testingJournal of Open Source Softwar · 2019 ↗ Noise2Self: Blind Denoising by Self-Supervision — ICML 2019 · 2019 · arXiv:1901.11365 · Joshua Batson, Loïc RoyerNoise2Self: Blind Denoising by Self-SupervisionICML 2019 · 2019 ↗ Falsify your Software: validating scientific code with property-based testing — Proceedings of the 19th Python in Scienc · 2020-07 · Zac Hatfield-DoddsFalsify your Software: validating scientific code withproperty-based testingProceedings of the 19th Python · 2020 ↗ Language Through a Prism: A Spectral Approach for Multiscale Language Representations — arXiv / NeurIPS 2020 · 2020-11 · arXiv:2011.04823 · Alex Tamkin, Dan Jurafsky, Noah GoodmanLanguage Through a Prism: A Spectral Approach for MultiscaleLanguage RepresentationsarXiv / NeurIPS 2020 · 2020 ↗ Semantic Segmentation of Histopathological Slides for the Classification of Cutaneous Lymphoma and Eczema — MIUA 2020 — Medical Image Understanding · 2020-09-10 · arXiv:2009.05403 · Jérémy Scheurer, Claudio Ferrari, Luis Berenguer Todo Bom, Michaela Beer et al.Semantic Segmentation of Histopathological Slides for theClassification of Cutaneous Lymphom…MIUA 2020 — Medical Image Unde · 2020 ↗ DABS: A Domain-Agnostic Benchmark for Self-Supervised Learning — arXiv / NeurIPS 2021 · 2021-11 · arXiv:2111.12062 · Alex Tamkin, Vincent Liu, Rongfei Lu, Daniel Fein, Colin Schultz, Noah GoodmanDABS: A Domain-Agnostic Benchmark for Self-Supervised LearningarXiv / NeurIPS 2021 · 2021 ↗ Deriving Semantics-Aware Fuzzers from Web API Schemas — CORRECTED — was 'arXiv · 2021-12-20 · arXiv:2112.10328 · Zac Hatfield-Dodds; Dmitry DygaloDeriving Semantics-Aware Fuzzers from Web API SchemasCORRECTED — was 'arXiv · 2021 ↗ Laplace Matching for fast Approximate Inference in Latent Gaussian Models — arXiv · 2021-05-07 · arXiv:2105.03109 · Marius Hobbhahn, Philipp HennigLaplace Matching for fast Approximate Inference in Latent GaussianModelsarXiv · 2021 ↗ Rissanen Data Analysis: Examining Dataset Characteristics via Description Length — arXiv; published at ICML 2021 · 2021-03-05 · arXiv:2103.03872 · Ethan Perez; Douwe Kiela; Kyunghyun ChoRissanen Data Analysis: Examining Dataset Characteristics viaDescription LengtharXiv · 2021 ↗ Topological Obstructions to Autoencoding — arXiv / published in JHEP 04 · 2021-02 · arXiv:2102.08380 · Joshua Batson, C. Grace Haaf, Yonatan Kahn, Daniel A. RobertsTopological Obstructions to AutoencodingarXiv / published in JHEP 04 · 2021 ↗ Viewmaker Networks: Learning Views for Unsupervised Representation Learning — arXiv / ICLR 2021 · 2020-10 · arXiv:2010.07432 · Alex Tamkin, Mike Wu, Noah GoodmanViewmaker Networks: Learning Views for Unsupervised RepresentationLearningarXiv / ICLR 2021 · 2021 ↗ What it Takes to Measure Reionization with Fast Radio Bursts — arXiv · 2021-07-29 · arXiv:2107.14242 · Stefan Heimersheim, Nina S. Sartorio, Anastasia Fialkov, Duncan R. LorimerWhat it Takes to Measure Reionization with Fast Radio BurstsarXiv · 2021 ↗ Action-modulated midbrain dopamine activity arises from distributed control policies — arXiv / NeurIPS 2022 · 2022-07 · arXiv:2207.00636 · Jack Lindsey, Ashok Litwin-KumarAction-modulated midbrain dopamine activity arises from distributedcontrol policiesarXiv / NeurIPS 2022 · 2022 ↗ Active Learning Helps Pretrained Models Learn the Intended Task — arXiv / NeurIPS 2022 · 2022-04 · arXiv:2204.08491 · Alex Tamkin, Dat Nguyen, Salil Deshpande, Jesse Mu, Noah GoodmanActive Learning Helps Pretrained Models Learn the Intended TaskarXiv / NeurIPS 2022 · 2022 ↗ Exponentially Improving the Complexity of Simulating the Weisfeiler-Lehman Test with Graph Neural Networks — arXiv · 2022-11-06 · arXiv:2211.03232 · Anders Aamand; Justin Y. Chen; Piotr Indyk; Shyam Narayanan; Ronitt Rubinfeld; Nichol et al.Exponentially Improving the Complexity of Simulating theWeisfeiler-Lehman Test with Graph Ne…arXiv · 2022 ↗ Fast Predictive Uncertainty for Classification with Bayesian Deep Networks — arXiv / UAI 2022 · 2020-03-02 · arXiv:2003.01227 · Marius Hobbhahn, Agustinus Kristiadi, Philipp HennigFast Predictive Uncertainty for Classification with Bayesian DeepNetworksarXiv / UAI 2022 · 2022 ↗ Feature Dropout: Revisiting the Role of Augmentations in Contrastive Learning — arXiv · 2022-12 · arXiv:2212.08378 · Alex Tamkin, Margalit Glasgow, Xiluo He, Noah GoodmanFeature Dropout: Revisiting the Role of Augmentations in ContrastiveLearningarXiv · 2022 ↗ First results on QCD+QED with C* boundary conditions — arXiv · 2022-09-27 · arXiv:2209.13183 · Lucius Bushnaq, Isabel Campos, Marco Catillo, Alessandro Cotellucci, Madeleine Dale et al.First results on QCD+QED with C* boundary conditionsarXiv · 2022 ↗ Instance-wise algorithm configuration with graph neural networks — arXiv only, v1 10 Feb 2022 · 2022-02-10 · arXiv:2202.04910 · Romeo Valentin, Claudio Ferrari, Jérémy Scheurer, Andisheh Amrollahi, Chris Wendler et al.Instance-wise algorithm configuration with graph neural networksarXiv only, v1 10 Feb 2022 · 2022 ↗ Merge what you can, fork what you can't: managing data integrity in local-first software — PaPoC '22 — 9th Workshop on Principles a · 2022-04-05 · Nicholas Schiefer; Geoffrey Litt; Daniel JacksonMerge what you can, fork what you can't: managing data integrity inlocal-first softwarePaPoC '22 — 9th Workshop on Pr · 2022 ↗ Learned Interpolation for Better Streaming Quantile Approximation with Worst-Case Guarantees — arXiv · 2023-04-15 · arXiv:2304.07652 · Nicholas Schiefer; Justin Y. Chen; Piotr Indyk; Shyam Narayanan; Sandeep Silwal; Tal et al.Learned Interpolation for Better Streaming Quantile Approximationwith Worst-Case GuaranteesarXiv · 2023 ↗ Flexible inference in heterogeneous and attributed multilayer networks — PNAS Nexus · 2024-05-31 · arXiv:2405.20918 · Martina Contisciani, Marius Hobbhahn, Eleanor A. Power, Philipp Hennig et al.Flexible inference in heterogeneous and attributed multilayernetworksPNAS Nexus · 2024 ↗ The Moral Inefficacy of Carbon Offsetting — Australasian Journal of Philosophy, vol. · 2024-04-30 · Tyler M. John; Amanda Askell; Hayden WilkinsonThe Moral Inefficacy of Carbon OffsettingAustralasian Journal of Philos · 2024 ↗ Longtermist Myopia — Oxford University Press, in 'Essays on L · 2025-08-25 · Amanda Askell; Sven NethLongtermist MyopiaOxford University Press, in 'E · 2025 ↗ MECHANISMHow a reading list grows a marshRoots › The marshThese are real papers by real people who appear in the rivers above. They arrivedbecause the list was built by following authors, not topics — so a researcher'sQCD paper, their chemistry paper and their philosophy paper came along too. Wekeep them, honestly labelled, as a lesson.SO WHAT: provenance is a security skill: know how every source got onto your listbefore you rely on it.authorsafety paper ✓safety paper ✓QCD boundary conditionschemical reaction networksinfinite ethicsdopamine & controlKEY RESULT · zoomAudit a source in three checks1. Does the title name an AI system, a risk, or a method used on models? 2. Does the abstract make a claim about safety or security? 3. Who put it on the list,and by what rule?applies to RAG indexes too
BENCH · Keep or cut?
Building a reading list for an agent-security team. Keep it or cut it?
Pick one — commit before you read the feedback.
 TODAY → Audit your sources: how did this paper get on the list? WHERE THE RIVERS MEET Today's agentic security, and which old river each part came from. Click a dot to jump to its river. Agent foundations & inner alignment Scheming & deceptive-alignment evals; model organisms ofmisalignment Threat Models · ring 4: the agent IS the adversary ↗ Scalable oversight · debate · ELK Human review at agent scale: LLM judges, CoT monitoring,decomposed review Guardrails & Runtime ↗ Interpretability: circuits → mech interp Probes and steering vectors used as runtime monitors andswitches Guardrails & Runtime ↗ AI control Untrusted-model protocols in agent harnesses: monitors,resampling, audit budgets Defense Frameworks ↗ Evals, scheming & honesty Agent benchmarks, red-team evals, time-horizon tracking Benchmarks & Evaluation ↗ Learning from humans & robustness Refusal training and adversarial robustness →prompt-injection robustness Prompt Injection ↗ Scaling & capabilities Why it is urgent: capability arrives on a schedule Fit to Deploy atlas ↗ Society, people & governance AI Act duties, third-party audits, incident reporting Governance & Assurance ↗