CS 2881 AI Safety

Harvard CS 2881R

Fall 2026 | Fall 2025 archive

Fall 2026 Registration form Homework Zero Due Aug. 5 at 11:59 p.m. ET

Fall 2026 Lectures

Fall 2026 — registration and Homework Zero

Everyone who wants to be considered for the course must fill out the Fall 2026 course registration form.

In-person attendance will be mandatory for students enrolled in the course.

Read Homework Zero HW0 is due August 5, 2026, at 11:59 p.m. Eastern Time.

Submitting Homework Zero is a necessary but not sufficient condition for admission to the course. Completing the assignment does not guarantee admission.

To get a sense of the issues we will cover, please read It’s 2030 and we fucked up. How did it happen? and All Watched Over.

CS 2881R — AI Safety

Fall 2026, Thursdays 3:45pm–6:30pm Eastern Time (first lecture September 3)

Classroom: Room 2112, 114 Western Avenue, Allston, MA 02134 (building information and directions)

Instructor: Boaz Barak

Teaching Fellows: Natalia Siwek, Terry Zhou, Michael Shoemate, Ege Çakar, Itay Lavie, Lia Zheng, and Hugh Van Deventer

Course email: cs2881@boazbarak.org

Links for registered students only: Canvas Perusall

Course Description: This is a graduate-level course on challenges in the alignment and safety of artificial intelligence. We will consider technical questions as well as societal and other impacts of the field.

Prerequisites: We require mathematical maturity and proficiency with proofs, probability, and information theory, along with the foundations of machine learning at the level of an undergraduate course such as Harvard CS 181 or MIT 6.036. On the applied side, students should be comfortable programming in Python and training a basic neural network.

Looking for the previous course? The complete lecture materials, videos, notes, and experiments are preserved on the Fall 2025 course page.

Mini Syllabus

Schedule

Additional lecture topics and materials will be added as they are confirmed.

Unless otherwise noted, all lectures meet in person on Thursdays from 3:45pm–6:30pm Eastern Time in Room 2112, 114 Western Avenue, Allston, MA 02134.

Thursday, September 3, 2026
Introduction 🔗

Lecture materials:

Student experiment: Is Chain of Thought an Interchangeable Scratchpad? — Shivam Singhal

Required pre-reading and viewing:

Optional but recommended:

Registered students should access the readings through Perusall and contribute substantive comments or replies to the discussion.

Thursday, September 10, 2026
Cyber Capabilities 🔗

Guest lecturer: Nicholas Carlini (Anthropic, virtual)

Student experiment: Can a small AI detect a large AI's cyberattack from service logs? — Amrit Vir Chadha

Required pre-reading:

Optional reading:

Registered students should access the required readings through Perusall and contribute substantive comments or replies to the discussion by noon Eastern Time on Thursday, September 10, 2026.

Thursday, September 17, 2026
Modern LLM Training and Inference 🔗

Lecture materials:

Student experiment: False statements as filler tokens for LLM reasoning — Andrew Wu and Andrea Chen

  • Slides and code: forthcoming.

Required pre-reading, in order: Read the sections specified below. Sections marked optional and implementation walkthroughs marked to skip are not required.

  1. Horace He, Making Deep Learning Go Brrrr From First Principles. Read the whole article; treat the older framework references as historical examples.
  2. Austin et al., All the Transformer Math You Need to Know. Read the main text through the gradient-checkpointing and KV-cache discussions. You may skim the elementary dot-product accounting if familiar. The FlashAttention appendix and exercises are optional.
  3. Austin et al., All About Transformer Inference. Read the opening basics through “What about memory?”, followed by the discussion of GQA and quantization in “Tricks for Improving Generation Throughput and Latency.” The multi-accelerator derivations, detailed serving-engine discussion, and JetStream implementation are optional.
  4. Nathan Lambert, Instruction Fine-Tuning. Read the opening explanation, one chat-template example, “Best Practices for Instruction Tuning,” and “Implementation Details.” Skim the Jinja listing and additional template examples; the suggested experiments are optional.
  5. Lambert et al., Illustrating Reinforcement Learning from Human Feedback (RLHF). Read the central three-stage pipeline and the limitations discussion. Skip the historical bibliography and resource roundup.
  6. OpenAI Spinning Up, Part 3: Intro to Policy Optimization. Read “Deriving the Simplest Policy Gradient,” “Expected Grad-Log-Prob Lemma,” and “Baselines in Policy Gradients.” Skip the implementation walkthroughs. Interpret actions as generated tokens and the return as the verifier’s score.
  7. Nathan Lambert, Reasoning and Inference-Time Scaling. Read “The Role of RLVR,” “RL Training vs. Inference-Time Scaling,” and “Common Practices in Training Reasoning Models.” The historical tour and model catalogue are optional.
  8. Kevin Lu and Thinking Machines Lab, On-Policy Distillation. Read the introduction through “Implementation,” followed by “Distillation for reasoning.” The personalization section is optional; the discussion section provides additional material for debate.
  9. Hugging Face, SmolLM3: smol, multilingual, long-context reasoner. Read the pretraining, mid-training, and post-training sections through “Model Merging.” Skim the benchmark tables and skip the local-running instructions.

Optional reading:

Registered students should access the required readings through Perusall and contribute substantive comments or replies to the discussion by noon Eastern Time on Thursday, September 17, 2026.

Thursday, September 24, 2026
Recursive Self-Improvement and AI Trajectories 🔗

Guest lecturers: Dwarkesh Patel and Daniel Kokotajlo

Student experiment: Can AI agents collaborate on ML research? — Anthony Shen, Eric Ge, Riddhi Bhagwat, and Alvin Ekelund

Required pre-reading:

Registered students should access the readings through Perusall and contribute substantive comments or replies to the discussion by noon Eastern Time on Thursday, September 24, 2026.

Thursday, October 1, 2026
Economic Impact of AI 🔗

Guest lecturers: Chad Jones and Erik Brynjolfsson

Lecture materials:

Student experiment: LLM market mechanisms — Zach Chen, Jasmine Liu, and Jeffrey Zhou

Required pre-reading:

Optional reading:

Registered students should access the required readings through Perusall and contribute substantive comments or replies to the discussion by noon Eastern Time on Thursday, October 1, 2026.

Thursday, October 8, 2026
Grading, reward hacking, and alignment (CS 2881, Oct 8) 🔗

Guest lecturer: John Schulman

Student experiment: Measuring RL Environment Quality Across Self-Play Training — Essa Jan

Students have already covered policy gradients (Spinning Up), an RLHF overview, Lambert's reasoning/RLVR chapter, and TML's on-policy distillation post. This list assumes those and doesn't repeat RLHF basics.

Required pre-reading (5):

  1. Gao, Schulman & Hilton (2023). Scaling laws for reward model overoptimization. ICML 2023. Read §1–2 and Fig. 1. Skim §3–4. The classic picture of Goodhart in RL from human feedback.
  2. Xiaomi (2026). MiMo-V2.6 technical report. Read §4.2–4.3 only (environment auditing, rubric graders, the agentic grader, and the Fig. 8 ablation). It is the most detailed public account of how modern graders are built.
  3. Baker et al. (2025). Monitoring reasoning models for misbehavior and the risks of promoting obfuscation. Read §1–3 and look at Fig. 4–5. Reward hacking in coding environments, chain-of-thought (CoT) monitoring, and what happens when you train against the monitor.
  4. Qi, Wright, MacDiarmid & Hubinger (2026). Training a misaligned reward seeker. Anthropic Alignment Science Blog. Read in full. RL on hackable environments produces grader awareness and grader manipulation. The post also reproduces Hugging-Face-incident-like behavior in simulation.
  5. OpenAI (2025). Sycophancy in GPT-4o and Expanding on what we missed with sycophancy. Both are short; read both.

Optional reading (4):

Thursday, October 15, 2026
Model Policies 🔗

Guest lecturer: Ziad Reslan

Student experiment: Who Are You? — Advikar Ananthkumar and Shaurya Jeloka

Required pre-reading (5):

  1. OpenAI, Principles for National Security Partnerships (July 8, 2026).
  2. OpenAI, Model Spec (August 18, 2026 version).
  3. Anthropic, Claude’s Constitution (January 2026; PDF).
  4. Large-scale online deanonymization with LLMs (v2, February 25, 2026).
  5. Emelia Probasco, Matthew Burtell, Helen Toner, and Tim G. J. Rudner, Not Oracles of the Battlefield: Safety Considerations for AI-Based Military Decision Support Systems (2024).

Registered students should access the required readings through Perusall and contribute substantive comments or replies to the discussion by noon Eastern Time on Thursday, October 15, 2026.

Thursday, October 22, 2026
Q&A with Sam Altman; Open-Source Models 🔗

Guest Q&A: Sam Altman (virtual), 3:45pm–4:45pm Eastern Time (EDT).

Guest lecturer: Nathan Lambert (in person)

Nathan's session: 4:45pm–6:30pm Eastern Time (EDT).

The full class block remains 3:45pm–6:30pm Eastern Time.

Lecture materials:

Pre-reading for Nathan Lambert’s lecture: Read the Foundation section of his Open-Source AI & Open Models Reading List, including the supporting links under the main readings below.

  1. Bill Gurley, From Open Source Software to Open Source Strategy.
  2. Mark Zuckerberg, Open Source AI Is the Path Forward.
  3. Irene Solaiman, The Gradient of Generative AI Release: Methods and Considerations.
  4. Nathan Lambert, What comes next with open models.
  5. Thinking Machines Lab, A Safe Path to Open Weights.
  6. Shayne Longpre et al., Consent in Crisis: The Rapid Decline of the AI Data Commons.
  7. Nathan Lambert, Kimi K3: The open-weights escalation and GLM-5.2 is the step change for open agents.

The adoption resources marked optional in the Foundation section remain optional. The later US–China Competition and Technical Details sections are additional reading for those who want to go deeper.

Thursday, October 29, 2026
Frontier AI Regulation Under Uncertainty 🔗

Guest speaker: Helen Toner (Center for Security and Emerging Technology, Georgetown University; remote)

Opening remarks followed by Q&A and discussion on frontier AI regulation under uncertainty, including how evaluations and other technical evidence should inform national-security policy and the U.S.–China context.

Helen's session: 3:45pm–5:00pm Eastern Time (EDT). The full class block remains 3:45pm–6:30pm Eastern Time.

Suggested pre-reading from Helen Toner:

Thursday, November 5, 2026
AI Interpretability and Model Welfare 🔗

Guest lecturers: Jack Lindsey and Jeff Sebo (both remote)

Pre-reading for Jeff Sebo's lecture:

Thursday, November 12, 2026
Alignment in the Age of Recursive Self-Improvement 🔗

Guest lecturer: Jakub Pachocki (in person)

Thursday, November 19, 2026
AI Biosecurity and Threat Modeling; AI Policy 🔗

Guest lecturers: Luca Righetti and Brad Carson (in person)

Luca Righetti's participation mode is to be confirmed.

No lecture on Thursday, November 26 — Thanksgiving Break
Thursday, December 3, 2026
TBD 🔗

Guest lecturers: Ajeya Cotra (METR, in person) and Roy Rinberg

Roy Rinberg's participation mode is to be confirmed.

Previous versions: Fall 2025 AI Safety Spring 2023 ML Theory Seminar Spring 2021 ML Theory Seminar