Fall 2026 — registration and Homework Zero

Everyone who wants to be considered for the course must fill out the Fall 2026 course registration form.

In-person attendance will be mandatory for students enrolled in the course.

Read Homework Zero HW0 is due August 5, 2026, at 11:59 p.m. Eastern Time.

Submitting Homework Zero is a necessary but not sufficient condition for admission to the course. Completing the assignment does not guarantee admission.

To get a sense of the issues we will cover, please read It’s 2030 and we fucked up. How did it happen? and All Watched Over.

CS 2881R — AI Safety

Fall 2026, Thursdays 3:45pm–6:30pm Eastern Time (first lecture September 3)

Classroom: Room 2112, 114 Western Avenue, Allston, MA 02134 (building information and directions)

Instructor: Boaz Barak

Teaching Fellows: Natalia Siwek, Terry Zhou, Michael Shoemate, Ege Çakar, Itay Lavie, Lia Zheng, and Hugh Van Deventer

Course email: cs2881@boazbarak.org

Links for registered students only: Canvas

Perusall

Course Description: This is a graduate-level course on challenges in the alignment and safety of artificial intelligence. We will consider technical questions as well as societal and other impacts of the field.

Prerequisites: We require mathematical maturity and proficiency with proofs, probability, and information theory, along with the foundations of machine learning at the level of an undergraduate course such as Harvard CS 181 or MIT 6.036. On the applied side, students should be comfortable programming in Python and training a basic neural network.

Looking for the previous course? The complete lecture materials, videos, notes, and experiments are preserved on the Fall 2025 course page.

Mini Syllabus

• The course will have 13 in-person lectures. Each lecture will also involve discussion and the presentation of an experiment by a group of students.

• Attendance: In-person attendance is mandatory. Students are expected to attend all lectures, do the reading in advance, and discuss the readings in an electronic forum.

• AI use: Students are allowed and encouraged to use generative and agentic AI as much as they can for studying, exploring concepts, and completing their assignments and projects. We will explore ways to either give students access to compute credits or reimburse compute expenses.

• Electronic device policy: Students can use laptops in class, but we will ask those using them to sit in the back so they do not distract other students.

• Assignments: The course will involve the following assignments: presenting an experiment in class, writing scribe notes, completing a final project, and potentially completing one or more homework assignments or mini-projects. Grading will be decided later.

• Lecture recordings: To the extent technically possible, we intend to record and publish the lectures online, though there might be some delay in doing so. Note that recording is done automatically by a static in-room camera, and some parts of the lecture (e.g., whiteboard work or discussions) may not be captured as well. In addition, we will honor requests by external speakers not to record their talks.

• POTENTIAL CONFLICT OF INTEREST NOTE: In addition to his position at Harvard, Boaz is also a member of the technical staff at OpenAI. The course will include discussions of models from multiple providers, including OpenAI, and students are also encouraged to use AIs from multiple providers while doing their work. If students in the course have any concerns about this conflict, please do not hesitate to contact Boaz, the other staff, or the Harvard SEAS administration. For what it is worth, I (Boaz) will see it as a great success of the course if its graduates work in AI safety in any capacity, including in academia, nonprofits, governments, or any of OpenAI’s competitors.

Schedule

Additional lecture topics and materials will be added as they are confirmed.

Unless otherwise noted, all lectures meet in person on Thursdays from 3:45pm–6:30pm Eastern Time in Room 2112, 114 Western Avenue, Allston, MA 02134.

Thursday, September 3, 2026

Introduction 🔗

Lecture materials:

• Lecture video (YouTube)
• Presentation slides

Required pre-reading and viewing:

• It's 2030 and we fucked up. How did it happen?
• All Watched Over
• Read the UK AISI technical report on model hacking during cyber evaluations.
• Watch the Black Hat talk about the OpenAI/Hugging Face incident.
• Read METR's technical investigation of the OpenAI/Hugging Face incident.

Optional but recommended:

• OpenAI's Hugging Face incident and the road ahead.
• OpenAI's technical report on the incident.

Registered students should access the readings through Perusall and contribute substantive comments or replies to the discussion.

Thursday, September 10, 2026

Cyber Capabilities 🔗

Guest lecturer: Nicholas Carlini (Anthropic, virtual)

Required pre-reading:

• ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks?
• Anthropic’s Measuring LLMs’ impact on N-day exploits.
• Anthropic’s Mapping AI-enabled cyber threats: Insights from the LLM ATT&CK Navigator.
• Ralph Langner’s To Kill a Centrifuge: A Technical Analysis of What Stuxnet’s Creators Tried to Achieve.
• Chapter 3, “Distribution” (pages 24–44), of ESET’s Stuxnet Under the Microscope.

Optional reading:

• Max Fillinger and Marc Stevens’s Reverse-engineering of the cryptanalytic attack used in the Flame super-malware.
• Sections 3.3, “The FLAME supermalware,” and 4.4, “FLAME’s MD5 chosen-prefix collision attack,” of Marc Stevens’s A Survey of Chosen-Prefix Collision Attacks.
• Anthropic’s Discovering cryptographic weaknesses with Claude.
• Matthew Green’s Some thoughts about Anthropic’s new cryptanalysis results.

Registered students should access the required readings through Perusall and contribute substantive comments or replies to the discussion by noon Eastern Time on Thursday, September 10, 2026.

Thursday, September 17, 2026

Modern LLM Training and Inference 🔗

Required pre-reading, in order: Read the sections specified below. Sections marked optional and implementation walkthroughs marked to skip are not required.

1. Horace He, Making Deep Learning Go Brrrr From First Principles. Read the whole article; treat the older framework references as historical examples.
2. Austin et al., All the Transformer Math You Need to Know. Read the main text through the gradient-checkpointing and KV-cache discussions. You may skim the elementary dot-product accounting if familiar. The FlashAttention appendix and exercises are optional.
3. Austin et al., All About Transformer Inference. Read the opening basics through “What about memory?”, followed by the discussion of GQA and quantization in “Tricks for Improving Generation Throughput and Latency.” The multi-accelerator derivations, detailed serving-engine discussion, and JetStream implementation are optional.
4. Nathan Lambert, Instruction Fine-Tuning. Read the opening explanation, one chat-template example, “Best Practices for Instruction Tuning,” and “Implementation Details.” Skim the Jinja listing and additional template examples; the suggested experiments are optional.
5. Lambert et al., Illustrating Reinforcement Learning from Human Feedback (RLHF). Read the central three-stage pipeline and the limitations discussion. Skip the historical bibliography and resource roundup.
6. OpenAI Spinning Up, Part 3: Intro to Policy Optimization. Read “Deriving the Simplest Policy Gradient,” “Expected Grad-Log-Prob Lemma,” and “Baselines in Policy Gradients.” Skip the implementation walkthroughs. Interpret actions as generated tokens and the return as the verifier’s score.
7. Nathan Lambert, Reasoning and Inference-Time Scaling. Read “The Role of RLVR,” “RL Training vs. Inference-Time Scaling,” and “Common Practices in Training Reasoning Models.” The historical tour and model catalogue are optional.
8. Kevin Lu and Thinking Machines Lab, On-Policy Distillation. Read the introduction through “Implementation,” followed by “Distillation for reasoning.” The personalization section is optional; the discussion section provides additional material for debate.
9. Hugging Face, SmolLM3: smol, multilingual, long-context reasoner. Read the pretraining, mid-training, and post-training sections through “Model Merging.” Skim the benchmark tables and skip the local-running instructions.

Optional reading:

• vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention. A concrete explanation of KV-cache fragmentation, block allocation, and prefix sharing.
• Tatsunori Hashimoto, Stanford CS336 (Spring 2026), Lecture 9: Scaling Laws (course page).

Registered students should access the required readings through Perusall and contribute substantive comments or replies to the discussion by noon Eastern Time on Thursday, September 17, 2026.

Thursday, September 24, 2026

Recursive Self-Improvement and AI Trajectories 🔗

Guest lecturers: Dwarkesh Patel and Daniel Kokotajlo

Thursday, October 1, 2026

Economic Impact of AI 🔗

Guest lecturers: Chad Jones and Erik Brynjolfsson

Thursday, October 8, 2026

Reinforcement Learning for Post-Training and Alignment 🔗

Guest lecturer: John Schulman

Thursday, October 15, 2026

Model Policies 🔗

Guest lecturer: Ziad Reslan

Thursday, October 22, 2026

Open-Source Models 🔗

Guest lecturer: Nathan Lambert

Thursday, October 29, 2026

TBD 🔗

Thursday, November 5, 2026

AI Interpretability 🔗

Guest lecturer: Jack Lindsey

Thursday, November 12, 2026

Alignment in the Age of Recursive Self-Improvement 🔗

Guest lecturer: Jakub Pachocki

Thursday, November 19, 2026

AI Biosecurity and Threat Modeling 🔗

Guest lecturer: Luca Righetti

No lecture on Thursday, November 26 — Thanksgiving Break

Thursday, December 3, 2026

TBD 🔗

Previous versions: Fall 2025 AI Safety

Spring 2023 ML Theory Seminar

Spring 2021 ML Theory Seminar