Weekly Schedule
The schedule along with the list of topics to be covered are tentative and subject to change.
| Week | Date | Module | Topic | Reading | Assignment | |
|---|---|---|---|---|---|---|
| 1 | T | 8/25 | Introduction | How AI fails, and what “trust” decomposes into (slides) | Post a note on Teams to introduce yourself. Review the Syllabus. | |
| R | 8/27 | Introduction | Convolutional networks and transformers (notebook) | Phuong & Hutter, Formal Algorithms for Transformers | Setup software and development environment for the course (see here) Assignment 0 out | |
| 2 | T | 9/1 | No class: Ravi traveling | |||
| R | 9/3 | No class: Ravi traveling | ||||
| 3 | T | 9/8 | Introduction | Transformer review and post-training (notebook) | Rafailov et al., Direct Preference Optimization (§3-4) | |
| R | 9/10 | Robustness Attacks | Adversarial examples as constrained optimization (notebook) | Goodfellow et al., Explaining and Harnessing Adversarial Examples; Madry et al., Towards Deep Learning Models Resistant to Adversarial Attacks (§2); Gilmer et al., Motivating the Rules of the Game for Adversarial Example Research | ||
| 4 | T | 9/15 | Robustness Attacks | Jailbreaking: GCG as discrete PGD (notebook) | Zou et al., Universal and Transferable Adversarial Attacks on Aligned Language Models | Assignment 0 due Assignment 1 out |
| R | 9/17 | Robustness Defenses | Empirical defenses, on images and on language models (notebook) | Madry et al., Towards Deep Learning Models Resistant to Adversarial Attacks; Sharma et al., Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teaming | ||
| 5 | T | 9/22 | Robustness Defenses | Empirical defenses, continued: evaluating a defense (notebook) | Athalye et al., Obfuscated Gradients Give a False Sense of Security; Carlini et al., On Evaluating Adversarial Robustness (§2-3) | |
| R | 9/24 | Robustness Guarantees | Certified robustness: bound propagation (notebook) | Gowal et al., On the Effectiveness of Interval Bound Propagation for Training Verifiably Robust Models; Singh et al., An Abstract Domain for Certifying Neural Networks (skim) | ||
| 6 | T | 9/29 | Interpretability | Input attribution (notebook) | Sundararajan et al., Axiomatic Attribution for Deep Networks | Assignment 1 due Assignment 2 out |
| R | 10/1 | Interpretability | Input attribution and its weaknesses, continued (notebook) | Adebayo et al., Sanity Checks for Saliency Maps | ||
| 7 | T | 10/6 | No class: Ravi traveling | |||
| R | 10/8 | Interpretability | Probes, concepts, and steering (notebook) | Arditi et al., Refusal in Language Models Is Mediated by a Single Direction | ||
| 8 | T | 10/13 | Interpretability | Probes, concepts, and steering, continued (notebook) | ||
| R | 10/15 | Interpretability | Sparse autoencoders and circuits (notebook) | Bricken et al., Towards Monosemanticity: Decomposing Language Models With Dictionary Learning; Wang et al., Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 Small | Assignment 2 due | |
| 9 | T | 10/20 | Midterm | |||
| R | 10/22 | Guest lecture by Prof. Aparna Bharati: Provenance and watermarking | Assignment 3 out | |||
| 10 | T | 10/27 | Training-time Failures | Data poisoning, backdoors, and sleeper agents (notebook) | Gu et al., BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain; Carlini et al., Poisoning Web-Scale Training Datasets is Practical; Hubinger et al., Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training | |
| R | 10/29 | Training-time Failures | Data poisoning, backdoors, and sleeper agents, continued (notebook) | |||
| 11 | T | 11/3 | Training-time Failures | Specification gaming (notebook) | Gao et al., Scaling Laws for Reward Model Overoptimization; Betley et al., Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs | Project proposal due |
| R | 11/5 | Uncertainty and Calibration | Does the model know when it is wrong? (part 1) (notebook) | Guo et al., On Calibration of Modern Neural Networks; Kadavath et al., Language Models (Mostly) Know What They Know; Farquhar et al., Detecting Hallucinations in Large Language Models Using Semantic Entropy | ||
| 12 | T | 11/10 | Uncertainty and Calibration | Does the model know when it is wrong? (part 2) (notebook) | ||
| R | 11/12 | Agents | Agent architecture and the agent threat model (notebook) | Willison, The lethal trifecta for AI agents; Hou et al., Model Context Protocol (MCP): Landscape, Security Threats, and Future Research Directions | ||
| 13 | T | 11/17 | Agents | Prompt injection (notebook) | Greshake et al., Not what you’ve signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection; Liu et al., Formalizing and Benchmarking Prompt Injection Attacks and Defenses | Assignment 3 due |
| R | 11/19 | Agents | Structural defenses and monitoring (notebook) | Costa et al., Securing AI Agents with Information-Flow Control; Palumbo et al., Formal Policy Enforcement for Agentic Systems | ||
| 14 | T | 11/24 | No class: Fall recess | |||
| R | 11/26 | No class: Fall recess | ||||
| 15 | T | 12/1 | Evaluation | Model and system cards | ||
| R | 12/3 | No class: Ravi traveling | ||||
| 16 | T | 12/8 | Wrap-up | Project presentations | ||
| R | 12/10 | Wrap-up | What we did not cover | Project report due | ||
| Finals | W | 12/16 | Final Exam in CSB 130 (6:20-8:20 pm) |