CS292F (Spring 2021) Statistical Foundation of Reinforcement Learning


Syllabus [ link ]

Instructor: Prof. Yu-Xiang Wang


Lecture Section: Monday/Wednesday 1:00-2:40 pm Location: on Zoom (link will be sent to you via email.)

Piazza: https://piazza.com/ucsb/spring2021/cs292/home
Piazza is our main channel of communication. Questions should be posted here.

Gradescope: https://www.gradescope.com/courses/258384
This is where you submit your homeworks and project reports.

Office hours: Instructor: by appointment.

Course evaluation: 40% Homework, 40% Project, 10% for attendance / Participation. 10% for scribing.

Scribing: Please volunteer here, use this latex template

Textbook:

Acknowledgments The instructor sincerely thanks Wen Sun, Nan Jiang and Sham Kakade for sharing
the homeworks and other materials from CS 6789 at Cornell/University of Washington and CS 598 at UIUC.

Course Schedule / Scribed Notes

DateLecturesReadingsAssignments
129-MarIntroduction and MDP basics [annotated, scribe]AJKS Ch 1.1-1.2HW0 out
231-MarMarkov Decision Processes I [annotated] AJKS Ch 1.3-1.5 
35-AprMarkov Decision Processes II [annotated, scribe] AJKS Ch 2HW1 out
47-AprMDP III and RL Algorithms I [annotated] SB Ch 5-6 
512-AprRL Algorithms II [annotated]SB Ch 9-10 HW0 due
614-Apr RL Algorithm III and Exploration I: MAB [annotated] SB Ch 13, AJKS Ch 9, AJKS Ch 5.1 
719-Apr Exploration I: MAB and Linear Bandits [annotated]AJKS Ch 5.1 Project proposal due
821-Apr Exploration II: Linear Bandits [annotated] AJKS Ch 5.2-5.3 
926-AprExploration: Tabular MDPs [annotated] AJKS Ch 6 HW2 out / HW1 due
1028-AprExploration: Linear MDP [annotated]AJKS Ch 7 
113-MayWrap up exploration, Intro to Offline RL [annotated] AJKS 7.3-7.4, Lihong's perspective article.Midterm report due
125-MayOffline RL: OPE in Bandits and RL (W., Agarwal, Dudik, 2016) (Jiang et al., 2016)  
1310-MayOffline RL: MIS and Fitted Q Iterations (Yin and W., 2019) (Duan and Wang, 2019) HW3 out / HW2 due
1412-MayOffline RL: Uniform OPE (Yin et al., 2020)  
1517-MayOffline RL: Pessimism in Value Iteration (Jin et al, 2021)  
1619-MayAdvanced topic / Student presentation 1  
1724-MayAdvanced topic / Student presentation 2 HW3 due
1826-MayAdvanced topic / Student presentation 3  
1931-MayNo lecture, Memorial Day  
202-JunAdvanced topic / Student presentation 4 Final project report due