CS292F (Spring 2021) Statistical Foundation of Reinforcement Learning

Syllabus [ link ]

Instructor: Prof. Yu-Xiang Wang

Lecture Section: Monday/Wednesday 1:00-2:40 pm Location: on Zoom (link will be sent to you via email.)

Piazza: https://piazza.com/ucsb/spring2021/cs292/home
Piazza is our main channel of communication. Questions should be posted here.

Gradescope: https://www.gradescope.com/courses/258384
This is where you submit your homeworks and project reports.

Office hours: Instructor: by appointment.

Course evaluation: 40% Homework, 40% Project, 10% for attendance / Participation. 10% for scribing.

Scribing: Please volunteer here, use this latex template


Acknowledgments The instructor sincerely thanks Wen Sun, Nan Jiang and Sham Kakade for sharing
the homeworks and other materials from CS 6789 at Cornell/University of Washington and CS 598 at UIUC.

Course Schedule / Scribed Notes

129-MarIntroduction and MDP basics [annotated, scribe]AJKS Ch 1.1-1.2HW0 out
231-MarMarkov Decision Processes I [annotated] AJKS Ch 1.3-1.5 
35-AprMarkov Decision Processes II [annotated, scribe] AJKS Ch 2HW1 out
47-AprMDP III and RL Algorithms I [annotated] SB Ch 5-6 
512-AprRL Algorithms II [annotated]SB Ch 9-10 HW0 due
614-Apr RL Algorithm III and Exploration I: MAB [annotated] SB Ch 13, AJKS Ch 9, AJKS Ch 5.1 
719-Apr Exploration I: MAB and Linear Bandits [annotated]AJKS Ch 5.1 Project proposal due
821-Apr Exploration II: Linear Bandits [annotated] AJKS Ch 5.2-5.3 
926-AprExploration: Tabular MDPs [annotated] AJKS Ch 6 HW2 out / HW1 due
1028-AprExploration: Linear MDP [annotated]AJKS Ch 7 
113-MayWrap up exploration, Intro to Offline RL [annotated] AJKS 7.3-7.4, Lihong's perspective article.Midterm report due
125-MayOffline RL: OPE in Bandits and RL (W., Agarwal, Dudik, 2016) (Jiang et al., 2016)  
1310-MayOffline RL: MIS and Fitted Q Iterations (Yin and W., 2019) (Duan and Wang, 2019) HW3 out / HW2 due
1412-MayOffline RL: Uniform OPE (Yin et al., 2020)  
1517-MayOffline RL: Pessimism in Value Iteration (Jin et al, 2021)  
1619-MayAdvanced topic / Student presentation 1  
1724-MayAdvanced topic / Student presentation 2 HW3 due
1826-MayAdvanced topic / Student presentation 3  
1931-MayNo lecture, Memorial Day  
202-JunAdvanced topic / Student presentation 4 Final project report due