There are two kinds of course projects: research projects and non-research projects. The primary goal of a non-research project is to reproduce the proof of a theoretical RL paper in order to better understand it. The primary goal of a research project is to investigate a novel research idea and ideally make a submission to some venues. In either case, you are expected to submit a proposal (5%), a progress report (5%), and a final write-up (30%) and deliver a project presentation (30%). There is no difference regarding the grades between the two types of projects.

Non-research project

I will provide a list of papers for you to choose from. You are also welcome to choose a paper not from the list but make sure to talk with me first for approval. You must complete the non-research project solo.

  • Proposal: You need to state what paper you choose to reproduce and give a tentative timeline for milestones (e.g., understanding the motivation, understanding the background, finishing Lemma 1, finishing Theorem 1, and finishing the write-up)
  • Progress report: What milestones have you achieved? Is there any obstacle?
  • Write-up:
    • The write-up is expected to be self-contained, i.e., a reader should be able to understand the write-up without referring to the original paper. To achieve this with minimal effort, the write-up should include the mathematical formulation of the problem the paper studies. Of course, you need to care about only the math part of the paper and feel free to ignore the introduction, related work, and experiments in the write-up. But in most cases you do need to understand the non-math part well to better understand the math part.
    • The write-up must be in your own words. In the write-up, you should try to make the equations as clear as possible. You should understand every equation you write and fix ambiguous arguments in the original paper (e.g., “it is easy to see”, “it is trivial that”). A reader from a CS department that has only undergraduate level probability and linear algebra knowledge should be able to understand the equations in your write-up. Annotate your equations well, e.g., if one inequality results from Cauchy–Schwarz or the extreme value theorem, you should spell it out.
    • Try to use the notations I use in the lectures. Make sure every notation you use is clearly defined in the write-up. Feel free to reorganize the original paper.
    • You are welcome to include some extensions of the existing results. Even very small extensions matter. Of course, extensions are just for bonus and are not necessary.
  • Presentation: Walk me and other students through the paper and your write-up. I might ask questions regarding the equations in your write-up.

Research project

I will provide a list of ideas for you to choose from. Each idea consists of several base papers and some possible directions for extensions. You are expected to understand the base papers and investigate the extension. I have only a limited number of ideas so it is on a first-come-first-served basis (I will discuss the details later). I will try my best but there is no guarantee that everyone can find a research project.

You are of course welcome to propose a new research project by yourself. Such new research projects even do not have to be theoretical but can also be empirical investigations / applications of some RL theories in the lectures. Make sure to talk with me first for the approval of your new research project.

Each group only needs to submit one proposal, one progress report, one write-up, and do one presentation.

  • Proposal: you need to state what idea you choose to investigate and the job assignment in your group (if you work in a group). You should give a tentative timeline for milestones (e.g., understanding the base papers, investigating the extensions, finishing the write-up)
  • Progress report: What milestones have you achieved? Is there any obstacle?
  • Write-up: The write-up is meant to be a paper that is ready for submission to some venues (e.g., TMLR, ICML). So it should have abstract, introduction, background, related work, new findings, conclusion, and references. Refer to the base paper to get an idea on what a research paper looks like.
  • Presentation: walk me and other students through your write-up. I might ask some questions.

In case the extension does not work, I expect the group (as a whole) to reproduce the base papers just like non-research projects and provide an additional report documenting the efforts they made. If the group has N students but the project has more than N base papers, the group needs to reproduce only N base papers. If the group has N students but the project has less than or equal to N base papers, the group needs to reproduce all base papers. So the research project is risk free in that you will anyway have a successful project for this course. Though the research projects are risk free, please take them seriously and start early. It is nontrivial to find ideas that are good enough, easy enough, and promising enough to fit into a course research project.