Course analysis for DD3501/FDD3501 Current Research in Proof Complexity, Winter 2011/12Course data
SummaryThis was an advanced course in proof complexity intended for PhD students and lastyear MSc students. The course focused on recent research and open problems in proof complexity as well as on connections to applied SAT solving. Quite excitingly, it so happened that for a number of open problems stated in the introductory lectures of the course, lectures towards the end of the course covered solutions to these problems discovered while the course was being given. The students found this to be a very challenging course (which was intentional) but also very rewarding. Comments in the course evaluation after the course (and in "opinion polls" conducted after some lectures during the course) were over all very positive. However, since this was the first time this course was given, some teething problems were inevitable. Many adjustments could be made already during the run of the course thanks to feedback from the opinion polls, but there is also room for improvement if/when the course is given next time.
StudentsThe course was attended by 6 MSc students, 4 of which dropped out at an early stage, and 10 PhD students, 2 of which also dropped out fairly early. In addition, there were 2 listeners who attended lectures but did not take the course for credit. The PhD students officially took the researchlevel course with course code FDD3501, but the requirements to pass were the same. The intention was that the PhD course should be graded pass/fail while the MSc course should have grades AF, but due to an administrative error it turned out that DD3501 had also been registered as pass/fail and apparently it was not possible to correct this afterwards.
Course goalsThis course was intended to
Actual course contentA fairly detailed description of the course content can be found in the schedule at www.csc.kth.se/~jakobn/teaching/proofcplx11/#schedule. After the second lecture, the course participants voted on how much connections to practical SAT solving they wanted (slightly more than proposed in the original course planning) and on what they thought about having guest lectures on SAT solving (there was strong support for this). During the early spring semester we arranged a round of voting on what topics to cover during the final third of the course (see www.csc.kth.se/~jakobn/teaching/proofcplx11/possiblethemes.php) and mostly followed the results of this voting. TeachingThe course was given in periods 2 and 3 during 2011/12. There were 11 lectures (of 2 x 45 minutes) in period 2 and 11 lectures in period 3. Out of the 22 lectures, there were a total of 4 guest lectures by
The students were very happy with the quality of the regular lectures. For the guest lectures the reviews were somewhat mixed (some were considered very good but some less so), but the students considered guest lectures as such to be a good idea. All lectures were given in English.
ExaminationThe course examination was done by 5 problem sets and by scribing of 12 lectures. Using problem sets is the traditional way of examining advanced TCS courses. The students found the problems challenging but interesting, and seemed mostly happy with this form of examination. A couple of students suggested that there should be more frequent and smaller problem sets rather than a few big ones. One slightly nonstandard feature of the problem sets was that we experimented with the possibility of "buying hints" for the problems. This setup was described as follows on the course webpage: "For most of the problems, it will be possible to purchase 'hints' for a certain deduction from the max score of the problem. In this way, you can configure yourself whether you want the problems to be more creative and openended, where sometimes a lot can depend on finding the right idea, or whether you want them to be more of guided exercises providing a useful workout on the concepts of proof complexity. Please feel free to use (or not use) this possibility to make the course more interesting and rewarding for you." This option was exercised for some problems by some students, but was not very frequently used. One comment in the course evaluation found that it was hard to know how to deal with the option of buying hints, although the idea as such seemed good. The decision to scribe lecture notes, and to have this as a form of examination, was taken after a vote by course participants after the second lecture on the course. An overwhelming majority of the students found the process of scribing fairly or very useful, and even more students found the notes very useful for purposes of selfstudy. One problem with having scribed lecture notes was that this turned out to be very labourintensive for the lecturer in the end, since even after a couple of iterations there were still many issues with the notes that needed to be fixed before the notes could be publicly published on the course webpage. For roughly the final third of lectures, the notes never made it into a polished enough state to publish, and were instead distributed informally to the students.
Course literatureThe course contents covered recent research in proof complexity, and there was no textbook, or even survey article, that could be used as course literature. Instead, lectures were based on research articles (published and in preparation) and as mentioned above students also used the scribe notes for selfstudy. PrerequisitesIn the prerequisites on the course webpage, it said that "[t]his course is open to anyone, but the main target audience are grad students and advanced undergrads. Some background in computational complexity theory and/or discrete mathematics will probably be helpful, but all that should really be needed is 'mathematical maturity' and a willingness to learn new stuff." Students found this to be a reasonable description.
Changes from last time the course was givenNot applicable — this was the first time the course was given.
Course evaluationThe course was evaluated by a course evaluation conducted after the course (to which 8 students answered, 7 of which had passed the course) as well as on a number of "opinion polls" conducted at various intervals during the (first half of) the course, namely on October 27, October 31, November 28, and December 5. These opinion polls were also used to adjust the course planning as we went along.The above documents contain a wealth of information concerning what students thought about the course. Below follows an attempt at a brief summary. (The percentages do not always round to 100% in the ACE evaluation tool used but have been left as computed by the tool.) 75% of respondents found the course hard or very hard, but 100% also found it fairly or very interesting. It was reasonably clear what the course goals and prerequisites were, and all students considered themselves to have the required prerequisites. 88% of students referred to the course webpage on a weekly basis or more often, and 100% found the webpages fairly or very good. 76% appreciated the course opinion polls and one commenter said it made the course feel configurable depending on student preferences, as the instructor had promised. Another comment was that it would have been good to have opinion polls also during the second half of the course (which we did not do, although we did have a vote on the content of the final third of the course as mentioned above). Lectures were wellattended — 75% of students attended more than 80% of the lectures. The number of lectures was considered to be appropriate. 88% found the regular (nonguest) lectures good or very good from a pedagogical point of view. 63% found the pace about right, 25% found it a bit too fast and 13% way too fast. Judging from the opinion polls, MSc students found the pace of the lectures a bit more challenging than the PhD students, which seems natural. Some lectures were given with slides but most were on the board. 50% wanted to keep it that way and 25% had no particular opinion (this is similar to the replies to the same question during an opinion poll barely halfway through the course). In the comments, the students especially liked brief recaps at the start of each lecture what we did last time. A summary of the lecture at the end would also have been nice, but often we struggled to find time for that. The students thought that it was good to have guest lectures. Matti Järvisalo got very good grades for his lecture; Chris Beck and Samuel Lundqvist slightly less so. The process of scribing lecture notes was very useful to 38% of the respondents and fairly useful to 50%. 75% found it very useful and 25% fairly useful to have the notes available for selfstudy. From the comments, however, it is clear that students found the task of scribing quite challenging, and one commenter raises the issue of whether the notes were worth the amount of time and work invested. The problem sets were considered to be fairly hard (63%) or very hard (38%). 63% considered the hardness level to be appropriate, however, while 25% found the problems a bit too hard and 13% far too hard. Again, MSc students appeared to feel that problems were harder than PhD students. The content of the problem sets was assessed to correspond well to the contents of the lectures. The number of problems and problem sets and the grading was "about right" according to a clear majority. The time given to solve the problems was about right or sometimes a bit too short (but we had votes about deadline extensions and extended accordingly when students so wished). The balance between theoretical and practical problems was about right. Most students assessed that they spent 3040 hours of work per problem set. One commenter suggested splitting the problem sets into more, smaller set. Two commenters would have wanted the first problem set to have appeared earlier in the course. 88% of the students spent 3050% of their total study/working time on the course while it was being given. 38% think that 9 ECTS credits was about right, but 63% feel that more than 9 credits would have been appropriate. 38% felt that a course like this should definitely be given on a regular basis (say, every second or third year) at KTH, and 25% thought that this should probably be the case. 25% were neutral and 13% thought that such a course should probably not be given on a regular basis. According to the replies in the course evaluation, most aspects of the course should stay the same if the course is to be given again. The same question was also asked in an opinion poll halfway through the course, and the replies there mentioned the scribing, the choice of topics, the quality and speed of presentation during lectures, and the problem sets were all aspects that should stay the same. As to suggestions for improvements, comments in the course evaluation mentioned that the number of ECTS credits could be increased. One commenter instead suggested lowering the ambitions of the course. One commenter was unhappy about the rules for the problem sets where students were allowed to work in pairs but not in larger groups. In the opinion poll halfway through the course, some suggestions for improvements were more frequent problem sets or fewer problems, recommendations on what to read before lectures, lowering the ambitions of the course a bit, and having less of recap at beginning of lectures (contradicting feedback from other students as mentioned above). Conclusions and future changesAll in all, this seems to have been a successful course. One central question, of course, is to what extent we could and should offer advanced, researchlevel courses at KTH CSC. But if we want MSc students to come into contact with research, which seems to be the officially stated goal, then that is an argument for giving this kind of courses. And given the result of the course evaluation and the opinion polls, it seems that things should be kept mostly the same if the course is to be given again. However, some changes should also be considered as follows:
One final remark is that the big challenge when organizing an advanced course like this is financing (and this holds true for other advanced TCS courses as well). This time, the course was paid for by the instructor's own research grants. It is not clear that this is a sustainable, longterm model of funding, but on the other hand it also seems hard to find money from the education budget for this kind of courses. This seems unfortunate, since one explicit goal of our education of MSc students is to get more connections to research in their courses, and this course was about as researchconnected as it gets.
