Course Analysis, 2D1380 Artificial intelligence, 4p

Course given 2006/2007 in study period 1

  • Course responsible:
    Patric Jensfelt

  • Lecturers:
    Patric Jensfelt, Danica Kragic

  • Lectures:
    12 lectures (24 hours) and 10 hours of for questions

  • Registered students:
      In total, 83 undergraduate students registered for the course, but 5 of these did not show up at all leaving 78 students starting the course.

  • Course material:
    Book:
    Artificial Intelligence: A Modern Approach (Second Edition) by Stuart J. Russell and Peter Norvig, Prentice Hall (2003), ISBN 0-13790-395-2
    All lectures were accessible for download.
    2 project descriptions and three homwork were posted on the web.

  • Examination:
    Assesment consisted of 3 homeworks (1p) and one project (3p). There was a deadline for the homeworks and the projects announced at the beginning of the course. Homework assignments were graded (0,1,2, 3,4,5) and together they accounted for 50% of the final course grade. The homeworks had to be completed individually and handed in on time. Homework received too late were be graded at all and received grade 0.

  • Course results:
    74 students completed all the neccessary parts by the end of the course and passed the course.
    2 students failed to complete the course
    2 students left the course half way through because they did not have enough time

  • "Prestationsgrad":
    (74*4)/(78*4) = 95%

  • "Examinationsgrad":
    74/78=95%

  • Relation to the previous years:
    The course was given for the second time but the lecturers changed since the first year. Henrik Christensen gave 6 of last years 12 lectures and this year these were taken over by Patric Jensfelt. In response to student requests from last year the part on using Bayes theory etc was expanded. This meant that Danica Kragic had to to restructure her lectures completely. The lectures on robotics and perception that was given 4 hours last year was now only given one lecture of 2h. Students last year thought that the course was too easy and therefore we added one more homework and also included problems to two of them that included implementing something. The idea was to get students to work with the material more hand on. A big change from last year was that all students were encouraged to select the same project. Last year we had a wide selection of them but that made it very difficult to compare them for the grading. We instead organized a competition between agents playing minesweep. As last year, the aim of the course was to give a broad overview of the problems and methods studied in the field of artificial intelligence to enable students to develop systems that utilize artificial intelligence.

  • Summary:
    12 lectures were held and most of them were followed by more than 60% of students. During the lectures, students were posing questions but more interaction has to be encouraged next year. Students were supposed to read the book on their own in parallel with lectures.

    Many of the lectures were new or given by a new lecturer this year.

    Three homework assignments were given out during the course. The last two of them contained problems which required the students to make an implementation. This had to be completed to have a chance at grade 5. The homeworks were completed individually except for the implementation paorblems that could be worked on in groups of 2 students.

    The minesweep project was selected by all but 2 students. The examination consited of an implementation of an agent that could play minesweep, a report and a presentation. The minesweep competition was organized for the first time and all students was that it was a great or at least a good idea. Things to think about for next year is to make sure that the students have agents that work onthe hardware and setup available in the computer rooms where the competition is held. It would have saved a lot of time during the actual competition. All students showed up for the presentations.


  • Results of students evaluation:
    26 students filled in the evaluation.

  • Course level:
    Most students think that the course level was medium. Since it is an introductory course, they felt that their prior knowledge was suitable to follow the course.

  • Planned changes:
    The questions in the homework have to be posed even more clearly. Look at the schedule for homework hand-ins to give a bit more time for the project. It is difficult though as there are great benefits in having the course finish before the next period starts.

    Questions to answer


  • How difficult was this course?

    1. 8% (2 st) Easy.
    2. 88% (23 st) Medium.
    3. 4% (1 st) Diffucult.


  • Was the aim of the course clear from the beginning?

    1. 81% (21 st) Yes.
    2. 19% (5 st) Not sure.
    3. 0% (0 st) No.


  • Was the course interesting and meaningful?

    1. 38% (10 st) Very much.
    2. 50% (13 st) Yes.
    3. 12% (3 st) Neutral.
    4. 0% (0 st) Not particularly.
    5. 0% (0 st) No.


  • Prerequisites for the course were courses in numerics and statistics. Do you think that your prior knowledge on this topics was enough to follow the course?

    1. 81% (21 st) Yes.
    2. 12% (3 st) Not sure.
    3. 8% (2 st) No.


  • Do you find course book by Russell and Norvig suitable?

    1. 42% (11 st) Yes.
    2. 15% (4 st) Not sure.
    3. 4% (1 st) No.
    4. 0% (0 st) Bought it but did not read it.
    5. 38% (10 st) Did not buy the book.

    Additional comments about the book:

    Good, although we only learnt a part of it.
    ---
    The book is quite good, but a little too lengthy now and then.
    ---
    Incredibly expensive, and not really necessary to follow the course.
    ---
    If according to you the book is useful you perhaps should tell the students which chapters or pages are the most important for the course and which one are in direct connection with such a lecture.
    ---
    Intressant och relativt lättläst bok i ämnet.
    ---
    I found the book very helpful throughout the course, especially during the probability section.
    ---
    boken var bra. Lättförståelig text med bra förklaringar och lagom pratig. Kul med skämt då och då ;-)
    ---
    Boken passade föreläsningarna, eller rättare sagt, det kändes mer som om de tidigare föreläsningarna passade boken väldigt bra, vilket gjorde det lätt att se vilka delar av boken som var mer relevanta för den här kursen. Som vanligt innehöll den mycket mer och var mycket djupare än vad vi hann gå in på.
    ---
    A too big book for this short period. Probably shortened summaries would be better. It is also very expensive.
    ---
    Because of the interesting contents of the course, I will buy the book to have as a reference for future work.
    ---
    One of the best course books I've had...
    ---
    Not a great boook. I agree it covers a lot of topics inthe course and maybe that's why it was chosen, but still it uses too many words to explain simple things and that too not-too-well.


  • How many lectures did you attend?

    1. 12% (3 st) Less than 30%.
    2. 19% (5 st) 30-60%.
    3. 19% (5 st) 60-80%.
    4. 50% (13 st) More than 80%.


  • What do you think about the quality of teaching? Were the topics clearly presented?

    1. 8% (2 st) Very good
    2. 62% (16 st) Good.
    3. 19% (5 st) Acceptable.
    4. 12% (3 st) Not so good.
    5. 0% (0 st) Bad.
    6. 0% (0 st) Did not attend.

    Additional comments - please, give us constructive feedback:

    Good, could've been nice with more demonstrations etc. like in the last lectures.
    ---
    Only attended Patrics lectures, but they were fine.
    ---
    Sometimes there was too many slides
    ---
    I find that there was a break between the courses given by the both teachers. The link between them was not enough clear according to me.
    ---
    Danica, dina föreläsningar om HMM var jättebra, det är svårt men jag tyckte du förklarade det bra.
    ---
    Patric has a tendency to re-explain what was said in the book, using the same examples. While this is helpful to understand the book, it would be nice to not have the concepts explained twice in the same way.
    Also, we spent at least the equivalent of a full lecture going over the problems that individual students were having. It would have been more beneficial to take care of those issues outside of lecture, instead of wasting the time of the other thirty students who had different problems.
    Both instructors were excellent when presenting material directly related to their fields (robotics and vision).

    ---
    Most was good from Patric, got nothing to comment there really. (read about your fair-drawing robot in ny teknik, good luck with that!)

    Danica did many things good also, but should have laid up the HMM lectures better. The subject should have had more examples and explanations of different things HMMs are used for. Why would we want to calculate the normalized joint probability of jointed probabilities jointed together, i never really understood during the lectures?? Give examples but more importantly explain why its interresting and useful.., dont throw formulas as the forward/backward algorithm as a solution to a problem that most (at least a few more than myself) didnt understand and/or why it should be solved.

    Also, felt like a bit of wasted time with lots of talk about homework 3. I agree it needed more explanation but.., felt like it took away two lectures that could have been spent better.

    ---
    The lectures about HMM weren't very good. Danica spent to much time covering the basics (markov models) and asking us if we understood - which at least I did - and thus lost focus on the relevant material.

    It would have, IMO, been more appropriate for her just to assume we know the basics and carry on with the material novel to us, i.e. the actual course content.

    Futhermore, I believe she spent far too much time covering the homework (homework 3) - questions and unclarities about it would have been much better handeled individually or posted on the web site (personally, I hadn't started on the homework by the time she spent more than an hour talking about it - thus any benefit for me was forfeit and the lecture time wasted).

    ---
    Teaching was quite good overall and easy to follow. A lot of effort was put in by the teacher which showed.
    One comment: I felt Danika was getting too defensive about the fact that she was teaching us a 'difficult' topic and spent a lot of lecture time catering to rather individual problems and justifying stuff. Yes it was slightly tough, but not THAT tough, and I felt students are lazy bums anyway, so if you want us to learn and really do something, you have to start by being stern and definite about it from the very start. I really admire her for her patience, flexibility and all the help she offered all the time, and I appreciate that since it was also needed. just that I'd wish she was bit more stern, though equally helpful, in the class. It helps students sit up and take things seriously from the beginning. We are in the class to learn, and learn an important thing. If it's tough, so what? We are responsible to put in proportional effort.


  • Did you read the relevant book chapters before they were presented at the lectures?

    1. 4% (1 st) Yes, always.
    2. 15% (4 st) Sometimes.
    3. 19% (5 st) Rearly.
    4. 62% (16 st) Never.


  • Did you think that the homeworks helped you learn the material in the course?

    1. 73% (19 st) Yes very much.
    2. 23% (6 st) Yes.
    3. 4% (1 st) Not really.
    4. 0% (0 st) Just a waste of time.

  • How much time did you spend on homework 1?

    1. 12% (3 st) Less than 2 hours.
    2. 58% (15 st) 2-5 hours.
    3. 31% (8 st) More than 5 hours.

    Comments about homework 1:

    The simplest homework. It was the direct application of the course.
    ---
    Some really ambigious questions. generally, HW1 is reasonable.
    ---
    This was a good and not too hard introduction to agent design and basic algorithms involved. Great as a first homework.
    ---
    Not as interesting as hw 2 and 3
    ---
    Easy, interesting.
    ---
    it is okay to warm up :)


  • How much time did you spend on homework 2?

    1. 4% (1 st) Less than 2 hours.
    2. 31% (8 st) 2-5 hours.
    3. 65% (17 st) More than 5 hours.

    Comments about homework 2:

    Nice to actually get to implement the stuff you've learnt.
    ---
    More stronger than the first. Very interesting, above all the last part which demanded a real involvement for the student, but relaively simple to solve thanks to the help that we can find on the internet, because it was classic subjects.
    ---
    Many ambigious questions in this one. Generally, HW2 is considered rather tough. Part C and D, were difficult to answer, not because I do not know the answers, but rahter more because I cannot understand what the question was asking.
    ---
    I enjoyed the n-queens assignment and feel that working on it was not only helpful in general, but also as a preparation for the group assignment.
    ---
    This homework was somewhat confusing. The logic part (with the family relations) was hard to understand.
    ---
    Det märks verkligen att ni lägger ner tid på att formulera hemuppgifterna. Just den här kanske var lite väl omfattande?
    ---
    My favorite homework because of the implementation challenge. It was nice to have a part of it dedicated to get an hands-on experience from using the CSP algorithms.
    ---
    Fun to do. A little tricky, esp the wordings of the questions. Thought the last section was a bit too much work to do.
    ---
    the last part (sudoku, 8queens implementation) should be more guided.
    i mean, students should know a bit more about what is required, or what is enough to get a good mark.


  • How much time did you spend on homework 3?:

    1. 0% (0 st) Less than 2 hours.
    2. 19% (5 st) 2-5 hours.
    3. 81% (21 st) More than 5 hours.

    Comments about homework 3:

    Homework 3 was significantly more difficult and demanding than the first two ones. I spent more time on homework 3 than on the project. This is one reason to question the course grading system.
    ---
    I find that the last part was very difficult, even if we had more time than for the others. It was not easy to solve the problem.
    ---
    I did this homework in less than 3 hours only because I did not do the final implementation part. That would have taken far too much time considering it is end of study period and work is piling up. I personally find this homework too tough to be finished reasonably.
    ---
    My statistics background was not high enough for this assignment.
    ---
    to hard!
    ---
    fun and challenging compared to the other homeworks (sudoku solver was fun as well). The work spent on it and the work required in being able to understand it were a lot more than the others so i think you will need to tweak that a bit...
    ---
    Implementing homework 3 was hard since the different papers provided different formulas and notations for the normalized algorithms. The short paper failed to mention the n_k factor, for example. The comprehensive one didn't even mention it and used totally different algorithms that weren't quite as easy to implement.

    Also, it wasn't clear that we needed normalization at all (our observation sequences were short, less than 10 observations) - the papers were particularly fuzzy about this.

    ---
    Vissa frågor var konstigt ställda. Följdfrågor var baserade på att man hade svarat på en tidigare på ett visst sätt, annars passade de inte riktigt in.
    ---
    Very time consuming in a period where I didn't have so much time to do it. So I had to skip part D without thinking about it.
    ---
    The theoretical part of this homework was fine and could be done with some help from the lecturers on clearing out some confusions. The implementation part felt a bit too hard to do, and involved lots of tricky maths and calculations that I had some hard time to tackle.
    ---
    I spent a lot of time on this one! I enjoyed learning the subject of HMMs, especially since it feels very central and applicable. This was the part of the course I enjoyed the most. I feel that I really learned something that I can use.
    ---
    tough job the last part. initial sections were interesting.
    ---
    in my opinion, we lack one of the most helpful (and time-saving) things: an example.
    when one has written a function that performs a large computation, but its outcome doesn't seem to be logical, it is frustrating to try to guess in which of its parts the mistake was: incorrect data preprocessing, any of the computation steps, ...
    i of course talk about the baum-welch method, as it uses the normalized alfas and betas, as well as the auxiliar gamma, eta, etc.. i experienced underflow problems after a few iterations, and spotting the errors was so hard, as i couldn't be sure about the correct operation of the sub-functions :S
    such a simple thing turns a 5 hours interesting task into a 3 days interesting, but extremely tiring, task

    anyway i must say the homework helps a lot to understand how these estimation algorithms work :)

    ---
    the final part is a little difficult to do
    ---
    We spent around 40 hours on homework 3.


  • How much time did you spend on the project?

    1. 0% (0 st) Less than 10 hours.
    2. 27% (7 st) 10-20 hours.
    3. 69% (18 st) More than 20 hours.

    Comments about the project:

    Interesting and thought provoking.
    ---
    Interesting to work with other people, it was a mean to better know who follow ths same course than us. About the subject (minesweeper for me) it was funny to get involve in the game for the competition.
    ---
    A little weird to make MineSweeper into a 2 player game.
    ---
    I liked it.
    ---
    Good project! Keept it!
    ---
    Fun with the competition, makes you work harder.
    ---
    det var för många med 5 i gruppen...
    ---
    Projektet var väldigt roligt, men det hade varit bra med lite fler hjälptillfällen så att man kunde fått feedback från kursledarna. Det kändes även som att man inte hann göra det man ville med projektet då hemläxorna tog lång tid. Det hade också varit roligare om man fått höra mer om hur andra grupper löst problemet. Det kanske finns andra sätt att redovisa om alla har samma problem? Tvärgrupper kanske är en idé? Ett annat förslag är att man slumpar ut vissa grupper som får redovisa inför alla/mindre grupper medan andra får ha samtal med kursledarna där lösning osv. motiveras.
    ---
    Mycket roligt projekt och jag hade tur med mina projektmedarbetare, som hade liknande ambitioner med projektet.
    ---
    Very interesting but I would have needed more time.
    ---
    Really interesting project (minesweeper) that combined the two major parts of the course and ended in a fitting roundup with the competition.
    ---
    It was really fun! I wish, however, that there would have been a little more time for the report. We didn't start to write it until the day before...It would really have improved the quality of the report if we had been given an extra 2 days or something.
    ---
    Fun!
    ---
    nice idea! minesweeper is a pretty good project to apply the concepts of the course.

    i think time should also be graded, specially in problems that can imply search strategies in such a big tree.


  • Did you attend the minesweep competition?

    1. 92% (24 st) Yes.
    2. 8% (2 st) No.

  • Was it a good idea to have a competition in minesweep?

    1. 85% (22 st) Yes, great idea.
    2. 15% (4 st) Yes.
    3. 0% (0 st) Not really.
    4. 0% (0 st) Just a waste of time.

    Suggested changes for minesweep:

    The server should have been able to handle more parts of the competition automatically.
    ---
    Perhaps a little more formal information regarding rules and behaviour of the game would be nice? It was pretty annoying to not really know what to expect from the competition mode. Other than that, no problems. It was great fun.
    ---
    We should perhaps have a more scheduled timetable because we spend a lot of time in the room just to wait that our opponent be free to play.
    ---
    Sätt en gräns för kvalificieringen, typ har man inte klarat qual-batch på 10min åker man ut. Tog för lång tid, annars kul. Lite mer avancerade regler skulle kunna göra det roligare för de som vill satsa lite mer typ passa ett drag per spel odyl.
    ---
    Better density value and/or rules for the tournaments.
    ---
    Document it a little better? Not everyone of us is Java expert and there was some difficulty in reading the codes to understand what it is doing in such a short span of time.
    ---
    It would have been very helpful to know how to use the minesweeper server and play 2-agent games before the day of the competition.
    ---
    be able to play against last year winner
    ---
    - better density choices, random gave many uninteresting boards.
    - another way to judge those that takes too long time.
    - make the start less dominating for the score.

    ---
    It would have been better if all the groups were able to finish the qualification round in a shorter period of time.. Also the rules and procuders were not really clear(at least for me.)
    ---
    Less buggy server.

    Stricter time limits - 5 minutes (?) is A LOT!

    ---
    Kanske går att snabba upp tävlingen lite?
    ---
    Automatisera hela tävlingen. Låt grupperna följa ett gränssnitt för hur man startar och stänger av programmen. Sedan kan man skicka in sina bidrag i början av en viss dag och sedan bara köra qualify och större gruppspel och finalspel och ha automatisk uppdatering av poäng under dagen, så slipper det bli trångt i en sal och man slipper hantera papper.
    ---
    a more formal specification of the rules would be nice, as one knows exactly what is expected from the agents performance
    ---
    Post this years winning client (the .class files) so that one can try out one's agent against last year's winner.


  • How many other courses did you attend in parallel to this course?

    1. 19% (5 st) One.
    2. 42% (11 st) Two .
    3. 27% (7 st) Three.
    4. 12% (3 st) More than three.


  • This is a 4p course. Compared to other similar courses, is 4p suitable?

    1. 85% (22 st) 4p is OK.
    2. 0% (0 st) Should be less than 4p.
    3. 8% (2 st) Should be more than 4p.


    things you liked/appreciated about the course:

    Teaches the fundamentals of AI. Got more knowledge of it, which also developed more interest in the subject.
    ---
    Interesting topics and ideas. HMM theory is very frequent, so that's a good aqcuintance.
    ---
    It was fun to get some insight into the AI field. I particularly enjoyed the parts that brushed on machine learning and gave more focus on the 'I' part of AI.
    ---
    I liked the part about robotic and perception at the end, because I'm inteteresting with cognitive science and human machine interaction.
    ---
    Bra exeminationsform, intressanta föreläsningar lagom svår.
    ---
    Teaching standard vary from class to class, it can be good on one day and boring on another day. For a new class with relatively young teaching staff, it is more than acceptable.
    ---
    Interesting programming assignments/project. Understandable book. Very interesting subject matter. Capable and helpful instructors.
    ---
    Many topics covered and felt like a good overview. Most lectures and homeworks held a good quality. The project was very nice as well.
    ---
    Rolig kurs som ger en bred introduktion till ämnet. Minesweeptävlingen var ett jätteroligt initiativ.
    ---
    Ämnena som gicks igenom var relevanta och ungefär så djupa som man har tid att gå igenom.

    Minesweeperprojektet och Homework 3 var bra för att få implementera de delar i kursen man har lärt sig.

    ---
    The project.
    ---
    I felt that both teachers was really dedicated to the course and knew a lot about the subject (or at least gave that impression =). The times I asked for help I really felt that I got it.
    ---
    Teachers; interesting slides/videos.
    ---
    . it was interesting, and the project quite stimulating
    . the relationship with the teachers was very good, more friendly than one usually expects

    ---
    the lecture is very compehensible


    things you would like to see changed next year:

    Nothing really, maybe some more fun videos/programs as examples during the lectures.
    ---
    The grading system is a mess. It should be changed to reflect the actual balance in the course. Information about the course was available on the homepage, but often difficult to find. It was evident that two people updated it. It should have been better structured.
    ---
    The project should be given less weight in the grading. It's really rough to know that your grade depends on your team-mates, more than it depends on yourself. Also, it was a bit of a bummer to find out so late that extra attention might be paid to good homework results when grading, when many of us have done homeworks tactically to get just the right amount of points for the grade we want.
    You should also consider being a bit more formal and strict in your material (lectures, notes and homeworks). Go right ahead and throw definitions, theorems and proofs at us, we're all used to it and it will, if done correctly, remove a lot of ambiguity in the material.
    Another thing that the course might benefit from is taking a closer look at what courses students have taken previously. You could probably fit more material into the course by spending less time repeating stuff that we already know (or should know) well, such as fundamental logic and search algorithms.

    ---
    The part about Markov Model was sizeable and I found that it was difficult to do the homework 3 with only the lecture. I know that there were some further information given for that. But I was not enough clear for me.
    ---
    En fortsättningskurs vore intressant.
    ---
    My main concern about this class is the way questions are asked in the homeworks. It makes the questions a lot tougher than they are.
    ---
    Please be more explicit about the statistics prerequisites on the study handbook page and at the beginning of the course. Use words like "Markov Models" to describe what we should know going in.
    When signing up for course, it is difficult to tell whether prerequisites are listed because the courses are in a sequence, even though the material is unused, or because the material is heavily relied upon, as it was in this course.
    This is especially true for the exchange students who are attempting to decide what their equivalent course history is.

    ---
    Homework 3 a bit and better presented HMMs.
    ---
    See the comments on HMMs.

    If a more focused approach to HMMs isn't viable, perhaps more time should be devoted to it as it's a very interesting technique.

    ---
    Den här kursen innehöll väldigt mycket arbetstunga uppgifter - kanske kan vara idé att minska lite på arbetsbördan/utöka antalet poäng?
    ---
    More lectures in the beginning of the period so you have more time for the project. You should also probably shorten the homework a little (less parts in each homework).
    ---
    Personally, I prefer a written exam. You always learn more with a written exam. The best setup would be an exam after 70% of the time and then the project the rest of the time.
    ---
    Less time on non-direct topics like perception. It's good but could be done faster. Include some more AI techniques
    ---
    . leave more time for the project! (or, what is almost the same, reduce the homeworks level of demand)
    ---
    nothing