DT2140 Multimodal interfaces

Gesture and voice recognition control of Google Earth using Kinect	Multiplayer gesture-controlled computer game using fiducials
DJ board in ReacTIVision using fiducial markers	Virtual Theremin

Introduction

Multimodal interaction and interfaces is a course at the Department of Speech, Music and Hearing (TMH), CSC, worth 7.5 ECTS credits. It is given in period 2.

The course is focused on the interaction between humans and computers, using interfaces that employ several modalities and human senses, such as speech, gestures and touch.

The course will give the students theoretical and practical introductions to different types of HCI interfaces for
- user input, such as speech recognition, motor sensors or eye and gesture tracking, and
- computer output, such as augmented reality representations, speech synthesis, sounding objects and haptic devices.

In particular the effects of combining different modalities are addressed.

Aim

After completing the course, the students should be able to
* Describe the functionality of state-of-the-art multimodal HCI interfaces
* Evaluate the strengths and weaknesses of existing or proposed multimodal interfaces
* Implement interfaces employing new interaction techniques with different modalities

Content

The course is focused around a group project to create, analyze and/or evaluate a multimodal or innovative interface for a given task.

In order to prepare for the project, introductory lectures and laboratory exercises present different interface technologies, and home assignments are solved to provide an adequate background and planning.

Some recent projects are presented in video illustrations on vimeo).

---

Course responsible: Olov Engwall, engwall@kth.se, 790 75 65