DT2140 Multimodal interfaces

Lectures

The objective of the lectures is to give an overview of different human-computer interfaces that can be used to improve the interaction compared to the traditional keyboard-mouse input and computer screen output.

This is a preliminary overview of the lectures and seminars in the course.

Teachers: OE=Olov Engwall, AF=Anders Friberg, RB=Roberto Bresin, ELS=Eva-Lotta SallnŠs, AL=Anders Lundstršm.

Lectures and seminars are in Fantum, if not otherwise stated.

Lecture 1: Introduction to multimodal interfaces, Q2
OE
The lecture will cover multimodality in human-human communication. How are different modalities used to transfer information? What is gained by combining or alternating different modalities? Why should human multimodal communication be considered for computer interfaces?

Literature: Part marked with "Multimodal interfaces" in the compilation in bilda.kth.se

Lecture notes can be found on Bilda

Lecture 2: Mixed Reality,
AL
The concept of Mixed Reality is to merge real and virtual worlds to empower the human senses and increase the understanding of the surroundings. The lecture will give an overview of the challenges and of the enabling technologies. We will look at numerous research projects that realize Mixed Reality in various ways.

Literature: The part marked with "Mixed visual interfaces" in the compilation in bilda.kth.se

Lecture notes can be found on Bilda

Lecture 3: Tabletops, Tangibles and Tracking,
AL
The vision of interactive digital surfaces is to enable manipulation of digital objects just as we would manipulate real-world objects, using fingers, hands and touch. Such systems typically employ rear-projection display and sensing of the user's hands and objects on the surface, to allow interaction and to support the augmentation of real-world artifacts with associated digital data. This lecture will look at the underlying technology in a number of example systems, as well as recent commerical products that realize these concepts.

Literature: The part marked with "Mixed visual interfaces" in the compilation in bilda.kth.se

Lecture notes can be found on Bilda

Lecture 4: Multimodal speech technology interfaces,
OE
The basics of multimodal spoken dialogue systems, speech recognition and text-to-speech synthesis will be outlined. What are the benefits of using speech technology? What are the limitations and problems? What type of speech synthesis should be used in different interfaces? What performance can be expected from state-of-the-art speech recognition? How can talking heads be used to improve HCI?

Literature: The part marked with "Spoken interfaces" in the compilation in bilda.kth.se
Gustafson, J. (2002) Developing Multimodal Spoken Dialogue Systems - Empirical Studies of Spoken Human-Computer Interactions".

Lecture notes can be found on Bilda

Lecture 5: Gesture-based interfaces,
AF
Perception of motion, motor control, examples of gesture interfaces.

Literature: The part marked with "Gesture-based interfaces" in the compilation in bilda.kth.se

Lecture notes can be found on Bilda

Lecture 6: Sound in interaction,
RB
The main topic of the lecture is about the use of sound in human-machine interaction. In particular it will be discussed how sounds have the potential to modulate human engagement (from isolation to arousal) in everyday environments. It will be shown how important is interdisciplinarity in the field of sound and interaction and which are the different disciplines contributing to it. Examples will be provided of sounds in fundamental interaction phenomena, in sound-augmented artefacts, and in sound-mediated cooperative work. Examples of European research projects will be given.

Literature:
The part marked with "Sound in interaction" in the compilation in bilda.kth.se, i.e.
Rocchesso, D., & Bresin, R. (2007). Emerging sounds for disappearing computers. In Streitz, N., Kameas, A., & Mavrommati, I. (Eds.), The Disappearing Computer (pp. 233-254). Berlin Heidelberg: Springer.

Lecture notes can be found on Bilda

Lecture 7: Haptic interfaces,
ELS
The basics of tactile perception and cognition will be outlined. haptic feedback devices and application development will be discussed. What can be achieved by integrating haptic feedback in user interfaces and what aspects for design have to be considered? What are the benefits of using haptic and auditory feedback in combination? Where are we in haptic visualisation today? The front line in today's research on haptic visualisation will be discussed.

Examples of research about haptic feedback in four different application areas will be covered: collaborative interfaces, interfaces for the blind, supporting medical procedures and information perceptualisation. What consequences do multimodal interfaces have for distributed collaboration between people? How does multimodal feedback support complex problem solving in single user and collaborative contexts? What can be communicated by vision, audio and haptic feedback respectively and in combination.

Literature:
Jonathan C. Roberts and Sabrina Panëels. Where are we with haptic visualization? In WorldHaptics 2007, Second Joint EuroHaptics Conference, 2007 and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, pages 316-321, Tsukuba, Japan, March 2007.
Pages 47-60 and 63-75 in: SallnŠs, Eva-Lotta (2004). The effect of modality on social presence, presence and performance in collaborative virtual environments. Ph.D. thesis, TRITA-NA-0404. NADA, KTH, Sweden.

Lecture notes can be found on Bilda

Lecture 8: Issues in combining modalities in human-computer-interfaces,
OE
How should different modalities be combined? Should they be used simultaneously or alternatively? What makes an interface efficient and how do we know? How is information stress avoided? Can and should interface designers recreate the adaptibility of human communication regarding user, environment, task progress?

Literature:
The part marked with "Combining interfaces" in the compilation in bilda.kth.se
- Oviatt S. (1999). Ten myths of multimodal interaction
- Oviatt S., Coulston, R., Lunsford, R. (2004). When do we interact multimodally?: cognitive load and multimodal communication patterns.

Lecture notes can be found on Bilda

---

Course responsible: Olov Engwall, engwall@kth.se, 790 75 65