EACare: Embodied Agent to support elderly mental wellbeing

The main goal of this multidisciplinary project is to develop a robot head with communicative skills capable of interacting with elderly people at their convenience.
The robot will be able to analyze their mental and psychological status via powerful audiovisual sensing and assessing their mental abilities to identify subjects in high risk or possibly at the first stages of depressive or dementing disorders, who would benefit from proper medical consultation. The framework will also provide tools for dementia preventive training.
We propose an ambitious, innovative and integrated approach that combines:
  • an easy-to-use patient-friendly and caregiver-friendly embodied agent to interact with patients via both verbal and non-verbal communication, including dialogue especially designed by psychiatrists to probe, measure, test, and monitor the patients mental state,
  • pioneering technological solutions at both the algorithmic and system level for multimodal information processing to achieve audiovisual human-computer interaction including action recognition, aural and facial expressions, and speech understanding,
  • systematic evaluation of the developed systems in both large and focused social groups, using feedback from each evaluation to guide further scientific research and design.


Department of Robotics, Perception and Learning (RPL)
KTH Royal Institute of Technology, Sweden
Project Director: Hedvig Kjellström
Department of Speech, Music and Hearing (TMH)
KTH Royal Institute of Technology, Sweden
Principal Investigator: Joakim Gustafson
Principal Investigator: Jonas Beskow
Department of Neurobiology, Care Sciences and Society (NVS)
Karolinska Institute, Sweden
Principal Investigator: Miia Kivipelto


Robot kan upptäcka demens tidigt, Framtidens Forskning, June, 2020.

Sverige behöver satsa på AI, IVA Aktuellt, 2:18-23, April, 2018.

Sweden: 'Social Robot' designed to recognise very early stages of Alzheimers, Ruptly TV, August, 2016.

Social robot ska kunna upptäcka demens tidigare, Dagens Nyheter, July, 2016.


K Håkansson, J Beskow, H Kjellström, J Gustafsson, A Bonnard, M Rydén, S Stormoen, G Hagman, U Akenine, K Morales Peres, G Henter, M Sundström, and M Kivipelto. Robot-assisted detection of subclinical dementia: progress report and preliminary findings. Alzheimer's Association International Conference (abstract), 2020.
Data collection and progress report

O Mikheeva, I Kazlauskaite, H Kjellström, and C H Ek. Bayesian nonparametric shared multi-sequence time series segmentation, arXiv:2001.09886, 2020.
Machine learning

T Kucherenko, P Jonell, S van Waveren, G Henter, S Alexanderson, I Leite, and H Kjellström. Gesticulator: A framework for semantically-aware speech-driven gesture generation, arXiv:2001.09326, 2020.
Non-verbal behavior production

T Kucherenko, D Hasegawa, G E Henter, N Kaneko, and H Kjellströom. Analyzing input and output representations for speech-driven gesture generation, ACM Intelligent Virtual Agents Conference, 2019.
Non-verbal behavior production

P Jonell. Using social and physiological signals for user adaptation in conversational agents. In International Conference on Autonomous Agents and Multiagent Systems, Doctoral Consortium, 2019.
Dialogue system design

T Kucherenko, D Hasegawa, N Kaneko, G E Henter, and H Kjellström. On the importance of representations for speech-driven gesture generation. In International Conference on Autonomous Agents and Multiagent Systems, Extended Abstracts, 2019.
Non-verbal behavior production

P Jonell, M Bystedt, F I Dogan, P Fallgren, J Ivarsson, M Slukova, U Wennberg, J Lopes, J Boye and G Skantze. Fantom: A crowd-sourced social chatbot using an evolving dialog graph. In 2nd Proceedings of Alexa Prize, 2018.
Dialogue system design

P Jonell, C Oertel, D Kontogiorgos, J Beskow, and J Gustafson. Crowd-sourced Multimodal Corpora Collection Tool. In LREC, 2018.
Data collection

T Kucherenko. Data driven non-verbal behavior generation for humanoid robots. In International Conference on Multimodal Interfaces, Doctoral Symposium, 2018.
Non-verbal behavior production

T Ngandu, F Mangialasche, and M Kivipelto. Lifestyle interventions to prevent cognitive impairment and dementia. Nature Neurology, (in press).
Clinical geriatrics

K Håkansson, T Ngandu, and M Kivipelto. The patient with cognitive impairment. In V Hachinsky (Ed), Treatable and Potentially Preventable Dementias, Cambridge University Press, 2018.
Clinical geriatrics

P Jonell, M Bystedt, P Fallgren, D Kontogiorgos, J Lopes, Z Malisz, S Mascarenhas, C Oertel, E Raveh, and T Shore. ARMI: An architecture for recording multimodal interactions. In Language Resources and Evaluation Conference, 2018.
Dialogue system design

P Jonell, C Oertel, D Kontogiorgos, J Beskow, and J Gustafson. Crowdsourced multimodal corpora collection tool. In Language Resources and Evaluation Conference, 2018.
Dialogue system design

O Mikheeva, C H Ek, and H Kjellström. Perceptual facial expression representation. In IEEE International Conference on Automatic Face and Gesture Recognition, 2018.
Machine learning

M Kivipelto and K Håkansson. A rare success against Alzheimer's. Scientific American (April):32-37, 2017.
Clinical geriatrics

P Jonell, J Mendelson, T Storskog, G Hagman, P Östberg, I Leite, T Kucherenko, O Mikheeva, U Akenine, V Jelic, A Solomon, J Beskow, J Gustafson, M Kivipelto, and H Kjellström. Machine Learning and Social Robotics for Detecting Early Signs of Dementia, arXiv preprint arXiv:1709.01613, 2017.
Position paper

P Jonell, C Oertel, D Kontogiorgos, J Beskow, and J Gustafson. Crowd-powered design of virtual attentive listeners. In International Conference on Intelligent Virtual Agents, 2017.
Dialogue system design

C Oertel, P Jonell, D Kontogiorgos, J Mendelson, J Beskow, and J Gustafson. Crowd-sourced design of artificial attentive listeners. In INTERSPEECH: Situated Interaction, 2017.
Dialogue system design

C Oertel, P Jonell, K El Addad, E Szekely, and J Gustafson. Using crowd-sourcing for the design of listening agents: Challenges and opportunities. In ACM SIGCHI International Workshop on Investigating Social Interactions with Artificial Agents, 2017.
Dialogue system design

K Håkansson, A Ledreux, K Daffner, Y Terjestam, P Bergman, R Carlsson, M Kivipelto, B Winblad, A-C Granholm, A K H Mohammed. BDNF responses in healthy older persons to 35 minutes of physical exercise, cognitive training, and mindfulness: Associations with working memory function. Journal of Alzheimer's Disease. 55(2):645-657, 2016.
Clinical geriatrics