VINST/ Visual Diaries

What Omid saw on his way to work on 3 different days.

Background:

Recent developments in proccessing units, memory and storage devices and image processing and computer vision algorithms makes it possible to process large amount of data automatically: using a normal PC, it is possible to store more than a thousand hours of videos in storage devices and process millions of images to find similar images automatically in a second. Also, the mass production and cheap prices of cameras has made it possible to collect large amounts of images and videos without any significant cost.

Utilizing such developments, we decided to collect, store and process videos similar to visual input of some subjects and study to what end we can use such a data to help people with limited memories. In other words, our long term goal is to answer the question of

What is worth remembering in a data set of every day life videos?

Long term issues

An answer to this question requires studies about

How and what the human brain chooses to remember?
How is this memory selection process affected by what people tend to find interesting/relevant/critical to remember?
How do people develop these interests?
How does such a memory selection affect our lives?

We believe the research carried out in such a direction will be beneficial for understanding the human memory selection process and could also be utilized to develop applications that help the memory impaired.

What we have done:

Initial starting point

Prior to such studies and we made the simple observation that

We tend to summarize the repeated events/activities and memorize those which are less frequent.

We are therefore currently aiming to detect unusual/novel activities/events/things as a preliminary and a necessary step for memory selection.

Novel Ego-Motion Detection

Our initial work in this project has been directly motivated by this observation. One of the members of the group recorded his journey to work for two weeks with a wearable camera (see the videos at the top of the page). Each row in the figure below displays a highly sub-sampled snap short from one disinct day. Ten days are shown. Then the novel events which occurred during these journeys were automatically detected. Please visit our website Novel Ego-Motion Detection to read more about the details of this work.

Keywords:

Wearable Vision, Memory Selection, Memory Management, Novelty Detection

People:

KTH
Professor
Stefan Carlsson
Assistant Professor
Josephine Sullivan
PhD Student
Omid Aghazadeh

Publications:

	"Novelty detection from an Egocentric perspective" Omid Aghazadeh , Josephine Sullivan , Stefan Carlsson . CVPR 2011 PDF Poster
	"Multi View Registration for Novelty/Background Separation" Omid Aghazadeh , Josephine Sullivan , Stefan Carlsson . CVPR 2012 PDF Poster

Resources:

W31 Dataset: This data set consists of 31 videos capturing the visual experience of a subject walking from a metro station to work, subsampled at 1 HZ. It consists of 7236 images in total. Each image is annotated with a location id which covers 9 unique labels in total. Temporal segments corresponding to novel ego motions are annotated as well. Download the data set from here and the annotations from here.
W12 Dataset: This data set consists of multiple images of 12 different places a subject goes through when walking from a metro station to work. It consists of 142 images in total. Each image is annotated with a location id. Objects/people which do not belong to the static environment are labeled in each image (segmentation ground truth). The dataset and annotations can be downloaded from here.

Links:

Computer Vision and Active Perception Laboratory