Human Action Recognition in realistic scenarios

BACK
In this project we are working on Automoatic Recognition of Human Actions in Realistic Scenarios using Contextual Priors. We extract motion features (capturing movements) as well as appearance features (capturing contextual structures). The goal is to combine information coming from actor bounding box with information coming from scene context in an optimal way and empirically study relative contribution of each information channel in recognition of several different action classes. Evaluations are done on HOHA2 dataset.

Proposal of the project as well as other relevant documents can be found below:
  • Literature survey (PPT)
  • Project proposal (PDF)
  • Project reports
    • Report #1 (PDF)
    • Report #2 (PDF)
    • Complementary report (baseline evaluation) (PDF)
  • Plans for the next milestone:
    • How to evaluate and compare results? (PDF)
Figure 1 illustrates different approaches which are evaluated, compared, and analyzed in this project. In fact, the goal is to classify the clip with label of the action of interest (yellow part). However, due to noisy alignment of video clips in HOHA2 there are some heading/trailing frames around the action of interest (gray frames).
For all of the approaches showed in Figure 1 we extract HOG, HOF, HOGHOF, and SIFT features and combine them using MKL. After all, we show that Approach-III outperforms all of the other methods which demonstrates usefulness of visual contextual features in recognition of human actions in realistic scenarios.
Figure 1. Overall scheme of five different methods for classification of human actions. At the left side a sample video clip is showed. Part of the clip showed in yellow corresponds to a temporally well-aligned period of time in which an action of interest is taking place. "A" represents Actor bounding box and "C" represents Context.
We have also annotated HOHA2 dataset by actor bounding boxes. Some examples from the annotations are listed in the following:

Acknowledgement: Special thanks go to Arnaud Ramey for his great job in developing the annotation tool.
Related publications: