TSC-DL: Unsupervised Segmentation of Surgical Demonstrations with Deep Learning

Adithyavairavan Murali, Animesh Garg, Sanjay Krishnan, Florian T. Pokorny, Pieter Abbeel, Trevor Darrell, Ken Goldberg
In IEEE ICRA, 2016


The growth of robot-assisted minimally invasive surgery has led to sizeable datasets of fixed-camera video and kinematic recordings of surgical subtasks. Temporal segmentation of these trajectories into meaningful contiguous sections is an important first step to facilitate human training and the automation of subtasks. Manual, or supervised, segmentation can be prone to error and impractical for large datasets. We present Transition State Clustering with Deep Learning (TSC-DL), a new unsupervised algorithm that leverages video and kinematic data for task-level segmentation, and finds regions of the visual feature space that mark transition events using features constructed from layers of pre-trained image classification Convolutional Neural Networks (CNNs). We report results on five datasets comparing architectures (AlexNet and VGG), choice of convolutional layer, dimensionality reduction techniques, visual encoding, and the use of Scale Invariant Feature Transforms (SIFT). TSC-DL matches manual annotations with up-to 0.806 Normalized Mutual Information (NMI). We also found that using both kinematics and visual data results in increases of up-to 0.215 NMI compared to using kinematics alone. We also present cases where TSC-DL discovers human annotator errors. Supplementary material, data and code is available at: http://berkeleyautomation.github.io/tsc-dl/


Download this publication


@inproceedings{murali2016a, title={TSC-DL: Unsupervised Segmentation of Surgical Demonstrations with Deep Learning}, author={Murali, Adithyavairavan and Garg, Animesh and Krishnan, Sanjay and Pokorny, Florian T. and Abbeel, Pieter and Darrell, Trevor and Goldberg, Ken}, booktitle = {IEEE ICRA}, url={http://goldberg.berkeley.edu/pubs/icra16-submitted-TSC-DL-Segmentation.pdf}, year = {2016}, }