Interactive Segmentation in Space-time Memory Networks

The introduction of memory networks was a major development in Semi-automatic Video Object Segmentation. In all these networks, frames are encoded into an embedding space using a convolutional neural network. Due to this, the encoded features are abstract and cannot be changed easily afterwards. Modifying the initial selection while retaining information from the already made memory to help in segmentation was previously not possible. In this research, two novel methods were devised that allow for the combining of memories from pre- and post modification of the tracked object. This allows the user to define (partial) objects that can be removed or added to the initially selected object. One of the methods combines in the encoded feature space whereas the other combines in the probability space of the decoder outputs. Tested on a made data set, both methods were able to segment in specific cases where the unmodified network fails, although the larger memory of the methods generally did not result in more accurate segmentation. The first devised method proved too inconsistent for practical use but the second performed well. The validity of using memory from one stream of stereo vision to segment both streams was also tested. Although the made data set was limited in size, results indicate this is possible.