Global spatiotemporal representations

and feature extraction in video sequences

ABSTRACT

This research evaluates the global motion representations of video sequences based on optical flow and motion history methods. The two most important motion features in a scene, moving speed and moving direction, are extracted from the spatiotemporal representation and are used to evaluate the performance of the representation. The speed feature is defined by the mean of foreground magnitudes and the direction feature is given by the entropy of directional angles for all pixels in the scene image.

    The optical flow techniques evaluated include the Horn-Schunck (H-S) and Lucas-Kanade (L-K) differential methods. They allow the direct extraction of speed and direction information of individual pixels, but cannot describe the complete cycle of an activity. The motion history techniques evaluated include Motion History Image (MHI) and exponential MHI. They do not give explicit motion features of speed and directions, but they can well represent the whole cycle of an activity. A hybrid spatiotemporal representation that incorporates the advantages of both optical flow and motion history is also proposed in this study. The applications of the motion representations and their extracted motion features for radical event detection and activity classification are demonstrated in this study.

    The video sequences with increasing speeds of movement and increasing complexity of moving directions and the public BEHAVE, Weizmann and KTH activity datasets are used for the test. Experimental results show the optical flow techniques can well describe speed and direction over consecutive images in the video and motion history techniques can better represent motion patterns and are good for activity recognition. The proposed hybrid representation gives overall the best performance.

Keywords: Motion representation; Motion features; Optical flow; Motion history image; Activity recognition