MULTI MODEL TRACKING OF MULTIPLE OBJECTS BASED ON SPEECH

Abstract

Object tracking has always been a very interesting and important research area in the field of Computer Vision and Artificial Intelligence. Tracking an object of interest is an application that can benefit from multiple sensing modalities. If the object of interest emits sound then information from both audio and video sensors can be fused together to remove effects of clutter and background noise. Therefore, the use of same visual and audio interface modalities that humans take for granted can make indoor spaces more intelligent. Audio and visual modalities complement each other when background noise impairs a single modality. This work presents a new approach for modeling and processing data from audio and visual sensors for tracking multiple objects simultaneously. This approach is based on graphical model for visual data and Time Delay of Arrival (TDOA) analysis for sound cue. Then both the cues are modeled by a data likelihood function. Finally, Particle Filtering for multiple target tracking and Dezert-Smarandache theory (DSmT) for fusing the information provided by audio-visual cues are combined. For modeling the visual cue, initially dominant motion is detected from the video frames. Then some dominant motion points are selected for depicting movements of target object. When some occlusion occurs, these motion points estimate the object position from a graphical model. As for modeling the sound cue, the Time Delay of Arrival (TDOA) which occurs between the two audio signals received by the two different microphones kept a fixed distance apart from each other provides an indication of the position of the sound source(s) relative to the microphone pair. This provides an estimate of the horizontal position of object in the image.