Tracking Multiple Instances of Retail Consumers from RGB and Thermal Images ^†

Abstract

Visual tracking of multiple objects is a challenging task that has gained considerable attention recently. The process involves several steps: detection, data association, and re-identification, each with its own set of challenges. Trending methods usually combine some of these steps and report impressive results in tracking benchmarks while running at practical speeds. In this work, we develop an online, real-time multi-object tracking approach that is also able to perform consumer tracking in indoor retail stores, without further tuning its design philosophy. Our model can utilize multi-modal inputs, such as RGB and thermal images when available, to improve its performance while maintaining acceptable speeds at 20FPS. Our key contribution to motion forecasting replaces the standard Kalman filters with an LSTM network to properly model long-term dependencies and target tracking. Numerical experiments of our method on the widely used MOT16 benchmark demonstrate its effectiveness. Additionally, qualitative results in in-house retail image data confirm the method’s capacity for practical multi-consumer tracking. Our proposed system is part of a research project aiming to capture retail consumers’ preferences, as a function of their relative position with respect to particular areas of a store that have been mapped according to the shelves, which are visible for each view area of the installed RGB and thermal cameras. Additionally, the system relies on accurate estimation of consumer age and gender using only body images in a multi-task deep learning framework in order to further taxonomize consumer preferences into age and gender groups. By these means, useful consumer meta-data can be extracted, ready to be employed for marketing campaign customization, which might improve sales and the consumer experience altogether.