A Novel Tracking Algorithm Using Thermal, mmWave Radar and Optical Sensors Fusion Imaging
Open Access
- Author:
- Iepure, Bogdan
- Graduate Program:
- Electrical Engineering
- Degree:
- Master of Science
- Document Type:
- Master Thesis
- Date of Defense:
- May 05, 2020
- Committee Members:
- Aldo W Morales, Thesis Advisor/Co-Advisor
Nashwa Nabil Elaraby, Committee Member
Sedig Salem Agili, Committee Member
Scott Van Tonningen, Program Head/Chair - Keywords:
- image fusion
sensor fusion
mmWave radar sensor fusion
thermal and optical image fusion
multi spetrum sensor fusion
radiometric thermal and optical sensor fusion
object tracking
moving object tracking
object detection
object classification
thermal object detection
thermal region of interest detection
radiometric reading region of interest
mmWave radar occlusion
mmWave radar range measurement
mmWave radar velocity measurement
mmWave radar angle of arrival measurement
tracking algorithm - Abstract:
- Moving object tracking in pan-tilt-zoom (PTZ) cameras is a critical feature for automated surveillance applications. The PTZ auto tracking function enables the camera to control electric motor mechanisms that perform pan-tilt-zoom tasks to adjust the course of the optical sensor field of view (FOV) of the camera to track moving objects automatically. The aim of moving object tracking is to extract a region of interest from a frame of sequenced images and to keep track of the region’s motion and position while avoiding occlusions. The preceding steps of tracking an object in a sequence of images are object detection and object classification. Object detection is performed to isolate and localize a region of interest from a frame of successive images. After identifying and curbing a region of interest in an image, the classification of the confined region is performed in order to implement a complete object tracking. Object classification is the process of categorizing the detected region of interest in unequivocal classes such that the region of interest can be discerned from other constituencies in a frame of consecutive images. In moving object tracking and preceding steps, the quality of the image plays an imperative role. The performance of an accurate object tracking implementation heavily relies on factors such as scene lighting, image background dynamic, the presence of shadows in the image, occlusions, or the existence of noise or image distortion. In order to enhance moving object tracking, the quality of the image in which the tracking is performed can be improved by fusing images from sensors with different wavelength spectrums such as thermal and optical sensors. In fusing such sensors, the image quality of an optical sensor can be drastically improved in low light or dark scene scenarios. The image fusion process is defined as congregating the significant information of two or more input images of a scene into a single image output conveying supplementary detailed information of the same scene, which ultimately is greater than the original information of the input images. In order to aid in avoiding occlusions that can pose difficulties in continuous object tracking, sensors that can measure the distance or angular velocity of the targeted region of interest such as millimeter wave (mmWave) based radars can be merged or fused together with thermal and optical image sensors. In this study, a comprehensive research of different methods of computer-vision object detection, object classification, and object tracking is presented. Further, a thorough examination of different methods of thermal and optical sensor image fusion is conducted. The advantages and drawbacks of the different methods presented are discussed and quantitatively evaluated. Moreover, the millimeter-wave radar technology for measuring the distance and angular velocity of an object of interest in the field of view of a camera sensor is explained. Finally, a proposed novel object tracking method based on the fusion of thermal and optical sensor images is implemented. Additionally, the fused image is merged with a millimeter wave radar raw data measurement. The implementation of the proposed method is performed on a small pan-tilt setup, which ultimately tracks the object of interest based on thermal data while keeping the optical image resolution the same as the thermal image resolution and provides valuable distance measurements to the target based on the millimeter-wave radar readings.