On board camera perception and tracking of vehicles

In this paper a visual perception system for Intelligent Vehicles is presented. The goal of the system is to perceive the surroundings of the vehicle looking for other vehicles. Depending on when and where they have to be detected (overtaking, at long range) the system analyses movement or uses a vehicle geometrical model to perceive them. Later, the vehicles are tracked. The algorithm takes into account the information of the road lanes in order to apply some geometric restrictions. Additionally, a multi-resolution approach is used to speed up the algorithm allowing real-time working. Examples of real images show the validation of the algorithm. 1 Perception in Intelligent Vehicles Human errors are the cause of most of traffic accidents, therefore can be reduced but not completely eliminated with educational campaigns. That is why the introduction of environment analysis by sensors is being researched. These perception systems receive the name of Advanced Driver Assistance Systems (ADAS) and it is expected that will be able to reduce the number, danger and severity of traffic accidents. Several ADAS, which nowadays are being researched for Intelligent Vehicles, are based on Computer Vision, among others Adaptive Cruise Control (ACC), which has to detect and track other vehicles. Now, commercial equipments are based on distance sensors like radars or LIDARs. Both types of sensors have the advantages of providing a direct distance measurement of the obstacles in front of the vehicle, are easily integrated with the vehicle control, are able to work under bad weather conditions, and lighting conditions do not affect them very much. The economical cost for LIDARs and a narrow field of view of radars are inconveniences that make Computer Vision (CV) an alternative or complementary sensor. Although it is not able to work under bad weather conditions and its information is much difficult to process, it gives a richer description of the environment that surrounds the vehicle. From the point of view of CV, the research on vehicle detection based on an onboard system can be classified in three main groups. Bottom-up or feature-based, where the algorithms looked sequentially for some features that define a vehicle. But they have two drawbacks: the vehicle is lost if one feature is not enough present in the image and false tracks can deceive the algorithm. Top-down or model-based, where there are one or several models of vehicles and the best model is found in the image through a likelihood function. They are more robust than the previous algorithms, but slower. The algorithm presented in this paper follows this approach. The third approach is learning-based. Mainly, they are based on Neural Networks (NN). Many images are needed to train the network. They are usually used together with a bottom-up algorithm to check if a vehicle has been actually detected. Otherwise, they have to scan the whole image and they are very slow. A previous detection of the road limits is done in [1]. After that, the shadow under the vehicles is looked for. Symmetry and vertical edges confirm if there is a vehicle. In [2] symmetry and an elastic net are used to find vehicles. Interesting zones in the image are localized in [3] using Local Orientation Coding. A Back-propagation NN confirms or rejects the presence of a vehicle. Shadow, entropy and symmetry are used in [4]. Symmetry is used in [5] to determine the column of the image where the vehicle is. After that, they look for an U-form pattern to find the vehicle. The tracking is performed with correlation. In [6] overtaking vehicles are detected through image difference and the other vehicles through correlation. Several 3D models of vehicles are used in [7]. The road limits are calculated and the geometrical relationship between the camera and the road is known. Preceding vehicles are detected in [8]. They calculate a discriminant function through examples. A different way of reviewing the research on vehicle detection based on optical sensors can be found in [9]. The review has shown some important aspects. First, the module in charge of detecting other vehicles has to exchange information with the lane detection module. The regions where vehicles can appear are delimited and some geometric restrictions can be applied. The number of false positives can be reduced and the algorithm speeds up. Moreover, the detection of road limits can be more robust as this module can deal with partial occlusions produced by vehicles. Second, vehicle appearance changes with distance and position respect to the camera. A model-based approach is not useful to detect over-taking vehicles which are not fully seen in the image, and a vehicle that is far away shows a low apparent speed in the image. Several areas in the image have to be defined in order to specify where, how and what is going to be looked for in the image. Third, the algorithm not only has to detect vehicles but to track them and specify their state. These three points define the structure of the paper. 2 Different Areas and Vehicle Appearance Different features define the same vehicle depending on the area of the image where it appears. As it is shown in Fig. 1, lateral areas of the images are the only ones where overtaking vehicles can appear. Depending on the country, overtaking vehicles will appear on the left/right lane, and overtaken vehicles on the right/left one. A modelbased approach is difficult to implement and it is better to use a feature-based approach, mainly taking movement into account. A different case is when the vehicle is in front of the camera. The rear part of the vehicle is full seen in the image and a model-based approach is possible. Beside these areas, there is another corresponding to the vehicles have just over-taken ours. The rear part of the vehicle is completely seen in the image, although a small deformation due to projective distortion appears. 58