Image processing in the car is becoming an increasingly important factor as a driver assistance system. In combination with other sensors and powerful pattern recognition methods, it will be a component in the success of autonomous driving. But where will the image processing take place, in the car or in the backend, i.e. the systems in the cloud?
Yesterday while out doing errands with the car, I deliberately paid attention to the pictures I saw during my trip. Actually, we should always drive with mindfulness, but honestly who does that? When we sit in the car, our focus is on specific events. So we register some things, such as road signs, traffic lights or vehicles ahead quite well, if we are not distracted and the weather conditions are pleasant. But how often do we manage to brake only at the last moment before we drive up to the car before us or cause an accident when people pop up out of nowhere by the roadside? A good image processing driver assistance system could help us to observe the environment carefully and to warn us of dangers earlier.
Image processing in the vehicle
Modern vehicles already have various camera systems (stereo camera, tracking assistant, parking assistants, virtual top-view, peeking around the corner) that help detect things, e.g. traffic signs. These systems make an all-round view possible in conjunction with radar, as well as image recognition in case of poor visibility, such as fog, rain or darkness. Since the camera itself is only a sensor that records information, there must be a component to process the images. It is an obvious choice to integrate this component directly into the vehicle. This way, images delivered by the camera can be analysed for patterns.
Now, however, it is not sufficient to have just one image processing component on board. Since the information imported from different cameras must also be analysed according to different patterns, several image processing components are necessary. These work alongside the images and each focuses on a specific pattern, such as the recognition of people. Pre-processing can recognise people as such, and post-processing can determine whether they are children. This creates a whole matrix of image processing components, which must be operated on one or more computer systems in the car.
Today, the sensors provide the intelligence. This also means that the image processing is a part of the camera, hence inflexible with respect to new pattern recognition algorithms. By means of a real-time operating system for the vehicle, the image acquisition sensors can be better decoupled from the processing components, and it becomes possible to update new pattern recognition processes.
Image processing in the backend
Relocating image processing to the backend entails a few interesting aspects. Thus, it is possible to use not only the images of a vehicle, but of several vehicles travelling at the same location, in order to evaluate a situation. The first vehicle recognises the person. In the time of about 0.5-1 seconds the vehicle has to recognise the person before it is past, it can take and transfer about 15-30 pictures. A second vehicle can now send further images of the person to the backend where the images of both vehicles are evaluated to create a motion vector of the person. Is the person approaching the street or walking away? In the first case, a third vehicle can be informed and its driver is given a warning.
Another advantage is that new processor units can be quickly integrated into new pattern analyses. As a result, the pattern recognition can be continuously improved. Also, the need for CPU and memory in the backend can theoretically be extended indefinitely. Onboard computer capacities are limited, so an expansion of image processing components requires the visit of a service company, in which case an additional CPU unit and memory must be installed.
As mentioned above, huge amounts of data are lost when transferring entire images to the backend. A typical onboard camera delivers colour images in the resolution 1344 x 1008 pixels, which corresponds to about 160 KB when compressed. If a vehicle transmits 30 frames per second of its image sensors to the back end, then approximately 11 MB/s of data transfer volume are required for five cameras (left, right, rear and stereo camera in front). With current 4G and soon 5G technologies, that might not sound like a lot, but the backend has to handle the data of tens of thousands, or even hundreds of thousands of vehicles, which quickly add up to 1-2TB/s. In the future, the image sensors will have a higher resolution, which will increase the image size again.
Smart car with powerful backend
Sending all images to the backend is not manageable due to the enormous amount of data. If the advantages of both possibilities – i.e. a little data transfer and the evaluation of the image information of several vehicles – are to be used, a compromise must be found.
The pre-processing of the images is carried out in the vehicle so that, as a result, vector-optimised images can be sent to the backend. Compared to the complete images, these are only a tiny fraction of the required data volume and can be handled easily in the backend. On the basis of these vector graphics, further analyses can be carried out on the backend side, as well as on the data of several vehicles.
In order to build up a smart car with a powerful backend, car-IT must work closely with IT, which is responsible for the backend. Unilateral definition of requirements, whether from backend to car-IT or vice versa, does not work. The pre-processing in the vehicle and the post-processing in the backend must be combined in order to be able to introduce new sample assessments.
Another important aspect is the introduction of an operating system that separates the hardware from the software, makes real-time processing possible, and allows easy installation of new pre-processor units. With AGL (Automotive-Grade Linux) or ROS (Robotic Operation System) two systems are up for the task. Currently, however, these operating systems are still heavily focused on the head unit and thus more on media applications.
With image processing in the smart car combined with a flexible data analysis and deep learning algorithms in the backend, the negative impact of an often occurring lack of awareness in traffic is reduced, thereby minimising the risk that we or other transport users come to harm. For the autonomous car this will be necessary anyway.
Latest posts by Eric Wirth-Durst (see all)
- Image processing in the car – smart car or smart backend? - 3. August 2017