INFORMATION ABOUT PROJECT,
SUPPORTED BY RUSSIAN SCIENCE FOUNDATION
The information is prepared on the basis of data from the information-analytical system RSF, informative part is represented in the author's edition. All rights belong to the authors, the use or reprinting of materials is permitted only with the prior consent of the authors.
Project titleModels and methods for intelligent driver support based on situation in vehicle cabin
Project LeadKashevnik Alexey
Implementation period2018 - 2020
Research area 01 - MATHEMATICS, INFORMATICS, AND SYSTEM SCIENCES, 01-201 - Artificial intelligence and decision-making
KeywordsContext, driver support, recommendation systems, ontologies, image recognition, personification
Last years research in the area of transport systems becomes more and more popular: autonomous vehicles, information support for a vehicle driver, driver behavior analysis in the cabin, analysis of a moving vehicle, and others. High level of accidents on public roads both in Russia and abroad causes the interest of research community to this area. At the same time, according to the forecast of McKinsey Global Institute, the average annual growth rate of the market for driver assistance systems up to 2020 will be more than 11 billion US dollars. Effective context-oriented support for the driver will allow both to prevent or reduce the impact of road accidents, and to increase the comfort of the driver when driving a vehicle. Within the framework of the project it is proposed to develop new models and methods for generating recommendations to the driver based on information from the camera and sensors of the mobile phone mounted on the vehicle windshield. In this case, the main difference from the existing Driver Assistance Systems (ADAS), which are integrated to vehicles at factories and available only in premium vehicle segments, the proposed system is designed for use in the mass market, since it does not require additional significant costs from use the smartphone driver to determine the situation in which it is located. Modern smartphones have small size but their functionality and computing power is quite high and evolving every day. Also, smartphones include a large number of built in different sensors: the front and rear cameras, the position sensor, the accelerometer (G-sensor), the gyroscope, the microphone can be included. Thus, depending on the situation in which the driver is located and his/her profile, the proposed system can generates recommendations that will reduce the accident rate and improve the comfort of driving.
The following main results are proposed in scope of the project. 1) Conceptual model of the intelligent driver support system. 2) Ontological model of the driver profile. 3) Method for determining dangerous states in the vehicle cabin. 4) Method for generating recommendations to the vehicle driver. 5) Method of the driver's intelligent support system self-learning. 6) Service-oriented architecture of the driver's intelligent support system. 7) Scenario model for preventing emergencies and monitoring the driver during the movement. 8) Research prototype of the intelligent driver support system. 9) Results of the assessment of the effectiveness of the definition of dangerous states and generation of the context-oriented recommendations to the driver of the vehicle. The nature of the presented results are both: fundamental and applied. On the one hand, it is proposed to develop new models and methods that increase the effectiveness of man-machine interaction through personalization, context management and self-learning technologies. On the other hand, the proposed results open new directions for using a personal driver's smartphone to prevent or reduce the road accidents, and to improve the comfort when driving a vehicle. The proposed results outperform the world level, as today there is no such full-featured system of context-oriented driver support based on information from the front camera and smartphone sensors. The practical utilisation of the project results is supposed in the social sphere: the driver installs the system to his/her smartphone and uses it together with the favorite navigation application.
Annotation of the results obtained in 2018
According to the report of World Health Organization on Road Safety there were more than 1.35 M human deaths and about 50 M injures registered due to road accidents for the period of 2018. The incredible progress on information and communication technologies for the last years allows to create intelligent driver support systems. Existing research efforts in the area of Advanced Driver Support Systems (ADAS) are focused on system operation as a part of software and hardware systems that are integrated into premium segment vehicles. The main difference of the proposed system from the existing ones is the usage of mobile phone (smartphone) as the main device for state monitoring in the vehicle cabin, This can reduce integration costs and therefore allow the system to spread in the market. During the reporting period all project participants have successfully finished planned research and the results have been published. The analysis of the state-of-the-art has been carried out for the areas of driver support systems, group recommender systems and context management. It has shown that the chosen direction of research is actual and a significant amount of problems are still open. It has also enabled to specify the following main requirements to systems of the considered class: (i) identification of dangerous states such as drowsiness, weakened attention, increased heart rate, driving under the influence of alcohol or drugs, aggressive driving, stress state; (ii) recommendation generation for the driver; (iii) the functioning of the system both in a regular mode (with access to the Internet) and in a limited mode (with no or limited access to the Internet); (iv) environment state information usage (road, traffic, weather, etc.) (v) support of personification mechanisms for interaction with the driver to properly identify dangerous states and generate recommendations; (vi) assessing the quality of recommendations in a broad sense (taking into account the diversity of recommendations); (vii) the use of hybrid schemes for the recommendation generation (viii) the use of metaheuristics. In addition the main approaches to personalization and adaptation of such a systems have been identified together with methods of personalization quality evaluation used by the scientific community. The scientific novelty of the proposed conceptual model of the intelligent driver support system can be stated in the following way: (i) the dangerous states of the vehicle driver are classified into online-detected and offline-detected ones (ii); the use of approaches to personification of the driver’s interaction with the system based on machine learning, both in determining online and offline dangerous states, and in recommendations generating to the driver; (iii) application of the open data concept to form the context (model of the situation inside and outside the vehicle). The online-detected dangerous states (drowsiness and distraction) should be detected directly on mobile phone, since the data transmission time to cloud is comparable to the driver reaction time to avoid traffic accident. It is taken into account that not all roads in Russia as well as in other countries are covered by communication network (cellular network) providing reliable access to the Internet, moreover many roads are completely out of range of cellular network. In turn offline-detected dangerous states (increased heart rate, driving under alcohol or drug, aggressive driving, stress) do not require instant reaction but significantly more computation power compared to online-detected dangerous states. In this regard, it was proposed to perform computations for the detection of offline-detected dangerous states using cloud technologies. The cloud service proposed within the conceptual model of the intelligent driver support system is aimed at detecting of offline-detected dangerous states, identifying characteristics of drivers, clustering drivers, and behavior patterns for each driver. The patterns are used for personalizing the driver’s interaction with the system, which consists in adapting modules for determining dangerous states (online and offline) for the individual characteristics of the driver, and in the generation of personalized recommendations. For this reason the personification module was allocated within the conceptual model of the system as well as its implementation was proposed based on the adaptive threshold mechanism. The role of personification based on the mechanism of adaptive thresholds at this stage of research is as follows: first, it is a model example that demonstrates the existence of all the links and functional blocks necessary for the implementation of personification mechanisms in the developed conceptual model; secondly, relatively simple personification using adaptive thresholds will later serve as a benchmark for comparison when evaluating more complex personification methods; thirdly, it is designed to increase the attractiveness of using the system, which is important, in particular, when collecting usage statistics for optimizing patterns of dangerous states recognition. For the recommendation generation project participants propose to use as information about recognized dangerous states calculated based on smartphone sensors analysis, as well as vehicle context calculated based on open data and linked data connected to the Internet To generate recommendations for a vehicle driver, it has been proposed to use both information about a certain dangerous condition and information calculated based on the analysis of smartphone sensor data (vehicle speed, acceleration, illumination in the cab, etc.), as well as the vehicle context calculated based on the analysis of information from available sources of linked open data that are connected to the Internet and related to transport (for example, traffic police statistics server, web cameras on the highway and at intersections, information about traffic jams, data from other vehicles, as well as information about roadside cafes and gas stations). In one of the publications prepared as part of the project, the authors proposed the term Internet of Transportation Things, the essence of which is that computing devices and services connected to the Internet and related to transport are used to form the context of the vehicle. Driver profile has been developed using the open source ontology editor Protégé. OWL language has been used for the ontology modelling. Ontological approach worked well during the last years what is supported by number of modern research papers. In the scope of the developed ontology model 117 axioms, 38 classes, 9 data properties, and 5 object properties have been defined. SPARQL has been used to acquire information and knowledge from the developed ontology. During the reported period the plan for next year has been clarified and detailed. In the beginning of 2019 project participants discussed potential collaboration related to drowsiness dangerous state investigation with the Department of Civil Engineering of the University of Porto, Portugal. Based on this discussion the preliminary arrangement has been achieved concerning the testing of the developed intelligent driver support system modules in the vehicle simulator available in the University of Porto. Also, during the second reporting period the project team plans to conduct the experiments based on the vehicle park of “XXI Century” medical clinic in St. Petersburg. Based on results of the first project phase 8 scientific papers have been prepared. 2 papers have been published in the proceedings of the international conferences indexed Scopus and Web of Science. Two papers have been published in journals (one indexed in Russian Science Citation Index and one - in Scopus). Three papers have been accepted to the international conferences indexed in Scopus (one will take place during the current reporting period and two - in the next one). One paper has been accepted to publuication in an international journal related to intelligent transportation systems (IEEE Transactions on Intelligent Transportation Systems), that is indexed in Scopus and Web of Science (Quartile: Q1, CiteScore: 5.14).
1. Кашевник А.М., Лашков И.Б. Decision Support System for Drivers & Passengers: Smartphone-Based Reference Model and Evaluation Proceedings of the 23rd Conference of Open Innovations Association FRUCT, сс. 166-171 (year - 2018).
2. Лашков И.Б., Кашевник А.М. Онтологическая модель системы предупреждения аварийных ситуаций на основе поведения водителя в кабине транспортного средства Интеллектуальные технологии на транспорте, №4, сс. 11-19 (year - 2018).
3. Лашков И. Подход к распознаванию стиля вождения водителя транспортного средства на основе использования сенсоров смартфона Информационно-управляющие системы, том 5, сс. 2-12 (year - 2018).
4. Кашевник А., Карельская К., Репп М. Dangerous Situations Determination by Smartphone in Vehicle Cabin: Classification and Algorithms Proceedings of the 24th Conference of Open Innovations Association FRUCT, cc. 130-139 (year - 2019).
5. Кашевник А., Лашков И., Гуртов А. Methodology and Mobile Application for Driver Behavior Analysis and Accident Prevention IEEE Transactions on Intelligent Transportation Systems, - (year - 2019).
6. Кашевник А., Лашков И., Тесля Н. Driver Intelligent Support System in Internet of Transportation Things: Smartphone-Based Approach 14th International Conference on System of Systems Engineering (SoSE), - (year - 2019).
7. Кашевник А., Лашков И. Intelligent Driver Decision Support System in Vehicle Cabin: Reference Model for Dangerous Events Recognition and Learning 15th IEEE International Conference on Control and Automation (IEEE ICCA 2019), - (year - 2019).
8. Лашков И.Б., Кашевник А.М. Smartphone-Based Intelligent Driver Assistant: Context Model and Dangerous State Recognition Scheme Proceedings of Intelligent Systems Conference (IntelliSys) 2019, - (year - 2019).
Annotation of the results obtained in 2019
Project team has got the following results in scope of the last reporting period of the project “Models and methods for intelligent driver support based on situation in vehicle cabin”. (1) method for dangerous state detection in the vehicle cabin was developed based on information from smartphone sensors mounted on the vehicle windshield; (2) method for recommendation generation to the driver based on the detected dangerous state and the current situation in the vehicle cabin has been developed; (3) method for the system to self-learning to generate group recommendations based on statistics of dangerous states detection and driver’s evaluation of the system operation correctnesshas been developed; (4) Implementation and testing basic modules prototypes of the main modules of the driver behavior monitoring system in the vehicle cabin have been implemented. Developed method for dangerous states detection in vehicle cabin is based on information from smartphone sensors mounted on vehicle windshield. It was proposed to recognize online-detected dangerous states, which include a dangerous state of drowsiness and distraction. Online-detected dangerous states are detected using a smartphone, it is required to have the real time system response. Moreover, to detect drowsiness dangerous state two algorithms have been proposed that can be used to adjust each other's accuracy depending on the current situation. The first algorithm is focused on detecting drowsiness dangerous state based psychophysiological parameters analysis of the driver’s face and determining such characteristics as the degree of closed eyes, blink rate per second, head rotation and tilt angles, degree of mouth openness. The second algorithm is focused on the classification of drivers based on speed analysis and its changes when driving the vehicle. So, the monotonous movement indicates the driver drowsiness, while speed constant changes indicate the exclusion of a state of drowsiness. The detection of the dangerous state of lowered attention is focused on the analysis of the dynamics of absolute values and changes in the rotation and inclination of the driver’s head, depending on the speed of the vehicle. It was proposed to dynamically calculate the time, which is critical for deviating the gaze of the driver from the road. So, for example, if the driver moves at low speed (for example, in the parking area, which forces him to constantly monitor vehicles on the right and left), then this time should be longer than he moves with the maximum allowed speed on the highway. Algorithms for recognizing off-line determined dangerous states have been developed, which include dangerous state of aggressive driving and increased heart rate (which can mean, depending on the degree of driving aggressiveness, such concomitant dangerous states as health problem, alcohol or drug intoxication, stress state). For each of the above dangerous states, a detection scheme has been developed based on threshold values, and a corresponding model functioning on the cloud server, that allows analyzing statistics from the driver’s smartphone using machine learning methods and improving threshold values of algorithms. The detection of the dangerous state of aggressive driving is based on the analysis of the vehicle’s acceleration and comparison of the obtained value with the average acceleration stored for this driver. The important fact about this algorithm is that it does not use the absolute value of this parameter, but the relative one, indicating how aggressively the driver drives the car at the current moment compared to the usual driving. The detection of the pulse from the images obtained from the smartphone’s camera is based on the remote photoplethysmography approach based on the skin color change during the passage of blood in the capillaries. Within the frame of the project, well-known remote photoplethysmography algorithms have been adapted and improved. The task was to determine whether the driver’s pulse is in the normal zone or is it higher than normal what may indicate that the driver is intoxicated or stressed. Within the framework of the developed method for generating hybrid group recommendations, the problem of detecting the best recommendation at a given time for a given user has been solved based on the operation of one of the following modules: (1) module of knowledge-based recommendation generation; (2) module for generating recommendations based on open data sources; and (3) module for machine learning-based recommendation generation. The knowledge-based recommendation generation module is the main one, since it does not depend of external factors and generates recommendations when a dangerous state occurs using the internal knowledge base. The module for generating recommendations based on open data sources generates recommendations when the dangerous state occurs or independently of it when there are data on road traffic accidents in the area where the vehicle is located. The module for generating recommendations based on machine learning methods is used to personify recommendations for a particular driver after enough information has been accumulated in the cloud server database to take into account the characteristics of the particular driver and his/her preferences regarding the receipt of certain recommendations. The developed self-learning method for generating group recommendations is aimed at grouping drivers of the monitoring system to use the experience of interacting with one driver when interacting with another. For this purpose, a number of models were proposed that make it possible to track if the driver follows certain recommendations during driving on an example of a dangerous state of drowsiness. In the framework of the study, recommendations were classified according to the degree of automation of checking if the driver follows these. The recommendations, the verification of following which can be easily done via analysis of information from the smartphone camera and sensors, were identified. Additionally, the recommendations, the verification of following which requires development of additional algorithms were identified as well: for example, recommendations related to determining the type of sounds in the vehicle interior, as well as recommendations, following which that can only be verified by indirectly. An algorithm has been also proposed for clustering drivers based on their reactions to recommendations. The implementation of the prototypes of the basic modules has been done for the Android operating system, which allowed them to be evaluated directly in the vehicle cabin while driving. The following modules have been implemented: the module for determining drowsiness, the module for determining low attention, the module for determining heart rate, and the module for automatic calibration of the angle of the driver head rotation. Testing and evaluating the implemented prototypes were carried out both on available datasets, and using own data collected during actual vehicle driving. Own data set collected during testing includes: 22,267 km of the distance traveled by vehicle drivers, who spent total of 463 hours driving. The total number of trips: 504 (trips of 5 km and longer lasting for at least 10 minutes after September 2019 were considered). To test the module for determining drowsiness, field experiments were conducted in the vehicle cabin, with the driver (having made sure of the safety of the performed actions) squinting his eyes (imitating the dangerous state of drowsiness) for three seconds. The approximate accuracy of the drowsiness determination module amounted to 69% in total according to the both criteria (5% of false positives). Similar experiments have been conducted to calculate the percentage of determined states of low attention. The driver’s task was to turn his head to the side for about 3 seconds. The approximate accuracy of the distraction determination module was 95% (2% of false positives). The heart rate detection module was tested on the Public Benchmark Dataset for Testing rPPG Algorithm Performance dataset. The experiments showed the accuracy of 80% -90% for different videos. The results of testing the module for automatic calibration of the drivers head rotation angle showed that about a minute of movement at a speed of 30 km / h is enough to determine the angle with an accuracy of several degrees, which acceptable for the task set.
1. Лашков И.Б., Кашевник А.М. Онтологическая модель системы предупреждения аварийных ситуаций на основе поведения водителя в кабине транспортного средства Интеллектуальные технологии на транспорте, Том 4, № 16. С. 11–19 (year - 2018).
2. Кашевник А., Карельская К., Репп М. Dangerous Situations Determination by Smartphone in Vehicle Cabin: Classification and Algorithms 24rd IEEE Conference of Open Innovations Association FRUCT, P. 130–139. (year - 2019).
3. Кашевник А., Лашков И., Гуртов А. Methodology and Mobile Application for Driver Behavior Analysis and Accident Prevention IEEE Transactions on Intelligent Transportation Systems, pp 1-10 (year - 2019).
4. Кашевник А., Лашков И., Тесля Н. Driver Intelligent Support System in Internet of Transportation Things: Smartphone-Based Scenario 14th International Conference on System of Systems Engineering (SoSE), pp. 1-6 (year - 2019).
5. Корзун Д., Баландина Е., Кашевник А., Баландин С., Виола Ф. Ambient Intelligence Services in IoT Environments IGI-Global, Hershey PA, USA, 213 p. (year - 2019).
6. Кашевник А., Лашков И. Intelligent Driver Decision Support System in Vehicle Cabin: Reference Model for Dangerous Events Recognition and Learning 15th IEEE International Conference on Control and Automation (IEEE ICCA 2019), pp. 27-31 (year - 2019).
7. Лашков И., Кашевник А. Smartphone-Based Intelligent Driver Assistant: Context Model and Dangerous State Recognition Scheme Proceedings of SAI Intelligent Systems Conference (IntelliSys 2019), pp. 152-165 (year - 2019).
8. Линдоу Ф., Кашевник А. Driver Behavior Monitoring Based on Smartphone Sensor Data and Machine Learning Methods Proceedings of the 25th IEEE Conference of Open Innovations Association FRUCT (FRCUT25), pp. 196-203 (year - 2019).
9. - Утро России: Будильник для водителя. Смартфон разбудит шофера, если он уснул за рулем Телеканал Россия, - (year - ).
10. - Не зевай, тебе говорю. Смартфон сможет предотвратить аварии на дороге Российская газета, - (year - ).
11. - Смартфон не даст заснуть за рулем. Ученые разработали приложение, чтобы водители оставались внимательными Коммерсант, - (year - ).
12. - Приложение для смартфона не позволит водителю быть невнимательным Indicator, - (year - ).
13. - Создано приложение, которое помогает водителю остаться внимательным на дороге РНФ, - (year - ).
14. - Российское приложение поможет водителям с безопасностью на дороге Поиск, - (year - ).
15. - Приложение для водителей, помогающее оставаться внимательным на дороге, разработали ученые Газета.ru, - (year - ).
16. - Создано приложение, которое помогает водителю остаться внимательным на дороге Наука тасс, - (year - ).
17. - В России появилось противоаварийное приложение для смартфонов Петербург 5 канал, - (year - ).
18. - В России создали систему, которая не даст заснуть за рулем РИА Новости, - (year - ).
19. - Создана облачная система контроля водителей для обеспечения безопасности на дорогах Indicator, - (year - ).
20. - Ученые создали облачную систему контроля водителей для обеспечения безопасности на дорогах Газета.RU, - (year - ).
21. Кашевник А., Лашков И., Пономарев А., Тесля Н., Гуртов А. Cloud-Based Driver Monitoring System Using Smartphone IEEE Sensors, - (year - 2020).
22. Кашевник А., Тесля Н., Пономарев А., Лашков И., Маятин А., Парфенов В. Driver Monitoring Cloud Organisation Based on Smartphone Camera and Sensor Data 17th International Conference on Information Technology–New Generations (ITNG 2020), - (year - 2020).
23. Кашевник А., Туан А. Evaluation of face analysis methods for personalization in driver monitoring systems Tools and Methods of Competitive Engineering (TMCE 2020), - (year - 2020).