Annotation of the results obtained in 2020
There are five scientific papers have been published in the reporting period of the project. The papers have been indexed in the Scopus and Web of Science citation systems. One paper has been published in the IEEE Access journal, which is included in the first quartile of journals on Web of Science. The paper was published together with Austrian colleagues (organization Virtual Vehicle Research GmbH), which emphasizes the relevance and significance of the research topic and the results obtained. Also on May 13, 2021, a project team together with Austrian colleagues, organized a workshop to discuss models and methods for monitoring driver behavior in the vehicle cabin. The project leader organized a special issue in Sensors journal (Q1, Web of Science) dedicated to the topic of driver monitoring in the vehicle cab, in which three scientific articles have already been published at the time of report writing. Among the main fundamental scientific results obtained, a hybrid method for determining the state of drowsiness, which consists in the use of machine learning technologies to classify this dangerous state. It was proposed to apply a gradient boosting method based on decision trees. At the same time, it was proposed to use as the primary data for the hybrid method: vehicle dynamics data (GPS / GLONASS coordinates, speed, accelerometer readings), facial characteristics (binary sign of face detection in the frame, the probability of a yawn, the probability of eye openness, the probability of a smile). The Karolinska Sleepiness Scale (KSS) was used as the target variable. Primary data were determined on the basis of studies carried out in the first and second years of the project, as well as experiments performed with accumulated driving statistics. Another fundamental result is the study of the possibility of using reinforcement learning to find a strategy for generating personalized recommendations to the driver, taking into account the driver's adherence to the recommendations and their impact on the driver's condition. The recommendation generated by the system has a definite effect on the driver's condition. In general, at a certain frequency, the warnings themselves are capable of focusing the driver's attention on the road situation for a certain time. However, if warnings are used too often, firstly, the driver can adapt to them and stop responding to them, and secondly, the driver may become annoyed (which, in turn, can lead to the disabling of the monitoring system). Among the practical results obtained, one can single out the information model of the cloud server, which supports the methods, models and algorithms proposed at the second stage of the project. The cloud server is focused on collecting statistics of vehicle drivers, analyzing it and presenting it to fleet dispatchers. Within the framework of the information model, a data model was designed, which allows storing statistics in such a form so that it can be used in the models and methods developed at the second stage of the project. The data model includes 18 tables containing both information from monitoring systems and personalized templates and settings, which are transferred from the cloud server to client devices. Also, a service-oriented architecture of the intelligent driver support system was developed using information from the sensors of a smartphone attached to the windshield of the vehicle. The architecture includes the following set of services: a service for collecting statistics of road accidents, a service for collecting driving statistics and its visualization, a service for analyzing vehicle driving data, a service for generating driver behavior patterns. The development of these services is of significant applied importance, since it made it possible to implement the models and methods developed at the previous stages of the project and evaluate their effectiveness and practical significance. Another practical result of the project is a scenario model of a driver monitoring system, within which five options for the roles of actors were identified, including a driver with a personal vehicle, an insurance company representative, a fleet administrator, a fleet dispatcher, and a driver from the fleet staff participating in the process of interacting with the system. For each case, the main scenarios for using the system were described, which laid the foundation for the developed service for collecting driving statistics and its visualization. One of the most important practical results of the project is the development of a research prototype of the driver monitoring system based on the developed methods and models. As part of the prototype, the following main modules of the driver monitoring system were implemented: a hybrid module for determining the state of drowsiness (both using video analysis and using vehicle speed analysis), a module for determining aggressive driving, a personification module (including determining the degree of eye closure individually for driver), module for determining turns and turns during vehicle movement, module for generating recommendations, module for analyzing the following recommendations by the vehicle driver, module for determining the context of vehicle movement (whether the car is moving on the highway or in the city). These modules were included in the Drive Safely mobile application for monitoring the state of the vehicle driver (https://play.google.com/store/apps/details?id=ru.igla.drivesafely), which was used by the team to conduct experiments. The development of the prototype made it possible to evaluate the effectiveness of the developed driver monitoring system for which field experiments were carried out. The experiments involved five drivers who traveled in vehicles for two weeks. Dangerous states of drowsiness and distraction were evaluated. The effectiveness of the driver monitoring system was assessed by recall and precision. At the same time, recall was determined by the driver himself in the vehicle cabin, who, if the system did not detect a dangerous state, covered the camera with his hand, which generated and sent the corresponding event to the server. Thus, within the framework of the experiment, 10 events were recorded in which the system did not detect a dangerous condition, and the driver believed that it should have been detected. The dangerous state of drowsiness was missed 8 times, and the dangerous state of distraction 2 times. Precision was assessed by manually reviewing all dangerous conditions identified by the monitoring system and viewing the accompanying video fragments and marking whether this dangerous condition really took place or whether it was a false alarm. Thus, the accuracy for determining sleepiness was 89%, and for distraction was 74%. Also, as part of the work on assessing the effectiveness of the developed monitoring system, work was carried out on load testing on the developed web server. For these purposes, 5,000 virtual drivers were generated, which sent statistics to the server in such a way as a real monitoring system would do. The increase in the load on the system was carried out gradually: every 100 drivers began to carry out their test trips with an interval of 10 seconds, after the completion of the trips, the driver carried out a new one. The average response time of the system reached 5 seconds, which is an acceptable value for data processing systems in real time. In some cases, the delay reached 30 seconds, which is explained by the logging of the incoming data and the non-use of direct data transfer between the backend and the database via an internal Unix socket. As a result, it can be concluded that the results obtained in terms of the accuracy of the system's operation, as well as in terms of the loading time of statistics, suggest that the methods and models developed within the framework of the project have prospects for practical use.

 

Publications

---

Annotation of the results obtained in 2018
According to the report of World Health Organization on Road Safety there were more than 1.35 M human deaths and about 50 M injures registered due to road accidents for the period of 2018. The incredible progress on information and communication technologies for the last years allows to create intelligent driver support systems. Existing research efforts in the area of Advanced Driver Support Systems (ADAS) are focused on system operation as a part of software and hardware systems that are integrated into premium segment vehicles. The main difference of the proposed system from the existing ones is the usage of mobile phone (smartphone) as the main device for state monitoring in the vehicle cabin, This can reduce integration costs and therefore allow the system to spread in the market. During the reporting period all project participants have successfully finished planned research and the results have been published. The analysis of the state-of-the-art has been carried out for the areas of driver support systems, group recommender systems and context management. It has shown that the chosen direction of research is actual and a significant amount of problems are still open. It has also enabled to specify the following main requirements to systems of the considered class: (i) identification of dangerous states such as drowsiness, weakened attention, increased heart rate, driving under the influence of alcohol or drugs, aggressive driving, stress state; (ii) recommendation generation for the driver; (iii) the functioning of the system both in a regular mode (with access to the Internet) and in a limited mode (with no or limited access to the Internet); (iv) environment state information usage (road, traffic, weather, etc.) (v) support of personification mechanisms for interaction with the driver to properly identify dangerous states and generate recommendations; (vi) assessing the quality of recommendations in a broad sense (taking into account the diversity of recommendations); (vii) the use of hybrid schemes for the recommendation generation (viii) the use of metaheuristics. In addition the main approaches to personalization and adaptation of such a systems have been identified together with methods of personalization quality evaluation used by the scientific community. The scientific novelty of the proposed conceptual model of the intelligent driver support system can be stated in the following way: (i) the dangerous states of the vehicle driver are classified into online-detected and offline-detected ones (ii); the use of approaches to personification of the driver’s interaction with the system based on machine learning, both in determining online and offline dangerous states, and in recommendations generating to the driver; (iii) application of the open data concept to form the context (model of the situation inside and outside the vehicle). The online-detected dangerous states (drowsiness and distraction) should be detected directly on mobile phone, since the data transmission time to cloud is comparable to the driver reaction time to avoid traffic accident. It is taken into account that not all roads in Russia as well as in other countries are covered by communication network (cellular network) providing reliable access to the Internet, moreover many roads are completely out of range of cellular network. In turn offline-detected dangerous states (increased heart rate, driving under alcohol or drug, aggressive driving, stress) do not require instant reaction but significantly more computation power compared to online-detected dangerous states. In this regard, it was proposed to perform computations for the detection of offline-detected dangerous states using cloud technologies. The cloud service proposed within the conceptual model of the intelligent driver support system is aimed at detecting of offline-detected dangerous states, identifying characteristics of drivers, clustering drivers, and behavior patterns for each driver. The patterns are used for personalizing the driver’s interaction with the system, which consists in adapting modules for determining dangerous states (online and offline) for the individual characteristics of the driver, and in the generation of personalized recommendations. For this reason the personification module was allocated within the conceptual model of the system as well as its implementation was proposed based on the adaptive threshold mechanism. The role of personification based on the mechanism of adaptive thresholds at this stage of research is as follows: first, it is a model example that demonstrates the existence of all the links and functional blocks necessary for the implementation of personification mechanisms in the developed conceptual model; secondly, relatively simple personification using adaptive thresholds will later serve as a benchmark for comparison when evaluating more complex personification methods; thirdly, it is designed to increase the attractiveness of using the system, which is important, in particular, when collecting usage statistics for optimizing patterns of dangerous states recognition. For the recommendation generation project participants propose to use as information about recognized dangerous states calculated based on smartphone sensors analysis, as well as vehicle context calculated based on open data and linked data connected to the Internet To generate recommendations for a vehicle driver, it has been proposed to use both information about a certain dangerous condition and information calculated based on the analysis of smartphone sensor data (vehicle speed, acceleration, illumination in the cab, etc.), as well as the vehicle context calculated based on the analysis of information from available sources of linked open data that are connected to the Internet and related to transport (for example, traffic police statistics server, web cameras on the highway and at intersections, information about traffic jams, data from other vehicles, as well as information about roadside cafes and gas stations). In one of the publications prepared as part of the project, the authors proposed the term Internet of Transportation Things, the essence of which is that computing devices and services connected to the Internet and related to transport are used to form the context of the vehicle. Driver profile has been developed using the open source ontology editor Protégé. OWL language has been used for the ontology modelling. Ontological approach worked well during the last years what is supported by number of modern research papers. In the scope of the developed ontology model 117 axioms, 38 classes, 9 data properties, and 5 object properties have been defined. SPARQL has been used to acquire information and knowledge from the developed ontology. During the reported period the plan for next year has been clarified and detailed. In the beginning of 2019 project participants discussed potential collaboration related to drowsiness dangerous state investigation with the Department of Civil Engineering of the University of Porto, Portugal. Based on this discussion the preliminary arrangement has been achieved concerning the testing of the developed intelligent driver support system modules in the vehicle simulator available in the University of Porto. Also, during the second reporting period the project team plans to conduct the experiments based on the vehicle park of “XXI Century” medical clinic in St. Petersburg. Based on results of the first project phase 8 scientific papers have been prepared. 2 papers have been published in the proceedings of the international conferences indexed Scopus and Web of Science. Two papers have been published in journals (one indexed in Russian Science Citation Index and one - in Scopus). Three papers have been accepted to the international conferences indexed in Scopus (one will take place during the current reporting period and two - in the next one). One paper has been accepted to publuication in an international journal related to intelligent transportation systems (IEEE Transactions on Intelligent Transportation Systems), that is indexed in Scopus and Web of Science (Quartile: Q1, CiteScore: 5.14).

 

Publications

---

Annotation of the results obtained in 2019
Project team has got the following results in scope of the last reporting period of the project “Models and methods for intelligent driver support based on situation in vehicle cabin”. (1) method for dangerous state detection in the vehicle cabin was developed based on information from smartphone sensors mounted on the vehicle windshield; (2) method for recommendation generation to the driver based on the detected dangerous state and the current situation in the vehicle cabin has been developed; (3) method for the system to self-learning to generate group recommendations based on statistics of dangerous states detection and driver’s evaluation of the system operation correctnesshas been developed; (4) Implementation and testing basic modules prototypes of the main modules of the driver behavior monitoring system in the vehicle cabin have been implemented. Developed method for dangerous states detection in vehicle cabin is based on information from smartphone sensors mounted on vehicle windshield. It was proposed to recognize online-detected dangerous states, which include a dangerous state of drowsiness and distraction. Online-detected dangerous states are detected using a smartphone, it is required to have the real time system response. Moreover, to detect drowsiness dangerous state two algorithms have been proposed that can be used to adjust each other's accuracy depending on the current situation. The first algorithm is focused on detecting drowsiness dangerous state based psychophysiological parameters analysis of the driver’s face and determining such characteristics as the degree of closed eyes, blink rate per second, head rotation and tilt angles, degree of mouth openness. The second algorithm is focused on the classification of drivers based on speed analysis and its changes when driving the vehicle. So, the monotonous movement indicates the driver drowsiness, while speed constant changes indicate the exclusion of a state of drowsiness. The detection of the dangerous state of lowered attention is focused on the analysis of the dynamics of absolute values and changes in the rotation and inclination of the driver’s head, depending on the speed of the vehicle. It was proposed to dynamically calculate the time, which is critical for deviating the gaze of the driver from the road. So, for example, if the driver moves at low speed (for example, in the parking area, which forces him to constantly monitor vehicles on the right and left), then this time should be longer than he moves with the maximum allowed speed on the highway. Algorithms for recognizing off-line determined dangerous states have been developed, which include dangerous state of aggressive driving and increased heart rate (which can mean, depending on the degree of driving aggressiveness, such concomitant dangerous states as health problem, alcohol or drug intoxication, stress state). For each of the above dangerous states, a detection scheme has been developed based on threshold values, and a corresponding model functioning on the cloud server, that allows analyzing statistics from the driver’s smartphone using machine learning methods and improving threshold values of algorithms. The detection of the dangerous state of aggressive driving is based on the analysis of the vehicle’s acceleration and comparison of the obtained value with the average acceleration stored for this driver. The important fact about this algorithm is that it does not use the absolute value of this parameter, but the relative one, indicating how aggressively the driver drives the car at the current moment compared to the usual driving. The detection of the pulse from the images obtained from the smartphone’s camera is based on the remote photoplethysmography approach based on the skin color change during the passage of blood in the capillaries. Within the frame of the project, well-known remote photoplethysmography algorithms have been adapted and improved. The task was to determine whether the driver’s pulse is in the normal zone or is it higher than normal what may indicate that the driver is intoxicated or stressed. Within the framework of the developed method for generating hybrid group recommendations, the problem of detecting the best recommendation at a given time for a given user has been solved based on the operation of one of the following modules: (1) module of knowledge-based recommendation generation; (2) module for generating recommendations based on open data sources; and (3) module for machine learning-based recommendation generation. The knowledge-based recommendation generation module is the main one, since it does not depend of external factors and generates recommendations when a dangerous state occurs using the internal knowledge base. The module for generating recommendations based on open data sources generates recommendations when the dangerous state occurs or independently of it when there are data on road traffic accidents in the area where the vehicle is located. The module for generating recommendations based on machine learning methods is used to personify recommendations for a particular driver after enough information has been accumulated in the cloud server database to take into account the characteristics of the particular driver and his/her preferences regarding the receipt of certain recommendations. The developed self-learning method for generating group recommendations is aimed at grouping drivers of the monitoring system to use the experience of interacting with one driver when interacting with another. For this purpose, a number of models were proposed that make it possible to track if the driver follows certain recommendations during driving on an example of a dangerous state of drowsiness. In the framework of the study, recommendations were classified according to the degree of automation of checking if the driver follows these. The recommendations, the verification of following which can be easily done via analysis of information from the smartphone camera and sensors, were identified. Additionally, the recommendations, the verification of following which requires development of additional algorithms were identified as well: for example, recommendations related to determining the type of sounds in the vehicle interior, as well as recommendations, following which that can only be verified by indirectly. An algorithm has been also proposed for clustering drivers based on their reactions to recommendations. The implementation of the prototypes of the basic modules has been done for the Android operating system, which allowed them to be evaluated directly in the vehicle cabin while driving. The following modules have been implemented: the module for determining drowsiness, the module for determining low attention, the module for determining heart rate, and the module for automatic calibration of the angle of the driver head rotation. Testing and evaluating the implemented prototypes were carried out both on available datasets, and using own data collected during actual vehicle driving. Own data set collected during testing includes: 22,267 km of the distance traveled by vehicle drivers, who spent total of 463 hours driving. The total number of trips: 504 (trips of 5 km and longer lasting for at least 10 minutes after September 2019 were considered). To test the module for determining drowsiness, field experiments were conducted in the vehicle cabin, with the driver (having made sure of the safety of the performed actions) squinting his eyes (imitating the dangerous state of drowsiness) for three seconds. The approximate accuracy of the drowsiness determination module amounted to 69% in total according to the both criteria (5% of false positives). Similar experiments have been conducted to calculate the percentage of determined states of low attention. The driver’s task was to turn his head to the side for about 3 seconds. The approximate accuracy of the distraction determination module was 95% (2% of false positives). The heart rate detection module was tested on the Public Benchmark Dataset for Testing rPPG Algorithm Performance dataset. The experiments showed the accuracy of 80% -90% for different videos. The results of testing the module for automatic calibration of the drivers head rotation angle showed that about a minute of movement at a speed of 30 km / h is enough to determine the angle with an accuracy of several degrees, which acceptable for the task set.

 

Publications

---