Anomaly detection and event recognition in cars based on multimodal sensor data interpretation

Abstract

The benefits of analyzing driving behavior extend across various sectors, including insurance, transportation planning, and autonomous vehicle development. Insurance companies can customize policies based on individual risk profiles, promoting safer driving habits. In fleet management, the analysis assists in risk control, regulatory compliance, and enhancing customer satisfaction. Additionally, it plays a pivotal role in detecting and preventing accidents, contributing to safer road environments. Over the past decades, significant advancements in Machine Learning (ML) techniques, particularly in learning relevant features for abstracted data representation, have been observed. The emergence of deep learning, utilizing Deep Neural Networks (DNNs), has further accelerated this trend, showcasing remarkable performance with ample data. This intersection has unlocked new possibilities for studying driving behavior, with ML playing a crucial role in extracting valuable insights from extensive driving data sets. However, applying ML in this domain presents challenges. This outcome can be attributed to a variety of influencing factors. Driving involves a complex blend of cognitive, psychomotor, and perceptual activities that are hard to quantify and model accurately. Therefore, in this work, in-car sensor data is employed, as it is cost-efficient, widely accessible, and provides access to comprehensive vehicle parameters (e.g., speed and acceleration) with real-time data precision. Driving behavior is inherently subjective, exhibiting significant variability among individuals and even within the same individual under different circumstances, which makes the labelling of data difficult and unreliable. To address this problem, this study leverages both supervised and unsupervised machine learning approaches and DNNs to detect all possible abnormal driving patterns in naturalistic driving patterns. Capturing comprehensive real-world driving data that can reflect the full range of these variables is a massive, if not impossible, task. Detailed recording of individual driving behaviors can raise significant privacy concerns, and a true ground truth of dangerous driving behavior raises ethical considerations. This work presents a naturalistic driving data set (performed by drivers, spanning basic to professional skill levels) carefully assembled and supervised by experienced driving instructors. This data set encompasses annotated hazardous driving patterns derived from in-car sensors, which mitigates privacy concerns inherent in radar or visually-based data modalities. Imbalanced data sets, a lack of positive (anomalous) samples, and interpretability issues in complex ML models further complicate the landscape. Therefore, comprehensive feature extraction methods using ML and DNNs are employed in this work to detect accidents within a naturalistic data set. This work aims to address these challenges and gaps in the literature, focusing on anomaly detection and event detection in driving behavior analysis. The investigation revolves around two categories of questions: 1. Utilizing primary in-car sensors using ML approaches: - Is it feasible to use primary in-car sensors with ML approaches to detect abnormal driving patterns? - Is there a benefit in employing unsupervised deep learning models for anomaly detection compared to traditional ML models? - Can the proposed solution be applied to a benchmark driving data set effectively, considering the lack of labeled driving patterns? 2. Detecting real-world accidents based on primary in-car sensor data: - Is it possible to detect real-world accidents using primary in-car sensor data? - What is the best feature extraction method for accident detection, and which features contribute significantly to the classification result? Each of these questions is examined in detail and has led to new insights, using advanced machine learning techniques to manage the complexity of detecting abnormal driving behaviour, including accidents. Chapter 2 unfolds into three significant sections, each offering valuable insights into anomaly detection in driving patterns. The initial segment introduces a foundational PRC framework for anomaly detection, achieving outstanding performance with supervised k-Nearest-Neighbors (kNN) and impressive results with unsupervised Gaussian Mixture Model (GMM). The second section delves into unsupervised anomaly detection using the proposed PRC framework on unlabeled Controller Area Network Bus (CAN-Bus) signal values, emphasizing the effectiveness of Autoencoders (AEs), particularly the noteworthy Long-Short-Term Memory Autoen coders (LSTM-AE), in detecting anomalous driving patterns based on speed and brake signals. The final section presents a benchmarking data set of naturalistic hazardous driving behavior, yielding remarkable results with Handcrafted Features (HC) classified by Support Vector Machine (SVM). In chapter 3, a framework for accident detection using in-car sensors from a naturalistic data set is presented, employing diverse ML approaches. Notably, the combination of Convolutional Neural Network (CNN) features and an SVM classifier stands out, achieving impressive accuracy and showcasing promising performance given the reliance on naturalistic accidents and the limited samples of severe accidents recognized by only four basic in-car sensors. Interpretability studies reveal the complementary nature of traditional feature engineering and DNNs in extracting optimal features from different sensor channels, enhancing overall effectiveness. Despite challenges in data acquisition and dealing with imbalanced data, this work significantly advances exploring and benchmarking various anomalies and accident detection in naturalistic driving data sets. Outstanding results indicate the strong potential for driving behavior analysis using ML and DNNs utilizing in-car sensor data.