ST AEKD-AICAR1 Bedienungsanleitung

Introduction

The AEKD-AICAR1 is a versatile deep learning system based on a long-short term memory (LSTM) recurrent neural network

(RNN). It is able to classify the car state:

•car parked

• car driving on a normal condition road

• car driving on a bumpy road

• car skidding or swerving

The main idea is to define an AI-car sensing node ECU with an embedded artificial intelligence processing.

The system hosts an SPC58EC Chorus 4M microcontroller, which is able to acquire discrete acceleration variations on a

three-axis reference system.

The AIS2DW12 motion sensor mounted on the AEK-CON-SENSOR1 board retrieves inertial data. The acquired data are

transmitted to the LSTM RNN, which classifies the car state. The classification result is shown on the AEK-LCD-DT028V1 LCD

touch display.

The LSTM RNN has been implemented and trained using the TensorFlow 2.4.0 framework (Keras) in the Google Colab

environment. The AI-SPC5Studio plug-in has been used to convert the resulting trained neural network into an optimized C

code library, which can run on an MCU with limited power computing resources.

Figure 1. AEKD-AICAR1 evaluation kit

The LSTM RNN training has been performed with several time-series acceleration waveforms recorded on a real vehicle in

motion. The resulting prediction accuracy, calculated by the confusion matrix, is about 93%. Field tests carried-out under all road

conditions with a sedan confirm the adherence of the computed results compared with the real road conditions.

Note: The AEKD-AICAR1 is an evaluation tool for R&D laboratory use only. It is not destined for use inside a vehicle.

Getting started with the AEKD-AICAR1 evaluation kit for car state classification

UM3053

User manual

UM3053 - Rev 1 - September 2022

For further information contact your local STMicroelectronics sales office. www.st.com

1Neural network basic principles

1.1 Artificial neural network

The artificial neural network works as the human brain neural network. Data are transferred to the neuron through

the inputs. Then, they are sent as an output after processing.

Figure 2. Artificial neural network

Artificial neural networks are built on layers of different neural units. Each unit consists of three parts:

• an input part that receives the data

• a hidden part that uses the neuron weight to calculate the result

• an output part that receives the calculation results and applies an eventual bias

The weight associated with each neuron determines its firing probability. The bias is the measure of assumption

made by the form of the output.

This architecture is typical for deep learning processes like data classification and pattern recognition neural

networks.

1.2 Long short-term memory recurrent neural network (LSTM RNN)

Traditional artificial neural networks keep no memory of what happened in the past. They take their decision only

on the data provided instant by instant. These architectures are well suited for data classification and pattern

recognition.

For other applications, like speech recognition, the proper classification requires a memory of the context, that is,

the prior words in the speech recognition application.

A recurrent neural network (RNN) is a class of artificial neural network that includes neurons connected in a loop.

UM3053

Neural network basic principles

UM3053 - Rev 1 page 2/39

Figure 3. Recurrent neural network

In most cases, the output values of an upper layer are used as the input for a lower level one. This

interconnection enables the use of one of the layers as the state memory and allows modeling a dynamic time

behavior dependent on the information previously received.

A typical recurrent neural network able to keep the information received is the long short-term memory (LSTM)

recurrent network. Unlike traditional artificial neural networks, LSTM has feedback connections. It can process not

only single data points but also entire sequences of data. For this reason, the LSTM neural network architecture is

the best candidate to model a deep learning system for an AI-car sensing node, using a collection of time-series

based on acceleration values.

UM3053

Long short-term memory recurrent neural network (LSTM RNN)

UM3053 - Rev 1 page 3/39

2Designing an AI-car sensing node

2.1 Tool-set introduction

An AI-car sensing node is an AI-deep learning system based on an LSTM recurring neural network, which can

provide a car state classification: parking, driving on a normal condition road, driving on a bumpy road, and car

skidding or swerving.

Figure 4. AI-car sensing node: car state classification

The LSTM RNN has been modeled with TensorFlow (Keras framework). This is an open-source software library

for machine learning, which provides optimized modules to implement AI algorithms related to the classification

problem.

A significant amount of computing power is required to implement sufficiently robust and efficient models. For

initial tests, we can rely on a standard machine. As the dataset size increases, the execution of complex training

algorithms becomes rapidly prohibitive.

To address this issue, there are several cloud services that offer computing power. Google Colab is an alternative

platform, which allows running the code directly on the cloud, even if with some limitations.

2.2 Creating a Google Colab notebook

To use the Google Colab platform, you need a Google account to login. After logging in, create a new “Colab

notebook” project file.

UM3053

Designing an AI-car sensing node

UM3053 - Rev 1 page 4/39

Figure 5. Project file

The document that you are creating is not a static web page. It is an interactive environment that lets you write

and execute your Python code to implement an LSTM RNN by using the TensorFlow framework.

2.3 Colab notebook setup and package importing

To implement and train an LSTM RNN, install the Tensorflow 2.4.0 framework on your Google Colab notebook by

using the packet index package (PIP) command.

%pip install tensorflow==2.4.0

Note: PIP is already installed on your Google Colab notebook.

Using the Python commands, import the following packages:

•Pandas

Open-source software library for data manipulation and analysis.

import pandas as pd

•Numpy

Open-source software library, adding support for large, multidimensional arrays and matrices, along with a

large collection of high-level mathematical functions to operate on these arrays.

import numpy as np

•Tensorflow

Open-source software library for machine learning, which provides optimized modules to implement artificial

intelligence (AI) algorithms.

import tensorflow as tf

•Sklearn

A package providing several common utility functions and transformer classes to change raw feature vectors

into a representation that is more suitable for the downstream estimators.

from sklearn.preprocessing import StandardScaler, MinMaxScaler

from sklearn.model_selection import train_test_split , StratifiedShuffleSplit

from sklearn.metrics import confusion_matrix

•OS

The Python language OS module has several useful functions to make the program interact with the

computer operating system.

import os

UM3053

Colab notebook setup and package importing

UM3053 - Rev 1 page 5/39

•Random

This module implements pseudo-random number generators for various distributions.

import random

•Seaborn

Seaborn is a Python data visualization library based on Matplotlib. It provides a high-level interface to draw

attractive and informative statistical graphics.

import seaborn as sn

•Matplot

Matplotlib is a comprehensive library to create static, animated, and interactive visualizations in Python.

import matplotlib.pyplot as plt

2.4 AI-car sensing node life cycle

The following steps define the life cycle of the AI-car sensing node implementation as a deep learning model for

classification:

1. Model definition

2. Model training

3. Model fitting and compilation

4. Model evaluation

2.4.1 Model definition

To define the model, we have chosen the topology of the LSTM network. The AI-car sensing node network is

based on the same network architecture of a speech recognition system. It consists of a neural convolution kernel

as an input layer, able to elaborate a convolution function with the input vector over a temporal dimension.

The input vector is a time sequence of TIMESERIES_LEN number of samples, which consist of discrete

acceleration variations on a three-axis (x, y, z) reference system: Δax, Δay, and Δaz.

TIMESERIES_LEN represents the minimum size of the temporal window for the car state classification. With an

acquisition sampling time equal to 100 msec, the value of TIMESERIES_LEN has been fixed to 50 samples (5

seconds per acquisition).

A dense function implements the output layer. This function can provide an output shape of four-dimensional

vectors (one for each expected status: parking, normal, bumpy, skid). The Softmax function activates this layer

and converts the output vector values to a probability distribution.

From an implementation perspective, this involves a model layer architecture built with a connection topology into

a cohesive model:

UM3053

AI-car sensing node life cycle

UM3053 - Rev 1 page 6/39

## Conv1D based model

model = tf.keras.models.Sequential([

tf.keras.layers.Conv1D(filters=16, kernel_size=3, activation='relu',

input_shape=(TIMESERIES_LEN, 3)),

tf.keras.layers.Conv1D(filters=8, kernel_size=3, activation='relu'),

tf.keras.layers.Dropout(0.5),

tf.keras.layers.Flatten(),

tf.keras.layers.Dense(64, activation='relu'),

tf.keras.layers.Dense(4, activation='softmax')

])

_________________________________________________________

Layer (type) Output Shape Param #

=================================================================

conv1d (Conv1D) (None, 48, 16) 160

conv1d_1 (Conv1D) (None, 46, 8) 392

dropout (Dropout) (None, 46, 8) 0

flatten (Flatten) (None, 368) 0

dense (Dense) (None, 64) 23616

dense_1 (Dense) (None, 4) 260

=================================================================

Total params: 24,428

Trainable params: 24,428

Non-trainable params: 0

As described in the above code block table, 24,428 trainable parameters define the AI-car sensing node network

model. These parameters are related to the LSTM RNN topology chosen.

2.4.2 Model training

The AI-car sensing node network training has been performed by acquiring datasets of acceleration variations in

the time domain:

•over a three-axis reference system (Δax, Δay, Δaz)

• with a frequency equal to 10 Hz

• a sample time of 100 msec

Each dataset targets a specific car state: parking, normal driving, bumpy driving or skidding.

The use of acceleration variations instead of raw acceleration data avoids misinterpretations of the car state. If

the car is parked on a road with a negative (or positive) slope (as shown in the figure below), the IMU registers

a nonzero X-axis acceleration that would not lead to the parking state. Instead, if we consider an acceleration

variation in the time domain, the result would be 0, leading to the correct parking state.

UM3053

AI-car sensing node life cycle

UM3053 - Rev 1 page 7/39

Figure 6. Acceleration variations

Car speed Vx = 0 Km/sec;

Car acceleration ax! = 0;

Temporal variation of acceleration Δax = 0

2.4.2.1 Training dataset

The acquisition of training datasets can be performed using a specific configuration of the AI-car sensing node

mock-up (acquisition mode). A proper source code #define enables the acquisition mode. In this mode, the

application sends each acceleration sample read from the sensor to a serial port with a baud rate set to 38400.

The serial data stream can be read through a common serial client (Tera Term or Putty). The data acquisition and

parsing in a CSV file are managed via a specific Python script (see Section Appendix C ).

The following figure shows an example of a time-based dataset generated with the Python script.

UM3053

AI-car sensing node life cycle

UM3053 - Rev 1 page 8/39

Figure 7. Time-based dataset example

The time column contains the recorded acceleration time samples. The other three columns contain the

acceleration values measured on the X, Y, and Z axis, respectively (without the gravity offset).

To prepare the file for the network training, you have to:

• Compute the acceleration variations (excel cell operation: B3 = B3–B2 for X axis).

UM3053

AI-car sensing node life cycle

UM3053 - Rev 1 page 9/39

• Add a status column for the car state for each specific acquisition (P = parking; N = normal; B = bumpy; S=

skid).

Figure 8. Adding the status column

• Create a CSV file for each car state through a specific acquisition task.

Figure 9. Creating CSV files

• Merge all CSV files by coping and pasting each row without changing the time column values.

• Add a new dummy row at the end of the file with the time value equal to 100.

• Add random values for Ax, Ay, and Ax status.

Note: A ready-to-use dataset (Diff_profile.csv) has been created to train the network for the car state classification.

2.4.2.2 Neural network training with Google Colab

Import the training dataset contained in the CSV file into the Colab notebook environment. Then, run a parser

function on the imported data to build an input vector compliant with the LSTM RNN.

The following code block shows the Google Colab script, which imports and loads the CSV file.

from google.colab import files

uploaded = files.upload()

db = pd.read_csv('Diff_profile.csv',sep=',')

Parse each column of the CSV file (status, Acc_x, Acc_y, Acc_z, time).

UM3053

AI-car sensing node life cycle

UM3053 - Rev 1 page 10/39

Inhaltsverzeichnis

Andere ST Hauptplatine Handbücher

ST STEVAL-MKI019V1 Bedienungsanleitung

ST STEVAL-ILH007V1 Bedienungsanleitung

ST STEVAL-MKI005V1 Bedienungsanleitung

ST UM2313 Bedienungsanleitung

ST STEVAL-MKI109V2 Bedienungsanleitung

ST EVB-LIV4F Bedienungsanleitung

ST STEVAL-IHM020V1 Bedienungsanleitung

ST STA333ML Montage- und Bedienungsanleitung

ST STEVAL-USBPD45H Administratorhandbuch

ST SPC58NHADPT176S Bedienungsanleitung

ST STM32MP157D-EV1 Bedienungsanleitung

ST STSW-STUSB002 Bedienungsanleitung

ST STEVAL-ISW001V1 Bedienungsanleitung

ST FP-SNS-6LPNODE1 Bedienungsanleitung

ST EVB-LIV3F Bedienungsanleitung

ST STEVAL-IHM022V1 Bedienungsanleitung

ST X-NUCLEO-CCA02M1 Bedienungsanleitung

ST STM32H757I-EVAL Bedienungsanleitung

ST STEVAL-AKI002V1 Bedienungsanleitung

ST EVAL-L9001 Bedienungsanleitung

ST SPC58XXADPT144S REV.B Bedienungsanleitung

ST EVAL-L9963-MCU Montage- und Bedienungsanleitung

ST STEVAL-IHM015V1 Bedienungsanleitung

ST STEVAL-AETKT2V1 Bedienungsanleitung

Beliebte Hauptplatine Handbücher anderer Marken

DFI

DFI SD170 Bedienungsanleitung

ASROCK

ASROCK ROMED8-2T Bedienungsanleitung

Asus

Asus P2L97A Bedienungsanleitung

Winmate

Winmate IC70 Bedienungsanleitung

Asus

Asus MES-B Bedienungsanleitung

Grantech

Grantech SYS8F377VGGA-3OMN Bedienungsanleitung

Telit Wireless Solutions

Telit Wireless Solutions SL869-3DR Bedienungsanleitung

Texas Instruments

Texas Instruments ADS8372EVM Bedienungsanleitung

Commell

Commell MS-C73 Bedienungsanleitung

Gigabyte

Gigabyte GA-9IVDT Bedienungsanleitung

IBT Technologies

IBT Technologies MB860 Bedienungsanleitung

Nvidia

Nvidia TEGRA DG-04927-001_V01 Bedienungsanleitung

Intel

Intel D33217CK Bedienungsanleitung

Asus

Asus F1A55-M LX PLUS R2.0 Bedienungsanleitung

EPC

EPC EPC9147C Bedienungsanleitung

Asus

Asus EX-B760M-V5 D4 Bedienungsanleitung

EPOX

EPOX MU-8KHA0010 Bedienungsanleitung

Ski-Doo

Ski-Doo Bombardier MPEM 219 700 090 Bedienungsanleitung