Artificial Intelligence to Localize Thanks to Echos As Bats

For the fun I wanted to design from scratch a home made robot that would be able to move autonomously inside my house.

Firstly I worked on mechanical, electronic and and software in order to be able to control the moves (for instance DC motor rotation or use of Inertial Measurement Unit). (Motion Control)

Even if this prototype need improvements, it is reliable enough to go from one determined place to a desired one.

But for the robot to run autonomously, the robot must be able to regularly answer the question : where am I ?

According to the amount of experiences we can find on internet,I should have tried with image recognition but I choose to try a other lead.

Echo sensors are frequently used for obstacle avoidance. I search over Internet about using those sensors for robot localization inside a space but I got nothing.

I wanted to explore the possibility to use echos to do so and this instructable is about work.

The 3 main questions are:

1. where am I ?
2. where am I going ?
3. how to get there ?

This instructable will ignore the second one and focus on the first one (red dots rectangle). It is important to understand that if the robot position is known precisely at the instant t0 each move, even with a high performance motion control, will add some imprecision in the futur location.

So the system to design must be enslaved to some feedbacks from real world in order to keep theoretical and real points of view as close as possible.

The idea is quite simple

1. set the robot in a predefined orientation using a compass

2. use echo sensors to get a set of distances 360° around the robot

3. build a 2D image with the set of echoes

4. compare this image with a set of previously learned ones

5. determine the current robot location

Space is divided in 30 x 30 cm squares.

Each square is divided into 9 sub-squares of 10x10cm. For each sub-square an integer defines the characteristics (roughly occupancy)

Location is defined by (x,y,h) coordinates in a 2D orthogonal system where h is the heading angle

The North orientation of the X and Y axis is know.

Robot location reference is in the middle of the wheels axes.

The robot heading inside the orthogonal system is defined by h angle
The robot absolute heading is defined by No angle

The system is made of 2 ultrasound sensors and a servomotor and is controlled by the Arduino Mega.

Ultrasonic sensors HY-SRF05 characteristics:

• · Sound frequency 40kHz
• · Measurement range 2cm to 450cm
• · Measurement resolution 0.3cm
• · Measurement angle 15°

Servomotor with metallic gears FT5316M

The system is used both for localization and obstacle detection.
For localization it makes scan by rotating the servomotor from 0 to 180° in 15 steps.

That means 13° rotation by step, a little bit less than the sensor measurement angle. 0° and 180° scans provide crossed front and back distances.

The 2 back to back HC-SRF05 sensors are alternatively triggered to provide a 360° scan.
Sensor returns 0 when it does not get any echo back to a trigger.

A tensorflow Conv 2D model has been trained to learn each 30x30cm squares 360° echoes scan.

the points to bear in mind for this instrutable is that the robot transfers the raw data of scans to a "server application" (Java + Octave code) that run on a Windows server.

The "server application" converts the raw data to an image and provides it to the tensorflow model that returns a list of square locations with associated probabilities.

Take into account the sensor limitations:

• the complexity of the beam
• the measurement range (400cm)

The complexity of echoes:

• depends of the raw material of the obstacle
• depends of the orientation of the obstacle
• depends of the orientation of the robot
• depends of the shape of the obstacles
• depends of the distance of the obstacles

In exactly the same condition, 2 consecutive scans provide both light and significative data differences.

But if we look at the data as an image they seems very close. That is why I thought images deep learning tools could be used as well for echoes recognition.

This picture shows the impact of a move of around 30cm all around a central position (184,184) on the image of the 360 echoes scan.

We can see when the robot is geometrically translated, new image is not a simple geometric translation of dots

1. be able to determine position when the robot is roughly at the learning conditions
2. be able to approximate position when the robot is some between learning places
3. be able to approximate position when robot heading précision is +-10°

I did some tries with simple logistic regression algorithm that were efficient to determine to face the challenge 1 only.

But with a Conv2D logistic regression algorithm it is possible to face all the challenges

Learning needs few epochs. The point is to avoid over fitting.

I stopped learning after reaching 95% of accuracy with training and validation data.

I got also over 95% of accuracy with the testing data

So theoretically it works but what about real world ?

The first tests are encouraging.

• Set the robot at a learned location but with a random heading
  • At first the robot is 0° oriented using compass. Due to the magnetic environment and compass limitation, the actual heading is +-8°.
  • Nevertheless the prediction is more than 90% the right one
• Set the robot at a random location and with a random heading
  
  • The prediction is more than 90% the one of the closest possibility

The above picture illustrates an example of determination result.

The robot has been set at a random position and then aligned to x axis with the compass.

There is a difference between theoretical and actual heading that will introduce noise in the measurements

• The best possible prediction is (X1,Y1,0) the closest from the robot
• The prediction is (X2,Y2,0)
• The robot actual position is (X0,Y0,6)

So the predicted position is rather close to the actual one.

But to be able to run autonomously the determined location must be more precise. If (X,Y) error could be rather easily corrected by doing consecutive moves & scans, it is much more difficult to deal with the heading error that impatcs next (X,Y).

I am working on 2 different directions.

• Firstly use particles filters as a mean to get location convergence with consecutive moves & scans.
• Secondly run the prediction on a microcontroller (Portenta) with tensorflow lite instead of a windows system.

I use Colab to developp and train the tensorflow model.

To reduce matrix size echo distances are divided by 10. That means the maximal X and Y values are 80 and the minimal are -80.

1 pixel = a 10x10cm square - above an example of reduced size image

Prediction code is written in python without Colab

The Model

_________________________________________________________________
Layer (type) Output Shape Param # =================================================================

conv2d_8 (Conv2D) (None, 80, 80, 8) 16 _________________________________________________________________

max_pooling2d_8 (MaxPooling2) (None, 40, 40, 8) 0 _________________________________________________________________

conv2d_9 (Conv2D) (None, 40, 40, 16) 1168 _________________________________________________________________

max_pooling2d_9 (MaxPooling2) (None, 20, 20, 16) 0 _________________________________________________________________

flatten_4 (Flatten) (None, 6400) 0 _________________________________________________________________

dense_4 (Dense) (None, 68) 435268 =================================================================