AI Smart Surveillance System

Today we'll be making a cheap DIY AI Smart Surveillance system, which is capable of not only detecting humans, but is also capable of recognizing human faces. The system is capable of rivaling commercial solutions, and unlike some commercial options, is completely free of charge!

For this we'll be using Home Assistant, so we are able to view the live notifications on our smartphone and view the clips and snapshots of the recorded events.

Additionally we'll also make a simple alarm system, using a microcontroller and a buzzer. But you could easily replace with a led light or some else.

And as an extra we'll also make a small scale showroom model to make your creation more presentable.

1. Materials needed for the Surveillance system

Raspberry Pi 5 (link to the one I used)

Active cooler for the Raspberry Pi (recommended if you will be using it for an extended period of time)

Charging cable for the Raspberry Pi (USB-C)

Router (which you have direct access to)

Ethernet cable (to connect the Raspberry Pi to the router)

Laptop

Smartphone (if you wish to receive notifications)

SD card burner (if your computer does not have one built in)

MicroSD card (32-64GB should be fine) (link to the one I bought)

MicroSD card to SD card adapter to plug in into your SD card reader.

IP camera (link to the one I used) or a USB camera (not recommended)

2. Materials needed for the alarm system:

NodeMCU v3 (link to the controller used in this project)

Buzzer (link to the ones I used in this project)

Old charger (micro-USB)

Soldering equipment (Soldering iron, and soldering tin)

3. Materials needed for a scale model:

Old cardboard box(es)

A hot glue gun (or equivalent)

White (spray) paint.

Scissors

Box cutter knife or scalpel blade

2M long folding ruler

Extra:

A color printer and paper (for example to print detailed elements)

Note the total cost of this project, excluding tools and delivery costs is about €226.

Install the Raspberry Pi Imager (which can be found here)

Plug your SD card into your SD card reader

Open the Raspberry Pi Imager

Select the Raspberry Pi 5

Go to Specific-purpose OS > Home assistants and home automation > Homeassistant

Select your SD card

Burn the Home Assistant Operating System (HA OS) on your microSD card. (Note this might take some time.)

Plug your microSD card into your Raspberry Pi 5. 

Turn on your Raspberry Pi by plugging it into the power cable.

Connect your Raspberry Pi 5 to your home router via an Ethernet cable.

Type "http://homeassistant.local:8123", or "http://homeassistant:8123" into your browser, but if that doesn't work you can also type the IP that the Raspberry Pi (which is now running home assistant OS) has received from the router in the URL bar of your browser, followed by ":8123", so for example "http://192.168.1.30:8123".

(The easiest way to find the IP address of the Pi is by typing the command ipconfig into the cmd of your computer which should be connected to the same network as the router. This command should return a list of IP addresses. You should use the one which was not there before. If you are not sure, just try out some of the the IP v4 addresses, until you find the right one.)

If all goes well you should be greeted by the following dashboard:

After the waiting is done you'll be able to create a homeassistant account (requiring a username and password) to be able to log into your home assistant server in the future.

It's also recommended to configure your location and as well as the other things for a better experience.

Go to settings > add-ons > add-on store 

And choose file editor (so we are able to edit the contents of files). And hit install.

Go to settings > add-ons > add-on store > select the top left icon > Repositories 

And add the URL of the following repository in it to access frigate: https://github.com/blakeblackshear/frigate-hass-addons

Then refresh the add-on store page, and you will see several Frigate versions you can install. Make sure to install the "Frigate (Full Access)", which is an NVR with real-time local object detection for IP cameras, designed for Home Assistant.

What does MQTT stand for? MQTT stands for Message Queuing Telemetry Transport. It is an extremely simple and lightweight messaging protocol (subscribe and publish) designed for limited devices and networks with high latency, low bandwidth or unreliable networks. And will allow us to create automations and notifications later on based on events that happen.

First install EMQX (which is an MQTT broker) via the add-on store.

Open EMQX, and login with the default username and password, which are: admin and public.

Change the default password.

Go to authentication and go to the top right and create a new authentication method, keep the default and click create.

Open the users tab, and create a new "frigate" and "mqtt" user.

Now go to settings > devices and services

search for MQTT and under broker paste your EMQX hostname, in our case it is: a0d7b954-emqx, and as a username enter "homeassistant" and your password.

When using a USB camera:

We first need to install the add-on MotionEye to turn the footage into a http stream, else we won't be able to process the footage in frigate.

Go to settings > add-ons > add-on store > Motion Eye 

And then install it.

Login with the username: admin and leave the password blank. (Note you can change this default username and password if you want.)

Create a new camera.

Select the correct device and the video resolution, as well as what kind of format it is in.

Go Video Streaming > Useful URL's 

and select streaming URL and copy it for later.

Note you could change the streaming frame rate and the stream quality to fit your needs. I'm using a framerate of 15 and a streaming quality of 75%.

When using an IP camera:

You will have to look up your manufacturers rtsp stream online, as it is different for each manufacturer.

It's recommended to install VLC player on your computer. And try to connect to the stream from there. It's not recommended to do it directly from within frigate, because each time you want to change the configuration file, frigate will have to restart and you will waste quite a bit of time constantly having to restart the system and waiting for frigate to come online.

Note this process can take quite some time and can be quite challenging to debug, as there is often very little documentation for some manufacturers. Even getting the IP address of your camera can be quite challenging in some cases. In my case, I needed to install software from the manufacturer on my pc which scanned the network in order to get the IP address.

In my case it turned out to be as follows, using an IMOU Range2 IP camera:

rtsp://admin:xxxxxxxx@192.168.xx.xxx:554/cam/realmonitor?channel=1&subtype=1

Note the username (in this case admin) and password, followed by the IP address of your camera on the port 554. And it uses channel 1 and subtype 1.

Note that it could be, depending on the camera you use that you need to install ONVIF and go to settings > devices and services and add the ONVIF device.

When using a USB camera:

As frigate is not able to process the USB camera directly, we will have to use the http stream from Motion-Eye that we saved in the previous step.

If the code below does not work directly and you get the error similar to: "No frames have been received, check the error logs" and you can not solve it through the error logs, you will need to install FFMPEG (via the following GitHub link: https://github.com/FFmpeg/FFmpeg). You should try to connect to the stream via the command line, and ask GPT to decode what it's saying. Most likely you will need to change the FFMPEG commands within frigate.

(Note that this part might be especially challenging and hard to debug, especially if you have no experience with using these tools, so please keep that in mind.)

frigate.yml file needs to look as follows: (Note you need to replace the x's with your own data)

mqtt:
 host: a0d7b954-emqx
 user: frigate
 password: xxxxxxx
detectors:
 cpu:
  type: cpu
  num_threads: 2

cameras:
 cam1:
  ffmpeg:
   inputs:
    - path: http://192.168.xx.xx:9081
     roles:
      - detect
      - rtmp
      - record
   input_args: -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt
   output_args:
    record: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v libx264 -an
    rtmp: -c:v libx264 -an -f flv
  detect:
   width: 640
   height: 480
   fps: 5
  record:
   enabled: true
  snapshots:
   enabled: true
  rtmp:
   enabled: true
  objects:
   track:
    - person
  motion:
   mask:
    - 570,480,640,480,640,457,567,456

Make sure to include your MQTT broker hostname and the user that we made in step 6. We'll be using 2 CPU cores for this. Note that I also added a motion mask over the timestamp, so the CPU has to do less work. (You can find more about it here.)

When using an IP camera:

You can keep the "mqtt:" and the "detectors:" the same, but under camera's make sure to add your rtsp stream.

This is the code I used:

 cam2:
  ffmpeg:
   inputs:
    - path: rtsp://admin:xxxxxxxx@192.168.xx.xxx:554/cam/realmonitor?channel=1&subtype=1
    
     roles:
      - detect
      - rtmp
      - record
  detect:
   width: 1280
   height: 720
   fps: 5
  record:
   enabled: true
  snapshots:
   enabled: true
  rtmp:
   enabled: true
  objects:
   track:
    - person
  motion:
   mask:
    - 0,0,214,0,423,0,648,0,568,291,394,290,196,590,0,720
    - 876,75,1232,83,1235,29,880,30

If you wish to do face recognition as well, add the following GitHub repository in your add-ons: https://github.com/jakowenko/double-take-hassio-addons

Now install both Deepstack (the AI engine) and DoubleTake (the UI to train our face recognition model.)

Add the following to config file of double take:

mqtt:
 host: a0d7b954-emqx
 user: doubletake
 password: xxxxxx
  
frigate:
 url: http://192.168.xx.xx:8123/ccab4aaf_frigate-fa/ingress
  
detectors:
 deepstack:
  url: http://192.168.xx.xx:5001
  key:

Also don't forget to create a new user in EMQX called 'doubletake'. Note that you might need to try some different url's, IP addresses and ports, as these can be different, and might require some troubleshooting.

If everything is set up correctly, when a person stands in front of the camera and Deepstack is able to detect there is a face, it will show up in the DoubleTake UI as unknown.

Now create a new class, for example naming it the person you would like to detect. 

You can upload personal images of the individuals you would like to detect, but note that often your model won't be able to detect the faces of the person accurately, especially if the size of the face in the image is too small, or the images the model trained on are too different from the ones from the camera.

Luckily we can also train the model almost in real time on the footage from the camera itself. We do this by selecting the image, selecting which person it is, and then hitting train. And when that is done you can press the retry button at the bottom of the card and you can check if it is now able to detect that person.

If you which to have a robust model it's highly recommended to leave your camera on for at least a couple days and train it on as many different facial expressions as possible, under varying lighting conditions.

For example with your eyes open or closed, leaning forward or backwards, or with attributes like sunglasses.

It's recommended to train your model each day at the end of the day on the detections it got wrong. If you repeat this process for many days you should be left with a relatively strong and robust model.

This video does an excellent job of explaining how it works.

If you wish to receive notifications on your smartphone you can download the Home Assistant app from the Google Play store.

Add the IP address of your Home Assistant server, or add your personal domain name if you have one. (Note we'll be discussing that last option in step 14.)

Go to settings > Automations & scenes 

and create a new automation called: Frigate

Make it so when an MQTT message has been received on the topic: "frigate/events", then Call a service 'Notifications: Send a notification'

Here you can find my sample code for the notification:

service: notify.notify
data:
 message: A {{trigger.payload_json["after"]["label"]}} was detected.
 data:
  tag: "{{ trigger.payload_json['after']['id'] }}"
  group: "{{ trigger.payload_json['after']['camera'] }}"
  when: "{{ trigger.payload_json['after']['start_time']|int }}"
  image: >-
   https://xxxxxxxxxxxxxxxxx/api/events/{{
   trigger.payload_json['after']['id'] }}/thumbnail.jpg?format=android

Making your home assistant dashboard accessible from outside of your home network.

We have several options we can choose from to do this:

1. Port forwarding via the router

2. Via a VPN (Virtual Private Network)

3. Via a personal domain name and a Cloudflare tunnel

Option 1 is the easiest and most straightforward, but quite insecure.

Option 2 is already a better solution and relatively simple, but requires one to install a VPN client on one's devices and as an add-on on home assistant.

Option 3 is the best option, as it allows one to access one's dashboard from anywhere in the world, from any device which has a browser, the only downside is that it requires quite a bit more time to set up.

Therefore, we'll be doing option 3:

Create a Cloudflare account

Get a free domain name. (For example via names.co.uk, note it might take quite a few days to get your your own domain name.)

Add your Cloudflare DNS servers to your domain name on the site you got your domain name from. (Note that it could take up to 12-48 hours before the DNS servers are updated.)

Install the Cloudflared add-on via this repository: https://github.com/brenner-tobias/ha-addons

Input the IP address of your Home Assistant server.

Add your personal domain name.

Now restart your Home Assistant server.

When you go to the logs you should hopefully get a link, as can be seen in the included image. Copy it and browse to it and authorize the Cloudflare tunnel.

Now you should be able to access your Home Assistant dashboard by entering: "https://" followed by your domain name in your browser. Once you hit enter you should see '400: Bad Request' error, this is actually good as we are getting a 400 response, it tells us the server is actually responding to our query and the tunnel is up.

We can solve this by going to the Cloudflared add-on documentation, and find the Home Assistant configuration example:

http:
 use_x_forwarded_for: true
 trusted_proxies:
  - 172.30.33.0/24

Add this to the bottom of your configuration.yml file and restart Home Assistant.

Now if you try to visit your personal domain name again you should be greeted with the home assistant login page, which means our tunnel is now up and working correctly.

This video might be helpful.

Take your buzzer and your NodeMCU and solder them together. Make sure the positive (+) sign on the buzzer is connected to GPIO pin 12 (D6) and the other side is connected to the Ground (G or GND).

The final product with the charger attached should look as follows:

Install Tasmotizer on your computer. (GitHub link: https://github.com/tasmota/tasmotizer)

Install the required drivers (CH340 or equivalent). A tutorial can be found there: https://learn.sparkfun.com/tutorials/how-to-install-ch340-drivers/all)

Connect the NodeMCU v3 to your computer using a microUSB cable. (Note you will need one which also can send data back and forth, the ones that only can deliver power won't work.)

Flash the Tasmota firmware on the NodeMCU. Note that it could be that you will have to press the reset (RST) button followed by the FLASH button on the NodeMCU.

Once the previous step is complete, a new network should pop up named tasmota. Connect to it, it should normally open a webpage at 192.168.4.1

Choose the right Wi-Fi network it should connect to and input the password of the Wi-Fi. Hit save. 

Normally, you should now get the IP of your Tasmota device, but if not the case, you can also get it's IP by clicking "Get IP" in Tasmotizer, and paste that into the browser instead.

Now you should be greeted with your Tasmota dashboard.

You should now see toggle on your screen. This button is by default connected to GPIO 12 or D6. 

Go to configuration and change the board layout to General (18), and select Buzzer for GPIO 12. Hit save.

Depending on what kind of buzzer you have it could be that you already get a continuous beep if you turn it on. But if you do not hear a continuous beep, go to configuration and change the board layout to General (18), and select PWM - 1 for GPIO 12. Hit save.

Now you can change the frequency with the slider. And when you click ON, it should hopefully beep continuously.

(Note you can also control the buzzer with commands from the terminal now, for example if you want to change the frequency or turn the power On or Off. This will come in handy in the following step.)

In order for us to trigger the alarm, we also need to configure MQTT on the device. 

Fill in the IP of the Home Assistant server, and leave the default port unchanged.

As the user fill in "alarm". (Note you will also need to create a new user named "alarm" in EMQX as well.)

And as the topic fill in "offbuzz".

And fill in the password.

And hit save.

We will now make an automation in Home Assistant, making it so when there is an event in frigate (a person is detected) it will buzz for 3 seconds.

So the automation will have to look as follows:

when an MQTT message has been received on the topic "frigate/events", we "call a service 'MQTT: Publish' " on the topic "cmnd/offbuzz/POWER" and we set the payload to "on". We add a delay of 3 seconds. And we publish a message again on the same topic with the payload being: "off".

Now when you walk in front of your camera your mini alarm system should make some noise.

(Note if you wish to turn off the alarm you can ways go to settings > Automations & scenes > and toggle on or off your automation.)

In order to make your project more presentable, you could create a miniature scale model of your house, or the environment you'll be surveilling.

Measure the dimensions of your home, using the 2M folding ruler or some other tool.

Note them down somewhere, for example on a piece of paper.

Decide what scale you want your model to be for example 1/50, or 1/100, or something else.

Convert the measurements you made into the right dimensions for your model. (For example something which is 3m in reality will become 3cm, if using a 1/100 scale.)

Take old cardboard boxes and cut it in pieces. (Make sure to not forget to find a big and strong piece which can function as the baseplate on which everything else can be glued.)

Spray-paint these pieces in the color that you like.

Make markings on the pieces where you need to make cuts.

Cut them out with a box cutter knife or a scalpel blade. Also don't forget to cut out the windows or doors in your model now, because doing it later will be a lot harder.

Additionally you could also add more detail to your model by printing things like laminate flooring on paper using a printer, and then gluing it onto your model.

Glue the first wall to the baseplate, and then follow up by gluing the other walls to both the first wall and the baseplate in order to keep the structure as strong as possible. And continue this process until all walls are put in place.

Now you can add additional details like furniture if you want to.

Making a small scale model of your camera, as well as a human could be useful to get an idea of scale for people in the audience.

The final product in my case looks as follows: