Privacy-protecting Family/Elderly Monitor Using Low-res Thermal Imaging (MLX90640/ESP32)

The write-up for this project is incomplete -

if there's enough interest, I'll finish it, or just contact me for the code and other details!

This project shows how to use relatively inexpensive thermal imaging sensors to watch over your loved ones. The low resolution of the image protects their privacy, and the thermal spectrum enables day/night operation and (to be blunt) confirms the living warmth of the subject.

Note this is NOT intended as a primary safety measure - it's a nice-to-have backup when grandma is not answering her phone, and you're worried she might have fallen or passed out. (It has saved us a few trips up to my father-in-law's when he forgets to charge his phone...)

The cameras require minimal assembly, just an ESP32 Supermini and the MLX90640 in a housing, plus a spare phone charger and cable. It'll be best if the connections are soldered, although if you're averse to soldering you can probably get away with jumper wires, tape and a bit of hoping.

The camera links to their WiFi and with some added setup can be viewed from anywhere, there are no monthly fees involved. You can use any web browser (use a personal VPN or Tailscale account to view remotely - details below), or you can link to a public or private MQTT broker and view the images in an Android app. There's even the option to record timelapse videos to a local computer using a Python script (still in development).

Now if grandma is not answering her phone, you can see she's just napping in her recliner!

A barebones project outline can be found here (Google Slides).

This project should never be relied upon as a medical safety device. It's intended for those who have an independent but frail family member living on their own, as an alternative means of checking on them when the usual channels are not available.

It should only be installed in areas where all occupants are aware of, and consent to, this type of video monitoring.

ESP32 C3 Supermini board (recommended, but almost any ESP32 should work with adaptations for the housing)

1. AliExpress: https://www.aliexpress.com/item/1005007446928015.html (AU$3)
2. Amazon (US): https://www.amazon.com/dp/B0D47G24W3/ ($14 for 3-pack)
3. Australian local dealer: https://core-electronics.com.au/esp32-c3-mini-dev-board-risc-v-processor-wi-fi-bt5-smd-compatible.html ($8)
4. eBay: https://www.ebay.com.au/itm/395294295533 or https://www.ebay.com.au/itm/395914094191

MLX90640ESF-BAB sensor (get the 55-degree version, the 110-degree wide angle is for closeup work)

1. AliExpress: https://www.aliexpress.com/item/32962087015.html (AU$43)
2. Amazon (US): https://www.amazon.com/dp/B0F2MYRJ4V?th=1 ($46)
3. Digikey https://www.digikey.com/en/products/detail/melexis-technologies-nv/MLX90640ESF-BAB-000-TU/8638464
4. NOTE: do NOT buy the MLX90614, it’s much cheaper but it’s a SINGLE-PIXEL sensor, intended for infrared thermometers!

Any USB power source, such as an old phone charger

Optional: 3D Printer for the housing - or DIY with cardboard? - and some kind of removable adhesive to affix camera to a wall or table (BluTack, 3M Command Adhesive strips)

The 3D printed housing design assumes an M6 bolt is used connect the base, you can easily adjust the holes in the design on Tinkercad to your preferred solution.

When soldering the sensor you should be very careful, and double-check yourself! The sensor is pricey and can be damaged by applying voltage to the wrong pins. (if you bought a version mounted on a breakout board it's easier, the pins will be labeled, but you'll need a bigger housing)

Attach short leads to each of the 4 pins, as shown in the diagram. I like to add a bit of heat-shrink tubing around the connection as well, though in a project this simple it's not really essential.

Use image search to find wiring diagrams and pinouts if the photos above are not sufficient (they probably aren't)

GND goes to the ground pin on the ESP32 and VDD goes to the 3V output pin (NOT the 5V pin! doublecheck...).

SDA and SCL get connected to the appropriate pins on your board - for mine, it's pins D4 and D5, but the ESP32 Supermini form factor comes in a few different flavors so check your own board, and modify the code as required.

Once you've doublechecked the wiring, check it again before applying power...

Note: if you're unfamiliar with the general process of loading Arduino IDE code onto an ESP32, there are many tutorials - this description won't attempt to include every detail of the process.

Using the Arduino IDE, install ESP32 board support and the following libraries:

1. PubSubClient
2. Adafruit_MLX90640
3. ArduinoOTA
4. ArduinoMDNS
5. WiFiManager.h

Modify the MyWiFi.h file with your WiFi SSID and password, as well as the address and credentials of your preferred MQTT server (if you're using the MQTT option - if not, un-comment the #noMQTT define in the main file).

Important: modify the "camNum" value for each board you load! This identifier is used not only with ClientID but also to identify the camera when you're viewing the page it serves. (yes I should make this better)

Assuming the code compiles without error, connect the board to your computer, and upload the code. Watch the Serial Monitor at 115200kbps for startup and troubleshooting info. If it can't connect to the camera, try one of the example sketches that come with the camera library. Doublecheck your SDA and SCL links, both in your wiring and in the code - the pin numbering can be confusing!

If it gets online and all is well, point a web browser at the IP address it gives - and enjoy! You should get at least 2-3fps - higher frame rates are possible, but these sensors get significantly more noisy when you boost their transmission rate.

You should be able to modify the code in-place via the OTA (over-the-air) update feature, as long as they're operating on your local network (the OTA update channel does not work over a VPN)

Even if you don't have a 3D printer, you may find one available at a nearby public library. Alternatively, I suggest looking online for a local 3d printer or maker group, you'll likely find someone happy to help. Or there are many commercial services where you can get small objects printed affordably.

If you're crafty, you can easily make a housing for this project yourself - out of cardboard, foamcore, Lego, whatever you like.

3D print files available here via TinkerCad. (includes an alternate, unfinished design allowing for the power plug to come in from the side, for wall mounting)

The ideal range for these cameras is between 1 and 4 meters from a subject's usual location in a room (a bed, a recliner, etc) though they can certainly cover a wider area, like a kitchen, with reasonable clarity.

Be aware of heat sources - a sunlit wall, a furnace vent, an oven. For example, there was a table in my father-in-law's living room perfect for the camera, but from that angle, the furnace was right behind his recliner, so it would have been hard to make him out from the background in cooler months.

You can use a heat source with the viewer to get an idea of the field of view for your camera - I just heated up a saucepan on the stove and moved it around the area while looking at the output.

Also of course you want to be near a wall outlet - you can get long USB cables, but make sure they're decent quality, as the cheap ones use thin wire that will lead to a voltage drop over the span. I have successfully used 6m cables that were sold for use with Wyze cameras.

The camera software is not at risk of corruption when the power is cut or the router rebooted. It reboots quickly, getting back online usually in seconds. Note that if you're using MQTT and the broker is inaccessible, it will reboot periodically after failing to connect a few times - if you've compiled with MQTT and then decide the browser option is better, update the camera code (which you can do via OTA from the Arduino IDE).

As most know, communication from a client on a home network to one on another has to pass through the NAT on each route. Unless you're network-savvy enough to open a port on your router safely, this pretty much requires a third-party server somewhere, to make the initial connection. There's many ways to do this, and I've implemented a couple in this project.

Most obviously, the camera puts up a basic web server and offers a page with an embedded Javascript applet that handles rendering the data. If you have a link to the local network somehow - a VPN, a Tailscale account with a client running on their end - just use that. (current plans are to include a feature in the web page code to add links to other cameras in the network)

Since the camera is low-resolution and thus has a low data rate, MQTT is an option. This is a lightweight messaging protocol mediated by a server called a broker, that can run in the cloud or in your home. You can use a free public broker (possibly with a lower frame rate as a drawback) or create your own. Then you can view the cameras in an Android app I've built using MIT App Inventor, which may be easier for some than using a browser.

A very simple MQTT broker can be loaded into an ESP8266 or ESP32, then you just set the router to forward that one port to the device. There's not a lot of potential in hacking an ESP8266 (especially if you haven't set up OTA) so I feel it's pretty safe to do this. I recommend using a free dynamic DNS server to enable easy access to the router - many home routers (ASUS, TPlink, Netgear, others) have a facility to manage that automatically.

Tailscale also lets you implement a very secure and efficient VPN between network clients. It can't be run from an ESP32 directly, but any other device on the target network - Mac/PC/Linux, Android or iOS phone/tablet - can run the client, and with the "local subnets" option turned on (per the tutorial on Tailscale) forward traffic to/from the cameras.

I've written up some more detailed notes on connectivity here - I hope to flesh out this part of the project once a few others have tried it out and provided feedback re: any problems or suggestions.

I began this project because I couldn't find any human-presence radar sensors which behaved to my satisfaction. However the product offerings have improved since then, and I will soon be testing a range of small inexpensive sensors to see which would be best. Some claim to sense breathing, others to auto-detect a fall, but I'll be happy if they just don't false-positive on sensing a body!

I plan to modify the project so that the IR sensor and the human presence radar work in tandem, to bolster the reliability of the system overall.

The cameras have a channel to respond to requests from a Python script running remotely, which is designed to store a frame every N seconds on local storage and compile them into a timelapse video, however this script is still in testing. It can be provided for those who wish to tinker with it.

The MQTT client plugin for MIT App Inventor is functional but offers little troubleshooting or debugging info, so it can be a pain to diagnose connection issues - and I've found it necessary to force-quit the app before trying to reconnect. I have paused development on the app in order to perhaps improve the web viewer to the point where it can be a full replacement. For starters, I plan to add the ability for cameras to find each other on a local network and provide links to each other on their own web pages.

There's also work to be done on how the thermal pixels are displayed, particularly in how the temperature range is mapped to colors. I'm using an adaptation of the Ironbow spectrum, commonly associated with FLIR cameras. Both the web page and Android app allow for adaptive and fixed temperature ranges which could benefit from some fine-tuning.

The camera sensor data sheet suggests that adding specific capacitors across the power lines may improve the noise levels, I'll be testing that as well as more of the selectable modes to find the best image quality.

Feedback from anyone trying this out will be greatly beneficial!