Espnow Based Sensors With MQTT/ Home Assistant

This is an instructable to get you started with espnow on ESP8266.

As per espressif website : ESP-NOW is yet another protocol developed by Espressif, which enables multiple devices to communicate with one another without using Wi-Fi. The protocol is similar to the low-power 2.4GHz wireless connectivity that is often deployed in a wireless mouse. So, the pairing between devices is needed prior to their communication. After the pairing is done, the connection is secure and peer-to-peer, with no handshake being required.

Now you would ask why would I need to move to espnow if I am happy with using WiFi on my ESP? There are a couple of reasons on why you may consider to do so:

1. It is very fast and light weight as compared to WiFi
2. Your battery based sensors can last for a longer time
3. No dependence on a WiFi router, can directly communicate with other peers
4. Although this too works on the 2.4 Ghz band, the range of communication in practicality is much better than WiFi.
5. You could increase the range of the sensors by putting in repeaters similar to how a mesh network does (not covered in this tutorial)

There may be many more....

And of course there are some limitations on using ESP-NOW:

1. Number of peers that a esp can connect to is limited to 6/20 depending on if it uses security for esp8266.
2. The max length of data you can send in a single message is limited to 250 bytes

In order to use espnow you need at least two nodes, one of them acting as the master or controller (one who sends the message) and the other one is called the slave ( one who receives the message). When I was learning to start off with espnow I saw many people using two ESPs for the gateway. One which would receive the espnow message and then it will relay the message to another ESP via Serial or other mechanism. The second ESP will then transmit the message to MQTT. This is not necessary, you can implement both the functionalities within the same ESP. So, what you need is:

1. First ESP as a sensor node - acting as a controller - I will use the example of a door sensor which will send the state of the door contact
2. Second ESP acting as a espnow slave (gateway) as well as as a WiFi client to connect to MQTT.

The gateway is a simple ESP8266 running the code for a slave as well as WiFi client.

The code for this can be accessed on github here

Before I go into the logic of the program, there are a few things to understand on how would we send messages from the sensor node to the gateway.

• The exchange of data between the sensor and gateway happens as a struct called espnow_message defined in espnowMessage.h. This struct is defined as a generic struct which can hold any int, float, string data apart from some fields like message_id and device_name
• The Gateway is agnostic of the data the sensor node is sending except that it is a struct. It simply takes the values from the struct and publishes them to MQTT. So adding new sensor types does not require modifying the gateway code. say, If we add a new temperature sensor tomorrow, it can be done easily.
• The WiFi channel number at which the sensor adds the peer (gateway) should be the same as the channel no for the gateway. Also, I am not sure if its a bug but the gateway cannot have a different channel for WiFi and espnow. Hence the sensor node also needs to be on the same channel as the SSID the gateway connects to.
• ESPNOW is capable of encrypting the messages via 16 bit keys but I was not able to make it work - something on the pending list.
• All properties specific to a particular gateway is defined in the file \include\Config.h
• I use a lot of util files from the folder \include which is outside of the project folder as this folder contains common files used across projects. Its path is included in the project in platform.ini as a build flag

The flow logic for the gateway is as below:

setup ()

1. Initializes WiFi properties , sets WiFi mode to WIFI_STA and connects to WiFi SSID access point
2. Initializes espnow properties, assigns self role as ESP_NOW_ROLE_SLAVE
3. Scans for the channel no of the SSID (This is very important , will talk about it more later)
4. Registers the peer (sensor node) via its mac address.
5. Registers callback to be called when the ESP receives a message and when it sends a message. We will only use the callback on receiving the message in this code.
6. Starts OTA for over the air updates
7. Publishes the startup health message to MQTT to indicate a successful startup

OnDataRecv()

1. called when the esp receives an espnow message, it simply copies the message to an internal queue.

publishHealthMessage()

1. publishes a health message to MQTT to a predefined path.
2. It includes stats like uptime, total messages published, queue length , rate of incoming messages etc

loop()

1. Connects to MQTT, if not connected
2. Handles OTA
3. Checks if the queue holding the sensor data has any sensor data. If yes, pop it out and publish it to MQTT
4. If message publish is unsuccessful then it retains the current message and retries the next time
5. Publish a health message to MQTT at certain interval - you can include any stats you want to be published in this message.

Please make or verify the following changes before compiling the code. You have to change the values of the following according to your needs:

• Define all devices you will have as gateways in Config.h , if only one define one of them. I have defined 2 of them as

#define GATEWAY_GF 1
#define GATEWAY_FF 2

• Set the properties of the gateway in Config.h like below

  #define DEVICE_NAME             "gateway_gf" //no spaces as this is used in topic names too
  #define MQTT_TOPIC              "home/espnow/" DEVICE_NAME
  #define MQTT_BASE_TOPIC          "home/espnow"
  #define ESP_IP_ADDRESS          IP_gateway_gf // from secrets.h
  #define WiFi_SSID               gf_ssid //from secrets.h
  #define WiFi_SSID_PSWD          gf_ssid_pswd //from secrets.h
  #define STATUS_LED              2

• mention the mac addresses of all sensor nodes which will send messages to this gateway in the file \include\secrets.h in the field CONTROLLERS

#define CONTROLLERS { \
    {0x4C, 0x64, 0x33, 0x22, 0x44, 0x2D} ,\
    {0x3C, 0x2D, 0x12, 0x73, 0x41, 0x88}\
    }

• mention the mac address of the gateway in secrets.h . You can get the mac address via the command WiFi.softAPmacAddress() . You will have to run a test code on the ESP to obtain this. Broadcast mac is a special mac as defined below, if the sensor sends the message to this address it is received by any gateway in range. For testing it might be good to first set the MAC as broadcast one.

#define GATEWAY_FF_MAC {0x22, 0x41, 0x44, 0x55, 0xA2, 0x8E} //- This is the SoftAP MAC addr of the ESP01 FF Gateway
#define GATEWAY_GF_MAC {0xEE, 0xFF, 0x12, 0x13, 0xF7, 0x5D} //- This is the SoftAP MAC addr of the ESP01 GF Gateway
#define BROADCAST_MAC  {0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF}// broadcast gateway - to everyone

• Set other values in secrets.h as per your needs

#define primary_ssid "YOUR SSID"
#define primary_ssid_pswd "YOUR PSWD"
#define default_gateway IPAddress(192,168,1,1)
#define default_dns IPAddress(192,168,1,1)
#define subnet_mask IPAddress(255,255,255,0)
#define mqtt_broker IPAddress(192,168,1,XXX)
#define mqtt_port 1883
#define mqtt_uname "XXX"
#define mqtt_pswd "XXXX"

• Define the static IP you want to assign to your ESP in the file static_ipaddress.h.

#define IP_gateway_gf IPAddress(192,168,1,11)

If you dont want to assign static IP to your ESP, you will have to comment out the following statement.

WiFi.config(ESP_IP_ADDRESS, default_gateway, subnet_mask);//from secrets.h

You can ignore the rest of the values in secrets.h. They are used in other programs

Before you hit compile you also have to ensure correct values of the environment you're compiling for in platform.ini

[env]
platform = espressif8266
framework = arduino
lib_deps = 
    bblanchon/ArduinoJson ; serilazation and desrerialization of json messages
    einararnason/ArduinoQueue @ ^1.2.5 ; for queue management of messages
    knolleary/PubSubClient
    bluemurder/ESP8266-ping @ ^2.0.1
build_flags =
  -I"C:/My Data/OneDrive/Documents/Circuits/home_automation/include"


[env:Gateway_GF]
board = esp01_1m
upload_port = COM5
upload_speed = 921600
monitor_port = COM5
monitor_speed=115200
build_flags = 
    ${env.build_flags}
    -DDEVICE=1

While the other settings are understandable , you have to pass the device you are compiling it for as a build flag eg. DDEVICE=1.

pass in the path of the include folder which contains all utility files, if you dont want to use an external folder you can copy all the files in this folder into this project's include folder.

That's it , hit compile and hopefully you should be able to upload the same on your ESP.

The gateway has Serial enabled so it should print messages on the serial window. On startup it will publish a startup message on the topic home/espnow/<gateway_name>/init , home/espnow/<gateway_name>/LWT & home/espnow/<gateway_name>/wifi

It then publishes health message every 30 sec (configurable via HEALTH_INTERVAL) as below

{
  "uptime": "1 06:28:51 ago",
  "mem_freeKB": 45,
  "msg_count": 1802,
  "queue_len": 0,
  "msg_rate": 0
}

Now that we have the gateway up and running we need a sensor node which can post messages to the gateway. For this we will use the code for a door sensor which is normally powered OFF. When power is supplied to it, the ESP wakes up and sends the state of the door. It then sends a signal to cut off its power supply.

The hardware for this is documented as a separate Instructable , you can access it here. The code here of course is independent of the hardware you use. It could be a simple reed switch tied to an ESP too.

The code for this door sensor can be accessed on github here

The flow logic for this code is as below:

setup()

• The ESP holds the HOLD PIN as LOW, this ensures that power to the ESP continues on till completion of its operation
• It reads the state of the door sensor
• Initializes espnow related operations
• registers callbacks for on_receive and on_send of messages
• refreshes the list of peers registered (slave in this case)
• constructs and posts the message to the slave
• sets the HOLD PIN HIGH cutting off power to itself

Initilize_espnow()

• reads the WiFi channel from EEPROM. If invalid, rescans the WiFi channel of the SSID (note this is a time consuming operation , may take 2-4 secs depending on the no of SSIDs found. As this is an expensive operation in time, the channel once got, is stored in the EEPROM to be read later. If the SSID channel changes, it will fail to send the message and again rescan the channel and update the EEPROM.
• sets the WiFi channel and initializes espnow

OnDataSent()

• Checks if the status of the sent message was successfully received by the Slave. If yes marks the status as success.

Set the following properties according to your needs before you compile the code:

in platform.ini define your environment. the program compiles for the device you pass here. The DEVICEs are defined in Config.h

[env:balconydoor]
board = esp01_1m
board_build.ldscript = eagle.flash.512k32.ld #https://github.com/esp8266/Arduino/tree/master/tools/sdk/ld
upload_port = COM5
upload_speed = 921600
monitor_port = COM5
monitor_speed = 115200
build_flags = 
    ${env.build_flags}
    -DDEVICE=3

Define properties of your device in Config.h , example below

at present the role of the sensor is only a controller (aka master, sensor). In future I plan to add code to 2 way communication to make use of the COMBO role.

#elif (DEVICE == BALCONY_DOOR)
  #pragma message "Compiling the program for the device: BALCONY_DOOR"
  #define DEVICE_NAME             "balcony_door"
  #define HOLD_PIN 5  // defines hold pin 
  #define SIGNAL_PIN 4 // indicates the message type
  #define MY_ROLE         ESP_NOW_ROLE_COMBO              // set the role of this device: CONTROLLER, SLAVE, COMBO
  #define RECEIVER_ROLE   ESP_NOW_ROLE_COMBO              // set the role of the receiver
  uint8_t gatewayAddress[] = GATEWAY_FF_MAC; //comes from secrets.h
  constexpr char WIFI_SSID[] = primary_ssid;// from secrets.h

Once you get the hang of ESPNOW , it is very fast , reliable and fun to work with as compared to WiFi. Espressif has of course built the ESPNOW mesh too which provides many more features but devices like ESP8266 will not support that.

Also, a point to note is that any of this can also be done on an ESP32, its just that its an overkill. Nevertheless, the ESPNOW code for ESP32 is a little different than for ESP8266, Espressif has changed some API names and some definitions which need to be taken into account. If people need it I can document that as a separate instructable.

Happy ESPNOW to you all !! Enjoy and ask for help if needed!