ESP32 Scraper-parser-mailer and Live Mapper

Hello World! This is my first Instructable! I'm Marco from Grosseto Italy, i'm not a developer, i'm not an electronic engineer but i'm an environmental guide in our local natural park (Maremma natural park).

In my cooperative we work a lot with canoe tours , naturalistic and calm tours on the Ombrone river for birdwatching and a bit of sense relaxing.

But the Ombrone river is also famous for its characteristic kind of flow: from summer "zero" to autumn-winter flow of several thousand cubic meters of water per hour.

For this reason, the Ombrone river is constantly monitored by the excellent regional hydrometric service (SIR), with a fantastic site that makes the collected data available.

Now we are waiting for the end of the season and for the first autumn floods for remove the dock and the canoes...

But when we start to talk about scraping parsing esp32ing etc? one moment more

During the italian lockdown i've done the first arduino blink lesson, cool... and i've contiunued to play around with sensors and arduino C. So if you find my code crude or full of nonsense logical gaps consider this!

I did a lot of things just to waste time but with this constant risk of flooding I decided to do something cool: thanks to an ESP32 with an arduino form factor (Wemos d1 r32) which I kept wrapped for months because I thought "difficult", I really had a lot of satisfactions:

I have done a scraper that bring and parse data from a SIR webpage and display the variables obtained on a cheap i2c LCD, and as brightness-frequency blinking on some leds on a realistic map.

so if a flood comes from the mountain side I will be warned by the increasing brightness (level change) and by the increasing frequency (level change in one hour) of the leds on the map... but the ESP32 can also sends emails! So why not do a function for this?

lets go to see the code below now!

• ESP32 based board: I have used Wemos d1 r32 cause have female pins and it's easier to handle
• leds, red is more "alert like"
• I2C 16x2 LCD iv'e only a blue one but the green is more stylish
• cheap cork board
• night time (optional)

online i have found only vapourous tutorials that uses external sites to parse data, so i tried to test the power of the esp32 cores parsing the data directly on the board... without problems!

so if you view the SIR page source can understand how its works: luckily they uses a "comma separated like" values for display their data, so with this code I search for an hydrographic station (code TOSnumber) and count the number of commas (as separator) that contain the value i want (and store it as variable), then remap it as led brightness and time in millis for "variation level" blinking

The core of this scraper is in http tab:

    ind = payload.indexOf("TOSstation_number");  //get the position (ind) from where start to count commas
    String my_var = getValue_ind(payload, ',', 8); //where 8 is the eighth cell defined by commas (separator)

"payload" is what i get from the web previously cleaned from "quotations" and "&nbsp";

and the getValue_ind function that is in the main tab

String getValue_ind(String data, char separator, int index) {
  int found = 0;
  int strIndex[] = {0, -1};
  int maxIndex = data.length() - 1;
  for (int i = ind; i <= maxIndex && found <= index; i++) { //from ind position
    if (data.charAt(i) == separator || i == maxIndex) {
      found++;
      strIndex[0] = strIndex[1] + 1;
      strIndex[1] = (i == maxIndex) ? i + 1 : i;
    }
  }
  return found > index ? data.substring(strIndex[0], strIndex[1]) : "";
} 

adapted from https://stackoverflow.com/a/14824108

the code asks for the page every 15min, it is about 44Kb, i don't think is a big load for the servers and for the ESP32.. To avoid too much scrolling i have divided the program in different functions, read the comments

here the code pasting is really a mess... i put the .ino files directly.

i've used the tabs feature of the arduino IDE so i have different .ino files.

this is the latest version of the code (?), there's a lot of efforts, so i'll appreciate any suggestion!

- for the straightening of the led brightness for low values,
- for the "map" function for floating point variables
- for the use of the second core on the ESP32,
- for understand how to print an html mail from a string with sprintf
I have spent a lot of true "nerding time" so read on the comments on the .ino files or on github!

as upload settings i've used Tools: Partition Scheme: NO OTA to have more space for this huge code

have fun!

click here for the code on github

this tut is incentred on the ESP32 code so i can give you only some keyword, links and tools used:

• the excellent service for extract vectorial areas form openstreetmap
  https://extract.bbbike.org/

• the "ostic" QGis for digest them
• search for attribute table for simplify your map
• create a print layout and export the map in svg format on A3 paper format
• have fun with the "smoother" illustrator
• print it

so i can suggest to die-cut the holes for the led and the display on the paper before fix it on the cork board

the wiring is really simple and and badly done by me :P

My soldering level: Pigeon

as you can see from the code the leds go to IO pins

#define LED_PIN0            14 //si<br>#define LED_PIN1            27 //buon
#define LED_PIN2            16 //sass
#define LED_PIN3            17 //tur
#define LED_PIN4            25 //ist
#define LED_PIN5            26 //berr

search for ESP32 pinout for understand

so this photo is not the best, but when we got some river flood i'll put a nice video!