IoT-Based Mole Detector: Smart Skin Health Monitoring System

This project was developed as a final assignment for an IoT course at Reichman University.In this project, we developed a smart IoT-based system that helps detect potentially dangerous moles using a machine learning model trained on medical data. With a simple ESP32-CAM and an ESP32 board, doctors can take pictures of moles, analyze them with AI, and instantly get results via a mobile app and speaker feedback. This project bridges hardware and AI for real-time medical support.

we decided to tackle the challenge of early skin cancer detection. We focused on using the ISIC 2019 dataset to create a system that’s not only innovative but medically impactful.

A video tutorial in this link:

https://youtube.com/shorts/ybsQDxSPdqE?feature=share

Hardware:

1. ESP32-CAM (1x)
2. ESP32 DevKit board (1x)
3. Mini speaker / buzzer (1x)
4. Breadboard and jumper wires
5. MicroSD card (optional)
6. USB cable (for ESP32)
7. Smartphone (for Blynk app)

Software & Services:

1. Arduino IDE
2. Python 3.x
3. TensorFlow / Keras
4. Flask (for local web server)
5. Blynk IoT platform
6. ISIC 2019 dataset

🚀 Arena and Motivation

Our chosen arena is healthcare — specifically aiding early diagnosis of melanoma and other skin conditions using accessible, affordable IoT tools. The fusion of embedded systems and machine learning can greatly empower physicians with real-time, automated analysis.

🛠️ System Architecture

1. ESP32-CAM captures an image of a skin mole.
2. The image is saved and uploaded to a local Flask server.
3. A trained CNN model classifies the mole into categories (benign, malignant, etc.).
4. Results are saved in a JSON file and served by the server.
5. ESP32 pings the server and retrieves the diagnosis.
6. Probabilities are displayed on Blynk and audio feedback is played.
7. A simulated email is "sent" to the patient.

This hybrid workflow unifies physical sensing, local processing, and cloud-style UI.

In this step, we focused on creating a simple yet functional module for capturing mole images using the ESP32-CAM board.

🎯 Objective

Capture high-quality images of skin moles and stream them to a browser for manual saving and later analysis.

🧠 Our Approach

We decided to base our camera interface on the official cameraWebServer example provided by the ESP32 Arduino core, since it offered fast setup for streaming and control. We then modified the code slightly to match our needs.

🛠️ What We Did

1. Installed the ESP32 board manager in Arduino IDE and selected “AI Thinker ESP32-CAM” as the board.
2. Opened the cameraWebServer example.
3. Modified the SSID and password in the code to connect to our network.
4. Adjusted the resolution settings to balance between image clarity and stability.
5. Used the built-in web interface to stream and manually save images for analysis.

🧩 Tips for Builders

1. Mount the camera module in a stable, well-lit environment to improve image clarity.

📝 Credits: Part of the data preprocessing and augmentation pipeline in this project is based on the implementation by wanghsinwei on GitHub, whose open-source ISIC-2019 project provided valuable tools and structure for working with medical image data.

In this step, we trained a custom image classification model to detect and classify mole images based on medical data from the ISIC 2019 dataset.

🎯 Objective

Build a reliable machine learning model to classify mole images into three categories: benign, malignant, and other.

🧠 Our Approach

We explored the ISIC dataset to identify relevant labels and images, then designed a training pipeline using TensorFlow and Keras.

🛠️ What We Did

1. Downloaded the ISIC 2019 dataset and filtered relevant image-label pairs.
2. Built a custom image data generator in datagenerator.py to:
3. Resize images to a fixed size (e.g. 224x224)
4. Normalize pixel values
5. Apply random data augmentation
6. Designed a CNN architecture in Train.py using Keras Sequential API.
7. Trained the model locally.
8. Saved the model weights to be loaded during inference.

To make predictions accessible to the ESP32, we built a lightweight local web server using Flask that serves prediction results via HTTP.

🎯 Objective

Host predictions from the trained model in a format the ESP32 can request and parse easily.

🧠 Our Approach

We created a Python-based Flask server that:

1. Listens for GET requests from ESP32
2. Reads prediction results from predictions.json
3. Returns the results in JSON format

🛠️ What We Did

1. Wrote server.py to initialize the Flask app and serve data.
2. Created main.py to handle the full prediction pipeline and store results.
3. Used Selector.py to determine output logic and label mapping.

Once the server was ready, we needed to fetch the predictions using the ESP32 and use them for user interaction.

🎯 Objective

Enable the ESP32 to request and parse predictions from the local Flask server and respond accordingly.

🧠 Our Approach

The ESP32 periodically sends HTTP GET requests to the Flask server and parses the JSON response. Based on the result, it updates a Blynk dashboard and triggers sound playback.

🛠️ What We Did

1. Wrote the Arduino sketch sketch_mar10c.ino to:
2. Connect to Wi-Fi
3. Send HTTP requests to http://<server_ip>/results
4. Parse the JSON for classification result
5. Send data to Blynk using auth token
6. Activate speaker output via digital pins

🎯 Objective

Allow the ESP32 to send prediction results via Gmail using secure authentication.

🧠 Our Approach

Since Google blocks less secure apps from accessing Gmail directly, we used a special App Password. This password is generated specifically for external apps and allows the ESP32 to authenticate securely when sending emails.

🛠️ What We Did

We created a Google App Password by following these steps:

1. Go to Google Account Security Settings

2. Make sure 2-Step Verification is turned on

3. Scroll down and click on App Passwords

4. Under “Select app”, choose Mail

5. Under “Select device”, choose Other and type a name like “ESP32 Project”

6. Click Generate

7. Copy the 16-character password and paste it in the Arduino sketch as follows:

#define AUTHOR_EMAIL "your.email@gmail.com"

#define AUTHOR_PASSWORD "xxxxxxxxxxxxxxxx" // Your App Password

🔐 Note: This App Password is used only for sending emails via Gmail. It can be revoked anytime from your Google account settings, and it keeps your main password safe.

To visualize the model’s results in real time, we created a Blynk interface for mobile feedback.

🎯 Objective

Display prediction probabilities on a smartphone in real time and alert users to results.

🧠 Our Approach

We used Blynk's IoT platform to create a simple dashboard and configured the ESP32 to send updates via Blynk’s API.

🛠️ What We Did

1. Installed Blynk library and web .
2. Created a new Blynk project and generated an auth token.
3. Added widgets:
4. Value Display to show class probabilities
5. Notification for alerts

⚙️ Sample Widget Setup

1. V0: Benign Probability
2. V1: Malignant Probability
3. V2: Other Class

To demonstrate the complete workflow, we created a full run-through of the system in action.

🎯 Objective

Show the system capturing an image, processing it, and displaying results across devices.

🎬 What’s Included in the Video

1. Doctor captures image using ESP32-CAM.
2. The image is saved locally.
3. The Python model performs inference and saves result.
4. ESP32 requests the result and updates Blynk.
5. Speaker plays corresponding sound.

📸 Additional Media

1. Screenshots of Blynk dashboard.
2. Code snippets from Arduino and Python.
3. Server terminal output.

1. Enable real email/SMS notifications using Make.com.
2. Host the server on the cloud (Heroku, AWS, etc.).
3. Integrate Node-RED for real-time analytics dashboard.
4. Expand training data for better accuracy.
5. Add multi-language UI support.
6. Replace polling with WebSockets or MQTT.

1. Arduino Sketch: SkinScan_Notifier.ino
2. Python Files:
3. main.py – system runner
4. server.py – Flask web server
5. Train.py – model training script
6. run_model.py – inference script
7. Selector.py – response logic
8. datagenerator.py – data prep script
9. pixels.py – image handler
10. Sample Output: predictions.json

1. Rony Rabinovitz
2. Asaf Ramati

For full build instructions, wiring diagrams, and code explanations, see the following steps.