XIAO ESP32S3 Sense-Powered Doorbell Enhanced with Machine Learning

Say goodbye to missed deliveries and unexpected guests! In today’s blog, we’re diving into an exciting home improvement project where a broken doorbell receives a remarkable upgrade, blending the power of Machine Learning with Telegram notifications. Discover how this transformation not only fixes a doorbell but also elevates it into a smart and responsive guardian of your front door

Seeed Hardware: Seeed Studio XIAO ESP32S3 Sense

Software: Arduino

Industry: Home Automation, Building Security

The Background

One of the remarkable aspects of encountering challenges is their potential to become fertile sources of inspiration. In Roni Bandini‘s situation, he encountered an issue with a traditional wired doorbell that consistently failed to function correctly. On occasion, it remained completely silent, and even when it did chime, its sound could not reach certain areas within my home. This predicament prompted Roni to contemplate a comprehensive solution: he conceived the idea of crafting a bespoke doorbell system using an ESP32, a push button, Wi-Fi notifications, and two or possibly three buzzers or speakers to address these shortcomings.

The Solution

Roni decided to address the issue by creating a custom-made doorbell system, utilizing an ESP32, a push button, WiFi alerts, and a pair of buzzers or speakers.

As he pondered the project further, he realized he could enhance its functionality by implementing facial recognition technology to capture visitors’ images, eliminating the need for relying solely on their voices. This innovation even made the push button redundant, thanks to the capabilities of Machine Learning in identifying faces at the door.

Roni found the XIAO ESP32S3 is a compact yet fully-equipped board and ideally suited for this task. This board featured a high-quality camera, WiFi connectivity, and a sufficiently powerful processor for Machine Learning applications.

Setup

• Snap the antenna to the board
• Snap the camera to the board
• Format the microSD as Fat32 and place it in the microSD slot

Software configuration

To set up Edge Impulse on Arduino IDE with an ESP32S3 board, follow these steps:

1. Launch the Arduino IDE, then navigate to “File” > “Preferences.” In the “Additional Boards Manager URL” field, add the following URL: https://raw.githubusercontent.com/espressif/arduino-esp32/gh-pages/package_esp32_dev_index.json
2. Proceed to “Tools” > “Board” > “Boards Manager.” Enter “esp32” in the search bar, select the ESP32 package, and install it.
3. Connect your ESP32S3 board using a USB-C cable. In the Arduino IDE, select “Board” and choose “XIAO ESP32S3.” Also, select “OPI PSRAM” from the available options.

1. To use the UniversalTelegramBot library, go to “Sketch,” then “Include Library,” and click on “Manage Libraries.”
2. Download the .ino file and the Machine Learning model. Under “Sketch,” choose “Include Library,” and then select “Add Zip File” to add the zipped model file.
3. Open the code file: Arduino/Documents/libraries/Face_detection_inferencing/scr/edge-impulse-sdk/classifier/ei_classifier_config.h
4. Locate the line containing “#define EI_CLASSIFIER_TFLITE_ENABLE_ESP_NN 1” and change the “1” to “0.”

Model training

How do we distinguish between one or multiple faces and the background or other objects in an image? Edge Impulse offers a powerful algorithm called FOMO (Faster Objects, More Objects) designed for precisely this purpose. FOMO is an innovative machine learning algorithm that enables object detection on resource-constrained devices. It empowers you to identify and count objects, pinpoint their locations within an image, and track multiple objects in real-time.

You don’t necessarily need to capture your own images for this task, as there are numerous face datasets readily available. I recently duplicated an existing Edge Impulse project that already included a dataset of faces, and that proved sufficient. However, if you prefer to use your own images and train a new model, here are the steps to follow:

1. Collect approximately 400 pictures of faces.
2. Annotate these images by selecting the facial region and labeling them as “face.”
3. Create an Impulse with image data sized at 96×96 pixels and set up an image processing block.
4. Implement object detection as a learning block within the Impulse.
5. Perform 70 training cycles with a learning rate of 0.00015.
6. Finally, deploy the trained model as an Arduino library and save the resulting Zip file.

By following these instructions, you can effectively train a custom model for face detection using your own images.

Software settings

Enter your Wi-Fi credentials

#define WIFI_SSID ""
#define WIFI_PASSWORD ""

• Go to Telegram App, search botfather, send /start send /newbot
• Go to t.me/sandoombot and get your token
• Search idBot in Telegram App
• Send /getId
• Send a message to the bot created searching by link
• Create a Group and add the bot
• Load https://api.telegram.org/botXXXXXXX:YYYYYYY/getUpdates
• Extract the ID, like id: -1234567890
• Enter Token and ID in these settings
• Upload the ino to the Xiao ESP32S3 Sense board.

The Conclusion

In summary, the XIAO ESP32S3 Sense device captures images, runs them through a facial detection model, and takes action when a face is found. If it detects a face, it lights up an LED, saves the image, and sends a notification to a family Telegram group.

These tasks may sound complex, but they were relatively easy to implement. This small device, with the help of Machine Learning, not only serves as a doorbell but also offers extra features.

While it doesn’t include a traditional doorbell sound yet, you can easily add one locally or remotely. You can also improve it by counting people at the door, recognizing friends, and playing a message for visitors.

More Information

Learn More Project Details on Hackster: Machine Learning doorbell with XIAO ESP32S3 Sense

Seeed Studio XIAO Series

Please feel free to reach out to [email protected] for any inquiries or if you’d like to engage in further project discussions. Your questions and interest are welcomed.

About Author