Empowering Travel Safety with XIAO ESP32S3 Sense, Round Display for XIAO, and TinyML: AI-Driven Keychain Detection for Instant Alerts and Location Request

Updated on Feb 28th, 2024

Introducing an AI-powered BLE Travel Emergency Assistant utilizing XIAO ESP32S3 Sense, Round Display for XIAO, and TinyML. Detect keychains for emergency alerts, notifying contacts via WhatsApp/SMS, with added functionality for requesting location info from the Maps API.

Seeed Hardware: Seeed Studio XIAO ESP32S3 Sense, Round Display for XIAO 

Software: Edge Impulse, Android Studio, MIT App Inventor 2, Twilio API for WhatsApp, Twilio SMS Messaging API, Google Maps, Autodesk Fusion 360, Ultimaker Cura, Microsoft Visual Studio 2017

Industry: Healthcare, Hospitality, Travel and Tourism, Transportation, Emergency Services, Insurance, Security

The Background

Travelers with disabilities are at greater risk of becoming victims of crimes such as violence, theft, verbal abuse, and neglect. They are at a higher risk, at least four to ten times, of being a victim of crime. To prevent these crimes, it is necessary to provide them with state-of-the-art devices. However, smartphones may not be suitable since they are time-consuming and arduous in a time of crisis for people with mobility impairments. Therefore, budget-friendly and accessible AIoT assistive devices should provide various features in a broader spectrum, considering people with disabilities and special needs.

The Challenge

To address this problem, Kutluhan Aktar created different keychains denoting emergencies and used a XIAO ESP32S3 Sense to capture and save images to an SD card. With Edge Impulse, he built an object detection model to identify the customized keychains and notify emergency contacts via WhatsApp/SMS. The results are displayed on a Round Display for XIAO through BLE, and emergency contacts can obtain location information from Maps API.

The Solution

Step 1: Designing and printing a smartwatch-inspired case

Focused on creating a budget-friendly assistive device, Kutluhan designed a versatile case that easily attaches to mobility aids for seamless communication with emergency contacts via WhatsApp or SMS. The main case, split into two parts for compatibility with different batteries, features snap-fit joints. To enhance functionality and prevent dust exposure, a modular part for XIAO ESP32S3 Sense and Round Display for XIAO was added. Dents in both parts allow flexible positioning of the built-in camera while scanning or collecting samples. The main case is adorned with the Seeed Studio logo, Twilio logo, and a traveling airplane icon, emphasizing the device’s capabilities.

Step 2: Developing a GPS, GPRS, and BLE-enabled Android application w/ the MIT APP Inventor

To ensure the assistive device functions without Wi-Fi, Kutluhan created an Android app using MIT App Inventor. Serving as the device’s interface, the app utilizes cellular network connections for data transfer via GPRS, obtains GPS-based location data, and communicates with XIAO ESP32S3 through BLE for model detection and data collection commands.

Step 3: Creating an account to utilize Twilio’s WhatsApp & SMS APIs

In order to notify the user’s emergency contacts about the latest detected keychain via the object recognition model through WhatsApp and SMS, Kutluhan integrated Twilio’s WhatsApp and SMS APIs. Twilio facilitates communication with a Twilio-verified phone through WhatsApp using a free webhook for trial accounts. Additionally, Twilio offers a trial text messaging service for international SMS transfer from a virtual to a verified phone number. Supporting official helper libraries in various programming languages, including PHP, Twilio ensures reliable API enforcement.

Step 4: Developing a web application to communicate w/ the Android app and process requests from WhatsApp

Needing to relay model detection results from XIAO ESP32S3 to inform emergency contacts, Kutluhan developed a basic web application utilizing Twilio’s WhatsApp and SMS APIs. The web app acts as a proxy between the Android application and emergency contacts, allowing the user to update contact information with a single HTTP GET request. Using stored data, the web app automatically notifies contacts via WhatsApp or SMS based on model detection results. Twilio’s APIs enable the web app to receive commands through WhatsApp for sending location inspections generated by Google Maps to the primary emergency contact. To meet Twilio’s requirements, he hosted the web app on an SSL-enabled server, but an HTTP tunneling tool like ngrok can also set up a public URL for the webhook.

Step 5: Setting up XIAO ESP32S3 Sense and Round Display on Arduino IDE

Utilizing the XIAO ESP32S3 Sense expansion board’s file-handling capability, Kutluhan opted to directly capture and save images from the built-in OV2640 camera to the microSD card. XIAO ESP32S3 communicates with the Android application via BLE to receive user commands and transmit model detection results.

Step 6: Capturing customized keychain (token) images w/ the built-in OV2640 camera

After configuring XIAO ESP32S3 with the round display and installing necessary libraries, Kutluhan programmed it to capture raw image buffers, convert them to JPG files, and save samples directly to the SD card on the expansion board.

To streamline the creation of a valid dataset for the object detection model, he used the Android application to send user commands over BLE to XIAO ESP32S3. This allowed capturing and saving samples to the SD card with the specified emergency class, eliminating the need for additional components and preventing an increase in the main case dimensions.

Step 7: Building an object detection (FOMO) model with Edge Impulse

After capturing images of customized keychains and storing them on the SD card, Kutluhan focused on developing an object detection (FOMO) model to identify keychains individually for immediate notification to emergency contacts via WhatsApp or SMS.

Opting for Edge Impulse, he leveraged its extensive microcontroller and development board support for model deployment. Edge Impulse’s advanced machine learning algorithm (FOMO) enabled the creation of more accessible and faster object detection models on edge devices like XIAO ESP32S3.

Step 8: Setting up the Edge Impulse FOMO model on XIAO ESP32S3 Sense

Having constructed, trained, and deployed the object detection model as an Arduino library on Edge Impulse, Kutluhan uploaded the generated library to XIAO ESP32S3. This allowed the model to run directly, ensuring the creation of an accessible assistive device with minimal latency, memory usage, and power consumption. Edge Impulse’s optimization and formatting of signal processing, configuration, and learning blocks into a single package simplified the import process for running inferences.

Step 9: Running the model and informing emergency contacts via WhatsApp & SMS

Following the successful setup of the Edge Impulse object detection (FOMO) model on XIAO ESP32S3:

1. Users can detect emergency-class keychains using the assistive device through BLE connection with the Android application.
2. XIAO ESP32S3 runs an inference every 30 seconds, notifying users upon detecting an emergency class through the XIAO round display.
3. Post-detection, XIAO ESP32S3 transfers model results to the Android app via BLE.
4. The Android app displays and transfers model results, current location, and date to the web app through GPRS.
5. The web app generates a Google Maps URL from decoded location information.
6. Using Twilio’s APIs, the web app notifies emergency contacts via WhatsApp or SMS based on the detected emergency class.

The Conclusion

In summary, integrating object detection models trained on customized keychains into assistive devices not only deters potential crimes against individuals with disabilities but also empowers those with mobility impairments in emergency situations. The system offers versatile notifications for various mobility aids, ensuring prompt communication of emergencies to contacts with precise location data. Additionally, it provides the ability to generate a travel itinerary of previously visited destinations upon request, enhancing its overall utility.

More Information

Learn More Project Details on Hackster: AI-driven BLE Travel Emergency Assistant w/ Twilio

Explore more TinyML stories:

• Revolutionizing Wildlife Monitoring: TinyML, IoT, and LoRa Technologies with XIAO ESP32S3 Sense and Wio E5 Module
• NMCS: Empowering Your Coffee Experience with Sound and Vision Classification
• IoT-Enabled Tree Disease Detection: Harnessing Vision AI, Wio Terminal, and TinyML
• Snake Recognition System: Harnessing LoRaWAN and XIAO ESP32S3 Sense for TinyML
• Innovative Community Projects That Utilized Grove-Vision AI Module: 11 Inspiring Stories

---

Choose the best tool for your TinyML projects:

Grove – Vision AI Module V2

It’s an MCU-based vision AI module powered by Himax WiseEye2 HX6538 processor, featuring rm Cortex-M55 and Ethos-U55. It integrates Arm Helium technology, which is finely optimized for vector data processing, enables:

• Award-winning low power consumption
• Significant uplift in DSP and ML capabilities
• Designed for battery-powered endpoint AI applications

With support for Tensorflow and Pytorch frameworks, it allows users to deploy both off-the-shelf and custom AI models from Seeed Studio SenseCraft AI. Additionally, the module features a range of interfaces, including IIC, UART, SPI, and Type-C, allowing easy integration with popular products like Seeed Studio XIAO, Grove, Raspberry Pi, BeagleBoard, and ESP-based products for further development.

Seeed Studio XIAO ESP32S3 Sense & Seeed Studio XIAO nRF52840 Sense

Seeed Studio XIAO Series are diminutive development boards, sharing a similar hardware structure, where the size is literally thumb-sized. The code name “XIAO” here represents its half feature “Tiny”, and the other half will be “Puissant”.

Seeed Studio XIAO ESP32S3 Sense integrates an OV2640 camera sensor, digital microphone, and SD card support. Combining embedded ML computing power and photography capability, this development board can be your great tool to get started with intelligent voice and vision AI.

Seeed Studio XIAO nRF52840 Sense is carrying Bluetooth 5.0 wireless capability and is able to operate with low power consumption. Featuring onboard IMU and PDM, it can be your best tool for embedded Machine Learning projects.

Click here to learn more about the XIAO family!

SenseCraft AI

SenseCraft AI is a platform that enables easy AI model training and deployment with no-code/low-code. It supports Seeed products natively, ensuring complete adaptability of the trained models to Seeed products. Moreover, deploying models through this platform offers immediate visualization of identification results on the website, enabling prompt assessment of model performance.

Ideal for tinyML applications, it allows you to effortlessly deploy off-the-shelf or custom AI models by connecting the device, selecting a model, and viewing identification results.

---

Please feel free to reach out to [email protected] for any inquiries or if you’d like to engage in further project discussions. Your questions and interest are welcomed.

About Author