NMCS: Empowering Your Coffee Experience with Sound and Vision Classification

Introducing NMCS, a device designed with both hearing and visual capabilities by harnessing TinyML technology, dedicated to ensuring your coffee-making ritual remains spill-free, guaranteeing a seamless start to your day.

Seeed Hardware: Wio Terminal, Grove-Vision AI Module

Software: Arduino, Edge Impulse, Roboflow

Industry: Smart Home

The Background

Sashrika Das, one of the winners of the “IoT in the Wild Contest 2022” by Seeed Studio, has recently developed a new device. The device, named “NMCS” (No More Coffee Spills), was inspired by her dad’s experience of spilling coffee while brewing it. NMCS aims to prevent such mishaps by checking for the presence of a cup before starting the brewing process.

The Challenge

The primary challenge NMCS tackles is the coordination of audio and visual data to ensure accurate predictions and minimize false alarms. To address this, she utilizes the Vision AI camera to visually confirm the presence of a cup, syncing this assessment with the audio input from the Wio Terminal’s microphone, which listens for the unmistakable sounds of coffee brewing. Should it detect brewing without a cup in place, the Wio Terminal promptly signals the user with an alert, preventing any potential coffee disasters.

The Solution

The choice of the Wio Terminal microcontroller stems from its familiarity to Sashrika and its built-in microphone, eliminating the need for an external audio sensor. The microphone employs audio classification to discern the distinct sounds of coffee preparation. Moreover, the Wio Terminal’s plug-and-play grove ports simplify sensor connections, eliminating the need for soldering. The Vision AI module seamlessly integrates into one of these grove ports. The device also boasts a user-friendly display, presenting real-time data on the coffee brewing prediction percentage and cup presence, mirroring the multisensory approach humans naturally employ to navigate their surroundings. By combining auditory and visual inputs, NMCS ensures both precision and reliability in its quest to save coffee and mornings alike.

Sashrika utilized Edge Impulse for two distinct purposes: audio classification and object detection, necessitating the creation of separate projects for each task. For the audio classification model, she amassed approximately 10 minutes and 10 seconds of data, with each dataset lasting around 5 seconds, encompassing background noise and the distinct sounds of a brewing coffee machine. After data collection, she created her impulse, configuring it with 100 training cycles and a learning rate of 0.005 before proceeding to deployment, ensuring seamless functionality on both mobile phones and computers before progressing to Arduino.

The preferred tool, Edge Impulse, also played a crucial role in her vision AI module project. Initially, she connected the Grove Vision AI module to Edge Impulse and gathered images directly from the module, facilitating impulse creation, training, and testing on her iPhone. To deploy the custom model to the Vision AI module, she encountered challenges but resolved them by acquiring additional modules and resorting to Roboflow for YOLO 5 training, successfully annotating 120 images of cups, resizing them to 192 x 192, and generating the dataset. She followed a guide to train the YOLO 5 model on Google Colab, eventually moving the trained model to the Vision AI module’s GROVEAI drive. Completing the device entailed adding a library to the code to display cup counts in the serial monitor, resulting in a scrumdiddlyumptious coffee experience devoid of waste, a perfect solution for any coffee enthusiast, including her dad, who enjoys coffee at any hour.

The Conclusion

This project faced its fair share of challenges, from struggles with deploying files to a module breakdown. Adapting to memory usage issues led to design modifications. With more time, Sashrika plans to optimize memory usage, add visual appeal to the display, and implement remote notifications for brewing without a cup. Exploring object detection within Edge Impulse aims to streamline future projects.

More Information

Learn More Project Details on Hackster: NMCS: No More Coffee Spills!

Seeed Studio TinyML Case Studies

Please feel free to reach out to [email protected] for any inquiries or if you’d like to engage in further project discussions. Your questions and interest are welcomed.

About Author