NVIDIA TAO Now Running on MCU – Grove Vision AI V2
Great news!! You can now use NVIDIA TAO on MCU – Grove Vision AI V2!
Short for Train, Adapt, Optimize, TAO is a set of open-source toolkit developed by NVIDIA. It is built on TensorFlow and PyTorch, utilizing transfer learning to simplify the model training process without compromising the inference throughput.
(TAO Overview Image, Credit: Nvidia https://docs.nvidia.com/tao/tao-toolkit/text/overview.html)
NVIDIA TAO provides a wide range of pre-trained models that can either be used directly for inference or be finetuned with custom datasets to train new models.
(Models on NVIDIA NGC Catalog, https://catalog.ngc.nvidia.com/models)
While there are many models available in the NVIDIA TAO model zoo, they are in ONNX format, which is more suitable for GPUs and CPUs. Due to the limitation of computational resources and memories, these TAO models need to be optimized for MCUs. Our team at Seeed Studio has been working on this, taking Grove Vision AI V2 as a starter. And it’s working without compromising the performance of the models.
ICYDK, Grove Vision AI V2 is a highly efficient MCU-based smart vision module driven by the Himax WiseEye2 HX6538 processor, featuring a dual-core Arm Cortex-M55 and integrated Arm Ethos-U55 neural network component.
Through the following steps, the pre-trained models at the NVIDIA NGC Catalog can be optimized and used on Grove Vision AI V2.
- Quantization: from floating-point ONNX models to TensorFlow Lite Int8 models.
- Conversion: optimize and convert the models into TFLM models with ARM’s Ethos-U Vela compiler. (This means the output models can also work on other products based on ARM Cortex-M Processors and Ethos-U NPU.)
- Publish the optimized models on SenseCraft AI platform as public models for anyone to use.
There are already several NVIDIA TAO models such Trafficcamnet Detection and Peoplenet Detection available on SenseCraft AI platform. You can simply connect Grove Vision AI V2 and try these models with an easy no-code experience on your Grove Vision AI V2 now!
We will continue adding models to SenseCraft AI Platform, including optimized NVIDIA TAO models. Welcome to comment and let us know the models you want the most. And of course, you are more than welcome to publish and share your models on SenseCraft AI platform!
Last but not least, our team will share a Wiki with step-by-step instructions for you to go through the above-mentioned process, so that you can use NVIDIA TAO models on Grove Vision AI V2 for your edge AI projects. Stay tuned!
About Author
The phrase “interface” has been replaced with “connect” for better understanding.
How can I interface Grove Vision module v2 with the unihiker screen from df robot? For displaying images on the screen