YOLOv7 TensorRT Benchmark from Cloud GPUs to Edge GPUs

How does the real-time computer vision inference performance vary from Cloud GPUs to Edge GPUs? 

Nilvana (an AI software platform company) conducted a series of performance testing of Object Detection, Anomaly Detection, and Instance Segmentation models on different devices, from cloud GPUs A100 to the latest tiny powerhouse Jetson AGX Orin (available now at Seeed!). Let’s take a look at the test results! 

Object Detection

Nilvana tested YOLOv7 variants models on A100, RTX 3060, Jetson AGX Orin, and Jetson Xavier NX (check out the reComputer J2012 based on Xavier NX!). Read more about this from the article at Nilvana Medium. 

What is Object Detection?

Image classification, object detection, and segmentation are three major tasks in computer vision. Image classification divides an image into a single category, usually corresponding to the most prominent object in the image. However, real world images usually contain more than one object. If the image classification model is used to assign a single label to the image, it can be rough and inaccurate. In such cases, an object detection model is needed to identify multiple objects in a picture and locate different objects (with bounding boxes). Object detection is used in real-world AI such as object tracking, video analytics, security, and autonomous driving.

YOLOv7

You Only Look Once (YOLO) is a state-of-the-art, real-time object detection system. The current and latest iteration, YOLOv7, infers faster and with great accuracy pushing Object Detection to newer heights. Wanna learn more about YOLO and get started with object detection? Check out our step-by-step tutorial using Roboflow and YOLOv5 to build the object detection model based on custom datasets and deploy it at NVIDIA Jetson-powered reComputer!

Test on DGX A100, RTX 3060, AGX Orin, and Xavier NX

According to the results table, Jetson Xavier NX can run the YOLOv7-tiny model pretty well. AGX Orin can even run the YOLOv7x model at more than 30 FPS!

Anomaly Detection 

What is Anomaly Detection?

Anomaly Detection is a step in data mining that identifies rare data points, events, and observations that deviate from a dataset’s normal behavior. Examples are technical glitches or potential opportunities such as a change in consumer behavior.  

Nilvana has tested out anomaly detection on the RTX 3060 and Jetson AGX Orin on the MVTec AD dataset. Read more about it in Nilvana’s medium article! They have tested training and inferencing using PyTorch Lightning by providing about 200 normal images and 20 abnormal images for each defect of the product. Trains the model with only normal images and evaluates the AUROC (Area Under the Receiver Operating Characteristic Curve) and F1 Score by a few abnormal images.

PaDiM is one of the famous anomaly detection algorithms on the leaderboard, the evaluation shows that PaDim is applicable to both large and small area defects.

Test on RTX 3060 and Jetson AGX Orin

Dataset: MVTec AD[3]

Class: Bottle

Training Image: 209 normal images, 20 abnormal images of each defect

Framework: PyTorch Lightning

Algorithm: PaDiM

The training time and inference FPS on NVIDIA RTX 3060 and Jetson AGX Orin respectively.

We can tell from NVIDIA Jetson AGX Orin can perform perspectively as well as RTX3060 and even A100. To train the model on cloud GPU and deploy it on an edge device is common practice, the above result of Jetson AGX Orin shows that performance on the edge device will be very reliable. We are excited to see more applications on this powerful tiny machine. Seeed is also working on new AGX Orin and upcoming Orin NX-powered edge devices. We released the reServer J5014, which is under development now. Let us know your thoughts and expectation on the discord edge ai channel!

reServer Jetson: Inference Center for the Edge

Powered by Jetson AGX Orin / NX

• Portable Edge AI center with up to 275 TOPS AI performance
• Chill 24/7 with Vapor Chamber heatsink (AGX Orin only), 40% more effective thermal performance
• Rapid network access and hybrid connectivity 2.5Gbes, 4G, 5G, LoRa, BLE, and WiFi (modules not included）
• Built-in storage for multiple concurrent AI applications
• Support Triton Inference Server and Pre-installed Jetpack
• Rich IOs ready for AIoT
• USB 3.2 Type-A port, USB 2.0 Type-A port, HDMI port, and DP port

reComputer Jetson: Real-World AI at the Edge, starts from $199

Built with Jetson Nano 4GB / Xavier NX 8GB/16GB

• Edge AI box fits into anywhere
• Embedded Jetson Nano/NX Module
• Pre-installed Jetpack for easy deployment
• Nearly the same form factor as Jetson Developer Kits, with a rich set of I/Os
• Stackable and expandable

About Nilvana

Nilvana is an innovative international brand dedicated to devising straightforward AI elements while showcasing them. They provide a simple, easy-to-use, and intuitive graphical interface which satisfies variable demands of AI development and inference. They mainly focus on the industries of smart city, smart transportation, manufacturing, and retail. 

Read More:

• All you need to know about Jetson AGX Orin
• Compare Jetson AGX Orin and AGX Xavier
• reServer J5014 – powered by AGX Orin
• Jetson AGX Orin Dev Kit – buy at Seeed Studio

About Author