AI-powered Depth Sensing with ZED camera and reComputer J2021 of Jetson Xavier NX

Depth sensing is a technology that allows a device to measure the distance to objects or surfaces in its environment. It is often used to create a 3D map or model of the environment. Depth sensing technology is used in a wide range of applications, including robotics, virtual and augmented reality, and computer vision. It can be used to create detailed 3D models of objects or environments, to enable robots to navigate and manipulate objects or to enable augmented reality experiences that are more realistic and immersive.

Depth sensing also includes stereoscopy and time-of-flight different measurements. Stereoscopy involves using two cameras to take images from slightly different angles, and then using the difference in the images to calculate the distance to objects in the scene. Time-of-flight (ToF) measures the time it takes for a pulse of light to travel to an object and back to the sensor. By measuring the time it takes for the light to travel, the distance to the object can be calculated.

Impelement 3D camera with NVIDIA Jetson

To use a 3D camera with the NVIDIA Jetson platform to achieve stereo depth sensing, you will need to select a suitable 3D camera. Today let’s take a look at how to install and configure the AI Stereo Camera via Cavedu‘s tutorial to install Stereolabs ZED with reComputer J2021 of Xavier NX.  ZED camera is filled with cutting-edge technology for spatial perception. ZED stereo cameras are modeled on human vision. Work with NVIDIA Jetson advanced embedded AI, two cameras (“eyes”) send video feeds to the GPU, and Stereolabs’ software calculates a depth map by measuring the difference between what it sees, similar to the human visual cortex. This technology allows the ZED to capture depth at up to 2.2k resolution and frame rates up to 120 fps. Developers can capture the 3D world and draw 3D models of indoor and outdoor scenes up to 20 meters away.

There is also a built-in IMU sensor, barometer sensor& magnetometer at ZED camera to help real-time synchronized inertial, elevation, and magnetic field data along with image and depth. Working with NVIDIA Jetson, developers can easily combine AI with 3D localization to create next-generation spatial awareness.

Stereolabs’ ZED cameras are high-definition 2K 3D cameras, with an SDK to deliver real-time depth sensing, tracking, and mapping of large spaces, as well as 3D Object detection and skeleton tracking. ZED features a USB3 interface and is plug-and-play with the Jetson platform. Stereolabs provides their optimized ZED SDK for NVIDIA Jetson, enabling developers to easily deploy 3D perception solutions for autonomous robotics, spatial analytics, and XR applications.

ZED contains minimal processing on-camera. Working with NVIDIA Jetson, ZED enables:

NEURAL DEPTH PERCEPTION

Using neural depth sensing, perceive your surroundings in 3D up to 20m distance, with increased accuracy in close range.

ENVIRONMENT 3D MAPPING

Localize your robot and capture a map of your environment using 3D SLAM. ROS compatible.

Being capable of 3D SLAM means that an autonomous machine can easily capture a detailed understanding of the world around it, opening new opportunities for any developer to navigate autonomously. With 3D depth cameras, machines can see the world almost like humans. Watch Stereolabs x Monarch Tractor integrated technology.

SPATIAL AI

Recognize and track 3D objects in real-time for navigation and manipulation.

Download and Install the ZED SDK

Download the ZED SDK for Jetson. Multiple versions of JetPack are supported, make sure to select the one that matches your system.

Go to the folder where the installer has been downloaded.

$ cd path/to/download/folder

Add execution permission to the installer using the chmod +x command. Make sure to replace the installer name with the version you downloaded.

$ chmod +x ZED_SDK_JP4.3_v3.0.run

Run the ZED SDK installer.

$ ./ZED_SDK_JP4.3_v3.0.run

On Jetson boards, CUDA is automatically installed with the JetPack so you’re now ready to use the ZED SDK.

Then let’s execute the ZED example on the reComputer. Go to the /usr/local/zed/tools/ directory and click the ZED_Depth_Viewer file to start this example. If you see the camera screen, it means that the ZED SDK is installed normally.

Please note: this example needs to be connected to a screen. SSH remote connection cannot successfully execute the example.

Find more tutorials at stereolabs and work with reComputer!

Image Capture	Shows how to open a stereo camera, capture an image and print its timestamp and image size in the terminal.
Depth Perception	Shows how to retrieve depth and point cloud of a scene, and print the distance of a given point in the terminal.
Camera Tracking	Shows how to enable positional tracking and get the position and orientation of the camera in real-time.
Spatial Mapping	Shows how to enable spatial mapping and capture a mesh or fused point cloud of the environment.
3D Object Detection	Shows how to detect objects in a scene and localize them in 3D (ZED 2 only).
Using Sensors	Shows how to acquire IMU, barometer and magnetometer data.

Resources

• Cavedu: [ZED深度攝影機系列] 如何在 Jetson Xavier NX 安裝 ZED SDK 與 Python開發環境範例 – 以 RK-reComputer J2012 為例
• stereolabs: Install ZED on Nvidia Jetson

Also read: DexForce’s NVIDIA Jetson-Powered 3D Camera Breaks The Cost Barrier to Industrial 3D Machine Vision with Seeed Fusion PCBA

Powered by NVIDIA Jetson Nano module, Xema is able to run 3D point cloud recognition algorithms and robotic arm control programs. Xema is also equipped with a DLP projector and a CMOS sensor, which enable the camera to perform fast imaging speed and strong anti-ambient light capability. It can generate high-resolution and precision point clouds of various objects such as reflective metal, black carbon fiber, thin cardboard, etc. On the working distance of 60mm, the Z-axis measurement accuracy can reach 50μm.

Seeed Jetson Ecosystem

Seeed is a Preferred Partner of NVIDIA Embedded for edge AI. Explore more carrier boards, full system devices, customization services, use cases, and developer tools at Seeed Jetson Ecosystem!

• You can also download our latest NVIDIA Jetson catalog.
• Latest update: we just announced reComputer of Orin NX and Orin Nano, let us know your interest and let us know your suggestions!

If you are working on edge AI solutions, join us as an Edge AI Partner

By consolidating our best-in-class hardware and cutting-edge technology from our software partners and the community, we aim at helping both developers and enterprises deploy ML in the real world across industries. Contact us at [email protected]

About Author