GDK Camera Interface Documentation (Python)

Overview

The Camera module provides the G02 robot with the ability to acquire real-time image data. Through the Python interface, developers can conveniently obtain the robot's visual perception data, suitable for object detection, image recognition, visual navigation, SLAM mapping, environment monitoring, and many other scenarios.

Interface Description

Camera Class

This class encapsulates the main data acquisition interfaces of the camera sensor.

1. get_latest_image()

• Function: Get the latest image data
• Parameters:

Parameter	Type	Description
type	CameraType	Camera type enum value
timeout	float	Timeout duration (milliseconds)

• Return value: An Image object, containing the following attributes:

Attribute	Type	Description	Unit
timestamp_ns	int	Timestamp of image acquisition	nanoseconds
width	int	Image width (pixels)	pixels
height	int	Image height (pixels)	pixels
encoding	str	Image encoding format	string
color_format	str	Image color format	string
bit_depth	int	Bits per pixel	bits
data	bytes	Raw pixel data of the image	bytes

Detailed Description of the Image Object

encoding (encoding format):

• Type: str
• Common values:
• "rgb8": RGB color image, 8 bits per channel
• "bgr8": BGR color image, 8 bits per channel
• "mono8": Grayscale image, 8 bits
• "mono16": Grayscale image, 16 bits
• "32FC1": Single-channel 32-bit float (depth map)

color_format (color format):

• Type: str
• Common values:
• "RGB": Red-Green-Blue
• "BGR": Blue-Green-Red
• "GRAY": Grayscale
• "DEPTH": Depth

bit_depth (bit depth):

• Type: int
• Common values:
• 8: 8-bit (0-255)
• 16: 16-bit (0-65535)
• 32: 32-bit (float)

data (image data):

• Type: bytes
• Description: Raw pixel data of the image
• Use: Image processing, display, saving
• Note: Needs to be parsed according to the encoding and dimensions

Camera types: - kHeadBackFisheye: Head rear fisheye camera - kHeadLeftFisheye: Head left fisheye camera - kHeadRightFisheye: Head right fisheye camera - kHeadStereoLeft: Head stereo left camera - kHeadStereoRight: Head stereo right camera - kHandLeftColor: Left hand color camera - kHandRightColor: Right hand color camera - kHeadColor: Head color camera - kHeadDepth: Head depth camera (outputs a depth map) - kHandLeftDepth: Left hand depth camera (outputs a depth map) - kHandRightDepth: Right hand depth camera (outputs a depth map) - kHandLeftUpperColor: Left hand upper color camera (reserved) - kHandRightUpperColor: Right hand upper color camera (reserved) - kHandLeftLowerColor: Left hand lower color camera (reserved) - kHandRightLowerColor: Right hand lower color camera (reserved) - kHandLeftUpperDepth: Left hand upper depth camera (outputs a depth map) (reserved) - kHandRightUpperDepth: Right hand upper depth camera (outputs a depth map) (reserved) - kHandLeftLowerDepth: Left hand lower depth camera (outputs a depth map) (reserved) - kHandRightLowerDepth: Right hand lower depth camera (outputs a depth map) (reserved)

• In standard mode, the head stereo left camera, head stereo right camera, left/right color cameras, right hand color camera, head color camera, and head depth camera are open by default; the remaining cameras are closed by default, and it is not recommended to enable the remaining cameras in standard mode

• The remaining cameras can be enabled or disabled in develop mode

• Example:

import agibot_gdk
import time

# Initialize the GDK system
if agibot_gdk.gdk_init() != agibot_gdk.GDKRes.kSuccess:
    print("GDK initialization failed")
    exit(1)
print("GDK initialized successfully")

camera = agibot_gdk.Camera()
time.sleep(3)  # Wait for the camera to initialize

image = camera.get_latest_image(agibot_gdk.CameraType.kHeadStereoLeft, 1000.0)
if image is not None:
    print(f"✅ Timestamp: {image.timestamp_ns}")
    print(f"Image size: {image.width} x {image.height}")
    print(f"Encoding format: {image.encoding}")
    print(f"Color format: {image.color_format}")
    print(f"Bit depth: {image.bit_depth}")
else:
    print("No image data obtained")

# Close the camera
camera.close_camera()

# Release GDK system resources
if agibot_gdk.gdk_release() != agibot_gdk.GDKRes.kSuccess:
    print("GDK release failed")
else:
    print("GDK released successfully")

2. get_nearest_image()

• Function: Get the image data nearest to a specified timestamp
• Parameters:

Parameter	Type	Description
type	CameraType	Camera type enum value
timestamp	int	Target timestamp (nanoseconds)
timeout	float	Timeout duration (milliseconds)

• Return value: An Image object, with the same structure as get_latest_image()

• Example:

import agibot_gdk
import time

# Initialize the GDK system
if agibot_gdk.gdk_init() != agibot_gdk.GDKRes.kSuccess:
    print("GDK initialization failed")
    exit(1)
print("GDK initialized successfully")

camera = agibot_gdk.Camera()
time.sleep(3)

# First get the latest image data
image = camera.get_latest_image(agibot_gdk.CameraType.kHeadStereoLeft, 1000.0)

if image is not None:
    # Get historical image data (1 second earlier)
    image_nearest = camera.get_nearest_image(
        agibot_gdk.CameraType.kHeadStereoLeft,
        image.timestamp_ns - 1000000000,
        1000.0
    )

    if image_nearest is not None:
        print(f"✅ Nearest image data: {image_nearest.timestamp_ns}")
        print(f"Image size: {image_nearest.width} x {image_nearest.height}")
        print(f"Encoding format: {image_nearest.encoding}")
    else:
        print("❌ No nearest image data found")
else:
    print("Failed to obtain the latest image data")

# Close the camera
camera.close_camera()

# Release GDK system resources
if agibot_gdk.gdk_release() != agibot_gdk.GDKRes.kSuccess:
    print("GDK release failed")
else:
    print("GDK released successfully")

3. get_image_shape()

• Function: Get the image data size
• Parameters:

Parameter	Type	Description
type	CameraType	Camera type enum value

• Return value: tuple, a tuple containing the image width and height (width, height)

• Example:

import agibot_gdk

# Initialize the GDK system
if agibot_gdk.gdk_init() != agibot_gdk.GDKRes.kSuccess:
    print("GDK initialization failed")
    exit(1)
print("GDK initialized successfully")

camera = agibot_gdk.Camera()

shape = camera.get_image_shape(agibot_gdk.CameraType.kHeadStereoLeft)
print(f"Image size: {shape[0]} x {shape[1]}")

# Close the camera
camera.close_camera()

# Release GDK system resources
if agibot_gdk.gdk_release() != agibot_gdk.GDKRes.kSuccess:
    print("GDK release failed")
else:
    print("GDK released successfully")

4. get_image_fps()

• Function: Get the image capture frame rate
• Parameters:

Parameter	Type	Description
type	CameraType	Camera type enum value

• Return value: float, the image frame rate (FPS)

• Example:

import agibot_gdk

# Initialize the GDK system
if agibot_gdk.gdk_init() != agibot_gdk.GDKRes.kSuccess:
    print("GDK initialization failed")
    exit(1)
print("GDK initialized successfully")

camera = agibot_gdk.Camera()

fps = camera.get_image_fps(agibot_gdk.CameraType.kHeadStereoLeft)
print(f"Image frame rate: {fps} FPS")

# Close the camera
camera.close_camera()

# Release GDK system resources
if agibot_gdk.gdk_release() != agibot_gdk.GDKRes.kSuccess:
    print("GDK release failed")
else:
    print("GDK released successfully")

5. get_image_latency()

• Note: Before obtaining image latency statistics, time synchronization must first be performed, otherwise the latency statistics results will be inaccurate
• Function: Get image latency statistics
• Parameters:

Parameter	Type	Description
type	CameraType	Camera type enum value
window_seconds	float	Statistics window duration (seconds)

• Return value: A LatencyStats object, containing latency statistics

• Example:

import agibot_gdk

# Initialize the GDK system
if agibot_gdk.gdk_init() != agibot_gdk.GDKRes.kSuccess:
    print("GDK initialization failed")
    exit(1)
print("GDK initialized successfully")

camera = agibot_gdk.Camera()

latency = camera.get_image_latency(agibot_gdk.CameraType.kHeadStereoLeft, 1.0)
print(f"Max latency: {latency.max_latency_ms} ms")
print(f"Average latency: {latency.avg_latency_ms} ms")

# Close the camera
camera.close_camera()

# Release GDK system resources
if agibot_gdk.gdk_release() != agibot_gdk.GDKRes.kSuccess:
    print("GDK release failed")
else:
    print("GDK released successfully")

6. get_camera_intrinsic()

• Function: Get camera intrinsic parameter information
• Parameters:

Parameter	Type	Description
type	CameraType	Camera type enum value

• Return value: A CameraIntrinsic object, containing camera intrinsic parameter information

• Note: Not all camera types support retrieval of intrinsic parameters. The camera types that support intrinsic parameters include: kHeadBackFisheye, kHeadLeftFisheye, kHeadRightFisheye, kHeadStereoLeft, kHeadStereoRight, kHandLeftColor, kHandRightColor, kHeadColor, kHeadDepth, kHandLeftDepth, kHandRightDepth. For unsupported camera types, calling this interface will throw an exception.

Detailed Description of the CameraIntrinsic Object

The CameraIntrinsic struct contains the following members:

Member	Type	Description	Unit
intrinsic	list[float]	Camera intrinsics [fx, fy, cx, cy]	pixels
distortion	list[float]	Distortion parameters [k1, k2, p1, p2, k3, k4, k5, k6]	unitless

Intrinsic parameter support for different camera types:

Camera type	intrinsic length	distortion length	Description
Stereo camera	4	8	fx,fy,cx,cy,k1,k2,p1,p2,k3,k4,k5,k6
RGBD camera	4	5	fx,fy,cx,cy,k1,k2,p1,p2,k3
Fisheye camera	4	6	fx,fy,cx,cy,k1,k2,p1,p2,k3,k4

• Example:

import agibot_gdk

# Initialize the GDK system
if agibot_gdk.gdk_init() != agibot_gdk.GDKRes.kSuccess:
    print("GDK initialization failed")
    exit(1)
print("GDK initialized successfully")

camera = agibot_gdk.Camera()

intrinsic = camera.get_camera_intrinsic(agibot_gdk.CameraType.kHeadStereoLeft)
print(f"Camera intrinsics:")
print(f"  fx: {intrinsic.intrinsic[0]}")
print(f"  fy: {intrinsic.intrinsic[1]}")
print(f"  cx: {intrinsic.intrinsic[2]}")
print(f"  cy: {intrinsic.intrinsic[3]}")

print(f"Distortion parameters:")
for i, dist in enumerate(intrinsic.distortion):
    print(f"  k{i+1}: {dist}")

# Close the camera
camera.close_camera()

# Release GDK system resources
if agibot_gdk.gdk_release() != agibot_gdk.GDKRes.kSuccess:
    print("GDK release failed")
else:
    print("GDK released successfully")

7. set_dev_camera_config()

• Function: Set the camera development configuration
• Parameters:

Parameter	Type	Description
cam_conf_path	str	Path to the camera configuration file

• Return value: None. Returns nothing on success; throws a std::runtime_error exception on failure

• Note:

• The configuration file path must exist, otherwise an exception will be thrown
• Each time you modify the custom camera configuration file and call this interface, you need to switch back to develop mode

Configuration options

GDK supports configuring the on/off state and frame rate of each camera. The customized camera configuration file is located under the deployment package, and its absolute path is typically ~/.cache/agibot/app/gdk/config/cam_config.json (where ~ denotes the user's home directory, e.g. /home/your_name) Set publish to true to enable the camera, false to disable it, and set fps to control the camera frame rate The specific configuration format for the camera is as follows

{
    "cam0": {
        "fps": "30",
        "name": "head_stereo_right",
        "publish": true
    },
    "cam3": {
        "fps": "30",
        "name": "head_stereo_left",
        "publish": true
    },
    "cam4": {
        "fps": "30",
        "name": "hand_left_depth",
        "publish": false
    },
    "cam5": {
        "fps": "30",
        "name": "hand_left_color",
        "publish": true
    },
    "cam6": {
        "fps": "30",
        "name": "hand_right_depth",
        "publish": false
    },
    "cam7": {
        "fps": "30",
        "name": "hand_right_color",
        "publish": true
    },
    "cam10": {
        "fps": "30",
        "name": "head_right_fisheye",
        "publish": false
    },
    "cam11": {
        "fps": "30",
        "name": "head_left_fisheye",
        "publish": false
    },
    "cam12": {
        "fps": "30",
        "name": "head_back_fisheye",
        "publish": false
    },
    "cam14": {
        "fps": "30",
        "name": "head_depth",
        "publish": true
    },
    "cam15": {
        "fps": "30",
        "name": "head_color",
        "publish": true
    }
}

• Example:

import agibot_gdk
import time

# Initialize the GDK system
if agibot_gdk.gdk_init() != agibot_gdk.GDKRes.kSuccess:
    print("GDK initialization failed")
    exit(1)
print("GDK initialized successfully")

camera = agibot_gdk.Camera()
time.sleep(1)  # Wait for the camera to initialize

# Set the camera configuration
config_path = "/home/<your_name>/.cache/agibot/app/gdk/config/cam_config.json"
try:
    camera.set_dev_camera_config(config_path)
    print("Camera configuration set successfully")
except RuntimeError as e:
    print(f"Failed to set camera configuration: {e}")

# Close the camera
camera.close_camera()

# Release GDK system resources
if agibot_gdk.gdk_release() != agibot_gdk.GDKRes.kSuccess:
    print("GDK release failed")
else:
    print("GDK released successfully")

Mode switching

cd ~/.cache/agibot/app/bin
./mode_switch --mode develop # Switch to develop mode

To switch back to the previous base mode, run

./mode_switch --mode base # Switch to base mode

8. close_camera()

• Function: Close the camera connection
• Parameters: None

• Return value: GDKRes, the operation result status code

• Example:

import agibot_gdk

# Initialize the GDK system
if agibot_gdk.gdk_init() != agibot_gdk.GDKRes.kSuccess:
    print("GDK initialization failed")
    exit(1)
print("GDK initialized successfully")

camera = agibot_gdk.Camera()

# Use the camera...

# Close the camera
result = camera.close_camera()
if result == agibot_gdk.GDKRes.kSuccess:
    print("Camera closed successfully")
else:
    print("Failed to close the camera")

# Release GDK system resources
if agibot_gdk.gdk_release() != agibot_gdk.GDKRes.kSuccess:
    print("GDK release failed")
else:
    print("GDK released successfully")

Usage Notes

1. GDK initialization: You must call agibot_gdk.gdk_init() to initialize the GDK system before using the Camera functionality
2. GDK release: You must call agibot_gdk.gdk_release() to release GDK system resources before the program ends
3. Initialization wait: After creating a Camera object, it is recommended to wait 3 seconds to ensure camera initialization completes
4. Timeout setting: Set an appropriate timeout duration based on actual needs to avoid long blocking periods
5. Data validity: Check whether the returned image data is None before use
6. Timestamp precision: The timestamp unit is nanoseconds, which can be used for precise time synchronization
7. Image processing: Image data volume is relatively large, so pay attention to memory usage during processing
8. Camera selection: Choose the appropriate camera type (fisheye/stereo/depth, etc.) based on the application scenario
9. Frame rate control: Be mindful of the camera's frame rate limits to avoid excessive requests
10. Camera intrinsics: Use get_camera_intrinsic() to obtain camera intrinsic parameters for image rectification and 3D reconstruction
11. Camera configuration: When using set_dev_camera_config() to set the camera configuration, make sure the configuration file path is valid
12. Resource management: Call close_camera() to release camera resources after use
13. Exception handling: Except for close_camera(), other interfaces throw a std::runtime_error exception on failure and need to be handled appropriately. close_camera() returns a GDKRes status code and does not throw exceptions
14. Camera switches: In standard mode/develop mode, enabling more cameras carries a performance risk

Application Scenarios

• Object detection: Using image data for object recognition and detection
• Visual navigation: Providing visual information for robot navigation
• SLAM mapping: Combining image data for simultaneous localization and mapping
• Environment monitoring: Real-time monitoring of changes in the surrounding environment
• Depth perception: Using the depth camera to obtain 3D environment information
• Stereo vision: Using the stereo camera for distance measurement
• Image recognition: Performing object classification and recognition
• Data fusion: Fusing with other sensor data to improve perception accuracy
• Camera calibration: Using intrinsic parameters for image rectification and distortion compensation
• 3D reconstruction: Combining intrinsic parameters for 3D point cloud reconstruction
• Visual measurement: Using camera intrinsic parameters for precise dimensional measurement
• AR/VR applications: Implementing augmented reality and virtual reality functionality based on camera intrinsic parameters