MAKIINASDK

GuidesSensors and recordingSee what the robot sees

See what the robot sees

Get the robot's camera frames as arrays in full resolution or foveated, point the high-detail foveas, turn the head, and tune the streams.

A robot with a head streams its cameras into every session. This guide shows how to get the frames as numpy arrays, how to choose between the two head camera modes, how to point the foveas, and how to turn the head. It applies to robots in your fleet; an arm on your PC has no cameras.

The Cameras panel in full res: Context L and Context R, the fovea card cut from the full picture, and Gripper R and Gripper L.

Get the frames

Python
frames = robot.sensors.get_camera_frames()      # {stream: HxWx3 RGB uint8}
sorted(frames)
# ['gripper_cam_left', 'gripper_cam_right', 'stereo_cam_left',
#  'stereo_cam_left_fovea', 'stereo_cam_right', 'stereo_cam_right_fovea']
frames["stereo_cam_left"].shape                 # (1296, 2304, 3)
frames["stereo_cam_left_fovea"].shape           # (640, 640, 3)

Each call returns the newest frame of every stream that has delivered one and drops the older ones, so a loop that reads frames is never behind.

StreamWhat it showsSize
stereo_cam_left, stereo_cam_rightthe two cameras in the head, each its whole field of view2304 x 1296 in full resolution, 1152 x 648 foveated
stereo_cam_left_fovea, stereo_cam_right_foveaa sharp window of each head camera, wherever you point it640 x 640, or 320 x 320 for eye tracking
gripper_cam_left, gripper_cam_righta camera on each gripper224 x 224

The head cameras run at 56 frames per second each. The gripper cameras come from a second computer on the robot. If it is off, the session runs without them and everything else works as usual. robot.sensors.camera_names() lists the streams this robot offers.

The frames are decoded on your PC by MAKIINA's own decoder in the native core that comes with the kit. A packet lost on the network damages a picture, and the damage stays in the frames after it until the robot's encoder refreshes that part. In full resolution your PC asks the robot for that refresh, and the head pictures are whole again within a few frames. In the foveated mode, and for the gripper cameras in either mode, the robot does not answer the request: the damage stays until the encoder's regular refresh, which starts every 560 frames, so for up to about 10 s. For a control-only session, skip the video altogether: Robot.connect("acme-robot-02", receive_cameras=False).

Choose full resolution or foveated

The head streams its two cameras in one of two modes:

  • Full resolution, the default in the SDK, the Robot Console and the VR app: each camera sends its whole picture, 2304 x 1296 at 56 frames per second. The fovea frames are windows your PC cuts out of the full pictures.
  • Foveated: each camera sends a 1152 x 648 context picture of its whole field of view, and the robot cuts a sharp fovea window out of the camera's 2304 x 1296 picture where you look and streams it next to the context. A picture that a lost packet damages stays so for up to about 10 s.

At the robot's starting bitrates both modes send the same, 8000 kbit/s per head camera (6000 for the context and 2000 for the fovea when foveated). To use less of your network, lower the bitrate: see Tune the streams.

Choose the mode when you connect, or switch it in a running session:

Python
robot = Robot.connect("acme-robot-02", video_layout="head")    # foveated
robot.control.set_video_layout("full")          # full resolution, the streams restart
robot.sensors.get_video_layout()                # what the robot streams now: "full" or "head"
robot.sensors.get_video_layout_status()         # "streaming", "preparing", "failed: <why>", ...

A switch restarts the head camera streams, so they blink for a moment. get_video_layout() tells you when the robot streams the new mode. A robot that runs older software streams foveated whatever you ask, and get_video_layout() then says "head".

Your code is the same in both modes: the stream names, the fovea sizes and the gaze calls do not change. Only the size of the context frames does, so read it from the frame rather than assuming it.

Show them

Python
import matplotlib.pyplot as plt

frames = robot.sensors.get_camera_frames()
names = ["stereo_cam_left", "stereo_cam_right", "stereo_cam_left_fovea"]
fig, axes = plt.subplots(1, 3, figsize=(15, 4))
for ax, name in zip(axes, names):
    ax.imshow(frames[name]); ax.set_title(name); ax.axis("off")
plt.show()

Point the foveas

Each fovea is a window of its camera at the camera's full working resolution, 2304 x 1296 across its field of view, about 18.6 pixels per degree. Wherever you point it, you get the full detail of a 2304 x 1296 camera per eye. Across both eyes, that is a combined 4608 x 1296, wider than a 4K frame. You choose where each fovea looks:

Python
robot.control.set_gaze([0.5, 0.5, 0.5, 0.5])    # right x, y, then left x, y
robot.control.set_gaze([0.7, 0.4, 0.7, 0.4])    # both a little right and up
robot.sensors.get_gaze()                        # what you last asked for

Coordinates are fractions of the context image: x from 0 (left edge) to 1 (right edge), y from 0 (top) to 1 (bottom). Gaze is not motion, so it works while the robot is disarmed. The window centres on the gaze and stops at the edges of the picture. In the full resolution mode your PC cuts the window, the same size and in the same place as the robot would. The window of every frame comes back with it:

Python
both = robot.sensors.get_camera_frames_and_metas()      # {stream: (RGB, meta)}
both["stereo_cam_left_fovea"][1]["crop"]                # the window's size and centre, and the gaze
robot.sensors.get_camera_frame_metas()                  # the metas alone: windows, head pose

In the console, clicking in a context view does the same.

Turn the head

Python
robot.sensors.get_head()                        # yaw in rad, 0 is straight ahead
robot.control.set_head(0.3)                     # positive turns to the robot's right

Turning the head is a motion command like any joint move, so the first set_head arms the session.

Tune the streams

In the full resolution mode each camera's picture has one bitrate:

Python
robot.control.set_video_bitrate(8000)                          # kbit/s per camera

In the foveated mode the context and the fovea streams have their own:

Python
robot.control.set_video_bitrates(context=6000, fovea=2000)     # kbit/s

Bitrates change at once, without a restart, and the robot keeps them for later sessions. The fovea stays sharp at a much lower bitrate than the context streams, so the two are set separately.

The gaze method sets the size of the fovea window:

Python
robot.control.set_gaze_method("EyeTrackingCombined")           # 320 px foveas
robot.control.set_gaze_method("HeadsetOrientation")            # 640 px foveas

In the foveated mode the robot restarts its camera pipelines to change the window, and the streams blink. In the full resolution mode only the window your PC cuts out changes.

The sensor settings apply in both modes:

Python
robot.control.set_camera_params({"ae_enable": False, "exposure_time": 8000,
                                 "analogue_gain": 4.0, "sharpness": 2.0})

exposure_time is in microseconds. The robot restarts its pipelines to apply them and keeps them afterwards. With auto exposure on, both modes give the same brightness and colour on the same scene.

Record frames with the state

To pair frames with joint states for learning, record whole sessions instead of saving frames yourself: the recorder keeps every frame of every stream together with every command and state, losslessly. See Record demonstrations.