GuidesSensors and recordingSee what the robot sees
See what the robot sees
Get the robot's camera frames as arrays in full resolution or foveated, point the high-detail foveas, turn the head, and tune the streams.
A robot with a head streams its cameras into every session. This guide shows how to get the frames as numpy arrays, how to choose between the two head camera modes, how to point the foveas, and how to turn the head. It applies to robots in your fleet; an arm on your PC has no cameras.

Get the frames
frames = robot.sensors.get_camera_frames() # {stream: HxWx3 RGB uint8}
sorted(frames)
# ['gripper_cam_left', 'gripper_cam_right', 'stereo_cam_left',
# 'stereo_cam_left_fovea', 'stereo_cam_right', 'stereo_cam_right_fovea']
frames["stereo_cam_left"].shape # (1296, 2304, 3)
frames["stereo_cam_left_fovea"].shape # (640, 640, 3)Each call returns the newest frame of every stream that has delivered one and drops the older ones, so a loop that reads frames is never behind.
| Stream | What it shows | Size |
|---|---|---|
stereo_cam_left, stereo_cam_right | the two cameras in the head, each its whole field of view | 2304 x 1296 in full resolution, 1152 x 648 foveated |
stereo_cam_left_fovea, stereo_cam_right_fovea | a sharp window of each head camera, wherever you point it | 640 x 640, or 320 x 320 for eye tracking |
gripper_cam_left, gripper_cam_right | a camera on each gripper | 224 x 224 |
The head cameras run at 56 frames per second each. The gripper cameras come from a second computer on the robot. If it is off, the session runs without them and everything else works as usual. robot.sensors.camera_names() lists the streams this robot offers.
The frames are decoded on your PC by MAKIINA's own decoder in the native core that comes with the kit. A packet lost on the network damages a picture, and the damage stays in the frames after it until the robot's encoder refreshes that part. In full resolution your PC asks the robot for that refresh, and the head pictures are whole again within a few frames. In the foveated mode, and for the gripper cameras in either mode, the robot does not answer the request: the damage stays until the encoder's regular refresh, which starts every 560 frames, so for up to about 10 s. For a control-only session, skip the video altogether: Robot.connect("acme-robot-02", receive_cameras=False).
Choose full resolution or foveated
The head streams its two cameras in one of two modes:
- Full resolution, the default in the SDK, the Robot Console and the VR app: each camera sends its whole picture, 2304 x 1296 at 56 frames per second. The fovea frames are windows your PC cuts out of the full pictures.
- Foveated: each camera sends a 1152 x 648 context picture of its whole field of view, and the robot cuts a sharp fovea window out of the camera's 2304 x 1296 picture where you look and streams it next to the context. A picture that a lost packet damages stays so for up to about 10 s.
At the robot's starting bitrates both modes send the same, 8000 kbit/s per head camera (6000 for the context and 2000 for the fovea when foveated). To use less of your network, lower the bitrate: see Tune the streams.
Choose the mode when you connect, or switch it in a running session:
robot = Robot.connect("acme-robot-02", video_layout="head") # foveated
robot.control.set_video_layout("full") # full resolution, the streams restart
robot.sensors.get_video_layout() # what the robot streams now: "full" or "head"
robot.sensors.get_video_layout_status() # "streaming", "preparing", "failed: <why>", ...A switch restarts the head camera streams, so they blink for a moment. get_video_layout() tells you when the robot streams the new mode. A robot that runs older software streams foveated whatever you ask, and get_video_layout() then says "head".
Your code is the same in both modes: the stream names, the fovea sizes and the gaze calls do not change. Only the size of the context frames does, so read it from the frame rather than assuming it.
Show them
import matplotlib.pyplot as plt
frames = robot.sensors.get_camera_frames()
names = ["stereo_cam_left", "stereo_cam_right", "stereo_cam_left_fovea"]
fig, axes = plt.subplots(1, 3, figsize=(15, 4))
for ax, name in zip(axes, names):
ax.imshow(frames[name]); ax.set_title(name); ax.axis("off")
plt.show()Point the foveas
Each fovea is a window of its camera at the camera's full working resolution, 2304 x 1296 across its field of view, about 18.6 pixels per degree. Wherever you point it, you get the full detail of a 2304 x 1296 camera per eye. Across both eyes, that is a combined 4608 x 1296, wider than a 4K frame. You choose where each fovea looks:
robot.control.set_gaze([0.5, 0.5, 0.5, 0.5]) # right x, y, then left x, y
robot.control.set_gaze([0.7, 0.4, 0.7, 0.4]) # both a little right and up
robot.sensors.get_gaze() # what you last asked forCoordinates are fractions of the context image: x from 0 (left edge) to 1 (right edge), y from 0 (top) to 1 (bottom). Gaze is not motion, so it works while the robot is disarmed. The window centres on the gaze and stops at the edges of the picture. In the full resolution mode your PC cuts the window, the same size and in the same place as the robot would. The window of every frame comes back with it:
both = robot.sensors.get_camera_frames_and_metas() # {stream: (RGB, meta)}
both["stereo_cam_left_fovea"][1]["crop"] # the window's size and centre, and the gaze
robot.sensors.get_camera_frame_metas() # the metas alone: windows, head poseIn the console, clicking in a context view does the same.
Turn the head
robot.sensors.get_head() # yaw in rad, 0 is straight ahead
robot.control.set_head(0.3) # positive turns to the robot's rightTurning the head is a motion command like any joint move, so the first set_head arms the session.
Tune the streams
In the full resolution mode each camera's picture has one bitrate:
robot.control.set_video_bitrate(8000) # kbit/s per cameraIn the foveated mode the context and the fovea streams have their own:
robot.control.set_video_bitrates(context=6000, fovea=2000) # kbit/sBitrates change at once, without a restart, and the robot keeps them for later sessions. The fovea stays sharp at a much lower bitrate than the context streams, so the two are set separately.
The gaze method sets the size of the fovea window:
robot.control.set_gaze_method("EyeTrackingCombined") # 320 px foveas
robot.control.set_gaze_method("HeadsetOrientation") # 640 px foveasIn the foveated mode the robot restarts its camera pipelines to change the window, and the streams blink. In the full resolution mode only the window your PC cuts out changes.
The sensor settings apply in both modes:
robot.control.set_camera_params({"ae_enable": False, "exposure_time": 8000,
"analogue_gain": 4.0, "sharpness": 2.0})exposure_time is in microseconds. The robot restarts its pipelines to apply them and keeps them afterwards. With auto exposure on, both modes give the same brightness and colour on the same scene.
Record frames with the state
To pair frames with joint states for learning, record whole sessions instead of saving frames yourself: the recorder keeps every frame of every stream together with every command and state, losslessly. See Record demonstrations.