[Question] Collect multiple images from different cameras

[zichunxx]

Question

Hi!

Is it possible to require multiple images from different cameras using Gymnasium API?

The list of camera names cannot be passed during registration and only one camera string is feasible.

Is there any elegant way to realize this feature?

Thanks.

[zichunxx]

I made a wrapper to collect images from different cameras:

class ImageCollection(gym.Wrapper, gym.utils.RecordConstructorArgs):
    def __init__(self, env: gym.Env, camera_names: list):
        gym.utils.RecordConstructorArgs.__init__(self, camera_names=camera_names)
        gym.Wrapper.__init__(self, env)

        assert env.render_mode == "rgb_array"
        self.unwrapped_model = env.model
        self.unwrapped_mujoco_renderer = env.mujoco_renderer
        self.camera_names = camera_names

    def render(self):
        frames = []
        for camera_name in self.camera_names:
            frames.append(self.unwrapped_mujoco_renderer.render(render_mode=self.render_mode, camera_name=camera_name))
        return frames

I am not sure if the above is the best way to realize this. Please feel free to offer some suggestions.

[pseudo-rnd-thoughts]

This solution looks like the best available currently

[zichunxx]

Thanks for your time to check the above code snippet.

[Kallinteris-Andreas]

The proper solution for MuJoCo Gymnasium would be to support multiple MujocoRendering objects in MujocoEnv, one for each camera, that will enable different configurations such as using "rgb_array" and "depth_array" cameras at the same time and multiple "human" render windows at the same time.

[zichunxx]

The proper solution for MuJoCo Gymnasium would be to support multiple MujocoRendering objects in MujocoEnv, one for each camera, that will enable different configurations such as using "rgb_array" and "depth_array" cameras at the same time and multiple "human" render windows at the same time.

That would be great if this is implemented.

[pseudo-rnd-thoughts]

@Kallinteris-Andreas The render api doesn't seem built to support as by default so would recommend that until this is a common feature wanted by many users, I wouldn't implement it within the codebase.

[zichunxx]

Hi! @pseudo-rnd-thoughts @Kallinteris-Andreas Is there any elegant method to collect multiple images with the human render mode when collecting expert demonstration?

[Kallinteris-Andreas]

For human (with mujoco I am assuming), no there is not a way to interact with it on the program's side (like "screenshotting" it), human is just for interfacing with us "humans"

taking rgb_arrays is the way to go