Specify a choice per label per video frame

[EricWiener]

Is your feature request related to a problem? Please describe.
I want to label videos of emergency vehicles and draw 2D boxes for each frame they are present, classify the type of vehicle, and then select a choice of whether the vehicle's emergency lights are flashing or not.

Right now I don't see a way to specify a choice per rectangle that allows me to change the choice selection per frame.

Here is my template:

<View>
   <Labels name="videoLabels" toName="video" allowEmpty="true">
     <Label value="kFireVehicle" background="#FF4D4F"/>
      <Label value="kMedicalVehicle" background="#FFA39E"/>
      <Label value="kPoliceVehicle" background="#1890FF"/>
      <Label value="kSchoolBus" background="#FADB14"/>
      <Label value="kGarbageTruck" background="#52C41A"/>
      <Label value="kConstructionVehicle" background="#FA8C16"/>
      <Label value="kRegularVehicle" background="#722ED1"/>
      <Label value="kTwoWheeledRegularVehicle" background="#13C2C2"/>
      <Label value="kUnknownVehicle" background="#8C8C8C"/>
      <Label value="kConstructionCone" background="#FF7A45"/>
      <Label value="kConstructionBarrier" background="#FFC53D"/>
      <Label value="kConstructionSign" background="#36CFC9"/>
      <Label value="kOtherConstruction" background="#F759AB"/>
   </Labels>
  
  <Choices name="choice" toName="video" perRegion="true" >
    <Choice value="Active"/>
    <Choice value="Inactive"/>
    <Choice value="Unknown"/>
  </Choices>
   
   <!-- Please specify FPS carefully, it will be used for all project videos -->
   <Video name="video" value="$video" framerate="25.0"/>
   <VideoRectangle name="box" toName="video" />
</View>

<!--{
 "video": "/static/samples/opossum_snow.mp4"
}-->

Describe the solution you'd like
Allow specifying choices per 2D rectangle per video frame.

Describe alternatives you've considered
I tried modifying the template to support this but had no luck.

Additional context
Attached is a video showing how changing the choice selection at one frame will change for all frames.

screenshot_2025-03-02_11_04_44.mp4

[heidi-humansignal]

Hello,

Thank you for contacting Label Studio,

Currently, Label Studio’s video annotation is designed so that a single VideoRectangle annotation (even when combined with a Choices tag using perRegion="true") applies its properties uniformly across the entire duration of the annotation. In other words, once you select a choice for a rectangle, that same choice is applied for all frames in which the rectangle appears. This is why, as you’ve observed, changing the choice on one frame affects all frames.
At the moment, there isn’t a built-in way to change the choice value on a per-frame basis for a single rectangle. The underlying design uses a sequence (or keyframe) mechanism to define the region over time, and the choice annotation is implicitly tied to that entire region rather than to individual frames.

A couple of workarounds you might consider are:

1. Splitting the Annotation Across Keyframes:

Use timeline functionality (if available in your project) to create separate keyframes with distinct choice values. This means you’d effectively split your annotation into segments (with potentially overlapping rectangles) so that you can assign different statuses for different frame ranges. Note that this would require additional configuration and might involve custom handling via the Label Studio SDK.
2. Custom Post-Processing:

If you’d like to retain a single annotation during labeling but need per-frame variations, you might consider exporting the annotation and then applying a custom script (outside of Label Studio) to adjust the choice values per frame as needed.

I realize this limitation can be a hurdle for detailed video labeling workflows like yours. We are aware of the demand for more granular video annotations and I will pass this usecase to our Product and Engineering team !

Have a good day!

Comment by Oussama Assili
Workflow Run