Panoptic Segmentation Annotator Request

[bhyun-kim]

Search before asking

• I have searched the Supervision issues and found no similar feature requests.

Description

It seems that supervision doesn't currently support Panoptic segmentation annotator. Is this feature included in your roadmap? If so, I would like to contribute.

Use case

No response

Additional

No response

Are you willing to submit a PR?

• Yes I'd like to help by submitting a PR!

[LinasKo]

Hi @bhyun-kim 👋

Sure, you can go ahead! Which model were you thinking of? I know transfomers has panoptic that we don't support yet, for example.

If I understand correctly, we'll need 2 types of id values - one for class and one for instance. If so, you may use Detections.class_id and create PANOPTIC_INSTANCE_ID_DATA_FIELD in supervision/config.py (it can be used in the Detections.data field).

[bhyun-kim]

Hi @LinasKo,

Sure, you can go ahead! Which model were you thinking of? I know transfomers has panoptic that we don't support yet, for example.

How about starting with 'from_mmdetection'? The panoptic segmentation prediction results in MMDetection include pred_panoptic_seg along with pred_instances, which is already supported. Huggingface's Transformers have various output format, so it might be difficult to handle them at once.

If I understand correctly, we'll need 2 types of id values - one for class and one for instance. If so, you may use Detections.class_id and create PANOPTIC_INSTANCE_ID_DATA_FIELD in supervision/config.py (it can be used in the Detections.data field).

Panoptic segmentation models return both semantic and instance segmentation results. Therefore, we need to store a numpy array that contains class information for each pixel in an image.

My concern is that Detections.data needs to be iterable so that it has the same length as Detections.xyxy or Detections.mask, as far as I understand the code. If so, this could be problematic because we only need to store one numpy array for the entire image. If not, we can implement it by storing the semantic segmentation in Detections.data as shown below.

@classmethod
def from_mmdetection(cls, mmdet_results) -> Detections:
    if hasattr(mmdet_results, 'pred_panoptic_seg'):
        return cls(
            xyxy=mmdet_results.pred_instances.bboxes.cpu().numpy(),
            confidence=mmdet_results.pred_instances.scores.cpu().numpy(),
            class_id=mmdet_results.pred_instances.labels.cpu().numpy().astype(int),
            mask=mmdet_results.pred_instances.masks.cpu().numpy()
            if "masks" in mmdet_results.pred_instances
            else None,
            data={'segmentation_map': mmdet_results.pred_panoptic_seg.sem_seg.cpu().numpy()}
        )
    else:
        return cls(
            xyxy=mmdet_results.pred_instances.bboxes.cpu().numpy(),
            confidence=mmdet_results.pred_instances.scores.cpu().numpy(),
            class_id=mmdet_results.pred_instances.labels.cpu().numpy().astype(int),
            mask=mmdet_results.pred_instances.masks.cpu().numpy()
            if "masks" in mmdet_results.pred_instances
            else None,
        )

[LinasKo]

Assigning transformers panoptic implementation to @onuralpszr.

[LinasKo]

@bhyun-kim, good point about the pixels.transformers, as far as I'm aware only return a single set of masks, so all we need is a second set of class labels.

Would you have some time to create a Colab for us, to see what kind of values panoptic MMDetection returns?

[bhyun-kim]

@LinasKo I am sharing the Colab notebook I created.

The issue is that the lengths of the semantic segmentation results and other outputs like masks, scores, labels (equivalent to classes), and bboxes are different. It seems that the data attribute of Detections needs to have the same length as masks, scores, and bboxes.