Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ability to use different Objectdetection models than FasterRCNN #2147

Open
julled opened this issue May 7, 2024 · 1 comment
Open

Add ability to use different Objectdetection models than FasterRCNN #2147

julled opened this issue May 7, 2024 · 1 comment

Comments

@julled
Copy link

julled commented May 7, 2024

🚀 Feature

It would be good to be able to use more up to date models than FasterRCNN in raster-vision.
Torchvision offers some possible dropin replacements / alternatives like RetinaNet or SSD based detectors.

Motivation

More up to date models promise better performance at lower GPU needs.

Pitch

The object_detection_learner in raster-vision should be able to accept a configuration parameter which allows a different model to be used, as written above.

Alternatives

The user could do this manually in the code, but this is not very scaleable and unclear how many manual changes are needed.

Additional context

@AdeelH
Copy link
Collaborator

AdeelH commented May 7, 2024

I agree that it would be useful to support other TorchVision object detection models. Adding something like an architecture field to ObjectDetectionModelConfig and then modifying ObjectDetectionModelConfig.build_default_model() might be the way to go.

That said, one way to do this without any changes to RV is to use the workaround described here:

if external_model:
"""This demonstrates how to use an external model for object detection,
but to successfully use this functionality with different settings, the
following things should be kept in mind:
(1) Torchvision does not expose its object detection models via
torch hub (https://github.com/pytorch/vision/issues/1945). So, to
use those, you will need to fork the torchvision repo and manually
add those models or corresponding factory functions to hubconf.py.
Example: github.com/AdeelH/vision/blob/det_hubconf_0.10/hubconf.py.
Further, you should ensure that the branch of the fork is the same
version as the version in Raster Vision's Docker image; or, if
using outside Docker, it should match the version of the local
torchvision installation.
(2) The external model should behave exactly like torchvision
object detection models. This includes, but might not be limited
to:
- Accepting targets as dicts with keys: 'boxes' and 'labels'.
- Accepting 1-indexed class labels.
- Computing losses internally and returning them in a dict
during training.
- Returning predictions as dicts with keys: 'boxes', 'labels',
and 'scores'.
"""
model = ObjectDetectionModelConfig(
external_def=ExternalModuleConfig(
github_repo='AdeelH/vision:det_hubconf_0.12',
name='ssd',
entrypoint='ssd300_vgg16',
force_reload=True,
entrypoint_kwargs={
# torchvision OD models need add an additional null class,
# so +1 is needed here
'num_classes': len(class_config.names) + 1,
'pretrained': False,
'pretrained_backbone': True
}))

You'll need to fork the torchvision repo and create a branch with hubconf.py modified like this: https://github.com/AdeelH/vision/blob/356a78a894acf2a377353a4cdb2df52c1cea84b4/hubconf.py#L68-L72

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants