Edge case: RandomWindowGeoDataset can sometimes cause the training to end prematurely

[AdeelH]

🐛 Bug

See the following discussion on Gitter: https://gitter.im/azavea/raster-vision?at=6298bff6ef00bd1dc60fc7ee

Whenever a RandomWindowGeoDataset for a scene fails to find an eligible chip in max_sample_attempts attempts, it throws a StopIteration error and causes the training to end, ignoring any other datasets that might still not be exhausted. This causes the training epoch to end prematurely.

To Reproduce

Steps to reproduce the behavior:

1. Use a training set with at least one scene where there are no eligible chips (e.g. due to AOI being too small) and at least one normal scene with multiple eligible chips.
2. Use a GeoDataConfig(methods=GeoDataWindowMethod.random) for both.
3. Attempt to train by running the train command.

The training should end immediately even though there are several unused chips in the second scene.

Expected behavior

The training should not end until all eligible chips have been used.

Environment

• How you installed and are running Raster Vision (pip install on local vs. inside Docker image): N/A
• Raster Vision version or commit: 0.13 and later
• OS (e.g., Linux): N/A
• Python version: N/A
• CUDA/cuDNN version if running on GPU: N/A
• Any other relevant information: N/A

Additional context

It is possible that using an IterableDataset might be more appropriate for RandomWindowGeoDataset.