_create_possibly_ragged_ndarray breaks if numpy 2.x.x is installed

[viktor-haag]

What happened + What you expected to happen

When you produce ragged ndarrays in map_batches, _create_possibly_ragged_ndarray will break if a numpy version with major version 2 is installed. The reason for that is an API change which was introduced with a major version bump in numpy.

relevant numpy changelog entry

Warnings and exceptions present in numpy.exceptions (e.g, ~numpy.exceptions.ComplexWarning, ~numpy.exceptions.VisibleDeprecationWarning) are no longer exposed in the main namespace.

As _create_possibly_ragged_ndarray is still accessing VisibleDeprecationWarning under the old path, this will fail with an AttributeError:

AttributeError: module 'numpy' has no attribute 'VisibleDeprecationWarning'

Suggestion: This can be fixed easily by replacing

ray/python/ray/air/util/tensor_extensions/utils.py

Line 49 in c1e7c45

warnings.simplefilter("ignore", category=np.VisibleDeprecationWarning)

with

major_version = int(np.__version__.split(".")[0])
if major_version > 1:
    visibleDeprecationWarning = np.exceptions.VisibleDeprecationWarning
else:
    visibleDeprecationWarning = np.VisibleDeprecationWarning
warnings.simplefilter("ignore", category=visibleDeprecationWarning)

Versions / Dependencies

ray==2.36.0
numpy==2.1.1

Reproduction script

import random
from typing import Dict

import numpy as np
import ray

context = ray.init()


class DummyMapper(object):
    def __call__(self, row: Dict[str, str]):
        batch_size = len(row["id"])
        dummy_data = [np.zeros(random.randint(1, 10)) for _ in range(batch_size)]
        return {"dummy_data": dummy_data, **row}


if __name__ == "__main__":
    samples = 10
    ds = ray.data.range(samples)
    ds.map_batches(DummyMapper, concurrency=1, batch_size=samples).write_json("./dummy_out")

Issue Severity

Low: It annoys or frustrates me.