[WIP] Add jazzy CI, migrate rolling to noble

[luca-della-vedova]

No description provided.

[maspe36]

Run sudo pip3 install git+https://github.com/colcon/colcon-cargo.git
error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
    python3-xyz, where xyz is the package you are trying to
    install.
    
    If you wish to install a non-Debian-packaged Python package,
    create a virtual environment using python3 -m venv path/to/venv.
    Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
    sure you have python3-full installed.
    
    If you wish to install a non-Debian packaged Python application,
    it may be easiest to use pipx install xyz, which will manage a
    virtual environment for you. Make sure you have pipx installed.
    
    See /usr/share/doc/python3.12/README.venv for more information.

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.

Seems to be a new thing as of Python 3.11. Now with Jazzy targeting 24.04, we will need to figure out how we want to handle this... probably change our README instructions as well.

Not sure why Rolling is still on 3.10, guess its still based on Ubuntu 22.04.

[luca-della-vedova]

Ah yes I forgot about that, there is a break-system-packages flag that we can use, i.e. ros-infrastructure/ros_buildfarm#1026.

There has been a fair bit of activity on the setup-ros side, the current status it that to get Jazzy up we should set a flag to use testing repos, instead of adding it and reverting it in 10 days I'd probably just leave it to be done when the actual stable release is out and leave this in draft for the time being.

[esteve]

@maspe36 @luca-della-vedova we could use pipx. pipx is available on Debian/Ubuntu via the pipx package

[esteve]

@luca-della-vedova I hope you don't mind, I've pushed a change to your fork that replaces sudo pip3 install with pipx. I wanted to submit a PR to your fork, but I pushed to the wrong remote. Let me know if you'd rather I submit a PR to your fork.

[arjo129]

PIPX won't work with colcon-cargo-ros as that is installed as a library/plugin and pipx is for installing standalone executables.

[luca-della-vedova]

I can patch it up by putting a --break-system-packages that only gets set for noble (since the argument doesn't exist in 22.04 pip3 versions).
I suspect that the real solution however, would be to distribute colcon-cargo and colcon-ros-cargo as binaries through the buildfarm, was there any reason this wasn't done in the past? They seem quite conventional Python packages (except colcon-cargo that includes minimal Rust test code) so I wonder if we can go that direction and remove the pip3 install, since it is in general not a great practice to recommend.

[mxgrey]

I suspect that the real solution however, would be to distribute colcon-cargo and colcon-ros-cargo as binaries through the buildfarm, was there any reason this wasn't done in the past?

I agree with this, but I think there may still be some loose ends to tie up before moving forward with this, like getting workspaces to build properly. I'm attempting to build a workspace with colcon-ros-cargo but it fails because the individual crates don't have an associated Cargo.lock file (it doesn't seem to know that it should defer to the workspace's Cargo.lock file).

If we can get that working, I'd be comfortable opening a PR to get the packages into rosdistro.

But until then, I'm definitely having some irritating problems with getting these colcon extensions to work in Ubuntu 24.04. It seems it's necessary to create a virtual environment and do a complete install of colcon and its extensions into the virtual environment, and build the colcon workspace with the virtual environment sourced. All of the dependencies needed by everything in the workspace also need to be manually installed into the virtual environment.

[esteve]

Are you sure the colcon extensions built by the buildfarm? I can't find colcon in https://build.ros2.org/ , I think they are released separate from the rest of ROS. And in fact, they are available via packagecloud, which does not contain any other ROS package.

@cottsay do you have more info about releasing colcon extensions as Debian packages? Thanks.

[mxgrey]

The colcon installation page indicates that debians for colcon get distributed from both the ROS 1 and ROS 2 apt repo.

The packagecloud repo is meant for users that don't want to depend on the ROS ecosystem.

[esteve]

@mxgrey I can't find colcon in the buildfarm nor in the rosdistro repo. Some of the colcon packages are however referenced in https://github.com/ros/rosdistro/blob/master/rosdep/python.yaml#L5312-L5319, but I'd say they are there just so rosdep can resolve the keys.

Ah, I just found an ticket I filed exactly for this years ago:

colcon/colcon-core#436 (comment)

And yeah, that confirms what I wrote about the extensions not being released via the buildfarm and instead via packagecloud, or at least at the time that's how it worked.

More info about releasing Debian packages for colcon can be found at:

https://colcon.readthedocs.io/en/released/developer/release.html#publishing-a-release

[cottsay]

The colcon packages published to packagecloud are imported into the ROS 1 and ROS 2 repositories: https://github.com/ros-infrastructure/reprepro-updater/blob/master/config/colcon.ubuntu.upstream.yaml

That said, colcon-rust has simply never built a deb before: https://packagecloud.io/app/dirk-thomas/colcon/search?q=rust&filter=all&dist=

[mxgrey]

I spent another day trying to work with colcon+cargo in 24.04 (Python 3.12). It seems the changes to how pip works are really problematic for colcon extensions. Here are the key bullet points:

• To use colcon-cargo or colcon-ros-cargo (without resorting to the --break-system-packages flag) users will have to create a Python virtual environment to install the extensions into
• If you create your virtual environment with the --system-site-packages flag, then you can use the system-installed colcon while your virtual environment is active, but colcon will not be able to discover the colcon-cargo or colcon-ros-cargo extensions which were installed to the virtual environment
• Combining the above two bullet points means that anyone who wants to use colcon-cargo or colcon-ros-cargo must manually pip install colcon and all the desired extensions into the same virtual environment in order for everything to work

Now that the latest main branch of cargo-ament-build was published to crates.io, the cargo extensions for colcon are able to build workspaces correctly. In my mind this was the only crucial blocker for releasing, so I would support releasing debs of the extensions now.

If no one has any objections, I'll begin that process later this week.

[esteve]

@mxgrey

If no one has any objections, I'll begin that process later this week.

Did you manage to build Debian packages for the colcon cargo extensions? It'd be nice to revisit this PR and add support for newer distributions.