Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CI: Add portability and integration tests with Sage #1760

Open
wants to merge 6 commits into
base: main
Choose a base branch
from

Conversation

mkoeppe
Copy link
Contributor

@mkoeppe mkoeppe commented Jan 30, 2024

As promised in #1166 (comment) @albinahlback @edgarcosta, here's an updated GH Actions workflow for portability and integration testing using the reusable SageMath workflows.

Currently configured to:

  • only run on tags, not branches / PRs; can also run manually
  • build FLINT from scratch and run its test suite on all Linux platforms supported by SageMath
  • build FLINT from scratch and run its test suite on some macOS configurations
  • build Sage incrementally from a prebuilt Docker image from ghcr.io for one Linux platform and run the Sage test suite.

Preview of the workflow run: https://github.com/mkoeppe/flint2/actions/workflows/ci-sage.yml

@mkoeppe mkoeppe force-pushed the ci_sage_2024 branch 2 times, most recently from cc82dd5 to eb35c62 Compare January 30, 2024 23:25
@mkoeppe mkoeppe changed the title Ci sage 2024 CI: Add portability and integration tests with Sage Jan 30, 2024
@albinahlback
Copy link
Collaborator

How long does each run take?

Also, if we would instantiate a new tag, the "damage" could already have been done months back with multiple commits in between. Tracking bugs and issues, and reverting commits with possible would be way harder in this case when the time of a new tag comes around. As such, this would preferably be run at least at every merge, but I suppose that this would take too long.

Now, this is hopefully mostly important at the moment as we do have some change in the interface from 3.0 to the upcoming 3.1. Don't know if this would impact Sage.

Other than that, this looks good!

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Jan 31, 2024

How long does each run take?

Each of the Linux runs is between 10 and 15 minutes, the macOS runs are 16 minutes.

The integration testing with Sage probably around 1.5 hours. Currently fails trying to rebuild Singular - https://github.com/mkoeppe/flint2/actions/runs/7718792281/job/21040792339

Also, if we would instantiate a new tag, the "damage" could already have been done months back with multiple commits in between. Tracking bugs and issues, and reverting commits with possible would be way harder in this case when the time of a new tag comes around. As such, this would preferably be run at least at every merge, but I suppose that this would take too long.

Sure, that's something for the project to figure out.

For the portability testing, it would would also be possible to only run a few selected platforms on each PR, but all platforms on tags.

Or it could be configured as a cron job that runs it nightly or weekly.

For the integration testing with Sage, we would be able to test some subset of the test suite instead of the whole test suite.

I'll happy to help with adjusting the workflow file to implement the desired policy.

@albinahlback
Copy link
Collaborator

albinahlback commented Jan 31, 2024

Each of the Linux runs is between 10 and 15 minutes, the macOS runs are 16 minutes.

Sounds good!

The integration testing with Sage probably around 1.5 hours. Currently fails trying to rebuild Singular - https://github.com/mkoeppe/flint2/actions/runs/7718792281/job/21040792339

We will look into this before releasing 3.1. Thanks!

Or it could be configured as a cron job that runs it nightly or weekly.

Seems like the best idea. I would prefer this.

For the integration testing with Sage, we would be able to test some subset of the test suite instead of the whole test suite.

This would be really good. Moreover, I don't think there is no need to test different OSs from our side as we have tests for different OSs already.

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Jan 31, 2024

For the failures on ubuntu-focal-standard, linuxmint-20.*-standard, fedora-{33,34,35}-standard, centos-stream-9-standard, almalinux-9-standard, I have opened:

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Jan 31, 2024

A failure:

https://github.com/mkoeppe/flint2/actions/runs/7718792281/job/21040792473#step:14:6853

  fmpz_mat_is_square...
  fmpz_mat_ismake[3]: *** [build/fmpz_mod_mat/test/main_TEST_RUN] Segmentation fault (core dumped)

@albinahlback
Copy link
Collaborator

A failure:

https://github.com/mkoeppe/flint2/actions/runs/7718792281/job/21040792473#step:14:6853

  fmpz_mat_is_square...
  fmpz_mat_ismake[3]: *** [build/fmpz_mod_mat/test/main_TEST_RUN] Segmentation fault (core dumped)

I don't see any specific reason to why this would fail. Looks like everything else works just fine. Moreover, I feel like this test is one of the last tests that would fail. I don't see anything compiler specific that could make this fail.

But if I understand this correctly, it is Ubuntu 22 with Ubuntu 14 toolchain? Why would someone use this combination?

@fredrik-johansson
Copy link
Collaborator

76b93b4 just fixed an uninitialized variable in the fmpz_mod_mat tests which potentially might have been responsible for that segfault.

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Jan 31, 2024

if I understand this correctly, it is Ubuntu 22 with Ubuntu 14 toolchain? Why would someone use this combination?

No, this is Ubuntu 14 with a backport toolchain https://github.com/sagemath/sage/blob/develop/tox.ini#L213 from https://launchpad.net/~ubuntu-toolchain-r/+archive/ubuntu/test
It's running in a container on a GitHub runner; the OS version of that runner is not relevant.

@albinahlback
Copy link
Collaborator

No, this is Ubuntu 14 with a backport toolchain https://github.com/sagemath/sage/blob/develop/tox.ini#L213 from https://launchpad.net/~ubuntu-toolchain-r/+archive/ubuntu/test It's running in a container on a GitHub runner; the OS version of that runner is not relevant.

Thanks for the clarification!

@edgarcosta
Copy link
Member

Looks like some of the tests are still failing: https://github.com/mkoeppe/flint2/actions/workflows/ci-sage.yml

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Mar 19, 2024

Rebased on main for another run.

@edgarcosta
Copy link
Member

You need to trigger the run on your repo

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Mar 21, 2024

Thanks for the reminder, I got distra...

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Mar 21, 2024

...cted, running now at https://github.com/mkoeppe/flint2/actions/runs/8382682177

@edgarcosta
Copy link
Member

it looks like all checks have failed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants