Travis & parallelize autogenerated versionCompatibility scenarios

[samselikoff]

Is there a way to use versionCompatibility with ember try:each and have the jobs run parallel on Travis?

Mirage's current .travis.yml looks like this and I'm wondering if there's a way to leverage the autogeneration piece from Ember Try, but still get parallel test runs in Travis.

Alternatively, is there a current "best practice" around these setups? I'm seeing 2 or 3 different configs being used across popular projects.

[kategengler]

Unfortunately, there's no way to hook into Travis' parallel infrastructure and use the scenario autogeneration. The only way to use Travis' parallel infrastructure is the setup with the environment variable for the scenario, as seen in the .travis.yml generated by ember addon <name>.

--skip-cleanup with Travis stages + deploying from Travis is risky -- you may be deploying the state of your package as tested with the last run scenario.

I personally recommend ember try:each because the matrix being in multiple places drives me crazy. Unfortunately, you lose Travis parallelization, the nice dashboard, and the ability to rerun just one scenario, but the build times seem to be similar.

[samselikoff]

Thanks for all the info!

I'm surprised that the test stages can leak into the release stage... I'll remove --skip-cleanup for sure then.

the build times seem to be similar

How is this possible? Maybe I'm missing an option. I tried this yesterday, here's the PR build. My try:each run got to 48 minutes and timed out. Here's my latest master build, and the versioned tests section took 7:53 of wall time.

[kategengler]

Yeah, it's a bit tricky that the stages work that way. Alternatively, you could have the deploy stage reset the repo.

I've found that it is often faster because it doesn't need to spin up all the containers + run an extra install for every container before ember-try can even be run.

The primary cause of the very long run is:

"versionCompatibility": {
   "ember": ">=1.13"
}

If you run ember try:config, you'll see that that expands to 28 scenarios ... the latest point release of every minor version of Ember between 1.13 and now + beta, canary, release, and the default scenario.

I recommend:

"versionCompatibility": {
   "ember": "2.12 || 2.16 || 2.18 || >=3.0"
}

Yes, I recommend dropping support for versions of Ember that Ember itself no longer supports. 2.12 LTS is no longer supported after this month This will expand to 11 scenarios -- canary, beta, release, default, latest 2.8, latest 2.12, latest 2.16, latest 2.18, latest 3.0, 3.1, 3.2, 3.3 and any new minor versions as they are released.

Some other things you can do to speed things up: I recommend only running lint as a stage and not as part of each ember test. I also see you are running test:browser which translates to ember test with floating dependencies, but ember-try floats dependencies and so that's just running the tests an extra time.

[samselikoff]

I see.

I think the linting tests are super fast, right? Also, how do you run your ember test suite and ignore the autogenerated linting tests?

I think test:browser is for a fast failure – if the floating deps cause an error I'd want to know after 1 build instead of running all versioned tests. I think I got the idea from Robert. Is it a good idea?

[kategengler]

They are pretty fast, but it seems wasteful to lint for every scenario. If you do it as a first stage with a fail fast, you can avoid running any scenarios if linting does fail. To run test without lint, run `ember test --query 'nolint'`. The first scenario run will be the `default` which is floated deps, though the rest of the scenarios will run in the `ember try:each` case; in the travis parallel case it would also fail fast.

…

On Tue, Aug 14, 2018 at 5:03 PM, Sam Selikoff ***@***.***> wrote: I see. I think the linting tests are super fast, right? Also, how do you run your ember test suite and ignore the autogenerated linting tests? I think test:browser is for a fast failure – if the floating deps cause an error I'd want to know after 1 build instead of running all versioned tests. I think I got the idea from Robert. Is it a good idea? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#202 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAbHOo9ot_05IK22kyN6f8lY0gRGVydiks5uQzsHgaJpZM4V64R4> .

[rwjblue]

FYI - the issue with linting is not about speed, its about ensuring that the linting runs with the specific versions in your lockfile. When you lint as part of your ember-try config, you end up getting trolled by things like patch releases of prettier/eslint-plugin-node/etc.