Put on conda-forge

[CJ-Wright]

Would it be possible to put all of this on conda-forge?

[pavoljuhas]

@CJ-Wright - that would be surely useful. Can you give it a start by trying to conda-build the whole diffpy-cmi suite using only the dependencies from the defaults and conda-forge channels? (i.e., no packages from the diffpy channel)

When I originally worked on conda recipies, the "boost" and "gsl" packages were either unavailable or broken in the defaults channel. I had to build those packages using recipes in diffpy/conda-recipes and they are available in the diffpy channel of Anaconda packages. It is likely the "boost" and "gsl" packages are now fixed in either of default or conda-forge channels.

The sequence to build the entire diffpy-cmi suite should be as follows:

• conda-recipes/pycifrw
• diffpy.structure/conda-recipe
• diffpy.utils/conda-recipe
• conda-recipes/periodictable
• libobjcryst/conda-recipe
• pyobjcryst/conda-recipe
• libdiffpy/conda-recipe
• diffpy.srreal/conda-recipe
• conda-recipes/srfit-sasview
• diffpy.srfit/conda-recipe

Let me know if you run into a problem building any of those.

[CJ-Wright]

Is there any way to contact the maintainer of pycifrw? It might be nice to have some CI and coverage on such a critical library.
@sbillinge thoughts?

[CJ-Wright]

conda-forge/staged-recipes#2544

[pavoljuhas]

Is there any way to contact the maintainer of pycifrw? It might be nice to have some CI and coverage on such a critical library.

@CJ-Wright, @sbillinge - I am in touch with James Hester and submitted a few PRs for pycifrw. He has recently asked me to check the merge of py2/py3 versions. I will do so and let him know about the conda-forge effort.

I am not sure if CI and especially coverage would be straightforward. PyCifRW sources are in "noweb" format which mixes source files with documentation. The build process first extracts ".py" files from the ".nw" sources and then uses them to produce the actual package.

We could suggest James to convert the library to standard ".py" sources with rst documentation, I am however not sure if we have time and resources to help him doing so.

[CJ-Wright]

At the very least we can build and test on a CI right?
I agree that we should suggest the shift over to a standard python library.

[pavoljuhas]

The main repo is on bitbucket (https://bitbucket.org/jamesrhester/pycifrw), which travis does not seem to support #667. Perhaps there are other alternatives that work there - would you mind to check what are the CI options for bitbucket? Another possibility is to mirror - with permission - on GitHub and setup the CI testing there.

[sbillinge]

If we are in touch with James Hester, then it is probably less work to seek permission to mirror on GitHub?

[pavoljuhas]

Perhaps we should refocus to the initial question - @CJ-Wright - did you have a chance to test out a full build sequence of conda packages for diffpy-cmi? I wonder if we can switch to the standard/conda-forge versions of the boost and gsl packages or if we need to continue building our own. The gsl and boost are required for libdiffpy, pyobjcryst and srreal, so these builds will show if standard versions work.

[CJ-Wright]

I don't know exactly what you mean, but using the source install we could change over the deps to conda-forge. I can't start building diffpy.structure until pycifrw is ready to go.

[pavoljuhas]

I meant to build all conda packages for CMI locally, install them with the --use-local option, and finally test them by running the run_test.py script. There is no need to put up anything on conda-forge yet. The full CMI suite requires the "gsl" and "boost" packages and I want to find out if they work. If you get a build failure with "gsl" and "boost" from conda-forge, we need to either fix those recipes or continue using our own (gsl and boost) versions from the diffpy channel. Let's do a Google Hangout if this is still confusing.

[pavoljuhas]

@CJ-Wright - any new progress with testing package build sequence locally w/r to conda-forge?
Can you assign to someone else in the group if you don't have time for that?

[CJ-Wright]

Sorry for the delay, I don't have a ton of time but I can try to get all the PRs in today. I usually test all the builds locally individually before sending them up to conda-forge. The tests will fail if there is a problem with the C++ libraries, right?

[pavoljuhas]

@CJ-Wright - Right - the idea is to test if a full package stack would work with gsl and boost dependencies from conda-forge. Perhaps it is better to reassign to someone else in the group and get more people experienced with building conda packages.

@dragonyanglong, @gzz0707, @justcalamari, @chiahaoliu - any volunteer to test a build of diffpy-cmi package stack w/r to conda-forge libraries?

[pavoljuhas]

I noticed the toolchain package for configuring conda-forge builds bumped up minimum Mac OS X version to 10.9 conda-forge/toolchain-feedstock@7a470c5. The stock conda-build uses 10.7.

Does anyone have access to a Mac with earlier OS, e.g., Mountain Lion 10.8?
If so - can you test if conda-forge python runs there?

conda create --name=cf27 --channel=conda-forge python=2.7
source activate cf27
# does this run or crash?
python

[CJ-Wright]

see conda-forge/staged-recipes#2669

[CJ-Wright]

If anyone else is interested in doing the packaging, they are more than welcome to it. I have posted all the package PRs in this issue, and mark off the ones that are accepted/exist.

[pavoljuhas]

see conda-forge/staged-recipes#2669

@CJ-Wright - I feel we should do a complete test of diffpy-cmi stack locally first and then decide if it is worth adding recipes to conda-forge.

@sbillinge - chances are conda-forge packages do not work on slightly older Mac-s which are still supported by stock Anaconda. Do we want to cut those users out by moving our packages to conda-forge or should we continue building packages on our own that work for them?

Also - distributing our packages from the diffpy channel makes it easier to see their download statistics. This would get more tricky if they are available at 2 channels (conda-forge and diffpy).