Create report containing information about packages installed

[fridex]

Is your feature request related to a problem? Please describe.

micropipenv provides easy to use tool to install application dependencies. It might be an interesting feature to let users know more about the installation process and provenance of installed packages. Consider a use-case when a user wants to know information about packages installed into the environment. The user can trigger:

pip list

This pip's package listing shows installed packages - only their names and versions. Sadly, the output does not capture additional information, such as which index was used to install packages or which wheel file was chosen.

Describe the solution you'd like

Provide --report option in micropipenv that would produce a report out of the installation process. This report can capture:

• which distribution was used (sdist/wheel)
• which index was used to obtain the package
• artifact hash used (even if users use lock files, lock files state multiple hashes matching multiple artifacts and the installation procedure picks the one matching environment)
• additional information about the artifact (such as size, full url following redirects, ...)
• environment information (ex. python interpreter version, markers evaluated, ...)

Additional context

The very first implementation could try to parse pip logs. Some instrumentation or monkey patching of pip might be needed to obtain relevant information.

[fridex]

CC @frenzymadness does this sound interesting feature to be implemented in micropipenv? It might be interesting to keep the report even in python s2i after the build to have a mechanism to introspect what exact packages are present after the image is built.

[frenzymadness]

Should this work during the installation or ex-post? I honestly think that micropipenv is verbose enough and you can always increase pip verbosity via an environment variable so I'm not sure I'd find a use case for such a feature. Moreover, with hashes in a dependency spec, you should be able to re-run the same installation and track down whatever you need to know. Have you ever been in need of such a feature?

[fridex]

The feature might be interesting to track down what the installation process brought to the environment. The lockfile states package versions (optionally index URL in case of Pipenv lockfiles) but it lists multiple hashes per package version. If one or more artifacts go missing, it might be hard to track down which artifacts were actually picked by pip and used during the installation process (similarly if more specific builds are added, not limited to manylinux standards). This additionally plays a role when one wants to snapshot used artifacts or things like SBOM. I agree that the pip verbosity can be increased, but having a report which artifacts (package name, package version, index url, artifact) really went to the installation process might be interesting to capture and keep track of.

[fridex]

BTW this might be interesting to discuss with pip upstream and see their vision in this area.

EDIT: A link to the discussion.
EDIT: related: pypa/pip-audit#170

[goern]

/kind feature
/priority important-soon

[sesheta]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[harshad16]

/lifecycle frozen

[fridex]

This resulted in PEP-710. Please feel free to participate in the discussion if you find this feature valuable. Thank you!

[frenzymadness]

Thanks for the info. So, the goal is to implement some kind of report functionality into micropipenv when the pip starts to provide files proposed in the PEP we can base the report on. Correct?

[fridex]

Thanks for the info. So, the goal is to implement some kind of report functionality into micropipenv when the pip starts to provide files proposed in the PEP we can base the report on. Correct?

As micropipenv uses pip, the provenance_url.json file would be automatically created when installing packages. Consumers can automatically take the file when needed. I don't think there is anything needed in micropipenv if the PEP is accepted.

[fridex]

Consumers can automatically take the file when needed.

If micropipenv upstream decides to be the consumer of this file, it can provide some kind of report subcomand. IMHO, it might be a good idea not to introduce such functionality into micropipenv. micropipenv could be kept lightweight and other tools can be used for this functionality (such as pip-preserve or others).