Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add simple download progress option #12586

Merged
merged 9 commits into from
Mar 27, 2024
Merged

Conversation

arenasys
Copy link
Contributor

Closes #11508, continuation of #12084.

GUI software using pip cant show download progress to the user since rich progress bars dont work through pipes, making large downloads quite unfriendly for users. These progress reports are the only information thats missing from pip outputs when running through pipes.

This PR adds a simple progress_bar option called 'raw', which outputs reports to stdout:

Downloading PyQt5-5.15.10-cp37-abi3-manylinux_2_17_x86_64.whl (8.2 MB)
Progress 0 of 8239447
Progress 10240 of 8239447
Progress 583680 of 8239447
Progress 1351680 of 8239447
Progress 2222080 of 8239447
...
Progress 8239447 of 8239447

These are rate limited to reduce spam.

@arenasys
Copy link
Contributor Author

With just these reports something like tqdm can wrap pip's downloads easily (and more importantly GUI's can).

import sys
import subprocess
import tqdm

args = [sys.executable, '-m', 'pip', 'install', '-I', '--no-cache-dir', 'pyqt5', '--progress-bar=raw']
process = subprocess.Popen(args, stdout=subprocess.PIPE, text=True)

bar, file, last = None, None, None
while line := process.stdout.readline():
    if line.startswith("Downloading"):
        file = line.split(' ', maxsplit=2)[1]
        if type(bar) != type(None):
            bar.close()
    if line.startswith("Progress"):
        _, current, _, total = line.split(' ')
        current, total = int(current), int(total)

        if current == 0:
            bar = tqdm.tqdm(desc=file, total=total, unit='iB', unit_scale=True, unit_divisor=1024)
            last = 0

        bar.update(current - last)
        last = current
PyQt5-5.15.10-cp37-abi3-manylinux_2_17_x86_64.whl: 100%|███████████████████████| 7.86M/7.86M [00:02<00:00, 3.47MiB/s]
PyQt5_Qt5-5.15.2-py3-none-manylinux2014_x86_64.whl: 100%|██████████████████████| 57.1M/57.1M [00:21<00:00, 2.77MiB/s]
PyQt5_sip-12.13.0-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.whl: 100%|███| 339k/339k [00:01<00:00, 251kiB/s]

markmc and others added 5 commits March 26, 2024 10:24
Fixes pypa#12577

This looks like it was an oversight in pypa#9450 - we should pass the
correct verbosity level to build env install subprocesses.

Tested with:

```
rm -rf ~/.cache/pip && rm -f *.whl && pip wheel --no-binary :all: hatchling
```

and all three verbosity levels, before and after this change, giving
the following logs:

```
     33 patched-verbosity0.log
   2549 patched-verbosity1.log
  11938 patched-verbosity2.log
     33 unpatched-verbosity0.log
     99 unpatched-verbosity1.log
   1030 unpatched-verbosity2.log
```

i.e. currently a lot of useful logs are being dropped from these install
subprocesess even with -vvv
* Specifically mention the --index-url and --extra-index-url both provide
  locations for pip to look for packages in the 'Finding Packages' section.
  However, explicitly state that there is no priority ordering for the search
  locations.
* This is a recurring point of confusion on the pip GitHub issue tracker as well
  as on https://github.com/pypa/packaging-problems/issues/, so additional
  clarification could help here.

Co-authored-by: Henry Schreiner <[email protected]>
Copy link
Member

@uranusjr uranusjr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure about naming this “raw” but I don’t have an immediate suggestion either.

@arenasys
Copy link
Contributor Author

Can guess from the name alone that its not for regular users and its probably for a more technical purpose, its literally the raw progress without any additional information. Name it 'simple' and users may expect something different. That's my reasoning anyway.

@uranusjr
Copy link
Member

Yeah I’m definitely not advocating for simple. Was more thinking about more like machine.

@arenasys
Copy link
Contributor Author

Could do. But none of Pip's outputs are guaranteed to be machine readable or to remain consistent, its all for humans to read. May give the wrong idea.

@joeyballentine
Copy link

I agree with using "raw", I think it makes the most sense for what it does.

@uranusjr uranusjr merged commit 4f3a184 into pypa:main Mar 27, 2024
24 checks passed
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Apr 11, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Feature Request] Machine-readable progress outputs
5 participants