Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Retry dataproc transient errors #1275

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Retry dataproc transient errors #1275

wants to merge 3 commits into from

Commits on Jul 8, 2024

  1. Pass Retry object to retry transient errors

    @OSalama
    OSalama committed Jul 8, 2024
    Configuration menu
    Copy the full SHA
    fab7f11 View commit details
    Browse the repository at this point in the history

  2. Increase wait time between polls when not running 

    Dataproc serverless batch jobs take at least 60 seconds to
start up, which leads to a lot of meaningless polling, and
increases the chance of getting a transient error.
In the absolute worst case this will add 9 seconds to
a model's runtime, but I would be very surprised if
people are using Python models to do less than
10 seconds of processing, and doubt there will be
any real world impact by this change
    @OSalama
    OSalama committed Jul 8, 2024
    Configuration menu
    Copy the full SHA
    35e2fac View commit details
    Browse the repository at this point in the history

  3. Add changelog entry

    @OSalama
    OSalama committed Jul 8, 2024
    Configuration menu
    Copy the full SHA
    869ce02 View commit details
    Browse the repository at this point in the history