Skip to content

Commit

Permalink
Increase wait time between polls when not running
Browse files Browse the repository at this point in the history
Dataproc serverless batch jobs take at least 60 seconds to
start up, which leads to a lot of meaningless polling, and
increases the chance of getting a transient error.
In the absolute worst case this will add 9 seconds to
a model's runtime, but I would be very surprised if
people are using Python models to do less than
10 seconds of processing, and doubt there will be
any real world impact by this change
  • Loading branch information
Omar Salama committed Jul 8, 2024
1 parent 48aa068 commit 4070539
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion dbt/adapters/bigquery/dataproc/batch.py
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ def poll_batch_job(
response = None
run_time = 0
while state in _BATCH_RUNNING_STATES and run_time < timeout:
time.sleep(1)
time.sleep(1) if state == Batch.State.RUNNING else time.sleep(10)
response = job_client.get_batch( # type: ignore
request=GetBatchRequest(name=batch_name), # type: ignore
retry=Retry()
Expand Down

0 comments on commit 4070539

Please sign in to comment.