Add support for custom containers while using dataproc serverless #650
Conversation
* allow custom containers for dataproc serverless jobs
|
This would very valuable for us in making dbt python models feasible in our stack. Are there any unmet requirements for this change to be reviewed? Anything I could contribute to help unblock this? |
|
@Fleid @nathaniel-may Just bumping this since there seems to be interest in this enhancement beyond myself |
|
+1 |
1 similar comment
|
+1 |
|
+1 Very eager. Implementation could be very similar to the new dataproc_batch-config? |
|
Is there any progress on this? This is something my team would really benefit from. What is still needed for this? Can I help somehow? |
|
This PR has been marked as Stale because it has been open with no activity as of late. If you would like the PR to remain open, please comment on the PR or else it will be closed in 7 days. |
|
Although we are closing this PR as stale, it can still be reopened to continue development. Just add a comment to notify the maintainers. |
resolves #642
Description
The current implementation only supports the standard spark runtime provided by dataproc serverless which does not allow installation of additional dependencies. The custom container can be defined in the runtime_config that is passed to the dataproc call. It expects an image that exists in Container Registry or Artifact Registry with a naming format like {hostname}/{project-id}/{image}:{tag}.
Checklist
changie newto create a changelog entry