Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Write microbatch compiled + run code to separate target files #10743

Open
wants to merge 4 commits into
base: main
Choose a base branch
from

Conversation

MichelleArk
Copy link
Contributor

@MichelleArk MichelleArk commented Sep 19, 2024

Resolves #10714

Problem

During execution of microbatch models, each batch is re-compiled with batch-level time filters. Currently, this results in the compiled/run .sql files becoming clobbered on each write when there should be one file per compiled/run batch.

Solution

Pass an additional argument, split_suffix (naming suggestions welcome!) that splits the write path by the suffix and writes to a top-level directory without the suffix.

Checklist

  • I have read the contributing guide and understand what's expected of me.
  • I have run this code in development, and it appears to resolve the stated issue.
  • This PR includes tests, or tests are not required or relevant for this PR.
  • This PR has no interface changes (e.g., macros, CLI, logs, JSON artifacts, config files, adapter interface, etc.) or this PR has already received feedback and approval from Product or DX.
  • This PR includes type annotations for new and modified functions.

@cla-bot cla-bot bot added the cla:yes label Sep 19, 2024
Copy link
Contributor

Thank you for your pull request! We could not find a changelog entry for this change. For details on how to document a change, see the contributing guide.

This comment was marked as outdated.

@MichelleArk MichelleArk changed the title first pass: split_suffix Write microbatch compiled + run code to separate target files Sep 20, 2024
@MichelleArk MichelleArk marked this pull request as ready for review September 20, 2024 16:27
@MichelleArk MichelleArk requested a review from a team as a code owner September 20, 2024 16:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Write compiled code to one file per batch during execution of microbatch models
1 participant