Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] Add ability to import/include YAML from other files #9695

Open
3 tasks done
b-per opened this issue Feb 28, 2024 · 6 comments · May be fixed by #10694
Open
3 tasks done

[Feature] Add ability to import/include YAML from other files #9695

b-per opened this issue Feb 28, 2024 · 6 comments · May be fixed by #10694
Labels
enhancement New feature or request paper_cut A small change that impacts lots of users in their day-to-day Refinement Maintainer input needed

Comments

@b-per
Copy link
Contributor

b-per commented Feb 28, 2024

Is this your first time submitting a feature request?

  • I have read the expectations for open source contributors
  • I have searched the existing issues, and I could not find an existing issue for this feature
  • I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

Describe the feature

I wonder if dbt would benefit from providing the ability to include YAML snippets from other files.

Home Assistant (a popular python tool, leveraging YAML for config), provides a few additional YAML constructors (quick overview of constructors).

The most interesting one would be !include which allows to define values of keys in YAML from other YAML files
The implementation of Home Assistant is in this source file.

While we can define some logic in Jinja in some YAML files, we are limited today to defining a single entry at a time (e.g. a string or a bool). With this approach we can also store config for nested fields.

Based on my first observations, this technique wouldn't mess with the current dbt code as the YAML is rendered before we start any "real" dbt parsing.

Describe alternatives you've considered

Copy pasting the same logic in YAML as we do today

Who will this benefit?

  • people who want to DRY their YAML files (e.g common tests for columns, common tags etc...)
  • people who want to define the config of dbt_project.yml from different files, with different code owners in git
  • people following this issue

Are you interested in contributing this feature?

Yes

Anything else?

I got an early prototype working locally and can share some of the code if we want to implement this feature.

@b-per b-per added enhancement New feature or request triage labels Feb 28, 2024
@graciegoheen graciegoheen added the paper_cut A small change that impacts lots of users in their day-to-day label Mar 25, 2024
@dbeatty10 dbeatty10 added Refinement Maintainer input needed and removed triage labels Apr 5, 2024
@databius
Copy link

Thanks @b-per for opening this feature. We really need it to make dbt project be DRY.

@b-per
Copy link
Contributor Author

b-per commented Aug 29, 2024

Hi team. Is it something that we'd be keen to merge if I write a PR for it?

@will-sargent-dbtlabs
Copy link

+1 for this feature

@alison985
Copy link

I've been talking about this under the keywords "YAML inheritance" for at least a year. We desperately need it, IMO. I wouldn't be able to contribute to a PR, but I'm 100% on board for anything else to support it.

@IL-Minh
Copy link

IL-Minh commented Sep 11, 2024

  • 1 for this feature

@b-per b-per linked a pull request Sep 11, 2024 that will close this issue
8 tasks
@b-per
Copy link
Contributor Author

b-per commented Sep 11, 2024

As a few people have shown interest, I created a draft PR with the code I have so far. It is not ready to be merged, but some people might be willing to take it over from here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request paper_cut A small change that impacts lots of users in their day-to-day Refinement Maintainer input needed
Projects
None yet
Development

Successfully merging a pull request may close this issue.

7 participants