Many fields in the table: jira__daily_issue_field_history

[RivkiHofman]

Hi

We use with the variable: issue_field_history_columns to hold the fields we need for the model.
For each field the history is stored in the history table: jira__daily_issue_field_history.
when we add fields and need to refresh the model full, it takes a few hours.
We have tens or hundreds of fields, it is very very heavy and every period we have to increase the warehose for this and the Snowflake's costs increase significantly.

It's quite unnecessary because in practice we don't need the history for the most of fields, from our point of view if the jira__issue_enhanced table contained the current state, the history table is unnecessary for the most of fields.

But in practice it seems that the jira__issue_enhanced table is based on the history table and brings the last record from it.

Is it possible to save resources and make the jira__issue_enhanced table or another table to bring us the latest information without having to save the history for all the fields we want to use?

Thanks for your answer and support.

Rivki

[fivetran-joemarkiewicz]

Hey @RivkiHofman thanks for posting here and sharing your experience and desired outcome. To make sure I am understanding your request:

You are wondering if there is a way we can decouple the issue_field_history_columns variable to allow the user to designate if they want the fields to have their history tracked (mainly for the jira__daily_issue_field_history model) or if they just want the latest record (mainly for the jira__issue_enhanced). This would allow you to include the thousands of fields in the variable, but not have to track the history if you do not care to track the history and just have the latest record show up in the jira__issue_enhanced model. This is mainly because you don't necessarily care to track the history for all the fields. However, the current build of the package forces the history to be tracked regardless if you want. This results in extra compute that you may not necessarily care to do if you are not concerned about the history of these additional fields.

Let me know if that is in alignment with what you are requesting. If that is, I believe this is a very reasonable feature request and one that I feel would help a lot of other users of this package to save on compute costs. I do want to note that this would likely be a larger update and may take more time to implement once accepted than our usual updates to this package.

[RivkiHofman]

Thanks for your answer @fivetran-joemarkiewicz .

That's exactly what I meant.
I understand that it is not easy to implement, but it is very important to us, we use a huge WAREHOUSE just because of this.
This jumps the increases for us by a margin, compared to other models.

I would appreciate an estimate of when this might happen so we can manage our budget.

[RivkiHofman]

Hi @fivetran-joemarkiewicz

Can we get an estimate for the development?
We are really looking forward to it.

Thanks.

[shirlil]

following

[elanfivetran]

Hey @RivkiHofman 👋,

We really do appreciate your continued engagement with the Jira package. I wanted to give an update that we’ve scoped this out as a feature we would like to add, however unfortunately I cannot give an exact date of when we expect to implement this as of right now. Our priority for new feature requests is determined by the impact, level of effort, and number of customers interested in the feature (meaning that if other customers are interested in this feature you should comment so below). 


That being said, we are an open source community and do encourage contributions to our packages! If you are interested, you are more than welcome to sign up for our office hours and we will help guide you on how to implement this feature improvement.