Merge pull request #69 from fivetran/feature/q3-2023-updates

Feature/q3 2023 updates
fivetran · Oct 18, 2023 · af98ad9 · af98ad9
2 parents d8ddbb9 + 7a359dc
commit af98ad9
Show file tree

Hide file tree

Showing 23 changed files with 1,113 additions and 164 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,6 +1,47 @@
-# dbt_stripe v0.UPDATE.UPDATE
+# dbt_stripe v0.11.0
 
- ## Under the Hood:
+[PR #69](https://github.com/fivetran/dbt_stripe/pull/69) contains the following updates:
+
+## 🚨 Breaking Changes 🚨
+
+  - Prefixed the following fields based on their corresponding upstream source to maintain clarity:
+
+| **Previous Name**                          | **New Name**                                                                                                                                                                                                                             |
+|--------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| created_at | balance_transaction_created_at
+| available_on | balance_transaction_available_on
+| currency | balance_transaction_currency
+| amount | balance_transaction_amount
+| fee | balance_transaction_fee
+| net | balance_transaction_net
+| type | balance_transaction_type
+| source | balance_transaction_source_id
+| reporting_category | balance_transaction_reporting_category
+| description | balance_transaction_description
+
+## Updates:
+- Introduced the following new models, named after the Stripe reports that they follow. These models help reproduce reports available in the [Stripe Reporting API](https://stripe.com/docs/reports/report-types). The reports introduced in this update include:
+  - stripe__activity_itemized_2
+  - stripe__balance_change_from_activity_itemized_3
+  - stripe__ending_balance_reconciliation_itemized_4
+  - stripe__payout_itemized_3
+
+- Updated the [`stripe__balance_transactions`](https://github.com/fivetran/dbt_stripe/blob/main/models/stripe__balance_transactions.sql) with the following changes:
+  - `reporting_category` has been updated to pull directly from the titular column. If no `reporting_category` exists, it then falls to sort based on balance transaction  `type` in accordance to the Stripe [documentation](https://stripe.com/docs/reports/reporting-categories).
+  - Added the following fields:
+    - dispute fields
+    - transfer fields
+    - additional payout fields
+    - additional customer fields
+    - additional card fields
+    - additional charge fields
+    - additional invoice fields
+  - Updated `customer_facing_amount` to include for refunds and disputes as well
+  - Updated `charge_id` to charge, refund, then dispute objects consecutively
+
+
+
+## Under the Hood:
 
 - Incorporated the new `fivetran_utils.drop_schemas_automation` macro into the end of each Buildkite integration test job.
 - Updated the pull request [templates](/.github).

diff --git a/README.md b/README.md
@@ -12,7 +12,7 @@
         href="https://fivetran.com/docs/transformations/dbt/quickstart">
         <img src="https://img.shields.io/badge/Fivetran_Quickstart_Compatible%3F-yes-green.svg" /></a>
 </p>
-
+ 
 # Stripe Transformation dbt Package ([Docs](https://fivetran.github.io/dbt_stripe/))
 # 📣 What does this dbt package do?
 - Produces modeled tables that leverage Stripe data from [Fivetran's connector](https://fivetran.com/docs/applications/stripe) in the format described by [this ERD](https://fivetran.com/docs/applications/stripe#schemainformation) and build off the output of our [stripe source package](https://github.com/fivetran/dbt_stripe_source).
@@ -32,7 +32,11 @@ The following table provides a detailed list of all models materialized within t
 | [stripe__invoice_line_item_details](https://fivetran.github.io/dbt_stripe/#!/model/model.stripe.stripe__invoice_line_item_details)      | Each record represents an invoice line item, enriched with details about the associated charge, customer, subscription, and pricing data.                                                                                 
 | [stripe__daily_overview](https://fivetran.github.io/dbt_stripe/#!/model/model.stripe.stripe__daily_overview)      | Each record represents, per day, a summary of daily totals and rolling totals by transaction type (balances, payments, refunds, payouts, and other transactions). You may use this model to roll up into weekly, quarterly, monthly, and other time grains. You may also use this model to create a MRR report.                                                  |
 | [stripe__subscription_details](https://fivetran.github.io/dbt_stripe/#!/model/model.stripe.stripe__subscription_details)    | Each record represents a subscription, enriched with customer details and payment aggregations.                                                                                                                                          |
-| [stripe__customer_overview](https://fivetran.github.io/dbt_stripe/#!/model/model.stripe.stripe__customer_overview)       | Each record represents a customer, enriched with metrics about their associated transactions.  Transactions with no associated customer will have a customer description of "No associated customer".                                                                                                                                                                   |
+| [stripe__customer_overview](https://fivetran.github.io/dbt_stripe/#!/model/model.stripe.stripe__customer_overview)       | Each record represents a customer, enriched with metrics about their associated transactions.  Transactions with no associated customer will have a customer description of "No associated customer".                                                                                                                                               |
+| [stripe__activity_itemized_2](https://fivetran.github.io/dbt_stripe/#!/model/model.stripe.stripe__activity_itemized_2)       |  This report displays balance transactions alongside associated customer, charge, refund, fee, and invoice details, useful for Interchange Plus (IC+) pricing users.                                                                                                                                             |
+| [stripe__balance_change_from_activity_itemized_3](https://fivetran.github.io/dbt_stripe/#!/model/model.stripe.stripe__balance_change_from_activity_itemized_3)       | This report functions like a bank statement for reconciling your Stripe balance, especially beneficial for treating Stripe as a bank account for accounting purposes. It offers a detailed breakdown of transactions affecting your balance.                                                                                                                                               |
+| [stripe__ending_balance_reconciliation_itemized_4](https://fivetran.github.io/dbt_stripe/#!/model/model.stripe.stripe__ending_balance_reconciliation_itemized_4)       | This reports models after Stripe's payout reconciliation reports, helping match bank account payouts with related transactions. It provides details for automatic payouts and transactions that hadn't settled as of the report's end date. This report is only available for users with automatic payouts enabled and optimized for those who prefer to reconcile the transactions included in each payout as a settlement batch.                                                                                                                                               |
+| [stripe__payout_itemized_3](https://fivetran.github.io/dbt_stripe/#!/model/model.stripe.stripe__payout_itemized_3)       | This report represents payouts with information on expected arrival dates and status, akin to a bank statement for reconciling your Stripe balance, particularly useful for accounting purposes.                                                                                                                                                      |
 <!--section-end-->
 
 # 🎯 How do I use the dbt package?
@@ -57,7 +61,7 @@ Include the following stripe package version in your `packages.yml` file:
 ```yaml
 packages:
   - package: fivetran/stripe
-    version: [">=0.10.0", "<0.11.0"]
+    version: [">=0.11.0", "<0.12.0"]
 
 ```
 Do **NOT** include the `stripe_source` package in this file. The transformation package itself has a dependency on it and will install the source package as well. 
@@ -232,7 +236,7 @@ This dbt package is dependent on the following dbt packages. Please be aware tha
 ```yml
 packages:
     - package: fivetran/stripe_source
-      version: [">=0.9.0", "<0.10.0"]
+      version: [">=0.10.0", "<0.11.0"]
 
     - package: fivetran/fivetran_utils
       version: [">=0.4.0", "<0.5.0"]

diff --git a/dbt_project.yml b/dbt_project.yml
@@ -1,7 +1,7 @@
 config-version: 2
 name: 'stripe'
 
-version: '0.10.1'
+version: '0.11.0'
 require-dbt-version: [">=1.3.0", "<2.0.0"]
 models:
   stripe:
@@ -16,6 +16,7 @@ vars:
     card: "{{ ref('stg_stripe__card') }}"
     charge: "{{ ref('stg_stripe__charge') }}"
     customer: "{{ ref('stg_stripe__customer') }}"
+    dispute: "{{ ref('stg_stripe__dispute') }}"
     fee: "{{ ref('stg_stripe__fee') }}"
     invoice: "{{ ref('stg_stripe__invoice') }}"
     invoice_line_item: "{{ ref('stg_stripe__invoice_line_item') }}"
@@ -26,3 +27,4 @@ vars:
     price_plan: "{{ ref('stg_stripe__price_plan') }}"
     refund: "{{ ref('stg_stripe__refund') }}"
     subscription: "{{ ref('stg_stripe__subscription') }}"
+    transfer: "{{ ref('stg_stripe__transfer') }}"
diff --git a/docs/catalog.json b/docs/catalog.json
diff --git a/docs/index.html b/docs/index.html
diff --git a/docs/manifest.json b/docs/manifest.json
diff --git a/docs/run_results.json b/docs/run_results.json
diff --git a/integration_tests/ci/sample.profiles.yml b/integration_tests/ci/sample.profiles.yml
@@ -16,13 +16,13 @@ integration_tests:
       pass: "{{ env_var('CI_REDSHIFT_DBT_PASS') }}"
       dbname: "{{ env_var('CI_REDSHIFT_DBT_DBNAME') }}"
       port: 5439
-      schema: stripe_integrations_tests_9
+      schema: stripe_integrations_tests_10
       threads: 8
     bigquery:
       type: bigquery
       method: service-account-json
       project: 'dbt-package-testing'
-      schema: stripe_integrations_tests_9
+      schema: stripe_integrations_tests_10
       threads: 8
       keyfile_json: "{{ env_var('GCLOUD_SERVICE_KEY') | as_native }}"
     snowflake:
@@ -33,7 +33,7 @@ integration_tests:
       role: "{{ env_var('CI_SNOWFLAKE_DBT_ROLE') }}"
       database: "{{ env_var('CI_SNOWFLAKE_DBT_DATABASE') }}"
       warehouse: "{{ env_var('CI_SNOWFLAKE_DBT_WAREHOUSE') }}"
-      schema: stripe_integrations_tests_9
+      schema: stripe_integrations_tests_10
       threads: 8
     postgres:
       type: postgres
@@ -42,13 +42,13 @@ integration_tests:
       pass: "{{ env_var('CI_POSTGRES_DBT_PASS') }}"
       dbname: "{{ env_var('CI_POSTGRES_DBT_DBNAME') }}"
       port: 5432
-      schema: stripe_integrations_tests_9
+      schema: stripe_integrations_tests_10
       threads: 8
     databricks:
       catalog: null
       host: "{{ env_var('CI_DATABRICKS_DBT_HOST') }}"
       http_path: "{{ env_var('CI_DATABRICKS_DBT_HTTP_PATH') }}"
-      schema: stripe_integrations_tests_9
+      schema: stripe_integrations_tests_10
       threads: 2
       token: "{{ env_var('CI_DATABRICKS_DBT_TOKEN') }}"
       type: databricks
diff --git a/integration_tests/dbt_project.yml b/integration_tests/dbt_project.yml
@@ -1,19 +1,20 @@
 config-version: 2
 
 name: 'stripe_integration_tests'
-version: '0.10.1'
+version: '0.11.0'
 
 profile: 'integration_tests'
 
 vars:
-  stripe_schema: stripe_integrations_tests_9
+  stripe_schema: stripe_integrations_tests_10
   stripe_source:
     stripe_account_identifier:                "account_data"
     stripe_group_identifier:                  "group_data"
     stripe_balance_transaction_identifier:    "balance_transaction_data"
     stripe_card_identifier:                   "card_data"
     stripe_charge_identifier:                 "charge_data"
     stripe_customer_identifier:               "customer_data"
+    stripe_dispute_identifier:                "dispute_data"
     stripe_fee_identifier:                    "fee_data"
     stripe_invoice_identifier:                "invoice_data"
     stripe_invoice_line_item_identifier:      "invoice_line_item_data"
@@ -26,6 +27,7 @@ vars:
     stripe_refund_identifier:                 "refund_data"
     stripe_subscription_history_identifier:   "subscription_history_data"
     stripe_subscription_identifier:           "subscription_data"
+    stripe_transfer_identifier:               "transfer_data"
     stripe_credit_note_identifier:            "credit_note_data"
     stripe_credit_note_line_item_identifier:  "credit_note_line_item_data"
 
@@ -51,6 +53,10 @@ seeds:
     customer_data:
       +column_types:
         description: "{{ 'varchar(500)' if target.type in ('redshift','postgres') else 'string'}}"
+    payout_data:
+      +column_types:
+        destination_bank_account_id: "{{ 'varchar(500)' if target.type in ('redshift','postgres') else 'string'}}"
+        destination_card_id: "{{ 'varchar(500)' if target.type in ('redshift','postgres') else 'string'}}"
 
 dispatch:
   - macro_namespace: dbt_utils

diff --git a/integration_tests/requirements.txt b/integration_tests/requirements.txt
@@ -1,7 +1,9 @@
 dbt-snowflake>=1.3.0,<2.0.0
 dbt-bigquery>=1.3.0,<2.0.0
-dbt-redshift>=1.3.0,<1.5.0
+dbt-redshift>=1.3.0,<2.0.0
 dbt-postgres>=1.3.0,<2.0.0
 dbt-spark>=1.3.0,<2.0.0
 dbt-spark[PyHive]>=1.3.0,<2.0.0
 dbt-databricks>=1.3.0,<2.0.0
+
+oscrypto @ git+https://github.com/wbond/oscrypto.git@d5f3437
diff --git a/integration_tests/seeds/dispute_data.csv b/integration_tests/seeds/dispute_data.csv
@@ -0,0 +1,11 @@
+id,_fivetran_synced,amount,balance_transaction,charge_id,connected_account_id,created,currency,evidence_access_activity_log,evidence_billing_address,evidence_cancellation_policy,evidence_cancellation_policy_disclosure,evidence_cancellation_rebuttal,evidence_customer_communication,evidence_customer_email_address,evidence_customer_name,evidence_customer_purchase_ip,evidence_customer_signature,evidence_details_due_by,evidence_details_has_evidence,evidence_details_past_due,evidence_details_submission_count,evidence_duplicate_charge_documentation,evidence_duplicate_charge_explanation,evidence_duplicate_charge_id,evidence_product_description,evidence_receipt,evidence_refund_policy,evidence_refund_policy_disclosure,evidence_refund_refusal_explanation,evidence_service_date,evidence_service_documentation,evidence_shipping_address,evidence_shipping_carrier,evidence_shipping_date,evidence_shipping_documentation,evidence_shipping_tracking_number,evidence_uncategorized_file,evidence_uncategorized_text,is_charge_refundable,livemode,metadata,reason,status
+du_0MmTsrig6peNXzgD4bsS3XWF,2023-06-02 08:23:57,1234,txn_0MmTt4ig6peNXzgDDEp8wsKh,ch_2MZdKRig6peNXzgD0EygIk7l,,2023-03-17 03:23:13,usd,,,,,,,,,,,2023-04-26 23:59:59.000000 UTC,true,false,1,,,,,,,,,,,,,,,,,,true,true,"""{""netsuite_credit_memo_id"":""3324234"",""netsuite_credit_memo_link"":""https://en.wikipedia.org/wiki/Brave_New_World"",""netsuite_customer_refund_id"":""3524872"",""netsuite_customer_refund_link"":""https://en.wikipedia.org/wiki/Aldous_Huxley""}""",product_unacceptable,won
+du_0Lvivfig6peNXzgD33SIekHq,2022-12-21 20:19:05,3333,txn_0LviwVig6peNXzgDkSPS1Y1C,ch_2LqbkJig6peNXzgD1jG4Xy7z,,2022-10-22 14:44:03,usd,,,,,,,,,,,2022-11-17 23:59:59.000000 UTC,false,false,1,,,,,,,,,,,,,,,,,,true,true,,general,won
+du_0KHowkig6peNXzgDyhndDHOe,2022-09-23 00:00:24,3331,txn_0KHoxcig6peNXzgDKWarDbF0,ch_2JtdoJig6peNXzgD1lCAvXwG,,2022-01-14 12:31:58,usd,,"1600 Pennsylvania Avenue NW, Washington, DC 20500",,,,,[email protected],Aldous Huxley,68.1.21.555,,2022-02-09 23:59:59.000000 UTC,false,false,1,,,,,file_0KHowoig6peNXzgDyeDmylk3,,,,,,,,,,,,,true,true,,general,won
+du_0M9SJTig6peNXzgD58OLM8Mz,2022-12-12 20:19:03,1099,txn_0M9SKaig6peNXzgDlb5cqlBg,ch_2M2d6mig6peNXzgD03nrNVYZ,,2022-11-29 11:49:23,usd,,,,,,,,,,,2022-12-25 23:59:59.000000 UTC,false,false,0,,,,,,,,,,,,,,,,,,false,true,,subscription_canceled,lost
+du_0LuGYgig6peNXzgDM8lPBTFC,2022-11-14 14:21:25,1222,txn_0LuGZyig6peNXzgDLgWzetHo,ch_2LUt28ig6peNXzgD0RdFu0qW,,2022-10-18 14:14:18,usd,,,,,,,,,,,2022-11-13 23:59:59.000000 UTC,false,false,0,,,,,,,,,,,,,,,,,,false,true,,subscription_canceled,lost
+du_0LyvFuig6peNXzgDyBMCjDi6,2022-11-27 14:20:24,221244,txn_0LyvHhig6peNXzgDg5LanDAq,ch_2Lqdeuig6peNXzgD0sA3iBuH,,2022-10-31 10:30:10,usd,,,,,,,,,,,2022-11-26 23:59:59.000000 UTC,false,false,0,,,,,,,,,,,,,,,,,,false,true,,subscription_canceled,lost
+du_0LiAl6ig6peNXzgDEejs24CE,2022-11-13 20:20:09,432532,txn_0Lumheig6peNXzgDSp1KIXph,ch_2LVoJuig6peNXzgD1DvyohjD,,2022-09-15 05:37:08,usd,,,,,,,,,,,2022-11-12 23:59:59.000000 UTC,false,false,0,,,,,,,,,,,,,,,,,,false,true,,general,lost
+du_0M9ScVig6peNXzgDkfd7tUNb,2022-12-12 20:19:03,1400,txn_0M9SdPig6peNXzgDgpb4aJDQ,ch_2LqcRzig6peNXzgD06FImTqU,,2022-11-29 12:09:03,usd,,,,,,,,,,,2022-12-25 23:59:59.000000 UTC,false,false,0,,,,,,,,,,,,,,,,,,false,true,,subscription_canceled,lost
+du_0LuGd7ig6peNXzgDUxXx85wF,2022-11-14 14:21:25,1400,txn_0LuGe5ig6peNXzgD7PMfTzJ6,ch_2LfkHjig6peNXzgD1e9ltqQt,,2022-10-18 14:18:53,usd,,,,,,,,,,,2022-11-13 23:59:59.000000 UTC,false,false,0,,,,,,,,,,,,,,,,,,false,true,,subscription_canceled,lost
+du_0LuGbbig6peNXzgDjHA4RfoT,2022-11-14 14:21:25,700,txn_0LuGcrig6peNXzgDKP5y95T9,ch_2LLwgKig6peNXzgD0Sr5X6kL,,2022-10-18 14:17:19,usd,,,,,,,,,,,2022-11-13 23:59:59.000000 UTC,false,false,0,,,,,,,,,,,,,,,,,,false,true,,subscription_canceled,lost
diff --git a/integration_tests/seeds/transfer_data.csv b/integration_tests/seeds/transfer_data.csv
@@ -0,0 +1,11 @@
+id,_fivetran_synced,amount,amount_reversed,balance_transaction_id,created,currency,description,destination,destination_payment,destination_payment_id,livemode,metadata,reversed,source_transaction,source_transaction_id,source_type,transfer_group
+po_0Njtvzig6peNXzgD6RPBasht,2023-08-29 02:19:08,1447318,0,txn_0Njtw0ig6peNXzgDXY3mJGwC,2023-08-28 01:08:03,usd,STRIPE PAYOUT,ba_0MkB7Nig6peNXzgDBFL0SIgk,,,true,"""{""netsuite_deposit_id"":""3975560"",""netsuite_deposit_link"":""https://en.wikipedia.org/wiki/Beyonc%C3%A9""}""",false,,,card,
+po_0NjXKxig6peNXzgDgn8MqQuZ,2023-08-28 08:20:02,2660278,0,txn_0NjXKyig6peNXzgDbmNVE84K,2023-08-27 01:00:19,usd,STRIPE PAYOUT,ba_0MkB7Nig6peNXzgDBFL0SIgk,,,true,,false,,,card,
+po_0NkGNTig6peNXzgDJWfkSimX,2023-08-30 02:19:14,760313,0,txn_0NkGNTig6peNXzgDTsXXisyE,2023-08-29 01:05:55,usd,STRIPE PAYOUT,ba_0MkB7Nig6peNXzgDBFL0SIgk,,,true,,false,,,card,
+po_0NkcrHig6peNXzgDwoDFU4Yi,2023-08-30 02:19:14,1944407,0,txn_0NkcrHig6peNXzgD7eFO2jc7,2023-08-30 01:06:11,usd,STRIPE PAYOUT,ba_0MkB7Nig6peNXzgDBFL0SIgk,,,true,,false,,,card,
+po_0Mbvvvig6peNXzgDNH6Wf3ha,2023-02-17 01:15:32,7251977,0,txn_0Mbvvvig6peNXzgDJAGJCdpN,2023-02-16 01:06:47,usd,STRIPE PAYOUT,ba_1S2vcDBpbM03IU,,,true,,false,,,card,
+po_0LunJvig6peNXzgD41QBrw1m,2022-10-21 08:21:56,-1426166,0,txn_0LunJwig6peNXzgDCnBflWPl,2022-10-20 01:13:15,usd,STRIPE PAYOUT,ba_1S2vcDBpbM03IU,,,true,,false,,,card,
+po_0MRQ1gig6peNXzgD138jhQS7,2023-01-19 02:19:35,7332491,0,txn_0MRQ1gig6peNXzgDchKTwIPQ,2023-01-18 01:01:16,usd,STRIPE PAYOUT,ba_1S2vcDBpbM03IU,,,true,,false,,,card,
+po_0MRmW1ig6peNXzgDUPQSnmYw,2023-01-20 02:19:18,6108653,0,txn_0MRmW1ig6peNXzgDFqR0yaFa,2023-01-19 01:02:05,usd,STRIPE PAYOUT,ba_1S2vcDBpbM03IU,,,true,,false,,,card,
+po_0MJ5C6ig6peNXzgDyic22YlD,2022-12-27 02:19:05,3456033,0,txn_0MJ5C7ig6peNXzgDfWTszRUw,2022-12-26 01:09:34,usd,STRIPE PAYOUT,ba_1S2vcDBpbM03IU,,,true,,false,,,card,
+po_0MDe3Sig6peNXzgDJHinDGmA,2022-12-12 02:18:56,3330213,0,txn_0MDe3Tig6peNXzgDSyxtRpRH,2022-12-11 01:10:10,usd,STRIPE PAYOUT,ba_1S2vcDBpbM03IU,,,true,,false,,,card,
diff --git a/models/docs.md b/models/docs.md
@@ -1,2 +1,2 @@
 {% docs source_relation -%} The source where this data was pulled from. If you are making use of the `union_schemas` variable, this will be the source schema. If you are making use of the `union_databases` variable, this will be the source database. If you are not unioning together multiple sources, this will be an empty string.
-{%- enddocs %}
+{%- enddocs %}