Add `kong2tf` command #1391

mheap · 2024-09-08T20:12:54Z

Adds the kong2tf command to convert a deck file to Terraform.

If --generate-imports-for-control-plane-id is provided, import blocks will be added to Terraform to adopt existing resources.

If --ignore-credential-changes is added, any credentials will be ignored until they are destroyed and recreated.

Tested with the following deck file which contains one of every entity, including plugin scopes.

_format_version: "3.0"
_konnect:
  control_plane_name: CP With Everything
ca_certificates:
  - cert: |-
      -----BEGIN CERTIFICATE-----
      MIIBwDCCAUagAwIBAgIUMDCK/a+EH6a9PvQeyFwHDALYrx0wCgYIKoZIzj0EAwIw
      FzEVMBMGA1UEAwwMa29uZ190ZXN0X2NhMB4XDTI0MDkwODE3NDUxMFoXDTI3MDkw
      ODE3NDUxMFowFzEVMBMGA1UEAwwMa29uZ190ZXN0X2NhMHYwEAYHKoZIzj0CAQYF
      K4EEACIDYgAEYEP63cMf47Xf6QcH9QT8ehjxBAbfF68PdCXOPgKZdSFvRYs+JZ+Y
      dIhWV4M06WiEP8Exn352xuzgZc4mVT9B6SpAsp+hsyJ9ZkuQkyb0lC1XodP5+ZlF
      yP0dZHkPYQ5Po1MwUTAdBgNVHQ4EFgQUYS5tJ4WfzVaNsB2oYrJOEf2WJ/UwHwYD
      VR0jBBgwFoAUYS5tJ4WfzVaNsB2oYrJOEf2WJ/UwDwYDVR0TAQH/BAUwAwEB/zAK
      BggqhkjOPQQDAgNoADBlAjBCIF9CB1CuRkdRpdjHq/x2G1TJkj622L80+Emvkctw
      IdyXs8Vp2gHrLjpbBAKoWAgCMQD0E/fjMNr0FAce66MlpKSxdKbeUGt7Zv3Gauld
      3jeXOUCitQiKp60wg0IhAslBoEM=
      -----END CERTIFICATE-----
    cert_digest: ef407ce6a2f2c0b68c24a31cf7c75dfd6fd850d631775545f17b6d67b4b93c64
    id: c878251f-35b1-403d-987f-89554e98f8bb
certificates:
  - cert: |-
      -----BEGIN CERTIFICATE-----
      MIIBxzCCAUygAwIBAgIUfCAyT/813MlPQFqasFetYcAdPVYwCgYIKoZIzj0EAwIw
      GjEYMBYGA1UEAwwPa29uZ19jbHVzdGVyaW5nMB4XDTI0MDkwODE3NDQzNloXDTI3
      MDkwODE3NDQzNlowGjEYMBYGA1UEAwwPa29uZ19jbHVzdGVyaW5nMHYwEAYHKoZI
      zj0CAQYFK4EEACIDYgAEygTwKjbWvTvj0AlmNfxXFaPFbOctV0LA9D4aUzdG8dPg
      3bZEco1wWKNY1ucs7sFO51T6oPuI77nHv+owCMLbS7Cx2m7Tdkdibp63ejggiogZ
      U6er9tNWI7kF3NJos8mdo1MwUTAdBgNVHQ4EFgQUrpk2dywGsqDcrr1lNPhEQT7n
      3OMwHwYDVR0jBBgwFoAUrpk2dywGsqDcrr1lNPhEQT7n3OMwDwYDVR0TAQH/BAUw
      AwEB/zAKBggqhkjOPQQDAgNpADBmAjEAuotEtPBtrU1loNBwxpWWbeJ3cH8yJVb7
      jtAqcs3KC2nZa+gDbzCQgePjL1nQ7M2yAjEAootOUs9SiFh89TbjEpjYX0JTV1sX
      3aHopOi30aG8yuGRkC8NAn85+YpjVvvk+dJC
      -----END CERTIFICATE-----
    id: f9a53112-2e73-4fc4-bb5f-4be86071e881
    key: |-
      -----BEGIN PRIVATE KEY-----
      MIG2AgEAMBAGByqGSM49AgEGBSuBBAAiBIGeMIGbAgEBBDCjxfLtYRbO+4501Zx4
      bwN2BK/yDugQ/p1uA+yz6Z02oOPbPq0eWl5iZTp7LsxpHP+hZANiAATKBPAqNta9
      O+PQCWY1/FcVo8Vs5y1XQsD0PhpTN0bx0+DdtkRyjXBYo1jW5yzuwU7nVPqg+4jv
      uce/6jAIwttLsLHabtN2R2Junrd6OCCKiBlTp6v201YjuQXc0mizyZ0=
      -----END PRIVATE KEY-----
    snis:
      - id: 68bd0d63-6ec7-4e6e-aa03-e7093e272c5b
        name: foo.example.com
    tags:
      - kong-clustering
consumer_groups:
  - id: 56d3cf4b-defc-4f55-a0d9-078b083bfe63
    name: gold
    plugins:
      - config:
          day: null
          error_code: 429
          error_message: API rate limit exceeded
          fault_tolerant: true
          header_name: null
          hide_client_headers: false
          hour: 10000
          limit_by: consumer
          minute: null
          month: null
          path: null
          policy: local
          redis:
            database: 0
            host: null
            password: null
            port: 6379
            server_name: null
            ssl: false
            ssl_verify: false
            timeout: 2000
            username: null
          redis_database: 0
          redis_host: null
          redis_password: null
          redis_port: 6379
          redis_server_name: null
          redis_ssl: false
          redis_ssl_verify: false
          redis_timeout: 2000
          redis_username: null
          second: null
          sync_rate: -1
          year: null
        id: 793f5bd6-8ec4-4fff-afeb-9787be181b49
        name: rate-limiting
    tags:
      - gold
      - alice
consumers:
  - acls:
      - group: group-a
        id: b482b4f7-069a-4ca8-876c-4e9dadbdcecc
        tags:
          - alice
          - group-a
    basicauth_credentials:
      - id: 43d95734-64cd-4c11-929d-c26d9e7f305a
        password: 60cf54018b3a0264fb19517f8214dddaa4d83ab7
        username: alice
    custom_id: alice123
    groups:
      - id: 56d3cf4b-defc-4f55-a0d9-078b083bfe63
        name: gold
        tags:
          - gold
          - alice
    hmacauth_credentials:
      - id: c85a1ce8-37f6-4114-a314-fcedae367b38
        secret: alicesecret
        username: aliceuser
    id: bdd14928-4c9c-4572-a266-29a6e1602663
    jwt_secrets:
      - algorithm: HS256
        id: ce19c10f-4a06-4016-b22f-07216c9a56b2
        key: alicejwtkey
        secret: alicejwtsecret
    keyauth_credentials:
      - id: faf3581a-d131-42d9-a0de-7c6dee9a072a
        key: alicekey
        tags:
          - alice
    plugins:
      - config:
          day: null
          error_code: 429
          error_message: API rate limit exceeded
          fault_tolerant: true
          header_name: null
          hide_client_headers: false
          hour: 10
          limit_by: consumer
          minute: null
          month: null
          path: null
          policy: local
          redis:
            database: 0
            host: null
            password: null
            port: 6379
            server_name: null
            ssl: false
            ssl_verify: false
            timeout: 2000
            username: null
          redis_database: 0
          redis_host: null
          redis_password: null
          redis_port: 6379
          redis_server_name: null
          redis_ssl: false
          redis_ssl_verify: false
          redis_timeout: 2000
          redis_username: null
          second: null
          sync_rate: -1
          year: null
        enabled: true
        id: bf7626c4-8383-4cda-b3f3-80fa75bd2121
        name: rate-limiting
        protocols:
          - grpc
          - grpcs
          - http
          - https
    username: alice
  - custom_id: bob456
    id: b2e8673d-8ed5-49cf-be9c-b3c93640c7a6
    tags:
      - bob
    username: bob
plugins:
  - config:
      anonymous: null
      hide_credentials: false
      key_in_body: false
      key_in_header: true
      key_in_query: true
      key_names:
        - apikey-global
      realm: null
      run_on_preflight: true
    enabled: true
    id: 600c38c5-da4b-4ec5-b3b8-70fc279a6503
    name: key-auth
    protocols:
      - grpc
      - grpcs
      - http
      - https
      - ws
      - wss
  - config:
      anonymous: null
      hide_credentials: false
      key_in_body: false
      key_in_header: true
      key_in_query: true
      key_names:
        - apikey-in-route
      realm: null
      run_on_preflight: true
    enabled: true
    id: efa261d5-7978-4c8f-84e5-c6b0387ba37c
    name: key-auth
    protocols:
      - grpc
      - grpcs
      - http
      - https
      - ws
      - wss
    route: route-in-service
    service: httpbin
routes:
  - https_redirect_status_code: 426
    id: 7a8141cc-aade-49b2-a6c0-a95b624c1b6f
    name: route-standalone
    path_handling: v0
    paths:
      - /
    preserve_host: false
    protocols:
      - http
      - https
    regex_priority: 0
    request_buffering: true
    response_buffering: true
    strip_path: true
services:
  - connect_timeout: 60000
    enabled: true
    host: httpbin.org
    id: 0cc39546-f5c9-4aae-898c-c63ef5b4a33b
    name: httpbin
    plugins:
      - config:
          anonymous: null
          hide_credentials: false
          key_in_body: false
          key_in_header: true
          key_in_query: true
          key_names:
            - apikey-service
          realm: null
          run_on_preflight: true
        enabled: true
        id: 44a6f30a-23d6-4522-9ce8-f56243a054e3
        name: key-auth
        protocols:
          - grpc
          - grpcs
          - http
          - https
          - ws
          - wss
    port: 443
    protocol: https
    read_timeout: 60000
    retries: 5
    routes:
      - https_redirect_status_code: 426
        id: 9ac61ee8-4de7-4d75-82af-b7624ec5fb33
        name: route-in-service
        path_handling: v0
        paths:
          - /
        preserve_host: false
        protocols:
          - http
          - https
        regex_priority: 0
        request_buffering: true
        response_buffering: true
        strip_path: true
    write_timeout: 60000
upstreams:
  - algorithm: round-robin
    hash_fallback: none
    hash_on: none
    hash_on_cookie_path: /
    healthchecks:
      active:
        concurrency: 10
        healthy:
          http_statuses:
            - 200
            - 302
          interval: 0
          successes: 0
        http_path: /
        https_verify_certificate: true
        timeout: 1
        type: http
        unhealthy:
          http_failures: 0
          http_statuses:
            - 429
            - 404
            - 500
            - 501
            - 502
            - 503
            - 504
            - 505
          interval: 0
          tcp_failures: 0
          timeouts: 0
      passive:
        healthy:
          http_statuses:
            - 200
            - 201
            - 202
            - 203
            - 204
            - 205
            - 206
            - 207
            - 208
            - 226
            - 300
            - 301
            - 302
            - 303
            - 304
            - 305
            - 306
            - 307
            - 308
          successes: 0
        type: http
        unhealthy:
          http_failures: 0
          http_statuses:
            - 429
            - 500
            - 503
          tcp_failures: 0
          timeouts: 0
      threshold: 0
    id: d8432ca0-b6c5-4c82-bad8-924c56fb2d04
    name: my-upstream
    slots: 10000
    targets:
      - id: eca40df5-a499-4611-86e8-b377d1b3acdb
        target: a.example.com:443
        weight: 100
      - id: 69f683b9-796f-4117-8461-7bc0a21e5d53
        target: b.example.com:443
        weight: 100
    use_srv_name: false
vaults:
  - config:
      prefix: none
    id: 5370372e-9bab-47c9-a8de-b6af35cfb170
    name: env
    prefix: my-env-vault

Steps to test yourself:

Configure a CP using deck or the UI
Run deck dump --konnect-token $KONNECT_TOKEN --konnect-control-plane-name "YOUR_CP_NAME" --with-id
Convert to Terraform with deck file kong2tf -s ./kong.yaml --generate-imports-for-control-plane-id "0dea9abf-074a-4988-bdd6-c9ea1ea25d4b" --ignore-credential-changes
Run terraform plan. It will show all entities being adopted

…ore blocks, plus appropriate switches; patches "route ID is read only field" bug

CLAassistant · 2024-09-08T20:13:00Z

All committers have signed the CLA.

mheap · 2024-09-08T20:13:39Z

@Prashansa-K I'm primarily looking for mistakes + potential bugs in this review. I know the code needs a ton of work to be up to standard. I'd like to submit that in a followup as this is a Summit feature.

codecov-commenter · 2024-09-08T20:15:57Z

Codecov Report

Attention: Patch coverage is 82.83828% with 104 lines in your changes missing coverage. Please review.

Please upload report for BASE (main@fb795ad). Learn more about missing BASE report.

Files with missing lines	Patch %	Lines
cmd/file_kong2tf.go	0.00%	50 Missing ⚠️
kong2tf/builder_default_terraform.go	89.14%	32 Missing and 5 partials ⚠️
kong2tf/generate_resource.go	91.70%	12 Missing and 4 partials ⚠️
cmd/root.go	0.00%	1 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #1391   +/-   ##
=======================================
  Coverage        ?   29.49%           
=======================================
  Files           ?       59           
  Lines           ?     5136           
  Branches        ?        0           
=======================================
  Hits            ?     1515           
  Misses          ?     3512           
  Partials        ?      109

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Prashansa-K · 2024-09-09T05:39:34Z

@mheap Do we want this command to be in prod before the summit? Or we doing a beta release first?

Prashansa-K · 2024-09-09T05:58:34Z

kong2tf/builder_default_terraform.go

+
+func (b *DefaultTerraformBuider) buildControlPlaneVar(content *file.Content, generateImportsForControlPlaneID *string, ignoreCredentialChanges bool) {
+	cpId := "YOUR_CONTROL_PLANE_ID"
+	if generateImportsForControlPlaneID != nil {


If generateImportsForControlPlaneID is nil, cpId is left to "YOUR_CONTROL_PLANE_ID". I am assuming that the user is supposed to replace this string with their own CP id in the generated Terraform file. Is that assumption correct?

Yes, that's correct. If they provide generateImportsForControlPlaneID we populate it for them

mheap · 2024-09-09T06:08:57Z

@Prashansa-K Before summit - the provider is already available as a beta

Prashansa-K · 2024-09-09T08:56:33Z

@mheap This looks good to me. The presence of exhaustive tests is giving me more confidence on this.
Just the lint failure and konnect integration test needs to be checked out.

Prashansa-K · 2024-09-09T10:00:39Z

Integration test seems like a flake.
Could be related to this issue #1365.

mheap · 2024-09-10T08:43:26Z

@Prashansa-K @battlebyte Can you work together to fix the lint error and release?

Prashansa-K · 2024-09-10T08:53:42Z

I am working on fixing the integration test errors here: #1396 PR is in review.
Once this is merged in main, we can merge it here.

For the release, I will add a changelog with this PR: #1395 which is already there for a pending release item.
@battlebyte Could you help me with a changelog entry?
Also, I believe now since we have a added a new command, we would have to bump the minor version?

battlebyte · 2024-09-11T10:06:23Z

@Prashansa-K @mheap PR for the docs here: Kong/docs.konghq.com#7899

battlebyte and others added 4 commits August 20, 2024 11:28

feat: Add new command for Kong to Terraform conversion

2bbd8a9

Granular testing

bf29df3

fix plugin route reference

621202d

feat(kong2tf): PROTOTYPE kong2tf with import blocks and lifecycle-ign…

283b09a

…ore blocks, plus appropriate switches; patches "route ID is read only field" bug

mheap requested a review from Prashansa-K September 8, 2024 20:12

Prashansa-K reviewed Sep 9, 2024

View reviewed changes

mheap force-pushed the feat/kong2tf branch from 25222bb to cb49a6c Compare September 9, 2024 06:07

Replace templated kong2tf with recursive implementation

92bcb48

mheap force-pushed the feat/kong2tf branch from cb49a6c to 92bcb48 Compare September 9, 2024 06:19

Prashansa-K and others added 2 commits September 10, 2024 15:03

Merge branch 'main' into feat/kong2tf

94c06a0

fix: linting issues

9a4bd9a

Prashansa-K approved these changes Sep 10, 2024

View reviewed changes

Prashansa-K merged commit a084b3c into main Sep 10, 2024
18 of 19 checks passed

Prashansa-K deleted the feat/kong2tf branch September 10, 2024 10:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add `kong2tf` command #1391

Add `kong2tf` command #1391

mheap commented Sep 8, 2024

CLAassistant commented Sep 8, 2024 •

edited

Loading

mheap commented Sep 8, 2024

codecov-commenter commented Sep 8, 2024 •

edited

Loading

Prashansa-K commented Sep 9, 2024

Prashansa-K Sep 9, 2024

mheap Sep 9, 2024

mheap commented Sep 9, 2024

Prashansa-K commented Sep 9, 2024

Prashansa-K commented Sep 9, 2024

mheap commented Sep 10, 2024

Prashansa-K commented Sep 10, 2024 •

edited

Loading

battlebyte commented Sep 11, 2024

Add kong2tf command #1391

Add kong2tf command #1391

Conversation

mheap commented Sep 8, 2024

CLAassistant commented Sep 8, 2024 • edited Loading

mheap commented Sep 8, 2024

codecov-commenter commented Sep 8, 2024 • edited Loading

Codecov Report

Prashansa-K commented Sep 9, 2024

Prashansa-K Sep 9, 2024

Choose a reason for hiding this comment

mheap Sep 9, 2024

Choose a reason for hiding this comment

mheap commented Sep 9, 2024

Prashansa-K commented Sep 9, 2024

Prashansa-K commented Sep 9, 2024

mheap commented Sep 10, 2024

Prashansa-K commented Sep 10, 2024 • edited Loading

battlebyte commented Sep 11, 2024

Add `kong2tf` command #1391

Add `kong2tf` command #1391

CLAassistant commented Sep 8, 2024 •

edited

Loading

codecov-commenter commented Sep 8, 2024 •

edited

Loading

Prashansa-K commented Sep 10, 2024 •

edited

Loading