Add OpenSearch Ingestion Pipeline Support

[blakeromano]

Description of your changes

Fixes #1446

I have:

• Read and followed Crossplane's contribution process.
• Run make reviewable to ensure this PR is ready for review.
• Added backport release-x.y labels to auto-backport this PR if necessary.

How has this code been tested

I took the example and modified line 26 to change it to my own AWS Account ID. I was able to validate the command:

make e2e UPTEST_EXAMPLE_LIST=examples/osis/v1beta1/pipeline.yaml

---
apiVersion: osis.aws.upbound.io/v1beta1
kind: Pipeline
metadata:
  annotations:
    crossplane.io/external-create-pending: "2024-09-12T16:23:43Z"
    crossplane.io/external-create-succeeded: "2024-09-12T16:23:43Z"
    crossplane.io/external-name: example
    crossplane.io/paused: "false"
    meta.upbound.io/example-id: osis/v1beta1/pipeline
    upjet.upbound.io/test: "true"
    uptest-old-id: example
  creationTimestamp: "2024-09-12T16:23:42Z"
  deletionGracePeriodSeconds: 0
  deletionTimestamp: "2024-09-12T16:29:49Z"
  finalizers:
  - finalizer.managedresource.crossplane.io
  generation: 3
  labels:
    testing.upbound.io/example-name: example
  name: example
  resourceVersion: "2581"
  uid: 351b183c-0dac-430e-ba5f-90aec59e402e
spec:
  deletionPolicy: Delete
  forProvider:
    maxUnits: 1
    minUnits: 1
    pipelineConfigurationBody: |
      version: "2"
      example-pipeline:
          source:
              http:
                  path: "/example"
          sink:
          - s3:
              aws:
                  sts_role_arn: "arn:aws:iam::ACCOUNT_ID:role/example-osi-pipeline-role"
                  region: "us-west-1"
              bucket: "example-upbound-osis-pipeline-example-blake"
              threshold:
                  event_collect_timeout: "60s"
              codec:
                  ndjson:
    pipelineName: example
    region: us-west-1
    tags:
      crossplane-kind: pipeline.osis.aws.upbound.io
      crossplane-name: example
      crossplane-providerconfig: default
  initProvider: {}
  managementPolicies:
  - '*'
  providerConfigRef:
    name: default
status:
  atProvider:
    id: example
    ingestEndpointUrls:
    - example-7myxz74qalrcrk5ilmygdzco2u.us-west-1.osis.amazonaws.com
    maxUnits: 1
    minUnits: 1
    pipelineArn: arn:aws:osis:us-west-1:ACCOUNT_ID:pipeline/example
    pipelineConfigurationBody: |
      version: "2"
      example-pipeline:
          source:
              http:
                  path: "/example"
          sink:
          - s3:
              aws:
                  sts_role_arn: "arn:aws:iam::ACCOUNT_ID:role/example-osi-pipeline-role"
                  region: "us-west-1"
              bucket: "example-upbound-osis-pipeline-example-blake"
              threshold:
                  event_collect_timeout: "60s"
              codec:
                  ndjson:
    pipelineName: example
    tags:
      crossplane-kind: pipeline.osis.aws.upbound.io
      crossplane-name: example
      crossplane-providerconfig: default
    tagsAll:
      crossplane-kind: pipeline.osis.aws.upbound.io
      crossplane-name: example
      crossplane-providerconfig: default
  conditions:
  - lastTransitionTime: "2024-09-12T16:29:42Z"
    reason: ReconcileSuccess
    status: "True"
    type: Synced
  - lastTransitionTime: "2024-09-12T16:29:42Z"
    reason: Available
    status: "True"
    type: Ready
  - lastTransitionTime: "2024-09-12T16:29:42Z"
    reason: UpToDate
    status: "True"
    type: Test
  - lastTransitionTime: "2024-09-12T16:29:42Z"
    reason: Success
    status: "True"
    type: LastAsyncOperation

[blakeromano]

/test-examples="examples/osis/v1beta1/pipeline.yaml"

[blakeromano]

So looking at logs https://github.com/crossplane-contrib/provider-upjet-aws/actions/runs/10835115001/job/30065908399 it seems like it actually does get ready but maybe I am missing something in terms of it not working or maybe it is just timing out too soon?

[mergenci]

First, I'm not an expert on Uptest logs 🙂 Second, the following line tells me that the Pipeline resource is not ready:

2024-09-12T17:13:53.1182356Z     case.go:363: command "${KUBECTL} wait pipeline.osis.aws.upbound.io/example --for=condition=Test --timeout 10s" exceeded 7 sec timeout, context deadline exceeded

When I check the resource output by grepping osis.aws.upbound.io in the Uptest log file I downloaded, I see that the resource is not ready:

- apiVersion: osis.aws.upbound.io/v1beta1
  kind: Pipeline
  metadata:
    annotations:
      crossplane.io/external-create-pending: "2024-09-12T17:13:44Z"
      crossplane.io/external-create-succeeded: "2024-09-12T17:13:44Z"
      crossplane.io/external-name: example
      meta.upbound.io/example-id: osis/v1beta1/pipeline
      upjet.upbound.io/test: "true"
    creationTimestamp: "2024-09-12T16:53:53Z"
    finalizers:
    - finalizer.managedresource.crossplane.io
    generation: 2
    labels:
      testing.upbound.io/example-name: example
    name: example
    resourceVersion: "4358"
    uid: 8b15e491-dd01-4096-9084-b373f7027be0
  spec:
    deletionPolicy: Delete
    forProvider:
      maxUnits: 1
      minUnits: 1
      pipelineConfigurationBody: |
        version: "2"
        example-pipeline:
            source:
                http:
                    path: "/example"
            sink:
            - s3:
                aws:
                    sts_role_arn: "arn:aws:iam::609897127049:role/example-osi-pipeline-role"
                    region: "us-west-1"
                bucket: "example-upbound-osis-pipeline-example"
                threshold:
                    event_collect_timeout: "60s"
                codec:
                    ndjson:
      pipelineName: example
      region: us-west-1
      tags:
        crossplane-kind: pipeline.osis.aws.upbound.io
        crossplane-name: example
        crossplane-providerconfig: default
    initProvider: {}
    managementPolicies:
    - '*'
    providerConfigRef:
      name: default
  status:
    atProvider: {}
    conditions:
    - lastTransitionTime: "2024-09-12T16:53:54Z"
      reason: Creating
      status: "False"
      type: Ready
    - lastTransitionTime: "2024-09-12T16:53:54Z"
      reason: ReconcileSuccess
      status: "True"
      type: Synced
    - lastTransitionTime: "2024-09-12T16:53:55Z"
      reason: Success
      status: "True"
      type: LastAsyncOperation

[blakeromano]

@mergenci sorry missed that 🤦‍♂️ what I am still confused by is why it is still in a creating state. I know OSI pipelines do take a long time to come up but I am able to get this same uptest command work locally on my AWS environment by just swapping the AWS Account ID in the pipeline body. Like I said I wonder if there we are just timing out on the test and it isn't an actual failure.

[turkenf]

Please rebase the PR and trigger again to get this fix, so that we can see if there is an error.

[blakeromano]

/test-examples="examples/osis/v1beta1/pipeline.yaml"

[blakeromano]

It looks like it is failing with this error:

AccessDeniedException: User: arn:aws:iam::153891904029:user/official-provider-testing is not authorized to perform: osis:CreatePipeline on resource: arn:aws:osis:us-west-1:153891904029:pipeline/example

So we may need to tweak the permissioning on the user to allow the test to pass.