Support non-AWS client generation via a Smithy SDK

[kggilmer]

This is the top-level issue to cover generating clients from any Smithy models, not just AWS service. Such a client generator will be referred to as a "Smithy SDK" in this issue.

Currently AWS SDKs for Kotlin are generated from a program consisting of three primary layers:

1. smithy-kotlin (codegen, rt)
2. aws-sdk-kotlin (codegen, rt)
3. aws-crt-kotlin (rt)

When an AWS SDK is generated in Kotlin, some runtime and codegen artifacts are produced specifically to support AWS features that would not be appropriate or generally applicable to non AWS services. Features such as AWS region, signing details, and AWS credential file parsing are all examples of functionality that non-AWS service clients would not need.

There are also missing aspects for general purpose clients. For example, the ability to specify endpoints or provide custom auth schemes.

Another complexity is in where various things are organized in our projects. At this time, there is no generic Smithy protocol. There is no spec or implementation that can provide serialization outside of what is specified by AWS protocols (defined in Smithy). This means that some AWS protocol logic will need to exist in the Smithy SDK, however our current architecture places all protocols in the aws-sdk-kotlin repo. This means that either the Smithy SDK will include code from aws-sdk-kotlin or the subset of protocols required to generate clients (restJson, awsJson1_0) will need to migrate down to smithy-kotlin.

AWS services have strong guarantees around how they are allowed to change over time, optimizing on the customer experience and avoiding situations which cause customers to have to update their code to work with new features provided by SDKs. This constrains us in some ways, for example how we treat required fields as being nullable in the AWS SDKs. As developers of various Smithy-based SDKs may have their own opinions or requirements about how Kotlin should be generated from service models, a more granular configuration needs to be expressed via the plugin settings.

Acceptance Critera

1. A combination of the specified repos can be used to produce working client SDKs from Smithy models. (Smithy SDK Client Generator)
2. The generated clients can be adapted to use auth schemes outside of AWS
3. The Smithy SDK allows customers to configure codegen in some specific ways that we currently do not support for AWS services, for example allowing required members to be generated as non-nullable types.
4. The Smithy SDK should be usable both by internal service teams as well as anyone that has a Smithy model and would like to generate a client in Kotlin.

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue, please leave a comment