How to accept requests that do not use json encoding?

[khanetor]

I have a request like this

POST /api/?method=some-method&encrypted=true HTTP/1.1
Host: axis.com
Content-Type: application/x-www-form-urlencoded
Content-Length: NNN
some_long_string_without_double_quotes_here

Due to the origin of this request, nothing can be done to change it to use json content-type. Regardless, I have a schema like this

@simpleRestJson
service CameraService {
    version: "1.0.0",
    operations: [Report]
}

@http(method: "POST", uri: "/api", code: 200)
operation Report {
    input: CameraQuery,
    output: AxisTimestamp
}

structure CameraQuery {
    @required
    @httpQuery("method")
    method: String,

    @required
    @httpQuery("encrypted")
    encrypted: Boolean,

    @required
    @httpPayload
    payload: String
}

When I submit the request above with double quotes around the body, then it works. But if I remove the double quotes, then I get 500, as expected because the body does not confirm to json format. I then tried changing the payload to Blob instead of String, and I get 500 error regardless.

How should I handle this case?

[daddykotex]

• If I understand correctly: you have this smithy spec
• You use a http4s client that is generated from this spec
• You're unable to make it work

See https://disneystreaming.github.io/smithy4s/docs/protocols/simple-rest-json/overview#semantics

The protocol specifically only issue application/json content-type requests. For example, if I take your smithy spec and build a project on it, with a main that looks like this:

  val client = EmberClientBuilder.default[IO].build.map { c =>
    val c2 = org.http4s.client.middleware
      .RequestLogger(true, true, logAction = Some(IO.println _))(c)
    SimpleRestJsonBuilder(CameraService).client(c2).resource
  }
  val run = (client, Routes.all).tupled.flatMap { case (buildClient, routes) =>
    for {
      client <- buildClient
      _ <- EmberServerBuilder
        .default[IO]
        .withPort(port"8080")
        .withHost(host"localhost")
        .withHttpApp(routes.orNotFound)
        .build
      _ <- client
        .report(
          "some-method",
          true,
          "some_long_string_without_double_quotes_here"
        )
        .toResource
    } yield ()
  }.use_
}

I will see HTTP/1.1 POST http://localhost:8080/api?method=some-method&encrypted=true Headers(Content-Length: 45, Content-Type: application/json) body=""some_long_string_without_double_quotes_here"" in the console

So:

• this is the wrong content-type, and unfortunately this is the way SimpleRestJson works
• because you encode the payload as a String, it is automatically double quoted

The request looks like this:

POST /api/?method=some-method&encrypted=true HTTP/1.1
Content-Type: application/json
Content-Length: 45
"some_long_string_without_double_quotes_here"

But if you're telling us that the following works (json payload with a different content-type):

POST /api/?method=some-method&encrypted=true HTTP/1.1
Content-Type: application/x-www-form-urlencoded
Content-Length: 45
"some_long_string_without_double_quotes_here"

Then you can probably implement this via a middleware. Just let us know and I'd be happy to show you

[khanetor]

So:

* this is the wrong content-type, and unfortunately this is the way `SimpleRestJson` works

Is there any other alternative than SimpleRestJson?

But if you're telling us that the following works (json payload with a different content-type):

POST /api/?method=some-method&encrypted=true HTTP/1.1
Content-Type: application/x-www-form-urlencoded
Content-Length: 45
"some_long_string_without_double_quotes_here"

Then you can probably implement this via a middleware. Just let us know and I'd be happy to show you.

I didn't mean that this works. I don't know how to set the Content-Type to application/x-www-form-urlencoded. I suppose we will need a decoder for that, and I hope you can show me how. I would need the content without double quotes as well, because we are integrating some really legacy system, where even a minute change is impossible. So, I hope to be able to consume a request like this

POST /api/?method=some-method&encrypted=true HTTP/1.1
Content-Type: application/x-www-form-urlencoded
Content-Length: 45
some_long_string_without_double_quotes_here

In this case, would the body payload be a Blob? I am fine with it being a string, because the legacy system sends plaint text over.

BR, Kha

[daddykotex]

Is there any other alternative than SimpleRestJson?
Currently there is none. It's possible to write one, we have one internally but it's irrelevant here.

So, I hope to be able to consume a request like this

To be clear: you want to build a server (using smithy4s) that accepts this requests:

POST /api/?method=some-method&encrypted=true HTTP/1.1
Content-Type: application/x-www-form-urlencoded
Content-Length: 45
some_long_string_without_double_quotes_here

Correct? (I ask because I had the impression you were trying to use the client but now I think I was wrong and you're trying to implement the service side of this)

[khanetor]

Oh sorry for not mentioning that :)

Yes, I am building a server, not a client. And yes, it is best to accept that request.

[kubukoz]

Without implementing a new protocol, you can work around this (not recommended, but it should work) by making your own SimpleRestJsonBuilder alternative:

object MyRestJsonBuilder
    extends SimpleProtocolBuilder[smithy4s.api.SimpleRestJson](
      // notable change from the definition of `SimpleRestJsonBuilder`
      CodecAPI.nativeStringsAndBlob(
        //
        smithy4s.http.json.codecs(
          smithy4s.api.SimpleRestJson.protocol.hintMask ++ HintMask(InputOutput)
        )
      )
    )

object Main extends ResourceApp.Forever {
  val impl: ReportService[IO] = new ReportService[IO] {
    def report(payload: RawPayload): IO[Unit] = IO.println(payload.value)
  }
  def run(args: List[String]): Resource[IO, Unit] =
    MyRestJsonBuilder
      .routes(impl)
      .resource
      .map(_.orNotFound)
      .flatMap(
        EmberServerBuilder
          .default[IO]
          .withHttpApp(_)
          .withPort(port"8080")
          .withHost(host"0.0.0.0")
          .build
      )
      .void
}

Then annotating your payload with @mediaType:

@mediaType("application/x-www-form-urlencoded;charset=utf-8")
string FormPayload

structure CameraQuery {
    @required
    @httpPayload
    payload: FormPayload
}

Full example

Do note the following though:

• as @daddykotex mentioned, the behavior of @simpleRestJson doesn't allow bodies like that. If you have to implement this particular interface, that means you're constrained to a different protocol. You would ideally implement a server for such a protocol on your own, as the semantics of @simpleRestJson don't fit your usecase
• if you have control over the schema and it's not a strict requirement to send the payload this way, I would advise to stick to JSON and have a top-level string or string member. It's better for schema evolution too, as you can add fields.
• the major problem is that none of the clients that know @simpleRestJson will be able to use this service without a similar workaround on their side.

[khanetor]

I see. I am also interested in developing a more official solution. Could you show me how to create a new protocol? For example, in this case, I want to create URLEncodedBuilder. To help make this easier, we can assume that the payload is a string, and not a blob.

[kubukoz]

Sorry for the long delay, I wanted to give you a proper answer and this would require some time.

To build a URLEncodedBuilder, I would say you'd need to build on the SimpleProtocolBuilder logic and implement a CodecAPI for URL forms.

CodecAPI is a generic construct (currently we only implement it for JSON) - it basically encapsulates the Array[Byte] => A and A => Array[Byte] serialization/deserialization, as well as an expected media type. I think you wouldn't need to build your own router/reverse router, just that CodecAPI and a protocol definition.

On the Smithy side, you would add a new @protocolDefinition. Then you'd refer to that protocol in your implementation of SimpleProtocolBuilder.

@Baccata, do you think this would be in scope for this repo? The CodecAPI implementation, specifically.

[Baccata]

@Baccata, do you think this would be in scope for this repo? The CodecAPI implementation, specifically.

I'd rather not tbh. Maintenance burden and all.

[khanetor]

@kubukoz Thank you so much! I shall try this approach!