Perform retries for certain HTTP status codes when submitting metrics to Datadog API

[thyandrecardoso]

This proposes adding some retries when Datadog responds with certain status codes. We're using Datadog API reporter and it is frequent to have some transient failures, which is the motivation for these changes.

I've also considered dealing with some Failures but, besides making testing harder, it is also not clear if those correspond to situations where the server did not receive the request at all and, thus, we might be duplicating data.

So, having retries only for the success case of the request and work with HTTP status codes seemed more sensible.

What do you think, does this make sense?

[ivantopo]

Yeap, I think the change makes sense. My only concern is whether the list of status codes that can be retried should be expanded/reduced. Is there any official Datadog documentation about when a client could/should retry requests?

[thyandrecardoso]

Well, I found these:

• https://github.com/DataDog/datadog-agent/blob/d341f634250822813bb64a2ed5fcf8d79c7911fb/pkg/trace/writer/sender.go#L370-L377
• https://github.com/DataDog/datadog-api-client-python/blob/9221073f63a3415bfa255fe0495bf54a0d32fed3/src/datadog_api_client/rest.py#L26
• https://github.com/DataDog/datadog-api-client-java/blob/master/src/main/java/com/datadog/api/client/ApiClient.java#L1614

I was unable to find anything in the official docs though :(

Nonetheless, the current list in this PR looks okish. We can also move these retriable status codes into configuration, but as we only have 1 endpoint and 1 verb being used against the public api, seems unlikely we will ever want to tweak those... 🤔