[Feature]: Support accepting OTLP via Kafka

[yurishkuro]

Requirement

As a user of OpenTelemetry Collector, I would like to send OTLP-encoded data to Jaeger.

Problem

Jaeger ingester today only understands 3 formats, json (Jaeger Protobuf as JSON), protobuf (Jaeger Protobuf), and zipkin-thrift.

Proposal

Add two more recognized encodings, otlp-json and otlp-proto.

It should be fairly simple to implement since we already import OTLP->Jaeger decoders when receiving OTLP via RPC.

Open questions

No response

[NonsoAmadi10]

Hello Yurishkuro, can I be assigned to this please? I want this to be my first open source contribution

[NonsoAmadi10]

Also what guide can I follow as to how I can make my Pull Request?

[NonsoAmadi10]

Quick question @yurishkuro , I noticed I would be requiring a marshal and an unmashaller for both encodings, Is this alright?

[yurishkuro]

Also what guide can I follow as to how I can make my Pull Request?

Are you asking in general how to do it on GitHub [1], or specifically in Jaeger project [2]?

I would be requiring a marshal and an unmashaller for both encodings, Is this alright?

I think you only need to unmarshall

jaeger/cmd/ingester/app/builder/builder.go

Line 35 in 3d67ac8

switch options.Encoding {

But when introducing a new encoding you probably don't want it to be available as output encoding of the collector, only the input encoding in the ingester. We probably don't support such asymmetry cleanly at the moment. The simplest solution is to fail collector at startup if otlp is selected at output encoding. But if you can implement bidirectional marshalling, thenwe could go with full symmetric support (OTEL contrib does have translation of OTLP to and from Jaeger proto).

[NonsoAmadi10]

Wow, alright. I would keep this in mind while implementing. Thank you Yuri!! I'd be raising my first PR soon for you to look and give feedback

[vishal-chdhry]

Hi! Can I take up this issue?

[yurishkuro]

@vishal-chdhry give it a try

[SaarthakMaini]

Hello,

Has this issue been resolved ?

If not , may I work on this issue?

Thank You!

[yurishkuro]

@SaarthakMaini there is a PR in progress, but the author did not come back to it since Dec.
If you want to pick it up, see #4049 (comment)

[metapox]

Hello, Can I give this a try?

[gebn]

It looks like there have been several false starts for this feature. Is there some inherent complexity with it? I'm definitely in a position to contribute if the logging library is the primary thing holding back #4333.

[yurishkuro]

afaik there are no specific blockers other than cleaning up the PR

[gebn]

Great! For posterity, did batching on the OTel Collector turn out not to be an issue? [1] Partitioning by trace ID sounds sensible.

[yurishkuro]

#4333 introduces batching API, but only within the Kafka consumer. I think it's a sufficient first step, and the overall Storage API batching could be done in the future if necessary/

[yurishkuro]

I am going to close this, it will be solved by jaeger-v2 using official OTEL Kafka receiver -- #4868.