Sharing retry topics across consumers

[ro0sterjam]

Hi all, from what I understand, the @RetryableTopic function expects that a single consumer (group) uses a dedicated set of retry topics.

This makes sense to me when we considering that each consumer has a different retry configuration, and thus different delays associated with each retry topic; as well as a different number of retry topics depending on how many attempts are allowed.

But the first point (different delays), from what I can tell, is actually a non problem when you consider that each consumer only cares about the messages intended for their consumption. All other messages can be, and should be ignored, in a hypothetical global retry topic. Something that can be easily achieved using a retry_topic-target-group-id header.

The second point is (different number of retry topics) is also a non problem, because the total number of "global retry topics" required is simply the max count across all consumers. If a consumer does not need, say, retry topics 5+, then it simply does not listen on them.

My question is, are there any arguments against using a "global retry topic(s)" pattern (or, rather, enabling the user to configure one)? This is particularly helpful when topic management is done manually and the individual prefers not to explode their topic count with numerous retry topics for each consumer.

[garyrussell]

Sorry, I am having trouble understanding what you are asking for.

Perhaps a concrete example showing a couple of @RetryableTopic listeners would help.

[ro0sterjam]

Apologies for any confusion. I think it'll be best to demonstrate via diagram.

Essentially, today every @RetryableTopic, as I understand it, will require a set of dedicated retry topics for that given topic and only be used by that given topic.

The concern I have is in the explosion of number of topics (N retry topics for every existing M topics).

I am essentially wondering if there were any specific concerns or reasons that we cannot simply use a single shared set of "global" retry topics?

Where the consumer for TopicA can simply ignore all the retry messages belonging to TopicB, and vice versa.

[garyrussell]

You will end up with mixed record types in the retry topics - how will you deal with deserialization of those different types and how will the unwanted records get discarded? For the latter, you would need to add a filter to each listener configuration.

However, you can do this today using a custom topic naming strategy.

See https://docs.spring.io/spring-kafka/docs/current/reference/html/#custom-naming-strategies

[garyrussell]

Also, you would need homogenous delays across all listeners, to avoid a short delay being stuck behind a longer one.

[ro0sterjam]

Both those issues be resolved using a targetGroupId header, wouldn't they?

Discard unwanted records via the header, not need to deserialize the payload if the header doesn't match. And since undesired records are discarded, these won't block on differing delays as long as we filter by targetGroupId acting on the delay/sleep.

The messages share the same topic, but they live in different dimensions and do not interact/interfere with each other.

[garyrussell]

Kafka provides no mechanism to "discard" records before deserialization; you could add a custom deserializer to return a null value and add a filter at the application level to discard the nulls.

But, yes, that should work, but you might need some trickery (e.g. examine the thread name) to make the discard decision at such a low level in the stack.