Does a paused consumer still send poll request to the broker in the background thread?

[Diyaa1]

Hi all,

I have a weird behavior when pausing a consumer, I'm using an FFI for librdkafka using ruby lang.

When I use below code to pause and sleep the processing for 15 minutes, the consumer group become empty after 5 minutes.

def stall_message_processing(message, consumer, stalling_time)
   partition_list = Rdkafka::Consumer::TopicPartitionList.new
   partition_list.add_topic(message.topic, [message.partition])
   consumer.pause(partition_list)
   sleep(15.minutes)
   consumer.resume(partition_list)
end

my question is that does librdkafka still send poll requests to the broker when we pause all partitions? (in my case I only have one partition)

appreciate any help, thank you. 😄

[mensfeld]

You're reaching the max poll interval. This is not a librdkafka issue nor rdkafka-ruby issue (maintainer here).

Let me expand the answer: pausing != "do whatever you want forever". You are still limited by Kafka protocol and max.poll.interval.ms.

If you are looking into processing long running jobs you may want to look here: https://karafka.io/docs/Pro-Long-Running-Jobs/

[Diyaa1]

Thanks a lot friend, I just wanted a confirmation for this, as I couldn't find any reference anywhere online 👍