Skip to content
This repository has been archived by the owner on Jun 17, 2024. It is now read-only.

Once rabbitmq is restarted, fluentBit pods are unable to connect to it #2

Open
SpoorthyHR opened this issue Dec 17, 2021 · 0 comments

Comments

@SpoorthyHR
Copy link

We are using this utility to read few component log files and filter for alarms and send the processed alarm entry to rabbitmq which will then be picked up by Alarm management system.

Observation :

  1. RabbitMQ (deployed as Microservice) was down and restarted..
  2. After rabbitMQ is up, a new component alarm got generated.
  3. This generated alarm logs are not pushed to Rabbitmq.

And notice following error

2021/11/24 13:21:15 Couldn't publish record: : Exception (504) Reason: "channel/connection is not open"

NOTE: This utility - fluent-bit-rabbitmq-output - is deployed as a daemonSet. Only after restart of the daemonSet, logs are forwarded to rabbitmq.

On further checking, we see that in output_rabbitmq.go following function is responsible for rabbitMQ connection -> func FLBPluginInit

And as per the comment in the code, this func is called only once during the instance startup.
// Gets called only once for each instance you have configured.

Could you please suggest a workaround for this issue ?

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant