-
Notifications
You must be signed in to change notification settings - Fork 79
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Prometheus: Out-of-order samples #471
Comments
Hey, yes it was introduced to have correct metrics independent of the pod size. If you need to customize this behavior I would appreciate your contributions. |
Hey,
replacing
with
While Grafana change is fully backward compatible (it will work the same if you dont have pod label) and will work with any number of pods, I dont believe pod label should be kept by default, since it might break a lot of dashboards/alerts people created.
Please let me know which option sounds better for you. I can bring PR if needed =) |
Hey, thanks a lot for your effort and details about solutions, very appreciate it. I will take a deeper look in the next days and reach back to you. |
I think we could introduce a "breaking change" in the 3.x major version and update metrics and dashboards accordingly to keep the pod label, also to make use of the other go metrics about the actual policy-reporter pod metrics. |
Hello,
We are using this exporter in HA mode (2 replicas) and scrape metrics using Prometheus (installed with kube-prometheus-stack)
We are seeing the following warns in prometheus logs due to duplicated metrics
The main reason for this is that in ServiceMonitor any label, which distinguish two targets, is dropped.
policy-reporter/charts/policy-reporter/charts/monitoring/templates/servicemonitor.yaml
Lines 49 to 57 in 3517577
And since labeldrop declared before
.Values.serviceMonitor.relabelings
used, user cant override this behavior.Looking at the code, I guess it was done this way for dashboards to work with any number of pods, but it triggers alerts on kube-prometheus-stack side https://github.com/prometheus-community/helm-charts/blob/68ba986b2a6283efd3f743f0cf7859d93b615b64/charts/kube-prometheus-stack/templates/prometheus/rules-1.14/prometheus.yaml#L339
I understand how current behaviour works for most cases, but I would like to have the ability to override it/disable labeldrop.
I can create PR if needed.
Thank you!
The text was updated successfully, but these errors were encountered: