Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OpenTelemetry endpoint is not capturing any k8sattributes like namespace, pod, container #672

Open
pavelpikatamcs opened this issue Aug 2, 2024 · 2 comments

Comments

@pavelpikatamcs
Copy link

pavelpikatamcs commented Aug 2, 2024

Our applications produce OTEL metrics and send them to local alloy endpoint exposed via this chart.

The metrics only contain attributes defined by the applications themselves, and any Kubernetes metadata is not included, e.g. namespace, pod, container are all missing, so it's very difficult to identify where the metrics are actually coming from.

Is there a way to configure this via this chart, or maybe have this enabled by default?

Chart version 1.4.4
Here is our helm values (traces and logs config removed as irrelevant):

cluster:
  name: eu1-aks-dev-1
alloy:
  alloy:
    stabilityLevel: "experimental"
    uiPathPrefix: /agent
    clustering:
      enabled: true
  controller:
    replicas: 1
    
  prometheus:
    host: "redacted"
    writeEndpoint: "/api/v1/push"
    tenantId: "redacted"
    authMode: basic
    basicAuth:
      usernameKey: username
      passwordKey: password
    secret:
      create: false
      name: external-mimir-creds
      namespace: monitoring
opencost:
  enabled: false
metrics:
  enabled: true
    
receivers:
  jaeger:
    grpc:
      enabled: true
    thriftBinary:
      enabled: true
    thriftCompact:
      enabled: true
    thriftHttp:
      enabled: true
  zipkin:
    enabled: true

Cluster info:
1.29.5, AKS with Overlay CNI

@pavelpikatamcs
Copy link
Author

Is it maybe possible to tweak pod_association setting of otelcol.processor.k8sattributes? I suspect that Azure CNI overlay may be messing with the collector

@pavelpikatamcs
Copy link
Author

@petewall Would you have a chance to take a guess what may be wrong here?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant