standard dashboard work incompletly on rke2 with cilium

[didlawowo]

Describe the bug

i'm using helm with vm K8s stack chart
grafana come with dashboard but some are not working correctly

To Reproduce

just install stack k8s

Version

latest

Logs

No response

Screenshots

No response

Used command-line flags

No response

Additional information

No response

[dmitryk-dk]

Hi @didlawowo ! What of the dashboard are you using?
VictoriaMetrics has its dashboards, and you can find them here.

[dmitryk-dk]

If you want to use this dashboard you should check the metrics which are used in that dashboard and probably correct them

[didlawowo]

i'm using dashboard provided by the k8s vm stack

[dmitryk-dk]

i'm using dashboard provided by the k8s vm stack

In the k8s stack, VictoriaMetrics exposes the dashboards that I shared before. As far as I can see from the domain you are using tailscale. So I think you should check which stack you are using

[dmitryk-dk]

Hi @didlawowo ! I reproduces your bug, need to check how to fix it

[dmitryk-dk]

Hi @didlawowo ! Can you check the vmagent on your installation? It will show you where is the problem with scrapes targets. If you fix it you should see all information into your dashboards.

[didlawowo]

thx you but could be more specific ? i'm not sure to understand

[didlawowo]

i'm using dashboard provided by the k8s vm stack

In the k8s stack, VictoriaMetrics exposes the dashboards that I shared before. As far as I can see from the domain you are using tailscale. So I think you should check which stack you are using

the tailscale service its just for exposing. no impact

[dmitryk-dk]

thx you but could be more specific ? i'm not sure to understand

Hi! We found a bug, and the dashboard should be updated. It happens because some kubernetes setup may missing image or container label
dotdc/grafana-dashboards-kubernetes#18 (comment)

As a small workaround you can install you kubelet with next configuration and check what the panels will have no data.

kubelet:
  spec:
    # drop high cardinality label and useless metrics for cadvisor and kubelet
    metricRelabelConfigs:
      - action: labeldrop
        regex: (uid)
      - action: labeldrop
        regex: (id|name)
      - action: drop
        source_labels: [__name__]
        regex: (rest_client_request_duration_seconds_bucket|rest_client_request_duration_seconds_sum|rest_client_request_duration_seconds_count)
      - target_label: image
        replacement: placeholder

[dmitryk-dk]

@Haleygo or @zekker6, can you take a look into the issue, please?

[didlawowo]

nice answer, i'm not sure how to config kubelet in rke2

https://docs.rke2.io/reference/windows_agent_config?_highlight=kubelet&_highlight=conf#windows-rke2-agent-cli-help

i take a look

[AndrewChubatiuk]

hey @didlawowo
what kubernetes version are you on?
what args are you passing to rke2 agent now?

[didlawowo]

i'm using rke2

with these parameters

write-kubeconfig-mode: "0600"
server: https://192.168.1.200:9345
token: 
tls-san:
  - "192.168.1.200"
# Make a etcd snapshot every 6 hours
etcd-snapshot-schedule-cron: "0 */6 * * *"
# Keep 56 etcd snapshorts (equals to 2 weeks with 6 a day)
etcd-snapshot-retention: 56
etcd-expose-metrics: true
cni:
  - cilium
disable:
  - rke2-ingress-nginx
  - rke2-canal
  - rke2-kube-proxy
disable-cloud-controller: true
disable-kube-proxy: true

v1.27.12+rke2r1

[AndrewChubatiuk]

hey @didlawowo
finally found time to test a case in this issue locally as we are not using RKE2 at all
was able to reproduce issues with scraping kube-scheduler, kube-controller-manager and etcd metrics.
All these services required additional configurations to become scrapable by vmagent

1. In server's /etc/rancher/rke2/config.yaml had to add additional values

etcd-expose-metrics: true
kube-scheduler-arg:
  - bind-address=0.0.0.0               # haven't checked how to pass there address from pod metadata instead
kube-controller-manager-arg:
  - bind-address=0.0.0.0               # haven't checked how to pass there address from pod metadata instead

2. also updated default scrape configs in k8s-stack chart

kubeControllerManager:
  vmScrape:
    spec:
      endpoints:
        - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
          port: http-metrics
          scheme: https
          tlsConfig:
            caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
            serverName: localhost          # maybe I've misconfigured something, but there was an issue until this value was set
            insecureSkipVerify: true        # haven't tried to pass automatically generated certificates in agent nodes
kubeEtcd:
  service:
    port: 2381
    targetPort: 2381
  vmScrape:
    spec:
      endpoints:  
        - port: http-metrics
          scheme: http
kubeScheduler:
  vmScrape:
    spec:
      endpoints:
        - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
          tlsConfig:
            caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
            serverName: 127.0.0.1
            insecureSkipVerify: true                # haven't tried to pass automatically generated certificates in agent nodes