Prometheus pending

[mhausenblas]

I have a 3 nodes EKS cluster (with managed node group) and all the pods come up but the Prometheus server. There's no indication for why, enough resources are available. What am I missing?

[julianocosta89]

If I am not mistaken Prometheus has a Volume configured.
If you are not deploying an EBS, you can simply update the Prometheus deployment to use an emptyDir.

@puckpuck maybe we could set emptyDir as default. What do you think?

[fatsheep9146]

I agreed with emptyDir volume as default.

[mhausenblas]

Thanks! So, where's the storage class to configure then?

[joshleecreates]

I think you would need to set storageClass under prometheus in your custom values, see https://github.com/prometheus-community/helm-charts/blob/main/charts/prometheus/values.yaml#L377

[mhausenblas]

OK, thanks @joshleecreates and it seems it's in the CRD. Is there an example anywhere or do I have to dig into that? :)

UPDATE: here's the fix …

1. Create a file called values.yaml with the following content:

prometheus:
  server:
    persistentVolume:
      enabled: false

2. Then execute:

$ helm upgrade my-otel-demo open-telemetry/opentelemetry-demo --values values.yaml

[puckpuck]

This should be a default on the Helm chart. I'll do some quick testing to make sure no weird side effects (ie: increased memory)

[puckpuck]

This PR should do what we need: open-telemetry/opentelemetry-helm-charts#661

[mhausenblas]

@joshleecreates can confirm that fixed it (with the content above in values.yaml and then a helm upgrade my-otel-demo open-telemetry/opentelemetry-demo --values values.yaml). Thank you so much, really appreciate it!

@puckpuck that's awesome, can't wait, thanks a bunch!

[julianocosta89]

Thanks everyone!
@joshleecreates I can't mark your reply as answer 😢

[mhausenblas]

No worries I already did ;)