[Feature] Emit metrics of cluster creation and other related metrics for observability

[ujjawal-khare-27]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

In production, ensuring that Ray cluster creation adheres to strict SLAs is crucial. Any unexpected delays in cluster creation can stem from various factors, such as prolonged image pull times, node creation delays, or suboptimal KubeRay settings. Currently, this process requires manual monitoring, which is cumbersome and prone to missing critical events or edge cases. Automating this monitoring is essential to improve reliability and efficiency.

Use case

Deeper insights and observability

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[kevin85421]

@ujjawal-khare-27 this makes sense to me. Would you mind providing a list of metrics that you think it's helpful for you? We will define the scope for v1.4.0. Maybe metrics about starting up time which are the missing pieces in Ray metrics.

[kevin85421]

I read the blog from @cchen777. The following picture is a good reference.

[win5923]

@kevin85421 can i take this?