Disabling pipeline level metric collection (metric.collect: false) is broken

[mashhurs]

Description of the problem including expected versus actual behavior:

• current issue discusses and fixes only pipeline level metric collection issues
• node level metric collection control needs discussion, see Disabling node level metrics collection issue #16887
  • which metrics (health-report, node, node-stats, stats) need to be enabled by default?
  • does metric collection signal apply for periodic ones?
  • what is actual intention and expectation with enabling/disabling metric collection?

Steps to reproduce:

Please include a minimal but complete recreation of the problem,
including (e.g.) pipeline definition(s), settings, locale, etc. The easier
you make for us to reproduce it, the more likely that somebody will take the
time to look at it.

1. 8.17 or compile LS from the main (HEAD: 274c212d9dc64cecbea59f04412c6ad011730a4f) branch
2. place metric.collect: false in pipeline config in pipelines.yml file
3. run LS: bin/logstash, got following errors

Provide logs (if relevant):

• sometimes

2025-01-06 22:37:58,653 pool-6-thread-1 ERROR An exception occurred processing Appender plain_console org.jruby.exceptions.TypeError: (TypeError) no implicit conversion of Hash into String
        at org.jruby.RubyKernel.inspect(org/jruby/RubyKernel.java:2365)
        at org.jruby.RubyHash.inspect(org/jruby/RubyHash.java:953)
        at org.jruby.RubyKernel.inspect(org/jruby/RubyKernel.java:2365)
        at org.jruby.RubyKernel.inspect(org/jruby/RubyKernel.java:2365)
        at org.jruby.RubyKernel.inspect(org/jruby/RubyKernel.java:2365)
        at org.jruby.RubyKernel.inspect(org/jruby/RubyKernel.java:2365)
        at org.jruby.RubyHash.inspect(org/jruby/RubyHash.java:953)
        at org.jruby.RubyHash.to_s(org/jruby/RubyHash.java:1019)
        at org.logstash.log.LoggerExt.error(org/logstash/log/LoggerExt.java:127)
        at RUBY.update(/Users/mashhur/Dev/elastic/logstash/logstash-core/lib/logstash/instrument/periodic_poller/base.rb:53)
        at RUBY.notify_to(/Users/mashhur/Dev/elastic/logstash/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/collection/copy_on_notify_observer_set.rb:102)
        at org.jruby.RubyHash.each(org/jruby/RubyHash.java:1615)
        at RUBY.notify_to(/Users/mashhur/Dev/elastic/logstash/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/collection/copy_on_notify_observer_set.rb:100)
        at RUBY.notify_observers(/Users/mashhur/Dev/elastic/logstash/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/collection/copy_on_notify_observer_set.rb:64)
        at RUBY.execute_task(/Users/mashhur/Dev/elastic/logstash/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/timer_task.rb:317)
        at RUBY.execute(/Users/mashhur/Dev/elastic/logstash/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/executor/safe_task_executor.rb:24)
        at RUBY.execute(/Users/mashhur/Dev/elastic/logstash/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/executor/safe_task_executor.rb:19)
        at RUBY.safe_execute(/Users/mashhur/Dev/elastic/logstash/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/ivar.rb:169)
        at RUBY.process_task(/Users/mashhur/Dev/elastic/logstash/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/scheduled_task.rb:285)
        at RUBY.process_tasks(/Users/mashhur/Dev/elastic/logstash/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/executor/timer_set.rb:165)
        at RUBY.run(/Users/mashhur/Dev/elastic/logstash/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/executor/java_executor_service.rb:79)

• mostly

// on console we see following lines:
[2025-01-08T13:20:46,785][WARN ][org.logstash.instrument.metrics.LazyInstantiatedFlowMetric] Underlying metrics for `queue_persisted_growth_events` not yet instantiated, could not capture their rates
			[2025-01-08T13:20:46,786][WARN ][org.logstash.instrument.metrics.LazyInstantiatedFlowMetric] Underlying metrics for `queue_persisted_growth_bytes` not yet instantiated, could not capture their rates

// when we hit localhost:9600/_node/stats or localhost:9600/_node/stats/flow
{
    "status": 500,
    "request_method": "GET",
    "path_info": "/_node/stats/flow",
    "query_string": "",
    "http_version": "HTTP/1.1",
    "http_accept": "*/*",
    "error": "Unexpected Internal Error",
    "class": "NoMethodError",
    "message": "undefined method `value' for nil:NilClass",
    "backtrace": [
        "/Users/mashhur/Dev/elastic/temp/logstash/logstash-8.17.0/logstash-core/lib/logstash/api/commands/stats.rb:35:in `block in queue'",
        "org/jruby/RubyArray.java:1981:in `each'",
        "/Users/mashhur/Dev/elastic/temp/logstash/logstash-8.17.0/logstash-core/lib/logstash/api/commands/stats.rb:33:in `queue'",
        "/Users/mashhur/Dev/elastic/temp/logstash/logstash-8.17.0/logstash-core/lib/logstash/api/modules/node_stats.rb:53:in `queue'",
        "/Users/mashhur/Dev/elastic/temp/logstash/logstash-8.17.0/logstash-core/lib/logstash/api/modules/node_stats.rb:34:in `block in NodeStats'",
        "org/jruby/RubyMethod.java:119:in `call'",
        "/Users/mashhur/Dev/elastic/temp/logstash/logstash-8.17.0/vendor/bundle/jruby/3.1.0/gems/sinatra-4.1.0/lib/sinatra/base.rb:1807:in `block in compile!'",
        "/Users/mashhur/Dev/elastic/temp/logstash/logstash-8.17.0/vendor/bundle/jruby/3.1.0/gems/sinatra-4.1.0/lib/sinatra/base.rb:1074:in `block in route!'",
        "/Users/mashhur/Dev/elastic/temp/logstash/logstash-8.17.0/vendor/bundle/jruby/3.1.0/gems/sinatra-4.1.0/lib/sinatra/base.rb:1092:in `route_eval'",
        "/Users/mashhur/Dev/elastic/temp/logstash/logstash-8.17.0/vendor/bundle/jruby/3.1.0/gems/sinatra-4.1.0/lib/sinatra/base.rb:1074:in `block in route!'",
        "/Users/mashhur/Dev/elastic/temp/logstash/logstash-8.17.0/vendor/bundle/jruby/3.1.0/gems/sinatra-4.1.0/lib/sinatra/base.rb:1123:in `block in process_route'",
        "org/jruby/RubyKernel.java:1426:in `catch'",
        "/Users/mashhur/Dev/elastic/temp/logstash/logstash-8.17.0/vendor/bundle/jruby/3.1.0/gems/sinatra-4.1.0/lib/sinatra/base.rb:1121:in `process_route'",
        "/Users/mashhur/Dev/elastic/temp/logstash/logstash-8.17.0/vendor/bundle/jruby/3.1.0/gems/sinatra-4.1.0/lib/sinatra/base.rb:1072:in `block in route!'",
        "org/jruby/RubyArray.java:1981:in `each'",
        "/Users/mashhur/Dev/elastic/temp/logstash/logstash-8.17.0/vendor/bundle/jruby/3.1.0/gems/sinatra-4.1.0/lib/sinatra/base.rb:1069:in `route!'",
        "/Users/mashhur/Dev/elastic/temp/logstash/logstash-8.17.0/vendor/bundle/jruby/3.1.0/gems/sinatra-4.1.0/lib/sinatra/base.rb:1193:in `block in dispatch!'",
        "org/jruby/RubyKernel.java:1426:in `catch'",
        "/Users/mashhur/Dev/elastic/temp/logstash/logstash-8.17.0/vendor/bundle/jruby/3.1.0/gems/sinatra-4.1.0/lib/sinatra/base.rb:1164:in `invoke'",
        "/Users/mashhur/Dev/elastic/temp/logstash/logstash-8.17.0/vendor/bundle/jruby/3.1.0/gems/sinatra-4.1.0/lib/sinatra/base.rb:1188:in `dispatch!'",
        "/Users/mashhur/Dev/elastic/temp/logstash/logstash-8.17.0/vendor/bundle/jruby/3.1.0/gems/sinatra-4.1.0/lib/sinatra/base.rb:1004:in `block in call!'",
        "org/jruby/RubyKernel.java:1426:in `catch'",
        "/Users/mashhur/Dev/elastic/temp/logstash/logstash-8.17.0/vendor/bundle/jruby/3.1.0/gems/sinatra-4.1.0/lib/sinatra/base.rb:1164:in `invoke'",
        "/Users/mashhur/Dev/elastic/temp/logstash/logstash-8.17.0/vendor/bundle/jruby/3.1.0/gems/sinatra-4.1.0/lib/sinatra/base.rb:1004:in `call!'",
        "/Users/mashhur/Dev/elastic/temp/logstash/logstash-8.17.0/vendor/bundle/jruby/3.1.0/gems/sinatra-4.1.0/lib/sinatra/base.rb:993:in `call'",
        "/Users/mashhur/Dev/elastic/temp/logstash/logstash-8.17.0/vendor/bundle/jruby/3.1.0/gems/rack-protection-4.1.0/lib/rack/protection/base.rb:53:in `call'",
        "/Users/mashhur/Dev/elastic/temp/logstash/logstash-8.17.0/vendor/bundle/jruby/3.1.0/gems/rack-protection-4.1.0/lib/rack/protection/xss_header.rb:20:in `call'",
        "/Users/mashhur/Dev/elastic/temp/logstash/logstash-8.17.0/vendor/bundle/jruby/3.1.0/gems/rack-protection-4.1.0/lib/rack/protection/path_traversal.rb:18:in `call'",
        "/Users/mashhur/Dev/elastic/temp/logstash/logstash-8.17.0/vendor/bundle/jruby/3.1.0/gems/rack-protection-4.1.0/lib/rack/protection/json_csrf.rb:28:in `call'",
        "/Users/mashhur/Dev/elastic/temp/logstash/logstash-8.17.0/vendor/bundle/jruby/3.1.0/gems/rack-protection-4.1.0/lib/rack/protection/base.rb:53:in `call'",
        "/Users/mashhur/Dev/elastic/temp/logstash/logstash-8.17.0/vendor/bundle/jruby/3.1.0/gems/rack-protection-4.1.0/lib/rack/protection/base.rb:53:in `call'",
        "/Users/mashhur/Dev/elastic/temp/logstash/logstash-8.17.0/vendor/bundle/jruby/3.1.0/gems/rack-protection-4.1.0/lib/rack/protection/frame_options.rb:33:in `call'",
        "/Users/mashhur/Dev/elastic/temp/logstash/logstash-8.17.0/vendor/bundle/jruby/3.1.0/gems/sinatra-4.1.0/lib/sinatra/middleware/logger.rb:17:in `call'",
        "/Users/mashhur/Dev/elastic/temp/logstash/logstash-8.17.0/vendor/bundle/jruby/3.1.0/gems/rack-3.1.8/lib/rack/head.rb:15:in `call'",
        "/Users/mashhur/Dev/elastic/temp/logstash/logstash-8.17.0/vendor/bundle/jruby/3.1.0/gems/sinatra-4.1.0/lib/sinatra/base.rb:227:in `call'",
        "/Users/mashhur/Dev/elastic/temp/logstash/logstash-8.17.0/vendor/bundle/jruby/3.1.0/gems/sinatra-4.1.0/lib/sinatra/base.rb:2138:in `call'",
        "/Users/mashhur/Dev/elastic/temp/logstash/logstash-8.17.0/vendor/bundle/jruby/3.1.0/gems/rack-3.1.8/lib/rack/urlmap.rb:76:in `block in call'",
        "org/jruby/RubyArray.java:1981:in `each'",
        "/Users/mashhur/Dev/elastic/temp/logstash/logstash-8.17.0/vendor/bundle/jruby/3.1.0/gems/rack-3.1.8/lib/rack/urlmap.rb:60:in `call'",
        "/Users/mashhur/Dev/elastic/temp/logstash/logstash-8.17.0/logstash-core/lib/logstash/api/rack_app.rb:75:in `call'",
        "/Users/mashhur/Dev/elastic/temp/logstash/logstash-8.17.0/logstash-core/lib/logstash/api/rack_app.rb:49:in `call'",
        "/Users/mashhur/Dev/elastic/temp/logstash/logstash-8.17.0/vendor/bundle/jruby/3.1.0/gems/rack-3.1.8/lib/rack/builder.rb:277:in `call'",
        "/Users/mashhur/Dev/elastic/temp/logstash/logstash-8.17.0/vendor/bundle/jruby/3.1.0/gems/puma-6.4.3-java/lib/puma/request.rb:100:in `block in handle_request'",
        "/Users/mashhur/Dev/elastic/temp/logstash/logstash-8.17.0/vendor/bundle/jruby/3.1.0/gems/puma-6.4.3-java/lib/puma/thread_pool.rb:378:in `with_force_shutdown'",
        "/Users/mashhur/Dev/elastic/temp/logstash/logstash-8.17.0/vendor/bundle/jruby/3.1.0/gems/puma-6.4.3-java/lib/puma/request.rb:99:in `handle_request'",
        "/Users/mashhur/Dev/elastic/temp/logstash/logstash-8.17.0/vendor/bundle/jruby/3.1.0/gems/puma-6.4.3-java/lib/puma/server.rb:464:in `process_client'",
        "/Users/mashhur/Dev/elastic/temp/logstash/logstash-8.17.0/vendor/bundle/jruby/3.1.0/gems/puma-6.4.3-java/lib/puma/server.rb:245:in `block in run'",
        "/Users/mashhur/Dev/elastic/temp/logstash/logstash-8.17.0/vendor/bundle/jruby/3.1.0/gems/puma-6.4.3-java/lib/puma/thread_pool.rb:155:in `block in spawn_thread'"
    ]
}