leader brokers without a matching listener

[barryzhounb]

I had a Kafka topic with 6 partitions, somehow Kafka producer threw out WARN to complain leader without listener.

I am wondering what possible reasons can cause them? Seems that Strimize cluster got trouble, not sure how to track and verify the real reason. Moreover, once it happens, does it lose messages? Or Does Strimiz cluster get self-heal?

2021-04-08 15:12:46,906 [kafka-producer-network-thread | dlcclient1] org.apache.kafka.clients.producer.internals.Sender: [WARN] [Producer clientId=dlcclient1] Got error produce response with correlation id 179686 on topic-partition dlake-topic-1, retrying (2147483645 attempts left). Error: NOT_LEADER_FOR_PARTITION

2021-04-08 15:12:46,906 [kafka-producer-network-thread | dlcclient1] org.apache.kafka.clients.producer.internals.Sender: [WARN] [Producer clientId=dlcclient1] Received invalid metadata error in produce request on partition dlake-topic-1 due to org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition.. Going to request metadata update now

2021-04-08 15:12:47,028 [kafka-producer-network-thread | dlcclient1] org.apache.kafka.clients.NetworkClient: [WARN] [Producer clientId=dlcclient1] 6 partitions have leader brokers without a matching listener, including [dlake-topic-0, dlake-topic-5, dlake-topic-4, dlake-topic-1, dlake-topic-2, dlake-topic-3]

2021-04-08 15:12:47,140 [kafka-producer-network-thread | dlcclient1] org.apache.kafka.clients.NetworkClient: [WARN] [Producer clientId=dlcclient1] 6 partitions have leader brokers without a matching listener, including [dlake-topic-0, dlake-topic-5, dlake-topic-4, dlake-topic-1, dlake-topic-2, dlake-topic-3]

[barryzhounb]

In the Strimiz clusteer, how to check cluster logs such as I want to see if broker was down or some errors exist in broker or zookeeper

[scholzj]

As with most Kube apps. The logs are sent to STDOUT where they are captured by your container runtime and ideally processed by tools such as Fluentd or Logstash and stored somewhere.

[barryzhounb]

After I read zookeeper and brokers logs, I saw a lot of errors in the zookeeper container and broker container. I just grab a couple of errors here.

Now I am thinking, in my OpenShift platform, although I installed Strimizi, but seem Strimizi is not enough stable, I installed Strimizi by Operator and automatically install without any custom configuration. My question is, how to install a stable Strimiz?

2021-04-09 13:08:32,703 WARN [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Connection to node 2 (dlake-cluster-kafka-2.dlake-cluster-kafka-brokers.abc.svc/10.254.27.71:9091) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient) [ReplicaFetcherThread-0-2]
2021-04-09 13:08:32,703 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Error sending fetch request (sessionId=271605607, epoch=INITIAL) to node 2: (org.apache.kafka.clients.FetchSessionHandler) [ReplicaFetcherThread-0-2]
java.io.IOException: Connection to dlake-cluster-kafka-2.dlake-cluster-kafka-brokers.abc.svc:9091 (id: 2 rack: null) failed.
	at org.apache.kafka.clients.NetworkClientUtils.awaitReady(NetworkClientUtils.java:71)
	at kafka.server.ReplicaFetcherBlockingSend.sendRequest(ReplicaFetcherBlockingSend.scala:103)
	at kafka.server.ReplicaFetcherThread.fetchFromLeader(ReplicaFetcherThread.scala:207)
	at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:300)
	at kafka.server.AbstractFetcherThread.$anonfun$maybeFetch$3(AbstractFetcherThread.scala:135)
	at kafka.server.AbstractFetcherThread.maybeFetch(AbstractFetcherThread.scala:134)
	at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:117)
	at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)
2021-04-09 13:08:32,704 WARN [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Error in response for fetch request (type=FetchRequest, replicaId=0, maxWait=500, minBytes=1, maxBytes=10485760, fetchData={dlake-topic-1=(fetchOffset=4321587, logStartOffset=0, maxBytes=1048576, currentLeaderEpoch=Optional[272]), dlake-topic-4=(fetchOffset=4342835, logStartOffset=0, maxBytes=1048576, currentLeaderEpoch=Optional[272])}, isolationLevel=READ_UNCOMMITTED, toForget=, metadata=(sessionId=271605607, epoch=INITIAL), rackId=) (kafka.server.ReplicaFetcherThread) [ReplicaFetcherThread-0-2]
java.io.IOException: Connection to dlake-cluster-kafka-2.dlake-cluster-kafka-brokers.abc.svc:9091 (id: 2 rack: null) failed.
	at org.apache.kafka.clients.NetworkClientUtils.awaitReady(NetworkClientUtils.java:71)
	at kafka.server.ReplicaFetcherBlockingSend.sendRequest(ReplicaFetcherBlockingSend.scala:103)
	at kafka.server.ReplicaFetcherThread.fetchFromLeader(ReplicaFetcherThread.scala:207)
	at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:300)
	at kafka.server.AbstractFetcherThread.$anonfun$maybeFetch$3(AbstractFetcherThread.scala:135)
	at kafka.server.AbstractFetcherThread.maybeFetch(AbstractFetcherThread.scala:134)
	at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:117)
	at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)
2021-04-09 13:08:32,731 INFO [ReplicaFetcher replicaId=0, leaderId=1, fetcherId=0] Partition dlake-topic-2 has an older epoch (253) than the current leader. Will await the new LeaderAndIsr state before resuming fetching. (kafka.server.ReplicaFetcherThread) [ReplicaFetcherThread-0-1]
2021-04-09 13:08:32,734 WARN [ReplicaFetcher replicaId=0, leaderId=1, fetcherId=0] Partition dlake-topic-2 marked as failed (kafka.server.ReplicaFetcherThread) [ReplicaFetcherThread-0-1]

Another errors:

2021-04-09 13:09:58,913 WARN Unable to reconnect to ZooKeeper service, session 0x3000a1dff88001b has expired (org.apache.zookeeper.ClientCnxn) [nioEventLoopGroup-24-1]
2021-04-09 13:09:58,913 WARN Exception caught (org.apache.zookeeper.ClientCnxnSocketNetty) [nioEventLoopGroup-24-1]
org.apache.zookeeper.ClientCnxn$SessionExpiredException: Unable to reconnect to ZooKeeper service, session 0x3000a1dff88001b has expired
	at org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:1381)
	at org.apache.zookeeper.ClientCnxnSocket.readConnectResult(ClientCnxnSocket.java:153)
	at org.apache.zookeeper.ClientCnxnSocketNetty$ZKClientHandler.channelRead0(ClientCnxnSocketNetty.java:507)
	at org.apache.zookeeper.ClientCnxnSocketNetty$ZKClientHandler.channelRead0(ClientCnxnSocketNetty.java:468)
	at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1533)
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1282)
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1329)
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:834)
2021-04-09 13:09:58,947 INFO channel is told closing (org.apache.zookeeper.ClientCnxnSocketNetty) [nioEventLoopGroup-24-1]
2021-04-09 13:09:58,915 INFO [ZooKeeperClient Kafka server] Session expired. (kafka.zookeeper.ZooKeeperClient) [zk-session-expiry-handler0]

[scholzj]

You do not share the full logs. You do not even explain what are the logs from. So it is to some extent just guessing. The first one is from Kafka ... that might look like some networking issue.

Now I am thinking, in my OpenShift platform, although I installed Strimizi, but seem Strimizi is not enough stable, I installed Strimizi by Operator and automatically install without any custom configuration. My question is, how to install a stable Strimiz?

Sorry, I'm not sure I follow what you mean with this.