Mirror Maker 2 TLS at the Edge | Authentication error due to SSL handshake and PKIX path validation failed

[bitvijays]

Dear Strimzi Team,

Hope you are doing well and had a good weekend. Thank you for your continuous support.

Currently, we have two Kafka clusters, one at the cloud and one at the edge (on a raspberry Pi) using internal listeners and then port forwarding them using ssh. We were able to create some test data into a topic on the edge cluster. Now, we want to run a mirrormaker 2 pod on the edge to replicate the data from the edge to the cloud. We are getting authentication error due to SSL handshake and PKIX path validation failed. We followed the Mirror-Maker-2-tls example. Could you please guide us on how this could be resolved. We have attached the error logs and MM2 yaml config. Do we need to somehow ignore the self-certificates?

Kafka

debian@cloudcore:~$ kubectl get kafka  -o wide -n my-kafka-project
NAME                   DESIRED KAFKA REPLICAS   DESIRED ZK REPLICAS   READY   WARNINGS
cloud-kafka            1                        1                     True    True
edge-kafka-node-0101   1                        1                     True    True

Kafka Pods

debian@cloudcore:~$ kubectl get pods  -o wide -n my-kafka-project
NAME                                                    READY   STATUS    RESTARTS   AGE   IP            NODE                        NOMINATED NODE   READINESS GATES
cloud-kafka-entity-operator-67bf57f65f-87qdh            3/3     Running   0          27h   10.244.1.49   kafka                       <none>           <none>
cloud-kafka-kafka-0                                     1/1     Running   0          27h   10.244.1.48   kafka                       <none>           <none>
cloud-kafka-zookeeper-0                                 1/1     Running   0          27h   10.244.1.47   kafka                       <none>           <none>
edge-kafka-node-0101-entity-operator-686766b7f7-r78xh   3/3     Running   0          27h   10.244.2.27   node-0101.bitvijays.local   <none>           <none>
edge-kafka-node-0101-kafka-0                            1/1     Running   0          27h   10.244.2.26   node-0101.bitvijays.local   <none>           <none>
edge-kafka-node-0101-zookeeper-0                        1/1     Running   2          27h   10.244.2.25   node-0101.bitvijays.local   <none>           <none>
mm2-node-0101-cloud-mirrormaker2-7c556689df-zd6dz       0/1     Error     241        25h   10.244.2.29   node-0101.bitvijays.local   <none>           <none> 

Secrets

debian@cloudcore:~$ kubectl get secrets  -o wide -n my-kafka-project
NAME                                               TYPE                                  DATA   AGE
cloud-kafka-clients-ca                             Opaque                                1      27h
cloud-kafka-clients-ca-cert                        Opaque                                3      27h
cloud-kafka-cluster-ca                             Opaque                                1      27h
cloud-kafka-cluster-ca-cert                        Opaque                                3      27h
cloud-kafka-cluster-operator-certs                 Opaque                                4      27h
cloud-kafka-entity-operator-certs                  Opaque                                4      27h
cloud-kafka-entity-operator-token-rtkdm            kubernetes.io/service-account-token   3      27h
cloud-kafka-kafka-brokers                          Opaque                                4      27h
cloud-kafka-kafka-token-9zrtl                      kubernetes.io/service-account-token   3      27h
cloud-kafka-zookeeper-nodes                        Opaque                                4      27h
cloud-kafka-zookeeper-token-25r6z                  kubernetes.io/service-account-token   3      27h
default-token-l79vb                                kubernetes.io/service-account-token   3      13d
edge-kafka-node-0101-clients-ca                    Opaque                                1      27h
edge-kafka-node-0101-clients-ca-cert               Opaque                                3      27h
edge-kafka-node-0101-cluster-ca                    Opaque                                1      27h
edge-kafka-node-0101-cluster-ca-cert               Opaque                                3      27h
edge-kafka-node-0101-cluster-operator-certs        Opaque                                4      27h
edge-kafka-node-0101-entity-operator-certs         Opaque                                4      27h
edge-kafka-node-0101-entity-operator-token-4lbjz   kubernetes.io/service-account-token   3      27h
edge-kafka-node-0101-kafka-brokers                 Opaque                                4      27h
edge-kafka-node-0101-kafka-token-bksb2             kubernetes.io/service-account-token   3      27h
edge-kafka-node-0101-zookeeper-nodes               Opaque                                4      27h
edge-kafka-node-0101-zookeeper-token-sb7ng         kubernetes.io/service-account-token   3      27h
mm2-node-0101-cloud-mirrormaker2-token-2fk4h       kubernetes.io/service-account-token   3      25h

Mirror Maker Yaml

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaMirrorMaker2
metadata:
  name: mm2-node-0101-cloud
spec:
  version: 2.6.0
  replicas: 1
  connectCluster: "cloud-kafka"
  template:
    pod:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                - key: kubernetes.io/hostname
                  operator: In
                  values:
                  - node-0101.bitvijays.local
  clusters:
  - alias: "edge-kafka-node-0101"
    bootstrapServers: node-0101:9093
    tls:
      trustedCertificates:
        - secretName: edge-kafka-node-0101-cluster-ca-cert
          certificate: ca.crt
  - alias: "cloud-kafka"
    bootstrapServers: kafka:9093
    tls:
      trustedCertificates:
        - secretName: cloud-kafka-cluster-ca-cert
          certificate: ca.crt
    config:
      config.storage.replication.factor: 1
      offset.storage.replication.factor: 1
      status.storage.replication.factor: 1
  mirrors:
  - sourceCluster: "edge-kafka-node-0101"
    targetCluster: "cloud-kafka"
    sourceConnector:
      config:
        replication.factor: 1
        offset-syncs.topic.replication.factor: 1
        sync.topic.acls.enabled: "false"
    heartbeatConnector:
      config:
        heartbeats.topic.replication.factor: 1
    checkpointConnector:
      config:
        checkpoints.topic.replication.factor: 1
    topicsPattern: ".*"
    groupsPattern: ".*"

Mirror Maker Error Log

{"log":"2021-03-28 21:39:03,562 INFO [AdminClient clientId=adminclient-1] Failed authentication with kafka/192.168.1.217 (SSL handshake failed) (org.apache.kafka.common.network.Selector) [kafka-admin-client-thread | adminclient-1]\n","stream":"stdout","time":"2021-03-28T21:39:03.566624141Z"}
{"log":"2021-03-28 21:39:03,572 ERROR [AdminClient clientId=adminclient-1] Connection to node -1 (kafka/192.168.1.217:9093) failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient) [kafka-admin-client-thread | adminclient-1]\n","stream":"stdout","time":"2021-03-28T21:39:03.573234717Z"}
{"log":"2021-03-28 21:39:03,578 WARN [AdminClient clientId=adminclient-1] Metadata update failed due to authentication error (org.apache.kafka.clients.admin.internals.AdminMetadataManager) [kafka-admin-client-thread | adminclient-1]\n","stream":"stdout","time":"2021-03-28T21:39:03.589230796Z"}
{"log":"org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed\n","stream":"stdout","time":"2021-03-28T21:39:03.589413053Z"}
{"log":"Caused by: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors\n","stream":"stdout","time":"2021-03-28T21:39:03.589632216Z"}
{"log":"\u0009at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)\n","stream":"stdout","time":"2021-03-28T21:39:03.589793677Z"}
{"log":"\u0009at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:349)\n","stream":"stdout","time":"2021-03-28T21:39:03.590070359Z"}
{"log":"\u0009at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:292)\n","stream":"stdout","time":"2021-03-28T21:39:03.591242306Z"}
{"log":"\u0009at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:287)\n","stream":"stdout","time":"2021-03-28T21:39:03.591302175Z"}
{"log":"\u0009at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1356)\n","stream":"stdout","time":"2021-03-28T21:39:03.591320156Z"}
{"log":"\u0009at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1231)\n","stream":"stdout","time":"2021-03-28T21:39:03.591337286Z"}
{"log":"\u0009at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1174)\n","stream":"stdout","time":"2021-03-28T21:39:03.591354026Z"}
{"log":"\u0009at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392)\n","stream":"stdout","time":"2021-03-28T21:39:03.591672744Z"}
{"log":"\u0009at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:443)\n","stream":"stdout","time":"2021-03-28T21:39:03.591694244Z"}
{"log":"\u0009at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1074)\n","stream":"stdout","time":"2021-03-28T21:39:03.591711373Z"}
{"log":"\u0009at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1061)\n","stream":"stdout","time":"2021-03-28T21:39:03.59172791Z"}
{"log":"\u0009at java.base/java.security.AccessController.doPrivileged(Native Method)\n","stream":"stdout","time":"2021-03-28T21:39:03.59174491Z"}
{"log":"\u0009at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1008)\n","stream":"stdout","time":"2021-03-28T21:39:03.591761595Z"}
{"log":"\u0009at org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:430)\n","stream":"stdout","time":"2021-03-28T21:39:03.59177815Z"}
{"log":"\u0009at org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:514)\n","stream":"stdout","time":"2021-03-28T21:39:03.591794557Z"}
{"log":"\u0009at org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:368)\n","stream":"stdout","time":"2021-03-28T21:39:03.59181115Z"}
{"log":"\u0009at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:291)\n","stream":"stdout","time":"2021-03-28T21:39:03.591837927Z"}
{"log":"\u0009at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:173)\n","stream":"stdout","time":"2021-03-28T21:39:03.591857353Z"}
{"log":"\u0009at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:547)\n","stream":"stdout","time":"2021-03-28T21:39:03.591874038Z"}
{"log":"\u0009at org.apache.kafka.common.network.Selector.poll(Selector.java:485)\n","stream":"stdout","time":"2021-03-28T21:39:03.591890667Z"}
{"log":"\u0009at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:544)\n","stream":"stdout","time":"2021-03-28T21:39:03.59190663Z"}
{"log":"\u0009at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1293)\n","stream":"stdout","time":"2021-03-28T21:39:03.591923389Z"}
{"log":"\u0009at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1224)\n","stream":"stdout","time":"2021-03-28T21:39:03.591940389Z"}
{"log":"\u0009at java.base/java.lang.Thread.run(Thread.java:834)\n","stream":"stdout","time":"2021-03-28T21:39:03.591957092Z"}
{"log":"Caused by: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors\n","stream":"stdout","time":"2021-03-28T21:39:03.591974092Z"}
{"log":"\u0009at java.base/sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:369)\n","stream":"stdout","time":"2021-03-28T21:39:03.592080961Z"}
{"log":"\u0009at java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:275)\n","stream":"stdout","time":"2021-03-28T21:39:03.592112997Z"}
{"log":"\u0009at java.base/sun.security.validator.Validator.validate(Validator.java:264)\n","stream":"stdout","time":"2021-03-28T21:39:03.592129349Z"}
{"log":"\u0009at java.base/sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:313)\n","stream":"stdout","time":"2021-03-28T21:39:03.592145312Z"}
{"log":"\u0009at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:276)\n","stream":"stdout","time":"2021-03-28T21:39:03.592178793Z"}
{"log":"\u0009at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:141)\n","stream":"stdout","time":"2021-03-28T21:39:03.592195478Z"}
{"log":"\u0009at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1334)\n","stream":"stdout","time":"2021-03-28T21:39:03.592211663Z"}
{"log":"\u0009... 19 more\n","stream":"stdout","time":"2021-03-28T21:39:03.592227718Z"}
{"log":"Caused by: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors\n","stream":"stdout","time":"2021-03-28T21:39:03.592242829Z"}
{"log":"\u0009at java.base/sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:158)\n","stream":"stdout","time":"2021-03-28T21:39:03.592639064Z"}
{"log":"\u0009at java.base/sun.security.provider.certpath.PKIXCertPathValidator.engineValidate(PKIXCertPathValidator.java:84)\n","stream":"stdout","time":"2021-03-28T21:39:03.592669564Z"}
{"log":"\u0009at java.base/java.security.cert.CertPathValidator.validate(CertPathValidator.java:309)\n","stream":"stdout","time":"2021-03-28T21:39:03.59268723Z"}
{"log":"\u0009at java.base/sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:364)\n","stream":"stdout","time":"2021-03-28T21:39:03.592704378Z"}
{"log":"\u0009... 25 more\n","stream":"stdout","time":"2021-03-28T21:39:03.592721267Z"}
{"log":"2021-03-28 21:39:03,595 INFO [AdminClient clientId=adminclient-1] Metadata update failed (org.apache.kafka.clients.admin.internals.AdminMetadataManager) [kafka-admin-client-thread | adminclient-1]\n","stream":"stdout","time":"2021-03-28T21:39:03.5965209Z"}
{"log":"org.apache.kafka.common.errors.TimeoutException: Call(callName=fetchMetadata, deadlineMs=1616967572445, tries=1, nextAllowedTryMs=-9223372036854775709) timed out at 9223372036854775807 after 1 attempt(s)\n","stream":"stdout","time":"2021-03-28T21:39:03.596614844Z"}
{"log":"Caused by: org.apache.kafka.common.errors.TimeoutException: The AdminClient thread has exited.\n","stream":"stdout","time":"2021-03-28T21:39:03.596635214Z"}
{"log":"2021-03-28 21:39:03,596 INFO [AdminClient clientId=adminclient-1] Metadata update failed (org.apache.kafka.clients.admin.internals.AdminMetadataManager) [kafka-admin-client-thread | adminclient-1]\n","stream":"stdout","time":"2021-03-28T21:39:03.59730876Z"}
{"log":"org.apache.kafka.common.errors.TimeoutException: Call(callName=fetchMetadata, deadlineMs=1616967573589, tries=1, nextAllowedTryMs=-9223372036854775709) timed out at 9223372036854775807 after 1 attempt(s)\n","stream":"stdout","time":"2021-03-28T21:39:03.597413073Z"}
{"log":"Caused by: org.apache.kafka.common.errors.TimeoutException: The AdminClient thread has exited.\n","stream":"stdout","time":"2021-03-28T21:39:03.597436258Z"}
{"log":"2021-03-28 21:39:03,614 ERROR Stopping due to error (org.apache.kafka.connect.cli.ConnectDistributed) [main]\n","stream":"stdout","time":"2021-03-28T21:39:03.624852422Z"}
{"log":"org.apache.kafka.connect.errors.ConnectException: Failed to connect to and describe Kafka cluster. Check worker's broker connection and security properties.\n","stream":"stdout","time":"2021-03-28T21:39:03.624931106Z"}
{"log":"\u0009at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:70)\n","stream":"stdout","time":"2021-03-28T21:39:03.624950347Z"}
{"log":"\u0009at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:51)\n","stream":"stdout","time":"2021-03-28T21:39:03.625016068Z"}
{"log":"\u0009at org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:95)\n","stream":"stdout","time":"2021-03-28T21:39:03.62503479Z"}
{"log":"\u0009at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:78)\n","stream":"stdout","time":"2021-03-28T21:39:03.62505179Z"}
{"log":"Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed\n","stream":"stdout","time":"2021-03-28T21:39:03.625068641Z"}
{"log":"\u0009at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)\n","stream":"stdout","time":"2021-03-28T21:39:03.625085123Z"}
{"log":"\u0009at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)\n","stream":"stdout","time":"2021-03-28T21:39:03.625101289Z"}
{"log":"\u0009at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)\n","stream":"stdout","time":"2021-03-28T21:39:03.625116974Z"}
{"log":"\u0009at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)\n","stream":"stdout","time":"2021-03-28T21:39:03.625133011Z"}
{"log":"\u0009at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:64)\n","stream":"stdout","time":"2021-03-28T21:39:03.625148899Z"}
{"log":"\u0009... 3 more\n","stream":"stdout","time":"2021-03-28T21:39:03.625164659Z"}
{"log":"Caused by: org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed\n","stream":"stdout","time":"2021-03-28T21:39:03.625180029Z"}
{"log":"Caused by: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors\n","stream":"stdout","time":"2021-03-28T21:39:03.625197714Z"}
{"log":"\u0009at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)\n","stream":"stdout","time":"2021-03-28T21:39:03.62521425Z"}
{"log":"\u0009at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:349)\n","stream":"stdout","time":"2021-03-28T21:39:03.625230398Z"}
{"log":"\u0009at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:292)\n","stream":"stdout","time":"2021-03-28T21:39:03.62524688Z"}
{"log":"\u0009at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:287)\n","stream":"stdout","time":"2021-03-28T21:39:03.625262842Z"}
{"log":"\u0009at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1356)\n","stream":"stdout","time":"2021-03-28T21:39:03.625278972Z"}
{"log":"\u0009at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1231)\n","stream":"stdout","time":"2021-03-28T21:39:03.625295805Z"}
{"log":"\u0009at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1174)\n","stream":"stdout","time":"2021-03-28T21:39:03.625316768Z"}
{"log":"\u0009at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392)\n","stream":"stdout","time":"2021-03-28T21:39:03.625334638Z"}
{"log":"\u0009at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:443)\n","stream":"stdout","time":"2021-03-28T21:39:03.625350878Z"}
{"log":"\u0009at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1074)\n","stream":"stdout","time":"2021-03-28T21:39:03.62536773Z"}
{"log":"\u0009at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1061)\n","stream":"stdout","time":"2021-03-28T21:39:03.625384822Z"}
{"log":"\u0009at java.base/java.security.AccessController.doPrivileged(Native Method)\n","stream":"stdout","time":"2021-03-28T21:39:03.625401192Z"}
{"log":"\u0009at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1008)\n","stream":"stdout","time":"2021-03-28T21:39:03.625453136Z"}
{"log":"\u0009at org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:430)\n","stream":"stdout","time":"2021-03-28T21:39:03.625476451Z"}
{"log":"\u0009at org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:514)\n","stream":"stdout","time":"2021-03-28T21:39:03.625493247Z"}
{"log":"\u0009at org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:368)\n","stream":"stdout","time":"2021-03-28T21:39:03.625509543Z"}
{"log":"\u0009at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:291)\n","stream":"stdout","time":"2021-03-28T21:39:03.625525542Z"}
{"log":"\u0009at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:173)\n","stream":"stdout","time":"2021-03-28T21:39:03.625541672Z"}
{"log":"\u0009at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:547)\n","stream":"stdout","time":"2021-03-28T21:39:03.625557746Z"}
{"log":"\u0009at org.apache.kafka.common.network.Selector.poll(Selector.java:485)\n","stream":"stdout","time":"2021-03-28T21:39:03.625574079Z"}
{"log":"\u0009at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:544)\n","stream":"stdout","time":"2021-03-28T21:39:03.625590005Z"}
{"log":"\u0009at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1293)\n","stream":"stdout","time":"2021-03-28T21:39:03.62560569Z"}
{"log":"\u0009at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1224)\n","stream":"stdout","time":"2021-03-28T21:39:03.625622504Z"}
{"log":"\u0009at java.base/java.lang.Thread.run(Thread.java:834)\n","stream":"stdout","time":"2021-03-28T21:39:03.625638782Z"}
{"log":"Caused by: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors\n","stream":"stdout","time":"2021-03-28T21:39:03.625655485Z"}
{"log":"\u0009at java.base/sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:369)\n","stream":"stdout","time":"2021-03-28T21:39:03.625672189Z"}
{"log":"\u0009at java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:275)\n","stream":"stdout","time":"2021-03-28T21:39:03.625688337Z"}
{"log":"\u0009at java.base/sun.security.validator.Validator.validate(Validator.java:264)\n","stream":"stdout","time":"2021-03-28T21:39:03.625704707Z"}
{"log":"\u0009at java.base/sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:313)\n","stream":"stdout","time":"2021-03-28T21:39:03.625729021Z"}
{"log":"\u0009at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:276)\n","stream":"stdout","time":"2021-03-28T21:39:03.625748169Z"}
{"log":"\u0009at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:141)\n","stream":"stdout","time":"2021-03-28T21:39:03.625764928Z"}
{"log":"\u0009at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1334)\n","stream":"stdout","time":"2021-03-28T21:39:03.625781317Z"}
{"log":"\u0009... 19 more\n","stream":"stdout","time":"2021-03-28T21:39:03.625797965Z"}
{"log":"Caused by: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors\n","stream":"stdout","time":"2021-03-28T21:39:03.625814002Z"}
{"log":"\u0009at java.base/sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:158)\n","stream":"stdout","time":"2021-03-28T21:39:03.625830168Z"}
{"log":"\u0009at java.base/sun.security.provider.certpath.PKIXCertPathValidator.engineValidate(PKIXCertPathValidator.java:84)\n","stream":"stdout","time":"2021-03-28T21:39:03.625846594Z"}
{"log":"\u0009at java.base/java.security.cert.CertPathValidator.validate(CertPathValidator.java:309)\n","stream":"stdout","time":"2021-03-28T21:39:03.625863038Z"}
{"log":"\u0009at java.base/sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:364)\n","stream":"stdout","time":"2021-03-28T21:39:03.62593037Z"}
{"log":"\u0009... 25 more\n","stream":"stdout","time":"2021-03-28T21:39:03.625958296Z"}

Certificate of Kafka

debian@cloudcore:~$ openssl s_client -connect kafka:9093 -showcerts
CONNECTED(00000003)
Can't use SSL_get_servername
depth=1 O = io.strimzi, CN = cluster-ca v0
verify error:num=19:self signed certificate in certificate chain
verify return:1
depth=1 O = io.strimzi, CN = cluster-ca v0
verify return:1
depth=0 O = io.strimzi, CN = cloud-kafka-kafka
verify return:1
---
Certificate chain
 0 s:O = io.strimzi, CN = cloud-kafka-kafka
   i:O = io.strimzi, CN = cluster-ca v0
-----BEGIN CERTIFICATE-----
MIIFADCCA+igAwIBAgIJAKRqkvEZdOLEMA0GCSqGSIb3DQEBCwUAMC0xEzARBgNV
BAoMCmlvLnN0cmltemkxFjAUBgNVBAMMDWNsdXN0ZXItY2EgdjAwHhcNMjEwMzI3
MTgyMDQ2WhcNMjIwMzI3MTgyMDQ2WjAxMRMwEQYDVQQKDAppby5zdHJpbXppMRow
GAYDVQQDDBFjbG91ZC1rYWZrYS1rYWZrYTCCASIwDQYJKoZIhvcNAQEBBQADggEP
ADCCAQoCggEBAMsRevDhM4qbav9K7MTsQY7QpIW3EGmCX6NlqWDpp1l7+1yXRbRp
rw4EJv1DdRmg1/QEAm8Wx3+xUQtGZwfyXGNiBLuM+0apMIyVgijMYYe0yuVqtDfH
rwfg6oEzbR0NnwtArBtLSd9SWiEzmYt73uTm3r0WwNyJ+6kppjY9OIUnRyJMZAWU
0nDNMF+xTRHy48QTLWLnitWwRvcJEr+U4ThxUfRANLgOuBK+buR+F6gVwehMiSeN
cpjz/nDV1nefsFtqj3VVKeMEfzSa3YSceNjAH1hfBlJRiOKq6xcnuvZXgTxbt63A
kJmwfikvIcMB3Rt6NcRrOsvxSoycH8A0c50CAwEAAaOCAh0wggIZMIICFQYDVR0R
BIICDDCCAgiCLmNsb3VkLWthZmthLWthZmthLWJyb2tlcnMubXkta2Fma2EtcHJv
amVjdC5zdmOCPGNsb3VkLWthZmthLWthZmthLWJyb2tlcnMubXkta2Fma2EtcHJv
amVjdC5zdmMuY2x1c3Rlci5sb2NhbIIZY2xvdWQta2Fma2Eta2Fma2EtYnJva2Vy
c4IqY2xvdWQta2Fma2Eta2Fma2EtYnJva2Vycy5teS1rYWZrYS1wcm9qZWN0glBj
bG91ZC1rYWZrYS1rYWZrYS0wLmNsb3VkLWthZmthLWthZmthLWJyb2tlcnMubXkt
a2Fma2EtcHJvamVjdC5zdmMuY2x1c3Rlci5sb2NhbIIwY2xvdWQta2Fma2Eta2Fm
a2EtYm9vdHN0cmFwLm15LWthZmthLXByb2plY3Quc3Zjgj5jbG91ZC1rYWZrYS1r
YWZrYS1ib290c3RyYXAubXkta2Fma2EtcHJvamVjdC5zdmMuY2x1c3Rlci5sb2Nh
bIIbY2xvdWQta2Fma2Eta2Fma2EtYm9vdHN0cmFwgixjbG91ZC1rYWZrYS1rYWZr
YS1ib290c3RyYXAubXkta2Fma2EtcHJvamVjdIJCY2xvdWQta2Fma2Eta2Fma2Et
MC5jbG91ZC1rYWZrYS1rYWZrYS1icm9rZXJzLm15LWthZmthLXByb2plY3Quc3Zj
MA0GCSqGSIb3DQEBCwUAA4IBAQB/RK+eZksUto8msXa4ZR9zvebTXXSlsEqcV4jL
Fy5A5lY2nKLnaDT+dzjbtn4X7xETntAbNA8XDR6Aa8jo6l7l5Q2NQyfpziUGSDD9
K7kK0UmIVqxaHXWcPJI7mpcd3BwI8wv5q/KPP/eWrWOxHgGaCm+yca0v7TLTSCdc
fbZe9INFc+U9wPMFra/ruWvphbPogdC0uuQVsNTlO9Ooo7RTs7XY3Ukz3qbhVTa2
AzccCmf+Zpgaa1vU9Ff0h/eSadWueLg9roslccni6FH1/yptNf7ULLJa5gGYadK/
jKw9FUckoSS8xnI6xwBd9gkaAqmnyI1PjNoxuWQMJR/xOSq9
-----END CERTIFICATE-----
 1 s:O = io.strimzi, CN = cluster-ca v0
   i:O = io.strimzi, CN = cluster-ca v0
-----BEGIN CERTIFICATE-----
MIIDLTCCAhWgAwIBAgIJAPNEvUiSQzO2MA0GCSqGSIb3DQEBCwUAMC0xEzARBgNV
BAoMCmlvLnN0cmltemkxFjAUBgNVBAMMDWNsdXN0ZXItY2EgdjAwHhcNMjEwMzI3
MTgyMDI3WhcNMjIwMzI3MTgyMDI3WjAtMRMwEQYDVQQKDAppby5zdHJpbXppMRYw
FAYDVQQDDA1jbHVzdGVyLWNhIHYwMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB
CgKCAQEAnZsyhEiEW5Hi291oBKXGt+DlNXoyURcTMGdvoKJSBOYpRBm+XeCfiqaw
e2G5aVYbzBgKe0+GhxElstYZhd85yutlhBvNOusvHVx00AWpVjclali1EQH71WyF
guapkBgUxukkOLX8DoMfne7fAuXpqf+XTZP/JsG0ZvVE9NgNkXPvuz74XohQbxQy
zAZcYsPEZc5kjJ1b78xuVVZvvvbxct/rSV5Oro2hhGSrTNYgNWerXBJUys9xEEh+
k4mymjR61LJwGaN79DUHvl2BngKwSEVCJKqzDKzrXmRqBe4Eq+kKhyNMzd1OHWxY
pIf0nnN9gvV1iR0kSwKsWwaIUU+hmwIDAQABo1AwTjAdBgNVHQ4EFgQUQ/os35Jj
JITtiC1lP/Ex6U79+14wHwYDVR0jBBgwFoAUQ/os35JjJITtiC1lP/Ex6U79+14w
DAYDVR0TBAUwAwEB/zANBgkqhkiG9w0BAQsFAAOCAQEAO9D/3BtJFG51jCjOc4Td
XL127REY25NPTsTM2T2cDl60+paLw21e8ghL2sKrCmCF0n4SR1r/Vjm13Y0W/9Dv
Iqd4eA6wSo9TN2OtS34L0fiwd6t43DULmwP4bGoIQvsvptPUE3OUh3eshMm2k/c5
/KZJNP5jZQdSBEpTxkwGr2zajOEEhz9YgH951uMABlZrxK7MQHHdq1tqR9dZRrWD
VeM9WuIL3d9FIeepZsh3JJuc0Ad93yf7Oe+DYV/rnIYOL0Bdj9BA6VC2pt04lt8a
/tbUJ4FZpTteWhcTkY1q46WKCzN0AQ6XlPEVhmTtT5e0EMfiCKV4cGV9UUXKYEGe
iw==
-----END CERTIFICATE-----
---
Server certificate
subject=O = io.strimzi, CN = cloud-kafka-kafka

issuer=O = io.strimzi, CN = cluster-ca v0

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 2638 bytes and written 363 bytes
Verification error: self signed certificate in certificate chain
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 19 (self signed certificate in certificate chain)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_256_GCM_SHA384
    Session-ID: 8211114F5FA50DF06BF2169D294D5017EAFC2192AAC56C9C4514B3BCB37E9B60
    Session-ID-ctx:
    Resumption PSK: 99491F3EC1F1ADF646475C831BC29E66968E62185B5645D84CBE2C15B2FA74428B92F05D0806E2DC83EAA8FED7EACD05
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 86400 (seconds)
    TLS session ticket:
    0000 - 02 6a ba c4 e3 7f 6f 5a-fe 7e 4d d6 53 08 47 a5   .j....oZ.~M.S.G.
    0010 - fd d3 c5 1e 7e 49 11 e4-b8 de fa 1b b6 2d f3 01   ....~I.......-..

    Start Time: 1616979759
    Timeout   : 7200 (sec)
    Verify return code: 19 (self signed certificate in certificate chain)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK

Certificate of Node-0101

debian@cloudcore:~$ openssl s_client -connect node-0101:9093 -showcerts
CONNECTED(00000003)
Can't use SSL_get_servername
depth=1 O = io.strimzi, CN = cluster-ca v0
verify error:num=19:self signed certificate in certificate chain
verify return:1
depth=1 O = io.strimzi, CN = cluster-ca v0
verify return:1
depth=0 O = io.strimzi, CN = edge-kafka-node-0101-kafka
verify return:1
---
Certificate chain
 0 s:O = io.strimzi, CN = edge-kafka-node-0101-kafka
   i:O = io.strimzi, CN = cluster-ca v0
-----BEGIN CERTIFICATE-----
MIIFdTCCBF2gAwIBAgIJALbBiiNh9oCmMA0GCSqGSIb3DQEBCwUAMC0xEzARBgNV
BAoMCmlvLnN0cmltemkxFjAUBgNVBAMMDWNsdXN0ZXItY2EgdjAwHhcNMjEwMzI3
MTgyMzMyWhcNMjIwMzI3MTgyMzMyWjA6MRMwEQYDVQQKDAppby5zdHJpbXppMSMw
IQYDVQQDDBplZGdlLWthZmthLW5vZGUtMDEwMS1rYWZrYTCCASIwDQYJKoZIhvcN
AQEBBQADggEPADCCAQoCggEBAN69HFCOnjLe/J8+JMhwFnwmgGKKOn5VjLgr/7rs
ru9Nj7RVv3x2hV5NkRo5mbxcM5/Rzx6O/Buz9iBvsxFJI0/FzwxG+v6HKIxUyi6Z
20sr63Y3kPpjctAkLcgog/ZyfD9FtgTJUmLpdhYTOV9XQ8QJFLvPAvlBHEFZrzxJ
f2o53adimrGd17W8B0GWIa1ikz/GlYxyCH6NS8cejvR1dLlaYsxaNY4RGcZXa2N0
mE0plqvdCD2bnHI2tNEQwsz/oxlzwdz8lFbzUgktZxK+YKhQNyl7RDzVMnKXEqry
KVPBmaG0WOqJtrFDSvPYyxxSpks84SgllJZo48+KX1+qNfcCAwEAAaOCAokwggKF
MIICgQYDVR0RBIICeDCCAnSCN2VkZ2Uta2Fma2Etbm9kZS0wMTAxLWthZmthLWJy
b2tlcnMubXkta2Fma2EtcHJvamVjdC5zdmOCRWVkZ2Uta2Fma2Etbm9kZS0wMTAx
LWthZmthLWJyb2tlcnMubXkta2Fma2EtcHJvamVjdC5zdmMuY2x1c3Rlci5sb2Nh
bIIiZWRnZS1rYWZrYS1ub2RlLTAxMDEta2Fma2EtYnJva2Vyc4IzZWRnZS1rYWZr
YS1ub2RlLTAxMDEta2Fma2EtYnJva2Vycy5teS1rYWZrYS1wcm9qZWN0gmJlZGdl
LWthZmthLW5vZGUtMDEwMS1rYWZrYS0wLmVkZ2Uta2Fma2Etbm9kZS0wMTAxLWth
ZmthLWJyb2tlcnMubXkta2Fma2EtcHJvamVjdC5zdmMuY2x1c3Rlci5sb2NhbII5
ZWRnZS1rYWZrYS1ub2RlLTAxMDEta2Fma2EtYm9vdHN0cmFwLm15LWthZmthLXBy
b2plY3Quc3ZjgkdlZGdlLWthZmthLW5vZGUtMDEwMS1rYWZrYS1ib290c3RyYXAu
bXkta2Fma2EtcHJvamVjdC5zdmMuY2x1c3Rlci5sb2NhbIIkZWRnZS1rYWZrYS1u
b2RlLTAxMDEta2Fma2EtYm9vdHN0cmFwgjVlZGdlLWthZmthLW5vZGUtMDEwMS1r
YWZrYS1ib290c3RyYXAubXkta2Fma2EtcHJvamVjdIJUZWRnZS1rYWZrYS1ub2Rl
LTAxMDEta2Fma2EtMC5lZGdlLWthZmthLW5vZGUtMDEwMS1rYWZrYS1icm9rZXJz
Lm15LWthZmthLXByb2plY3Quc3ZjMA0GCSqGSIb3DQEBCwUAA4IBAQAQ8ZLiW0ct
JXzXWFnzxo1u1f7Y6nGAXd/fB6ppTOBtU8ExhiPW9dX4wq7xKakkCQGMclYxCABX
qd+Qv5xn1ZJlTeZMLMuO2Zg4BtY3FZtTMDJIvBav/mKHptOHch1qBgJT72G0khcz
dN2XzWghDVsRVRq56TK4KUlaFkACwLqEWdgOpZD2jdleJ5UNg59HirScxL7u5aLc
SrKKPxtxSGtSRc2SmS/63AC0EITFQoJATQk3Pe/BRW0gVkW8BNsQHz9t/FElsmXu
CWJViuVH3V4ttedsDkHOtuZ/cAAvM1ZsRAmboavRTk5zhyRgGdmLBAQ2Bo3Rm8dT
YPl2GXkSbfJ2
-----END CERTIFICATE-----
 1 s:O = io.strimzi, CN = cluster-ca v0
   i:O = io.strimzi, CN = cluster-ca v0
-----BEGIN CERTIFICATE-----
MIIDLTCCAhWgAwIBAgIJAL5df9kpas3RMA0GCSqGSIb3DQEBCwUAMC0xEzARBgNV
BAoMCmlvLnN0cmltemkxFjAUBgNVBAMMDWNsdXN0ZXItY2EgdjAwHhcNMjEwMzI3
MTgyMjA3WhcNMjIwMzI3MTgyMjA3WjAtMRMwEQYDVQQKDAppby5zdHJpbXppMRYw
FAYDVQQDDA1jbHVzdGVyLWNhIHYwMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB
CgKCAQEA0EJXruOnTxQx4IgsP6iERtgM4BpBCriFP5fSTb87DnXq8XG6GXeGGX1M
//2YdyItgLl6iWXORoWMZaEAeYVBZtK50w0ncam4dXfr2d0fJ4B9RDp4BVmdGQH1
0Cgz7LLYqTTixtLtlVwpKxrDo7xYJ8Q/cC++bC50/omyUxdSxJOtwvP5lulNllKP
t9DooXbebef0AK7qUqaGIJIBpFCo4IxGPR34duSWZ5kcvb3XtQVKLacxWRO6i9+a
RXwXzckX2FDTW6AiN1N1s+YVWSYaYeo/qhr79pM1JVwH01u/FaxvhyO2OkqqE/ZF
MOgTYNaQ5HdsaxoCElNX5xZo2fMNnwIDAQABo1AwTjAdBgNVHQ4EFgQUuIv9zBvg
vbPU3rFq/xi+vG9khuIwHwYDVR0jBBgwFoAUuIv9zBvgvbPU3rFq/xi+vG9khuIw
DAYDVR0TBAUwAwEB/zANBgkqhkiG9w0BAQsFAAOCAQEAmTJuCuOSP6HFuKDF1AfH
P0x+d76q5OkQcSKJcXZZr9TtX03LmoaAwig8vHk8bkqSFaAfkyJmNOyD9hP6YMKH
JxnffiPDR+XUNrBJDjzG0mm6z3nVrb9jxn0vYyAA7wVO5yip8YcUEdLaMdRyMNBr
om5Q2puMp5HfNNi3kVH1FQhfUYdY713CU1FPNfHDen84J2WuY9VYFgsqBv56HxH0
+hlFimdOt+HPP5ebJPho5yyLjh4j/HwGD7E/53P3uC+6BO4wyBvCTG0m7cpy/zPO
ZOV8gylx5FdSWuconVjpFHiFEqqLbgLnP9VrjxkBSO9jjMECY6v1lbopnksWbRji
jQ==
-----END CERTIFICATE-----
---
Server certificate
subject=O = io.strimzi, CN = edge-kafka-node-0101-kafka

issuer=O = io.strimzi, CN = cluster-ca v0

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 2755 bytes and written 363 bytes
Verification error: self signed certificate in certificate chain
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 19 (self signed certificate in certificate chain)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_256_GCM_SHA384
    Session-ID: A98DC5263AF8E73025F7B38AD3362D486A2E0B35FE988B71FB170E13240A5406
    Session-ID-ctx:
    Resumption PSK: D60CF335013001E8DF0AFBD22F2CE698F9C8B5881E08647B3DE77AEADD38A4370B9CA1EE3D3704F9CFE346E9AB50F756
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 86400 (seconds)
    TLS session ticket:
    0000 - f8 d1 c7 ce 99 81 03 08-41 4a c0 b5 0b 67 31 6e   ........AJ...g1n
    0010 - 77 e5 10 a6 e6 37 41 d4-f2 1e 10 f0 e3 9f a8 79   w....7A........y

    Start Time: 1616979943
    Timeout   : 7200 (sec)
    Verify return code: 19 (self signed certificate in certificate chain)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK

[scholzj]

I think the PKIX Path Validation suggests that the ca.crt you configure it with does not match the one used by the broker. Your bootstrapServers are also non-standard. So you will probably have also issues with TLS hostname verification. You should probably compare what you have in the secret with what is the openssl command return you.

[scholzj]

Sorry, but what p12 are you comparing it with?

[bitvijays]

Hi @scholzj If we compare the ca-certificate of the cloud-kafka found with openssl s_client -connect kafka:9093 -showcerts with the certificate stored in p12 of cloud-kafka-cluster-ca-cert, it doesn't match.

Q1: It seems that both the certificates of cloud-kafka and edge-kafka are signed with different ca. They should have been signed with the same ca. Right?
Q2: If there's a Kafka cluster (let's say Cloud) signed by a cloud-ca , the certificate in p12 of Cloud-cluster-ca-cert should match Right?

ca.crt of cloud-kafka from kafka:9093

MIIDLTCCAhWgAwIBAgIJAPNEvUiSQzO2MA0GCSqGSIb3DQEBCwUAMC0xEzARBgNV
BAoMCmlvLnN0cmltemkxFjAUBgNVBAMMDWNsdXN0ZXItY2EgdjAwHhcNMjEwMzI3
MTgyMDI3WhcNMjIwMzI3MTgyMDI3WjAtMRMwEQYDVQQKDAppby5zdHJpbXppMRYw
FAYDVQQDDA1jbHVzdGVyLWNhIHYwMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB
CgKCAQEAnZsyhEiEW5Hi291oBKXGt+DlNXoyURcTMGdvoKJSBOYpRBm+XeCfiqaw
e2G5aVYbzBgKe0+GhxElstYZhd85yutlhBvNOusvHVx00AWpVjclali1EQH71WyF
guapkBgUxukkOLX8DoMfne7fAuXpqf+XTZP/JsG0ZvVE9NgNkXPvuz74XohQbxQy
zAZcYsPEZc5kjJ1b78xuVVZvvvbxct/rSV5Oro2hhGSrTNYgNWerXBJUys9xEEh+
k4mymjR61LJwGaN79DUHvl2BngKwSEVCJKqzDKzrXmRqBe4Eq+kKhyNMzd1OHWxY
pIf0nnN9gvV1iR0kSwKsWwaIUU+hmwIDAQABo1AwTjAdBgNVHQ4EFgQUQ/os35Jj
JITtiC1lP/Ex6U79+14wHwYDVR0jBBgwFoAUQ/os35JjJITtiC1lP/Ex6U79+14w
DAYDVR0TBAUwAwEB/zANBgkqhkiG9w0BAQsFAAOCAQEAO9D/3BtJFG51jCjOc4Td
XL127REY25NPTsTM2T2cDl60+paLw21e8ghL2sKrCmCF0n4SR1r/Vjm13Y0W/9Dv
Iqd4eA6wSo9TN2OtS34L0fiwd6t43DULmwP4bGoIQvsvptPUE3OUh3eshMm2k/c5
/KZJNP5jZQdSBEpTxkwGr2zajOEEhz9YgH951uMABlZrxK7MQHHdq1tqR9dZRrWD
VeM9WuIL3d9FIeepZsh3JJuc0Ad93yf7Oe+DYV/rnIYOL0Bdj9BA6VC2pt04lt8a
/tbUJ4FZpTteWhcTkY1q46WKCzN0AQ6XlPEVhmTtT5e0EMfiCKV4cGV9UUXKYEGe
iw==

ca.crt stored in p12 of cloud-kafka-cluster-ca-cert

MIIDLTCCAhWgAwIBAgIJAL5df9kpas3RMA0GCSqGSIb3DQEBCwUAMC0xEzARBgNV
BAoMCmlvLnN0cmltemkxFjAUBgNVBAMMDWNsdXN0ZXItY2EgdjAwHhcNMjEwMzI3
MTgyMjA3WhcNMjIwMzI3MTgyMjA3WjAtMRMwEQYDVQQKDAppby5zdHJpbXppMRYw
FAYDVQQDDA1jbHVzdGVyLWNhIHYwMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB
CgKCAQEA0EJXruOnTxQx4IgsP6iERtgM4BpBCriFP5fSTb87DnXq8XG6GXeGGX1M
//2YdyItgLl6iWXORoWMZaEAeYVBZtK50w0ncam4dXfr2d0fJ4B9RDp4BVmdGQH1
0Cgz7LLYqTTixtLtlVwpKxrDo7xYJ8Q/cC++bC50/omyUxdSxJOtwvP5lulNllKP
t9DooXbebef0AK7qUqaGIJIBpFCo4IxGPR34duSWZ5kcvb3XtQVKLacxWRO6i9+a
RXwXzckX2FDTW6AiN1N1s+YVWSYaYeo/qhr79pM1JVwH01u/FaxvhyO2OkqqE/ZF
MOgTYNaQ5HdsaxoCElNX5xZo2fMNnwIDAQABo1AwTjAdBgNVHQ4EFgQUuIv9zBvg
vbPU3rFq/xi+vG9khuIwHwYDVR0jBBgwFoAUuIv9zBvgvbPU3rFq/xi+vG9khuIw
DAYDVR0TBAUwAwEB/zANBgkqhkiG9w0BAQsFAAOCAQEAmTJuCuOSP6HFuKDF1AfH
P0x+d76q5OkQcSKJcXZZr9TtX03LmoaAwig8vHk8bkqSFaAfkyJmNOyD9hP6YMKH
JxnffiPDR+XUNrBJDjzG0mm6z3nVrb9jxn0vYyAA7wVO5yip8YcUEdLaMdRyMNBr
om5Q2puMp5HfNNi3kVH1FQhfUYdY713CU1FPNfHDen84J2WuY9VYFgsqBv56HxH0
+hlFimdOt+HPP5ebJPho5yyLjh4j/HwGD7E/53P3uC+6BO4wyBvCTG0m7cpy/zPO
ZOV8gylx5FdSWuconVjpFHiFEqqLbgLnP9VrjxkBSO9jjMECY6v1lbopnksWbRji
jQ==

[scholzj]

The p12 file is not used for anything here. What matters is the .crt file. So does the ca.crt file from the secret correspond to the CA you get from OpenSSL or not?

[bitvijays]

Yes @scholzj . The contents of .crt file in cloud-kafka-cluster-ca-cert matches with the ca-cert obtained from openssl.

[scholzj]

Then it is maybe just the hostnames what are the problem? The JSON log is quite hard to read. Have you tried to disable the hostname verification / add the bootstrap hostnames to the certificates using the alternativeNames option?