External listener with type ingress does not works with class: istio

[naddame]

Hi,

I have installed strimzi kafka in the namespace kafka-uat
I have already using istio (gateway) in all other projects. Istio is installed in the namespace istio-system

I want to expose the kafka cluster to external world using istio gateway. Creating istio ingress is so easy, as we have juste to define ingressClassName: istio in the spec of ingress yaml file.

Following your documentation, and as i specified class: istio in configuration it suppose to create a simple ingress file with ingressClassName: istio but it continue to try using nginx. See errors bellow.

Questions:

• Why the operator does not create a simple ingress using ingressClassName: istio ? if it can, what wrong in my config ?
• What the behaviour of disabling external ingress listener and creating manually external services and expose them using istio gateway?
• Enabling the ingress listener generate external services but no ingress created. We can still have unwanted error logs bellow, but what the behaviour if i create the ingress manually for each broker and boostrap ?

Error:
2021-05-14 11:28:02 INFO AbstractOperator:255 - Reconciliation #17(watch) Kafka(kafka-uat/kafka-cluster): Kafka kafka-cluster will be checked for creation or modification 2021-05-14 11:28:04 ERROR AbstractOperator:276 - Reconciliation #17(watch) Kafka(kafka-uat/kafka-cluster): createOrUpdate failed io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://10.0.0.1/apis/networking.k8s.io/v1beta1/namespaces/kafka-uat/ingresses. Message: Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": Post https://ingress-nginx-controller-admission.ingress-nginx.svc:443/networking/v1beta1/ingresses?timeout=10s: service "ingress-nginx-controller-admission" not found. Received status: Status(apiVersion=v1, code=500, details=StatusDetails(causes=[StatusCause(field=null, message=failed calling webhook "validate.nginx.ingress.kubernetes.io": Post https://ingress-nginx-controller-admission.ingress-nginx.svc:443/networking/v1beta1/ingresses?timeout=10s: service "ingress-nginx-controller-admission" not found, reason=null, additionalProperties={})], group=null, kind=null, name=null, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": Post https://ingress-nginx-controller-admission.ingress-nginx.svc:443/networking/v1beta1/ingresses?timeout=10s: service "ingress-nginx-controller-admission" not found, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=InternalError, status=Failure, additionalProperties={}).

Config of external listener:

`- name: external

    port: 9094
    type: ingress
    tls: true
    # Extract the public certificate of the broker certificate authority
    # kubectl get secret KAFKA-CLUSTER-NAME-cluster-ca-cert -o jsonpath='{.data.ca\.crt}' | base64 -d > ca.crt
    authentication:
      type: tls
    configuration:
      **class: istio**
      bootstrap:
        host: {{ current_service.hosts.bootstrap }}
        annotations:
          external-dns.alpha.kubernetes.io/hostname: {{ current_service.hosts.bootstrap }}.
          external-dns.alpha.kubernetes.io/ttl: "60"
      brokers:
      - broker: 0
        host: {{ current_service.hosts.borker_0 }}
        annotations:
          external-dns.alpha.kubernetes.io/hostname: {{ current_service.hosts.borker_0 }}.
          external-dns.alpha.kubernetes.io/ttl: "60"
      - broker: 1
        host: {{ current_service.hosts.borker_1 }}
        annotations:
          external-dns.alpha.kubernetes.io/hostname: {{ current_service.hosts.borker_1 }}.
          external-dns.alpha.kubernetes.io/ttl: "60"
      - broker: 2
        host: {{ current_service.hosts.borker_2 }}
        annotations:
          external-dns.alpha.kubernetes.io/hostname: {{ current_service.hosts.borker_2 }}.
          external-dns.alpha.kubernetes.io/ttl: "60"`

[scholzj]

Strimzi is written and tested to work with Kubernetes Nginx Ingress. The different Ingress controllers are not fully compatible. I know from our Slack channel and from other issues that there were users trying to get this working and some were successful, but it is not completely trivial I guess.

You also did not share absolutely no information about what versions are you using. But in general:

Why the operator does not create a simple ingress using ingressClassName: istio ? if it can, what wrong in my config ?

It is used as the Ingress class name.

What the behaviour of disabling external ingress listener and creating manually external services and expose them using istio gateway?

You can create internal listener for example, override the advertised hostnames and port and create the Ingress resources manually.

Enabling the ingress listener generate external services but no ingress created. We can still have unwanted error logs bellow, but what the behaviour if i create the ingress manually for each broker and boostrap ?

The error suggests that you have Nginx Ingress Validation Webhook configured but not running or not working. This is issue of your environment which you would need to solve.

[naddame]

Thank you for your quick response.
I'm using the latest operator version 0.23.0. and kafka version 2.8.0.

You can create internal listener for example, override the advertised hostnames and port and create the Ingress resources manually.

Have you any example, link or demo of how to do that overrides? Is it session safe?

Thanks in advance

[scholzj]

I do not have example of what exactly you are asking for. But the example of the advertised host / port is here: https://strimzi.io/docs/operators/latest/full/using.html#type-GenericKafkaListenerConfigurationBroker-reference

I'm not sure what exactly you mean with session safe - it really just lets you manually defined the advertised listeners address in the Kafka brokers to use any address you provide (so for example the address of the manually created Ingress resources).

[naddame]

I have deleted the nginx ValidatingWebhookConfiguration that cause the errors above and ingress are created successfully.

Thanks

[naddame]

For information: i can't connect to the cluster with tls + authentication enabled using default generate ingress with class istio.

Instead i have deleted all generated ingresses and created istio gateway and virtualservice to be able to connect successfully.

Here is an example : I'm using ansible to generate yml file, so just change variables {{....}} with correct informations

---

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
name: {{ current_service.name }}-gateway
namespace: {{ kafka_namespace }}
spec:
selector:
istio: ingressgateway # use istio default ingress gateway
servers:
- port:
number: 443
name: https
protocol: HTTPS
tls:
mode: PASSTHROUGH
hosts:
- {{ current_service.hosts.bootstrap }}
- {{ current_service.hosts.borker_0 }}
- {{ current_service.hosts.borker_1 }}
- {{ current_service.hosts.borker_2 }}

---

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: {{ current_service.name }}-ingress
namespace: {{ kafka_namespace }}
labels:
service: {{ current_service.name }}-bootstrap-ingress
spec:
hosts:
- {{ current_service.hosts.bootstrap }}
- {{ current_service.hosts.borker_0 }}
- {{ current_service.hosts.borker_1 }}
- {{ current_service.hosts.borker_2 }}
gateways:

• {{ current_service.name }}-gateway
  tls:
• match:
  • port: 443
    sniHosts:
    • {{ current_service.hosts.bootstrap }}
      route:
  • destination:
    host: {{ current_service.name }}-kafka-external-bootstrap
    port:
    number: 9094
• match:
  • port: 443
    sniHosts:
    • {{ current_service.hosts.borker_0 }}
      route:
  • destination:
    host: {{ current_service.name }}-kafka-0
    port:
    number: 9094
• match:
  • port: 443
    sniHosts:
    • {{ current_service.hosts.borker_1 }}
      route:
  • destination:
    host: {{ current_service.name }}-kafka-1
    port:
    number: 9094
• match:
  • port: 443
    sniHosts:
    • {{ current_service.hosts.borker_2 }}
      route:
  • destination:
    host: {{ current_service.name }}-kafka-2
    port:
    number: 9094