Strimzi
Problem to Get Strimzi KafkaConnect to work with AWS MSK Kafka #4983
Answered
by
jinzishuai
jinzishuai
asked this question in
Q&A
-
|
Hi there, I am working on a PoC where
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
name: dv-msk
annotations:
strimzi.io/use-connector-resources: "true"
spec:
image: <redacted: customer image based on strimzi-0.23-kafka-0.28 with debzium mysql plugin>
replicas: 1
bootstrapServers: <redacted: info from AWS MSK>
config:
config.storage.replication.factor: 1
offset.storage.replication.factor: 1
status.storage.replication.factor: 1
config.providers: file
config.providers.file.class: org.apache.kafka.common.config.provider.FileConfigProvider
externalConfiguration:
volumes:
- name: secrets
secret:
secretName: kafka-connect
metricsConfig:
type: jmxPrometheusExporter
valueFrom:
configMapKeyRef:
name: kafka-connect-metrics
key: metrics-config.yml
apiVersion: "kafka.strimzi.io/v1beta2"
kind: "KafkaConnector"
metadata:
name: test-msk-mysql-1621283499
labels:
strimzi.io/cluster: dv-msk
spec:
class: io.debezium.connector.mysql.MySqlConnector
tasksMax: 1
config:
database.hostname: "<my-sql ip>"
database.port: "3306"
database.user: "${file:/opt/kafka/external-configuration/secrets/tenant.mysql:username}"
database.password: "${file:/opt/kafka/external-configuration/secrets/tenant.mysql:password}"
database.server.id: "1621283499"
database.server.name: "msk"
database.history.kafka.bootstrap.servers: <redacted: info from AWS MSK>
database.history.producer.security.protocol: PLAINTEXT
database.history.kafka.topic: "single-msk"
database.history.store.only.monitored.tables.ddl: "true"
include.schema.changes: "true"
decimal.handling.mode: "string"
database.whitelist: "test_test1_main"
include.query: "true"
inconsistent.schema.handling.mode: "warn"
event.processing.failure.handling.mode: "warn"
database.history.skip.unparseable.ddl: "true"
table.whitelist: >
^test_test1_main.user$,
^test_test1_main.site$When the above are apply, the I do see a topic Thanks a lot. |
Beta Was this translation helpful? Give feedback.
Answered by
jinzishuai
May 18, 2021
Replies: 2 comments 10 replies
-
|
I think it is this annotation changing it to |
Beta Was this translation helpful? Give feedback.
9 replies
-
|
Well, that is expected ... setting the annotation to |
Beta Was this translation helpful? Give feedback.
-
|
right, I have set it back to true and the KCTR are shown as state of RUNNING but then I am at getting lot of |
Beta Was this translation helpful? Give feedback.
-
|
I think that means that it cannot create the topic / find the leader. But no idea what would be the issue ... does the Kafka cluster show something more in logs? |
Beta Was this translation helpful? Give feedback.
-
|
no, I cannot see anything useful from the MSK log |
Beta Was this translation helpful? Give feedback.
-
|
@scholzj does the debzium plugins log anywhere else? |
Beta Was this translation helpful? Give feedback.
-
|
I finally got it. In fact it is the first line in the MSK config that disables topic auto creation. All I needed is |
Beta Was this translation helpful? Give feedback.
1 reply
-
|
Hmm, interesting. I thought Debezium creates its own topics directly. Maybe they changed it. Anyway, glad it works for you. |
Beta Was this translation helpful? Give feedback.
Answer selected by
jinzishuai
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I finally got it. In fact it is the first line in the MSK config that disables topic auto creation.
All I needed is