chore(outputs.sql): Make default create table template suitable for ClickHouse #16464
Conversation
…lickHouse
This changes the default TableTemplate of the SQL output plugin so that
it can be used with ClickHouse. Previously it was invalid for ClickHouse
because it didn't include a PRIMARY KEY or ORDER BY clause, which is
required for ClickHouse.
This then also allows to remove the custom TableTemplate from the
ClickHouse unit test.
This also adds another placeholder {TAG_COLUMNS} that allows to get a
list of all tag columns in a TableTemplate. This can then be used to use
the tag columns as a primary or sort key, especially in ClickHouse.
telegraf-tiger
bot
added
area/sql
chore
plugin/output
AndreKR
changed the title
|
Download PR build artifacts for linux_amd64.tar.gz, darwin_arm64.tar.gz, and windows_amd64.zip. 📦 Click here to get additional PR build artifactsArtifact URLs |
There was a problem hiding this comment.
@AndreKR thanks for your effort! Why don't we simple modify the default table_template for Clickhouse to
## Note: The default template for clickhouse is
## table_template = "CREATE TABLE {TABLE}({COLUMNS}) ORDER BY (<tag columns>, <time column>)"
## for all other drivers the default is
# table_template = "CREATE TABLE {TABLE}({COLUMNS})"
That's because I didn't find a good way to make the default table template dependent on the driver. The default table template comes from |
|
By the way, this (the test to be precise) depends on #16462, so marking this as a draft for now. |
Summary
The SQL output plugin automatically creates the required table in the target database. For this it uses a default create table template that currently has the value
CREATE TABLE {TABLE}({COLUMNS})where{TABLE}is the metric name and{COLUMNS}is a list of column name/type pairs. The plugin has a settingtable_templateto override this template.ClickHouse requires that in a CREATE TABLE statement a primary key or a sort key is given, thus the default template above does not work with the ClickHouse database.
There is a section about ClickHouse-specific configuration in the README, but it doesn't mention a mandatory
table_template.The integration test solves this issue by hardcoding an override for the template that reads:
CREATE TABLE {TABLE}({COLUMNS}) ENGINE MergeTree() ORDER by timestamp. In other words, it specifies the timestamp as the sort key.Specifying only the timestamp as sort key is a valid approach, but it is not recommended in the ClickHouse world because the timestamp has a high cardinality.
This PR makes the following changes:
CREATE TABLE {TABLE}({COLUMNS})toCREATE TABLE {TABLE}({COLUMNS}) {SORT_KEY_CLAUSE}. The placeholder{SORT_KEY_CLAUSE}will contain a list of all tag columns when the database is ClickHouse. It will be removed (replaced with an empty string) for any other database.{TAG_COLUMNS}that contains the list of tag columns. You can use this in yourtable_templatefor example to build a CREATE TABLE statement with extra columns, like this:CREATE TABLE {TABLE} ({COLUMNS}, my_extra_column int) ORDER BY ({TAG_COLUMNS}, my_extra_column)table_templateoverride from the integration test because it is no longer needed to make the test work.Checklist
Related issues
resolves #16463