Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make SQL output plugin work with ClickHouse out of the box #16463

Open
AndreKR opened this issue Feb 2, 2025 · 0 comments · May be fixed by #16464
Open

Make SQL output plugin work with ClickHouse out of the box #16463

AndreKR opened this issue Feb 2, 2025 · 0 comments · May be fixed by #16464
Labels
feature request Requests for new plugin and for new features to existing plugins

Comments

@AndreKR
Copy link
Contributor

AndreKR commented Feb 2, 2025

Use Case

The SQL output plugin automatically creates the required table in the target database. For this it uses a default create table template that currently has the value CREATE TABLE {TABLE}({COLUMNS}) where {TABLE} is the metric name and {COLUMNS} is a list of column name/type pairs. The plugin has a setting table_template to override this template.

Expected behavior

I would expect this to work with every supported database, including ClickHouse.

Actual behavior

ClickHouse requires that in a CREATE TABLE statement a primary key or a sort key is given, thus the default template above does not work with the ClickHouse database.

There is a section about ClickHouse-specific configuration in the README, but it doesn't mention a mandatory table_template.

The integration test solves this issue by hardcoding an override for the template that reads: CREATE TABLE {TABLE}({COLUMNS}) ENGINE MergeTree() ORDER by timestamp In other words, it specifies the timestamp as the sort key.

Specifying only the timestamp as sort key is a valid approach, but it is not recommended in the ClickHouse world because the timestamp has a high cardinality.

Additional info

What I would like to see (and provide in the PR I created) is this:

  • The default template works out of the box with ClickHouse.
  • When setting your own table_template you have access to a list of tag columns that you can use as a low-cardinality key instead or in addition to the timestamp.
@AndreKR AndreKR added the feature request Requests for new plugin and for new features to existing plugins label Feb 2, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request Requests for new plugin and for new features to existing plugins
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant