Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Tag name prevents write? #25915

Open
danatinflux opened this issue Jan 26, 2025 · 2 comments
Open

Tag name prevents write? #25915

danatinflux opened this issue Jan 26, 2025 · 2 comments
Labels
v3

Comments

@danatinflux
Copy link

When I try to write the following LP, I get gateway errors and it fails to write to the db:

alblogs,lb=test1,request=foobar rectime="2025-01-24T22:19:27.441768Z",client="xx.xx.xx.140:55678",target="xx.xx.xx:8181",request_ptime=0.001,target_ptime=0.001,response_ptime=0.000,elb_status_code=400,target_status_code=400,recd_bytes=304,sent_bytes=263,request="POST",agent="http:__backend.stickontheice.com:8181_write?db=telegraf_test&precision=s",action="forward"

However, if I change the tag from 'request' to 'requestname' it goes right through:

alblogs,lb=test1,requestname=foobar rectime="2025-01-24T22:19:27.441768Z",client="xx.xx.xx.140:55678",target="xx.xx.xx.193:8181",request_ptime=0.001,target_ptime=0.001,response_ptime=0.000,elb_status_code=400,target_status_code=400,recd_bytes=304,sent_bytes=263,request="POST",agent="http:__backend.stickontheice.com:8181_write?db=telegraf_test&precision=s",action="forward"

Test environment:

AWS ALB -> nginx (to provide password protection) -> two EC2 instances running the software in docker.

Command (password implemented in Nginx):

curl -iv  -X POST  -H "Authorization: Bearer TOKEN"  "http://backend.stickontheice.com:8181/write?db=testdb2&precision=ns" --data-binary @${1}

Results from sending LP with the 'request=foobar' tag:

ddobbs@carbonara:/u01/aws_v3_setup/write_tests$ ./sendit.sh bad_lp.txt 
Note: Unnecessary use of -X or --request, POST is already inferred.
* Host backend.stickontheice.com:8181 was resolved.
* IPv6: (none)
* IPv4: 52.86.239.136, 18.233.59.62
*   Trying 52.86.239.136:8181...
* Connected to backend.stickontheice.com (52.86.239.136) port 8181
> POST /write?db=testdb2&precision=ns HTTP/1.1
> Host: backend.stickontheice.com:8181
> User-Agent: curl/8.9.1
> Accept: */*
> Authorization: Bearer TOKEN
> Content-Length: 372
> Content-Type: application/x-www-form-urlencoded
> 
* upload completely sent off: 372 bytes
< HTTP/1.1 502 Bad Gateway
HTTP/1.1 502 Bad Gateway
< Date: Sun, 26 Jan 2025 20:32:46 GMT
Date: Sun, 26 Jan 2025 20:32:46 GMT
< Content-Type: text/html
Content-Type: text/html
< Content-Length: 166
Content-Length: 166
< Connection: keep-alive
Connection: keep-alive
< Server: nginx/1.24.0 (Ubuntu)
Server: nginx/1.24.0 (Ubuntu)
< 

<html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.24.0 (Ubuntu)</center>
</body>
</html>
* Connection #0 to host backend.stickontheice.com left intact

I reattempted to ensure it failed on both gateways.

Results when that tag is changed:

ddobbs@carbonara:/u01/aws_v3_setup/write_tests$ ./sendit.sh good_lp.txt 
Note: Unnecessary use of -X or --request, POST is already inferred.
* Host backend.stickontheice.com:8181 was resolved.
* IPv6: (none)
* IPv4: 52.86.239.136, 18.233.59.62
*   Trying 52.86.239.136:8181...
* Connected to backend.stickontheice.com (52.86.239.136) port 8181
> POST /write?db=testdb1&precision=ns HTTP/1.1
> Host: backend.stickontheice.com:8181
> User-Agent: curl/8.9.1
> Accept: */*
> Authorization: Bearer TOKEN
> Content-Length: 376
> Content-Type: application/x-www-form-urlencoded
> 
* upload completely sent off: 376 bytes
< HTTP/1.1 204 No Content
HTTP/1.1 204 No Content
< Date: Sun, 26 Jan 2025 20:31:54 GMT
Date: Sun, 26 Jan 2025 20:31:54 GMT
< Connection: keep-alive
Connection: keep-alive
< Server: nginx/1.24.0 (Ubuntu)
Server: nginx/1.24.0 (Ubuntu)
< 

* Connection #0 to host backend.stickontheice.com left intact

Script to create docker instance:

NAME=$(cat /etc/hostname)
BUCKET="v3-mybucket-test"
KEY="MYKEY"
SECRET="SECRET"
REGION="us-east-1"

OPTIONS="--mode read_write --plugin-dir /plugins --read-from-node-ids "
REPLICAS="v3-test-data3,v3-test-data4"


### run with -it to get license, then remove -it and add --detach afterwards
#
#docker run -it --name ${NAME} -p 8181:8181 -v ./plugins:/plugins influxdb3-enterprise serve  --object-store s3 --bucket ${BUCKET} \
#     	--aws-access-key-id ${KEY}   --aws-secret-access-key ${SECRET}  --aws-default-region ${REGION} --node-id ${NAME} ${OPTIONS} ${REPLICAS}


docker run -d --restart=always --name ${NAME} -p 8181:8181 -v ./plugins:/plugins influxdb3-enterprise serve  --object-store s3 --bucket ${BUCKET} \
     	--aws-access-key-id ${KEY}   --aws-secret-access-key ${SECRET}  --aws-default-region ${REGION} --node-id ${NAME} ${OPTIONS} ${REPLICAS}

I'm thinking this might be the error on the server side:

2025-01-26 12:32:46.604	
2025-01-26T20:32:46.604222Z ERROR panic_logging: Thread panic panic_type="unknown" panic_message="invalid column id in series key definition" panic_file="influxdb3_catalog/src/catalog.rs" panic_line=1028 panic_column=22
2025-01-26 12:32:46.604	
invalid column id in series key definition
2025-01-26 12:32:46.604	
thread 'InfluxDB 3 Enterprise Tokio IO 2' panicked at influxdb3_catalog/src/catalog.rs:1028:22:
2025-01-26 12:32:46.604	
2025-01-26T20:32:46.604133Z  INFO influxdb3_server::http: write_lp to testdb2
@pauldix
Copy link
Member

pauldix commented Jan 26, 2025

What happens if you write directly to the server? Can you capture what's going to the InfluxDB server? This looks like it might be a problem with the software you have in the middle (ALB or NGINX)

@pauldix pauldix added the v3 label Jan 26, 2025
@danatinflux
Copy link
Author

I spun up a whole new pair of servers, and attempted to write directly to them. This was the curl I used:

curl -iv  -X POST  "http://54.185.76.103:8181/write?db=testdb1&precision=ns" --data-binary @${1}

With the 'good' example, this was the output:

ddobbs@carbonara:/u01/aws_v3_setup/write_tests$ ./sendit.sh good_lp.txt 
Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying 54.185.76.103:8181...
* Connected to 54.185.76.103 (54.185.76.103) port 8181
> POST /write?db=testdb1&precision=ns HTTP/1.1
> Host: 54.185.76.103:8181
> User-Agent: curl/8.9.1
> Accept: */*
> Content-Length: 376
> Content-Type: application/x-www-form-urlencoded
> 
* upload completely sent off: 376 bytes
< HTTP/1.1 204 No Content
HTTP/1.1 204 No Content
< date: Sun, 26 Jan 2025 21:48:28 GMT
date: Sun, 26 Jan 2025 21:48:28 GMT
< 

* Connection #0 to host 54.185.76.103 left intact

I changed to 'testdb2' to ensure I didn't get 'tag poisoned' and executed the same command with a different payload:

ddobbs@carbonara:/u01/aws_v3_setup/write_tests$ ./sendit.sh bad_lp.txt 
Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying 54.185.76.103:8181...
* Connected to 54.185.76.103 (54.185.76.103) port 8181
> POST /write?db=testdb2&precision=ns HTTP/1.1
> Host: 54.185.76.103:8181
> User-Agent: curl/8.9.1
> Accept: */*
> Content-Length: 372
> Content-Type: application/x-www-form-urlencoded
> 
* upload completely sent off: 372 bytes
* Empty reply from server
* shutting down connection #0
curl: (52) Empty reply from server

It created the db successfully, but sent no data.

It appears to get a very similar error, though:

2025-01-26 13:49:12.233	
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
2025-01-26 13:49:12.233	
invalid column id in series key definition
2025-01-26 13:49:12.233	
thread 'InfluxDB 3 Enterprise Tokio IO 1' panicked at influxdb3_catalog/src/catalog.rs:1028:22:
2025-01-26 13:49:12.233	
2025-01-26T21:49:12.233615Z ERROR panic_logging: Thread panic panic_type="unknown" panic_message="invalid column id in series key definition" panic_file="influxdb3_catalog/src/catalog.rs" panic_line=1028 panic_column=22
2025-01-26 13:49:12.233	
2025-01-26T21:49:12.233572Z  INFO influxdb3_catalog::catalog: return new db testdb2
2025-01-26 13:49:12.233	
2025-01-26T21:49:12.233541Z  INFO influxdb3_server::http: write_lp to testdb2

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
v3
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@pauldix @danatinflux