Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](nereids) fix range inference generate an empty range #47434

Open
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

yujun777
Copy link
Collaborator

@yujun777 yujun777 commented Jan 24, 2025

What problem does this PR solve?

for a >= 8 and a <8 will generate a range(a>=8) interset range(a <8) = range[8, 8), this is a empty range.

PR #46303 introduce a RangeSet to hold the union of RangeValues, for a empty range[8, 8), RangeSet.asRanges() will return empty. Then cause a index out of bound exception.

Caused by: java.lang.IndexOutOfBoundsException: Index 0 out of bounds for length 0
        at jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:64) ~[?:?]
        at jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:70) ~[?:?]
        at jdk.internal.util.Preconditions.checkIndex(Preconditions.java:266) ~[?:?]
        at java.util.Objects.checkIndex(Objects.java:359) ~[?:?]
        at java.util.ArrayList.get(ArrayList.java:427) ~[?:?]
        at org.apache.doris.nereids.rules.expression.rules.RangeInference.simplify(RangeInference.java:166) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.rules.expression.rules.RangeInference.visitOr(RangeInference.java:130) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.rules.expression.rules.RangeInference.visitOr(RangeInference.java:59) ~[doris-fe.jar:1.2-SNAPSHOT]

Fix: for a empty RangeValue like [8, 8), replace it with an EmptyValue.

What's more, this PR will also replace singleton RangeValue like ([8, 8]) to DiscreteValue([8]), so after this PR, simplify range will rewrite a >= 8 and a <= 8 to a = 8. This is just BetweenToEqual rule do. After this PR, we can remove rule BetweenToEqual.

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Jan 24, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@yujun777
Copy link
Collaborator Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32516 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 58d5288a12bc13424a5db61d37d2364d557474c0, data reload: false

------ Round 1 ----------------------------------
q1	17609	5598	5531	5531
q2	2047	326	185	185
q3	10682	1282	736	736
q4	10238	998	550	550
q5	7935	2438	2178	2178
q6	209	175	136	136
q7	914	774	601	601
q8	9244	1501	1268	1268
q9	5276	4946	5129	4946
q10	6915	2351	1876	1876
q11	477	276	249	249
q12	358	392	223	223
q13	17789	3903	3107	3107
q14	233	235	207	207
q15	527	460	462	460
q16	622	641	587	587
q17	566	908	334	334
q18	6898	6486	6290	6290
q19	2038	1037	567	567
q20	326	328	187	187
q21	2883	2276	1982	1982
q22	366	338	316	316
Total cold run time: 104152 ms
Total hot run time: 32516 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5802	5639	5670	5639
q2	243	337	232	232
q3	2232	2678	2326	2326
q4	1440	1803	1391	1391
q5	4336	4790	4854	4790
q6	190	166	131	131
q7	2082	2004	1775	1775
q8	2690	2910	2799	2799
q9	7354	7136	7273	7136
q10	3028	3320	2906	2906
q11	585	506	524	506
q12	685	804	613	613
q13	3542	3982	3336	3336
q14	276	311	271	271
q15	524	471	481	471
q16	670	692	643	643
q17	1277	1750	1263	1263
q18	7733	7511	7366	7366
q19	817	1219	1216	1216
q20	2061	2083	1884	1884
q21	6141	5127	5119	5119
q22	636	617	631	617
Total cold run time: 54344 ms
Total hot run time: 52430 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 191969 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 58d5288a12bc13424a5db61d37d2364d557474c0, data reload: false

query1	1322	950	914	914
query2	6201	2203	2204	2203
query3	10941	4370	4428	4370
query4	33095	23341	22894	22894
query5	3632	612	452	452
query6	287	201	195	195
query7	3983	478	311	311
query8	305	239	226	226
query9	9251	2622	2604	2604
query10	459	314	245	245
query11	17728	15190	15102	15102
query12	151	106	101	101
query13	1553	504	386	386
query14	9289	7384	6785	6785
query15	256	220	184	184
query16	7869	620	468	468
query17	1565	759	577	577
query18	2145	419	327	327
query19	201	187	163	163
query20	125	115	121	115
query21	209	126	102	102
query22	4536	4471	4482	4471
query23	34426	33953	33460	33460
query24	6648	2321	2270	2270
query25	455	466	422	422
query26	1207	274	148	148
query27	2334	478	349	349
query28	4944	2525	2440	2440
query29	664	563	438	438
query30	215	184	151	151
query31	950	910	835	835
query32	71	64	57	57
query33	534	364	310	310
query34	757	899	537	537
query35	819	850	766	766
query36	1030	1049	992	992
query37	114	100	74	74
query38	4376	4368	4384	4368
query39	1484	1457	1432	1432
query40	217	119	106	106
query41	53	55	55	55
query42	115	100	102	100
query43	563	540	487	487
query44	1353	835	817	817
query45	187	183	167	167
query46	871	1041	675	675
query47	1892	1901	1825	1825
query48	382	434	322	322
query49	762	521	457	457
query50	666	693	409	409
query51	4307	4293	4237	4237
query52	112	106	96	96
query53	231	257	197	197
query54	498	514	432	432
query55	89	85	77	77
query56	260	263	267	263
query57	1205	1224	1153	1153
query58	245	248	234	234
query59	3282	3406	3231	3231
query60	269	289	300	289
query61	128	113	120	113
query62	812	758	686	686
query63	224	198	199	198
query64	3935	1035	653	653
query65	3301	3226	3240	3226
query66	951	392	306	306
query67	15491	15469	15427	15427
query68	2257	818	551	551
query69	411	312	322	312
query70	1228	1150	1141	1141
query71	301	286	266	266
query72	4960	3788	3790	3788
query73	647	749	367	367
query74	10096	9134	8948	8948
query75	3176	3149	2657	2657
query76	2006	1185	775	775
query77	326	370	282	282
query78	10133	10180	9289	9289
query79	1001	858	597	597
query80	631	535	458	458
query81	491	274	249	249
query82	1193	153	120	120
query83	277	170	161	161
query84	280	90	76	76
query85	739	357	300	300
query86	331	321	303	303
query87	4533	4694	4410	4410
query88	3087	2238	2168	2168
query89	401	329	296	296
query90	1738	190	197	190
query91	135	140	106	106
query92	63	58	54	54
query93	932	924	556	556
query94	481	387	287	287
query95	333	272	260	260
query96	491	593	280	280
query97	2822	2887	2762	2762
query98	220	198	195	195
query99	1305	1410	1283	1283
Total cold run time: 275557 ms
Total hot run time: 191969 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.27 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 58d5288a12bc13424a5db61d37d2364d557474c0, data reload: false

query1	0.04	0.03	0.02
query2	0.08	0.04	0.03
query3	0.23	0.07	0.07
query4	1.62	0.10	0.10
query5	0.44	0.42	0.40
query6	1.14	0.66	0.67
query7	0.03	0.02	0.02
query8	0.04	0.03	0.03
query9	0.58	0.52	0.49
query10	0.56	0.57	0.56
query11	0.14	0.11	0.11
query12	0.14	0.12	0.11
query13	0.61	0.60	0.60
query14	2.76	2.72	2.72
query15	0.90	0.82	0.83
query16	0.39	0.38	0.39
query17	1.01	1.07	1.04
query18	0.23	0.20	0.21
query19	1.85	1.81	2.05
query20	0.02	0.01	0.01
query21	15.36	0.94	0.58
query22	0.77	0.78	0.68
query23	15.24	1.37	0.56
query24	3.66	0.67	1.46
query25	0.17	0.15	0.05
query26	0.28	0.14	0.13
query27	0.05	0.05	0.04
query28	13.93	1.01	0.44
query29	12.64	3.98	3.24
query30	0.25	0.08	0.06
query31	2.84	0.58	0.39
query32	3.22	0.55	0.46
query33	2.96	2.99	3.03
query34	16.52	5.19	4.54
query35	4.56	4.47	4.51
query36	0.65	0.49	0.50
query37	0.10	0.06	0.06
query38	0.05	0.04	0.04
query39	0.04	0.03	0.02
query40	0.16	0.14	0.12
query41	0.07	0.02	0.02
query42	0.03	0.03	0.02
query43	0.03	0.02	0.02
Total cold run time: 106.39 s
Total hot run time: 30.27 s

@yujun777
Copy link
Collaborator Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32335 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f86c29f1ef1a9c9bee52624238b69d0004f2bbcd, data reload: false

------ Round 1 ----------------------------------
q1	17574	5466	5405	5405
q2	2040	309	164	164
q3	10439	1285	782	782
q4	10226	960	521	521
q5	7754	2421	2178	2178
q6	199	178	134	134
q7	892	761	590	590
q8	9230	1362	1164	1164
q9	5166	4903	4950	4903
q10	6862	2350	1899	1899
q11	488	282	250	250
q12	350	358	218	218
q13	17770	3746	3151	3151
q14	221	224	211	211
q15	522	483	468	468
q16	632	610	605	605
q17	582	870	323	323
q18	6996	6451	6319	6319
q19	1901	993	571	571
q20	312	315	187	187
q21	2976	2216	1989	1989
q22	378	338	303	303
Total cold run time: 103510 ms
Total hot run time: 32335 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5639	5520	5533	5520
q2	242	328	231	231
q3	2255	2697	2314	2314
q4	1448	1849	1404	1404
q5	4334	4857	4749	4749
q6	182	179	141	141
q7	2090	1992	1856	1856
q8	2672	2953	2752	2752
q9	7430	7306	7463	7306
q10	3105	3345	2871	2871
q11	595	530	508	508
q12	700	796	632	632
q13	3745	4000	3387	3387
q14	285	295	281	281
q15	513	470	470	470
q16	649	687	664	664
q17	1230	1753	1276	1276
q18	7716	7546	7516	7516
q19	833	1138	1156	1138
q20	2061	2056	1976	1976
q21	5754	5272	4979	4979
q22	593	580	600	580
Total cold run time: 54071 ms
Total hot run time: 52551 ms

@yujun777
Copy link
Collaborator Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32536 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ccffd6ea7d5768bfaefca539131a13180e39a09a, data reload: false

------ Round 1 ----------------------------------
q1	17594	5547	5393	5393
q2	2043	312	174	174
q3	10406	1318	741	741
q4	10226	968	533	533
q5	7536	2450	2188	2188
q6	197	164	139	139
q7	925	764	637	637
q8	9226	1375	1167	1167
q9	5302	4913	4904	4904
q10	6893	2354	1909	1909
q11	494	282	251	251
q12	348	357	219	219
q13	17769	3734	3133	3133
q14	241	224	215	215
q15	528	488	480	480
q16	639	631	597	597
q17	573	867	314	314
q18	6977	6471	6501	6471
q19	1207	947	531	531
q20	319	330	191	191
q21	2942	2481	2029	2029
q22	381	339	320	320
Total cold run time: 102766 ms
Total hot run time: 32536 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5552	5459	5552	5459
q2	237	323	232	232
q3	2281	2670	2368	2368
q4	1401	1836	1386	1386
q5	4363	4775	4703	4703
q6	170	157	123	123
q7	2009	1871	1787	1787
q8	2555	2741	2687	2687
q9	7056	6918	6928	6918
q10	2938	3183	2698	2698
q11	560	506	482	482
q12	633	711	567	567
q13	3360	3756	3133	3133
q14	289	293	263	263
q15	515	466	459	459
q16	633	697	673	673
q17	1191	1726	1228	1228
q18	7276	7177	6946	6946
q19	802	1082	1087	1082
q20	1930	1988	1838	1838
q21	5590	5152	4838	4838
q22	604	600	558	558
Total cold run time: 51945 ms
Total hot run time: 50428 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 185759 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ccffd6ea7d5768bfaefca539131a13180e39a09a, data reload: false

query1	947	397	386	386
query2	6292	2098	2061	2061
query3	6153	219	215	215
query4	33418	23259	23301	23259
query5	4444	618	450	450
query6	300	203	200	200
query7	3996	511	316	316
query8	294	248	225	225
query9	8212	2693	2679	2679
query10	443	342	256	256
query11	17491	15148	14874	14874
query12	155	108	106	106
query13	1272	534	404	404
query14	9293	7479	6983	6983
query15	239	195	187	187
query16	8294	577	455	455
query17	1694	714	545	545
query18	1901	385	294	294
query19	216	196	158	158
query20	119	115	111	111
query21	214	119	100	100
query22	4377	4318	4226	4226
query23	33955	33198	33081	33081
query24	6870	2261	2269	2261
query25	494	441	378	378
query26	777	263	156	156
query27	1988	460	330	330
query28	5302	2479	2447	2447
query29	609	545	425	425
query30	238	185	155	155
query31	960	877	799	799
query32	81	59	57	57
query33	512	361	284	284
query34	736	837	510	510
query35	785	810	735	735
query36	1006	1049	957	957
query37	119	105	83	83
query38	4236	4281	4020	4020
query39	1483	1403	1426	1403
query40	206	123	103	103
query41	61	67	63	63
query42	117	104	98	98
query43	524	522	499	499
query44	1339	814	818	814
query45	175	172	162	162
query46	857	1032	652	652
query47	1799	1851	1777	1777
query48	393	395	333	333
query49	747	490	423	423
query50	610	645	400	400
query51	4173	4239	4093	4093
query52	104	107	96	96
query53	233	262	188	188
query54	488	498	404	404
query55	82	85	85	85
query56	257	268	248	248
query57	1144	1164	1099	1099
query58	245	238	227	227
query59	2942	3064	2960	2960
query60	285	274	262	262
query61	121	114	116	114
query62	773	743	650	650
query63	220	192	189	189
query64	3085	1069	703	703
query65	3235	3154	3163	3154
query66	857	412	309	309
query67	15696	15610	15466	15466
query68	2345	849	566	566
query69	417	301	256	256
query70	1237	1144	1158	1144
query71	327	290	281	281
query72	6104	3844	3851	3844
query73	634	777	360	360
query74	9643	9231	8839	8839
query75	3145	3169	2685	2685
query76	2134	1199	787	787
query77	337	366	282	282
query78	10116	10266	9356	9356
query79	999	888	617	617
query80	1273	539	453	453
query81	538	272	232	232
query82	709	157	125	125
query83	243	190	157	157
query84	244	93	70	70
query85	791	406	305	305
query86	395	317	303	303
query87	4544	4599	4485	4485
query88	3318	2236	2183	2183
query89	403	340	289	289
query90	1789	194	197	194
query91	132	139	106	106
query92	58	59	58	58
query93	967	858	542	542
query94	671	394	293	293
query95	342	266	261	261
query96	507	605	291	291
query97	2793	2825	2794	2794
query98	219	198	198	198
query99	1282	1394	1261	1261
Total cold run time: 269339 ms
Total hot run time: 185759 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.51 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ccffd6ea7d5768bfaefca539131a13180e39a09a, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.03	0.03
query3	0.25	0.07	0.07
query4	1.61	0.11	0.10
query5	0.42	0.42	0.41
query6	1.13	0.67	0.66
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.59	0.51	0.49
query10	0.56	0.56	0.56
query11	0.15	0.11	0.10
query12	0.14	0.11	0.11
query13	0.62	0.60	0.61
query14	2.80	2.74	2.70
query15	0.90	0.85	0.83
query16	0.39	0.38	0.39
query17	1.07	1.04	1.03
query18	0.23	0.22	0.20
query19	1.92	1.82	2.05
query20	0.02	0.01	0.01
query21	15.38	0.88	0.58
query22	0.76	0.75	0.74
query23	15.31	1.44	0.52
query24	3.04	1.67	1.72
query25	0.16	0.17	0.07
query26	0.31	0.15	0.14
query27	0.06	0.05	0.03
query28	14.03	0.94	0.44
query29	12.60	3.93	3.22
query30	0.25	0.09	0.07
query31	2.81	0.62	0.39
query32	3.23	0.55	0.48
query33	3.03	3.05	3.06
query34	16.49	5.21	4.60
query35	4.54	4.59	4.56
query36	0.66	0.50	0.48
query37	0.10	0.06	0.06
query38	0.05	0.03	0.03
query39	0.04	0.03	0.02
query40	0.17	0.13	0.12
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 106.14 s
Total hot run time: 31.51 s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants