Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[feat](spill) spill and reserve #47462

Draft
wants to merge 1 commit into
base: master
Choose a base branch
from
Draft

Conversation

mrhhsg
Copy link
Member

@mrhhsg mrhhsg commented Jan 26, 2025

What problem does this PR solve?

Problem Summary:

A brand-new spilling triggering strategy:

  1. Use workload groups to control/manage the memory usage of queries.
  2. Trigger spilling when the memory reservation attempt fails.

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

Squash from origin commit: 17bc0cc7ee6e9a436c87ce164f0989d8759d1dd7
@Thearas
Copy link
Contributor

Thearas commented Jan 26, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@mrhhsg mrhhsg marked this pull request as draft January 26, 2025 10:20
@mrhhsg
Copy link
Member Author

mrhhsg commented Jan 26, 2025

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32548 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit bbb25f1b0e644ce1b6368ccaba3ed10d144321ce, data reload: false

------ Round 1 ----------------------------------
q1	17797	5532	5403	5403
q2	2073	305	174	174
q3	10877	1213	726	726
q4	10219	961	546	546
q5	7652	2390	2168	2168
q6	200	172	134	134
q7	931	763	600	600
q8	9225	1383	1171	1171
q9	5228	4953	4898	4898
q10	6854	2344	1901	1901
q11	473	282	260	260
q12	346	365	215	215
q13	17759	3731	3151	3151
q14	230	229	216	216
q15	547	461	469	461
q16	638	627	598	598
q17	555	879	333	333
q18	7246	6703	6558	6558
q19	2001	965	539	539
q20	303	320	190	190
q21	2870	2166	1983	1983
q22	377	341	323	323
Total cold run time: 104401 ms
Total hot run time: 32548 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5571	5487	5449	5449
q2	242	339	243	243
q3	2267	2659	2350	2350
q4	1445	1811	1418	1418
q5	4341	4762	4714	4714
q6	170	165	126	126
q7	2077	2007	1857	1857
q8	2659	2890	2689	2689
q9	7349	7233	7324	7233
q10	3065	3337	2763	2763
q11	609	518	487	487
q12	657	739	612	612
q13	3606	4025	3296	3296
q14	293	296	282	282
q15	535	480	480	480
q16	660	714	667	667
q17	1263	1741	1263	1263
q18	7794	7586	7479	7479
q19	800	851	1069	851
q20	2027	2102	1920	1920
q21	5876	5238	5022	5022
q22	634	626	587	587
Total cold run time: 53940 ms
Total hot run time: 51788 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 193144 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit bbb25f1b0e644ce1b6368ccaba3ed10d144321ce, data reload: false

query1	1309	933	949	933
query2	6247	2093	2102	2093
query3	11101	4725	4800	4725
query4	32690	23660	23070	23070
query5	4694	593	458	458
query6	304	199	195	195
query7	3994	493	320	320
query8	300	252	237	237
query9	9296	2656	2635	2635
query10	473	307	277	277
query11	17877	15362	14970	14970
query12	171	116	106	106
query13	1575	528	408	408
query14	9577	7163	7283	7163
query15	243	210	185	185
query16	7572	672	517	517
query17	1550	754	572	572
query18	1449	387	313	313
query19	202	219	174	174
query20	120	114	117	114
query21	214	133	108	108
query22	4606	4550	4389	4389
query23	34401	33484	33503	33484
query24	6430	2361	2369	2361
query25	497	469	407	407
query26	741	279	160	160
query27	2143	505	343	343
query28	5274	2506	2471	2471
query29	550	564	430	430
query30	221	189	161	161
query31	956	895	819	819
query32	84	58	59	58
query33	479	360	316	316
query34	786	899	529	529
query35	832	875	771	771
query36	1020	1060	974	974
query37	127	106	88	88
query38	4405	4158	4247	4158
query39	1492	1455	1456	1455
query40	215	127	110	110
query41	60	57	56	56
query42	124	109	105	105
query43	524	541	499	499
query44	1352	851	865	851
query45	183	186	172	172
query46	879	1066	666	666
query47	1901	1948	1871	1871
query48	374	423	333	333
query49	715	551	436	436
query50	642	699	409	409
query51	4337	4355	4311	4311
query52	104	100	96	96
query53	248	278	192	192
query54	508	529	448	448
query55	89	79	88	79
query56	283	287	251	251
query57	1188	1184	1139	1139
query58	252	236	249	236
query59	3236	3204	3027	3027
query60	289	286	265	265
query61	126	124	121	121
query62	808	736	692	692
query63	236	205	192	192
query64	2993	1057	693	693
query65	3473	3272	3299	3272
query66	759	397	295	295
query67	16158	15588	15530	15530
query68	8486	902	555	555
query69	477	288	257	257
query70	1185	1170	1105	1105
query71	438	279	256	256
query72	5830	3961	4086	3961
query73	655	756	370	370
query74	10293	9180	8833	8833
query75	3975	3170	2656	2656
query76	3552	1177	781	781
query77	762	434	275	275
query78	9942	10230	9266	9266
query79	3655	843	588	588
query80	692	516	450	450
query81	507	276	242	242
query82	656	148	125	125
query83	200	183	154	154
query84	285	100	71	71
query85	767	363	311	311
query86	364	317	277	277
query87	4412	4441	4391	4391
query88	4789	2181	2141	2141
query89	417	335	287	287
query90	1805	191	191	191
query91	135	141	106	106
query92	65	57	50	50
query93	2322	928	546	546
query94	662	417	298	298
query95	338	263	261	261
query96	493	626	284	284
query97	3382	3396	3310	3310
query98	217	200	191	191
query99	1506	1405	1261	1261
Total cold run time: 291322 ms
Total hot run time: 193144 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.37 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit bbb25f1b0e644ce1b6368ccaba3ed10d144321ce, data reload: false

query1	0.03	0.04	0.03
query2	0.08	0.05	0.05
query3	0.23	0.05	0.06
query4	1.65	0.08	0.09
query5	0.55	0.55	0.54
query6	1.19	0.72	0.74
query7	0.02	0.01	0.02
query8	0.07	0.05	0.05
query9	0.56	0.50	0.51
query10	0.56	0.56	0.56
query11	0.17	0.12	0.13
query12	0.15	0.12	0.13
query13	0.61	0.61	0.59
query14	2.71	2.86	2.88
query15	0.92	0.86	0.83
query16	0.38	0.39	0.40
query17	1.04	1.04	1.07
query18	0.19	0.18	0.19
query19	1.94	1.88	1.97
query20	0.01	0.01	0.01
query21	15.41	0.98	0.65
query22	0.77	0.80	0.72
query23	14.96	1.50	0.68
query24	2.22	0.36	0.23
query25	0.15	0.09	0.08
query26	0.27	0.19	0.17
query27	0.08	0.08	0.08
query28	13.42	1.29	0.55
query29	12.66	4.11	3.41
query30	0.24	0.08	0.07
query31	2.85	0.62	0.40
query32	3.23	0.59	0.49
query33	3.01	3.04	3.05
query34	16.43	5.20	4.52
query35	4.64	4.68	4.58
query36	0.62	0.49	0.48
query37	0.20	0.16	0.16
query38	0.15	0.15	0.15
query39	0.05	0.05	0.04
query40	0.18	0.14	0.12
query41	0.09	0.05	0.05
query42	0.07	0.05	0.05
query43	0.05	0.04	0.05
Total cold run time: 104.81 s
Total hot run time: 31.37 s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants