Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[enhancement](cloud) file cache evict in advance #47473

Open
wants to merge 6 commits into
base: master
Choose a base branch
from

Conversation

freemandealer
Copy link
Contributor

evict in advance if current cache size is over threshold to avoid sync evict during query, which may affect query performance.

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@freemandealer
Copy link
Contributor Author

run buildall

gavinchou
gavinchou previously approved these changes Jan 27, 2025
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 27, 2025
Copy link
Contributor

PR approved by at least one committer and no changes requested.

Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 42.07% (10974/26086)
Line Coverage: 32.35% (92799/286887)
Region Coverage: 31.50% (47582/151075)
Branch Coverage: 27.54% (24108/87524)
Coverage Report: http://coverage.selectdb-in.cc/coverage/b684c8dc6cb3875c0ef29e1f984ddcaa2b067cab_b684c8dc6cb3875c0ef29e1f984ddcaa2b067cab/report/index.html

@gavinchou
Copy link
Contributor

run buildall

@freemandealer
Copy link
Contributor Author

run performance

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 42.07% (10974/26086)
Line Coverage: 32.35% (92820/286887)
Region Coverage: 31.50% (47589/151075)
Branch Coverage: 27.54% (24101/87524)
Coverage Report: http://coverage.selectdb-in.cc/coverage/b684c8dc6cb3875c0ef29e1f984ddcaa2b067cab_b684c8dc6cb3875c0ef29e1f984ddcaa2b067cab/report/index.html

dataroaring
dataroaring previously approved these changes Jan 29, 2025
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@freemandealer
Copy link
Contributor Author

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Feb 1, 2025
@doris-robot
Copy link

TPC-H: Total hot run time: 32452 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0a01d0fac37c1e80dc6ec73558cbf85b5900f96a, data reload: false

------ Round 1 ----------------------------------
q1	17571	5465	5347	5347
q2	2054	314	166	166
q3	10471	1230	760	760
q4	10210	997	537	537
q5	7533	2423	2203	2203
q6	197	164	133	133
q7	892	786	582	582
q8	9221	1372	1169	1169
q9	5221	4923	4983	4923
q10	6828	2343	1878	1878
q11	477	288	255	255
q12	339	349	218	218
q13	17777	3672	3106	3106
q14	229	250	213	213
q15	514	468	475	468
q16	651	633	591	591
q17	574	874	322	322
q18	7185	6562	6598	6562
q19	1682	951	528	528
q20	304	330	188	188
q21	2765	2189	1982	1982
q22	368	342	321	321
Total cold run time: 103063 ms
Total hot run time: 32452 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5479	5497	5485	5485
q2	238	327	234	234
q3	2215	2637	2255	2255
q4	1411	1868	1336	1336
q5	4328	4762	4596	4596
q6	159	157	122	122
q7	2110	1899	1824	1824
q8	2645	2826	2777	2777
q9	7344	7232	7305	7232
q10	3017	3280	2785	2785
q11	592	516	507	507
q12	633	745	583	583
q13	3432	3905	3345	3345
q14	289	291	266	266
q15	513	492	465	465
q16	653	691	635	635
q17	1242	1741	1268	1268
q18	7699	7517	7441	7441
q19	793	1164	1067	1067
q20	2012	2023	1920	1920
q21	5715	5368	4896	4896
q22	630	631	599	599
Total cold run time: 53149 ms
Total hot run time: 51638 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 191066 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0a01d0fac37c1e80dc6ec73558cbf85b5900f96a, data reload: false

query1	1288	992	940	940
query2	6107	2006	2014	2006
query3	11130	4605	4636	4605
query4	61733	29308	22970	22970
query5	5630	611	450	450
query6	437	198	181	181
query7	5543	527	303	303
query8	338	258	236	236
query9	8247	2593	2567	2567
query10	462	300	246	246
query11	17307	15122	15549	15122
query12	171	114	110	110
query13	1487	531	428	428
query14	10972	7463	7171	7171
query15	214	197	189	189
query16	7322	613	519	519
query17	1116	751	586	586
query18	1921	424	338	338
query19	212	186	167	167
query20	124	126	116	116
query21	209	130	103	103
query22	4516	4653	4423	4423
query23	34682	33102	33186	33102
query24	5635	2358	2407	2358
query25	465	444	407	407
query26	639	243	157	157
query27	1681	475	331	331
query28	4049	2498	2432	2432
query29	541	531	426	426
query30	214	187	153	153
query31	921	897	811	811
query32	71	55	55	55
query33	459	362	304	304
query34	728	859	516	516
query35	845	841	750	750
query36	1025	1048	934	934
query37	114	98	76	76
query38	4450	4372	4253	4253
query39	1479	1457	1448	1448
query40	204	122	103	103
query41	54	50	52	50
query42	125	106	106	106
query43	528	521	485	485
query44	1287	816	833	816
query45	186	175	170	170
query46	882	1056	651	651
query47	1928	1921	1872	1872
query48	381	393	312	312
query49	765	490	386	386
query50	632	678	393	393
query51	4278	4340	4183	4183
query52	107	109	100	100
query53	245	261	189	189
query54	478	501	420	420
query55	85	80	82	80
query56	263	269	251	251
query57	1185	1186	1140	1140
query58	241	240	280	240
query59	3010	3269	2940	2940
query60	303	272	268	268
query61	118	122	124	122
query62	745	742	653	653
query63	240	194	191	191
query64	1290	1045	642	642
query65	3231	3174	3185	3174
query66	733	400	294	294
query67	16033	15894	15391	15391
query68	5029	819	529	529
query69	474	303	259	259
query70	1197	1176	1140	1140
query71	394	289	255	255
query72	6304	3892	3889	3889
query73	798	740	358	358
query74	9743	9238	8732	8732
query75	3234	3164	2625	2625
query76	3696	1174	764	764
query77	489	365	280	280
query78	10174	10006	9394	9394
query79	3301	781	587	587
query80	1688	527	482	482
query81	562	273	236	236
query82	439	149	126	126
query83	273	176	152	152
query84	292	96	73	73
query85	781	341	304	304
query86	458	322	312	312
query87	4395	4594	4327	4327
query88	4549	2136	2099	2099
query89	394	328	285	285
query90	1552	185	187	185
query91	137	135	110	110
query92	73	58	54	54
query93	2870	859	529	529
query94	782	411	297	297
query95	334	262	255	255
query96	488	614	281	281
query97	2816	2935	2792	2792
query98	216	201	205	201
query99	1319	1335	1243	1243
Total cold run time: 313292 ms
Total hot run time: 191066 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.18 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0a01d0fac37c1e80dc6ec73558cbf85b5900f96a, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.25	0.06	0.06
query4	1.61	0.10	0.10
query5	0.42	0.43	0.41
query6	1.17	0.66	0.66
query7	0.03	0.01	0.02
query8	0.03	0.03	0.03
query9	0.60	0.50	0.50
query10	0.55	0.57	0.55
query11	0.14	0.11	0.10
query12	0.14	0.11	0.10
query13	0.62	0.59	0.60
query14	2.77	2.76	2.73
query15	0.88	0.83	0.83
query16	0.40	0.40	0.39
query17	1.10	1.06	1.09
query18	0.23	0.20	0.20
query19	1.86	1.90	2.02
query20	0.02	0.01	0.01
query21	15.35	0.91	0.58
query22	0.76	0.74	0.63
query23	15.32	1.45	0.52
query24	2.63	1.36	0.32
query25	0.23	0.19	0.13
query26	0.34	0.16	0.14
query27	0.06	0.03	0.06
query28	13.42	1.11	0.44
query29	12.59	3.99	3.26
query30	0.25	0.09	0.08
query31	2.82	0.61	0.39
query32	3.24	0.56	0.46
query33	2.96	3.13	3.02
query34	16.87	5.18	4.53
query35	4.56	4.57	4.55
query36	0.67	0.50	0.49
query37	0.09	0.06	0.06
query38	0.04	0.04	0.04
query39	0.03	0.02	0.03
query40	0.18	0.14	0.13
query41	0.09	0.03	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.49 s
Total hot run time: 30.18 s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 42.08% (10995/26131)
Line Coverage: 32.36% (92893/287019)
Region Coverage: 31.52% (47633/151138)
Branch Coverage: 27.54% (24108/87544)
Coverage Report: http://coverage.selectdb-in.cc/coverage/0a01d0fac37c1e80dc6ec73558cbf85b5900f96a_0a01d0fac37c1e80dc6ec73558cbf85b5900f96a/report/index.html

dataroaring
dataroaring previously approved these changes Feb 2, 2025
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Feb 2, 2025
Copy link
Contributor

github-actions bot commented Feb 2, 2025

PR approved by at least one committer and no changes requested.

be/src/io/cache/block_file_cache.cpp Show resolved Hide resolved
be/src/common/config.cpp Outdated Show resolved Hide resolved
be/src/common/config.cpp Show resolved Hide resolved
evict in advance if current cache size is over threshold to avoid sync
evict during query, which may affect query performance.

Signed-off-by: zhengyu <[email protected]>
Signed-off-by: zhengyu <[email protected]>
Signed-off-by: zhengyu <[email protected]>
@freemandealer
Copy link
Contributor Author

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Feb 6, 2025
@freemandealer
Copy link
Contributor Author

run performance

Signed-off-by: zhengyu <[email protected]>
@freemandealer
Copy link
Contributor Author

run buildall

Signed-off-by: zhengyu <[email protected]>
@freemandealer
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 31863 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 54f641b4f766dfd9b28dbb13fefa255095df7d60, data reload: false

------ Round 1 ----------------------------------
q1	17627	5319	5097	5097
q2	2054	301	180	180
q3	10476	1239	794	794
q4	10230	1030	552	552
q5	7602	2386	2396	2386
q6	193	169	132	132
q7	923	777	586	586
q8	9307	1319	1121	1121
q9	4898	4640	4689	4640
q10	6833	2304	1897	1897
q11	483	290	254	254
q12	357	356	224	224
q13	17777	3783	3119	3119
q14	226	231	209	209
q15	511	479	461	461
q16	635	616	583	583
q17	593	886	344	344
q18	6777	6232	6317	6232
q19	1798	949	567	567
q20	321	326	188	188
q21	2783	2211	1998	1998
q22	362	335	299	299
Total cold run time: 102766 ms
Total hot run time: 31863 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5197	5129	5096	5096
q2	234	335	233	233
q3	2205	2740	2329	2329
q4	1441	1842	1423	1423
q5	4288	4182	4165	4165
q6	206	163	127	127
q7	1854	1837	1759	1759
q8	2618	2714	2632	2632
q9	7136	7111	7076	7076
q10	3053	3214	2818	2818
q11	596	513	500	500
q12	692	829	650	650
q13	3471	3825	3262	3262
q14	289	312	274	274
q15	522	468	459	459
q16	653	704	638	638
q17	1165	1634	1336	1336
q18	7666	7461	7383	7383
q19	838	841	848	841
q20	1981	2032	1873	1873
q21	5669	5104	4879	4879
q22	623	653	546	546
Total cold run time: 52397 ms
Total hot run time: 50299 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 190755 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 54f641b4f766dfd9b28dbb13fefa255095df7d60, data reload: false

query1	1334	972	937	937
query2	6242	1954	1907	1907
query3	10968	4405	4446	4405
query4	55900	25611	23056	23056
query5	5102	573	483	483
query6	352	201	193	193
query7	4927	508	287	287
query8	302	240	231	231
query9	5935	2577	2593	2577
query10	434	324	247	247
query11	15258	15048	15005	15005
query12	166	111	109	109
query13	1100	523	414	414
query14	10865	7141	6637	6637
query15	207	207	197	197
query16	7091	644	494	494
query17	1085	743	572	572
query18	1610	431	319	319
query19	215	204	171	171
query20	142	127	121	121
query21	215	123	112	112
query22	4299	4444	4218	4218
query23	33961	33386	33384	33384
query24	5617	2457	2440	2440
query25	474	475	403	403
query26	691	287	162	162
query27	1722	494	331	331
query28	2831	2475	2426	2426
query29	591	559	446	446
query30	211	191	157	157
query31	913	919	840	840
query32	82	64	59	59
query33	450	382	296	296
query34	771	866	522	522
query35	797	837	780	780
query36	964	1030	930	930
query37	145	109	82	82
query38	4358	4347	4284	4284
query39	1516	1427	1468	1427
query40	215	126	115	115
query41	53	51	56	51
query42	128	105	106	105
query43	503	516	505	505
query44	1328	806	798	798
query45	193	173	171	171
query46	863	1067	678	678
query47	1867	1884	1775	1775
query48	392	418	308	308
query49	684	517	406	406
query50	710	765	422	422
query51	4350	4306	4276	4276
query52	117	111	96	96
query53	232	263	193	193
query54	493	510	438	438
query55	82	82	81	81
query56	279	306	279	279
query57	1167	1190	1139	1139
query58	257	258	244	244
query59	2853	3013	2910	2910
query60	299	286	262	262
query61	122	123	134	123
query62	758	778	699	699
query63	229	204	198	198
query64	1707	1024	664	664
query65	3334	3184	3134	3134
query66	719	394	295	295
query67	15980	15505	15551	15505
query68	5478	785	513	513
query69	521	301	261	261
query70	1230	1092	1080	1080
query71	456	302	284	284
query72	5974	3733	3801	3733
query73	1274	750	356	356
query74	9007	8916	8933	8916
query75	3185	3143	2696	2696
query76	3860	1182	759	759
query77	532	391	285	285
query78	9879	10090	9266	9266
query79	2347	809	609	609
query80	672	565	459	459
query81	498	276	242	242
query82	267	150	128	128
query83	179	174	153	153
query84	292	102	74	74
query85	778	347	302	302
query86	432	291	305	291
query87	4502	4543	4347	4347
query88	3565	2265	2207	2207
query89	419	322	285	285
query90	1724	195	194	194
query91	143	145	109	109
query92	71	63	60	60
query93	2073	1025	583	583
query94	679	426	304	304
query95	356	275	258	258
query96	502	563	277	277
query97	2841	2823	2713	2713
query98	230	203	206	203
query99	1355	1403	1277	1277
Total cold run time: 295078 ms
Total hot run time: 190755 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.3 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 54f641b4f766dfd9b28dbb13fefa255095df7d60, data reload: false

query1	0.03	0.03	0.04
query2	0.07	0.03	0.03
query3	0.23	0.07	0.06
query4	1.61	0.10	0.10
query5	0.41	0.41	0.41
query6	1.15	0.66	0.66
query7	0.03	0.02	0.01
query8	0.04	0.04	0.03
query9	0.60	0.50	0.53
query10	0.57	0.57	0.57
query11	0.15	0.10	0.10
query12	0.14	0.11	0.11
query13	0.64	0.60	0.60
query14	2.81	2.73	2.72
query15	0.92	0.84	0.85
query16	0.37	0.38	0.38
query17	1.01	1.05	1.04
query18	0.22	0.20	0.19
query19	1.93	1.83	1.99
query20	0.01	0.01	0.01
query21	15.36	0.91	0.56
query22	0.73	1.30	0.92
query23	14.69	1.41	0.64
query24	6.79	1.66	1.20
query25	0.52	0.26	0.09
query26	0.54	0.15	0.14
query27	0.04	0.05	0.05
query28	9.74	0.90	0.44
query29	12.56	4.04	3.36
query30	0.25	0.08	0.06
query31	2.83	0.59	0.40
query32	3.22	0.54	0.47
query33	2.99	2.97	2.98
query34	15.80	5.13	4.48
query35	4.49	4.50	4.50
query36	0.66	0.49	0.51
query37	0.09	0.06	0.07
query38	0.05	0.04	0.03
query39	0.04	0.02	0.02
query40	0.16	0.15	0.12
query41	0.08	0.03	0.03
query42	0.04	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 104.65 s
Total hot run time: 31.3 s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 42.09% (11021/26187)
Line Coverage: 32.38% (93115/287608)
Region Coverage: 31.55% (47766/151394)
Branch Coverage: 27.56% (24170/87690)
Coverage Report: http://coverage.selectdb-in.cc/coverage/54f641b4f766dfd9b28dbb13fefa255095df7d60_54f641b4f766dfd9b28dbb13fefa255095df7d60/report/index.html

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants