Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[enhance](runtime filter) impl partition pruning in runtime filer #47025

Open
wants to merge 21 commits into
base: master
Choose a base branch
from

Conversation

suxiaogang223
Copy link
Contributor

@suxiaogang223 suxiaogang223 commented Jan 15, 2025

This PR implements partition pruning through runtime filters. When executing a SQL query like:

SELECT count(*) 
FROM int_partition_table 
WHERE partition_col = (
    SELECT partition_col 
    FROM int_partition_table 
    GROUP BY partition_col 
    HAVING count(*) > 0 
    ORDER BY partition_col DESC 
    LIMIT 1
)

During execution, the backend (BE) will receive a dynamic runtime filter condition partition_col = xxx. Since partition_col is a partitioning column, we can use its value to determine if the partition can be pruned.

Additionally, this mechanism also supports filtering queries like:

SELECT count(*) 
FROM int_partition_table 
WHERE func(partition_col) = xxx

If func cannot be evaluated at the frontend (FE), the frontend will not perform partition pruning. However, since the backend can compute func, this mechanism allows us to handle pruning scenarios that are not possible at the frontend, providing a more efficient pruning process on the backend side.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Jan 15, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@suxiaogang223 suxiaogang223 marked this pull request as draft January 15, 2025 08:25
@suxiaogang223 suxiaogang223 marked this pull request as ready for review January 16, 2025 08:10
@suxiaogang223
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32093 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 96860e7a6a5b3a9bbb5bf9afdb10e953ab8cc502, data reload: false

------ Round 1 ----------------------------------
q1	17584	5485	5353	5353
q2	2046	296	175	175
q3	10429	1182	730	730
q4	10210	961	554	554
q5	7590	2330	2144	2144
q6	191	164	135	135
q7	899	761	598	598
q8	9228	1331	1119	1119
q9	5111	4807	4785	4785
q10	6804	2337	1875	1875
q11	475	260	258	258
q12	339	346	213	213
q13	17755	3686	3126	3126
q14	221	236	202	202
q15	518	466	472	466
q16	641	618	591	591
q17	566	839	323	323
q18	6874	6442	6431	6431
q19	1226	953	533	533
q20	315	316	190	190
q21	3091	2175	1983	1983
q22	367	336	309	309
Total cold run time: 102480 ms
Total hot run time: 32093 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5513	5515	5404	5404
q2	252	332	234	234
q3	2426	2730	2420	2420
q4	1489	1903	1459	1459
q5	4246	4745	4673	4673
q6	163	158	129	129
q7	1982	1979	1801	1801
q8	2605	2816	2688	2688
q9	7276	7231	7243	7231
q10	3046	3286	2805	2805
q11	575	516	498	498
q12	659	737	573	573
q13	3553	3904	3297	3297
q14	279	304	271	271
q15	499	487	469	469
q16	650	709	656	656
q17	1239	1747	1269	1269
q18	7638	7512	7322	7322
q19	832	1019	1120	1019
q20	1990	1994	1885	1885
q21	5669	5232	4864	4864
q22	628	603	563	563
Total cold run time: 53209 ms
Total hot run time: 51530 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 41.53% (10824/26064)
Line Coverage: 31.92% (91470/286527)
Region Coverage: 31.07% (46865/150823)
Branch Coverage: 27.16% (23731/87374)
Coverage Report: http://coverage.selectdb-in.cc/coverage/96860e7a6a5b3a9bbb5bf9afdb10e953ab8cc502_96860e7a6a5b3a9bbb5bf9afdb10e953ab8cc502/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 195472 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 96860e7a6a5b3a9bbb5bf9afdb10e953ab8cc502, data reload: false

query1	1287	950	933	933
query2	6472	2060	2016	2016
query3	11005	4553	4564	4553
query4	60974	28853	23002	23002
query5	5557	602	446	446
query6	433	198	190	190
query7	5531	520	302	302
query8	336	247	236	236
query9	8184	2683	2687	2683
query10	462	314	264	264
query11	17296	15208	15566	15208
query12	155	104	116	104
query13	1379	580	423	423
query14	11545	7280	7302	7280
query15	211	210	190	190
query16	6648	601	477	477
query17	1102	716	556	556
query18	1070	376	303	303
query19	213	182	165	165
query20	122	113	114	113
query21	207	126	104	104
query22	4611	4379	4711	4379
query23	34054	33576	33818	33576
query24	5681	2386	2351	2351
query25	461	467	411	411
query26	656	256	159	159
query27	1666	478	332	332
query28	4070	2567	2485	2485
query29	564	574	467	467
query30	212	194	165	165
query31	928	904	822	822
query32	82	64	60	60
query33	439	390	316	316
query34	747	826	527	527
query35	838	872	831	831
query36	997	1036	958	958
query37	125	96	80	80
query38	4315	4396	4192	4192
query39	1486	1437	1452	1437
query40	207	119	104	104
query41	50	51	48	48
query42	121	107	106	106
query43	508	503	498	498
query44	1385	834	822	822
query45	189	173	168	168
query46	880	1061	662	662
query47	1912	1932	1820	1820
query48	404	410	374	374
query49	721	508	409	409
query50	685	662	403	403
query51	7107	7073	7012	7012
query52	110	103	96	96
query53	229	258	186	186
query54	497	502	430	430
query55	96	83	89	83
query56	262	309	267	267
query57	1195	1206	1142	1142
query58	259	248	260	248
query59	3058	2973	2943	2943
query60	283	265	250	250
query61	119	113	117	113
query62	756	696	648	648
query63	221	194	186	186
query64	1224	1047	662	662
query65	3318	3142	3179	3142
query66	752	399	321	321
query67	15790	15537	15645	15537
query68	5081	841	536	536
query69	502	312	268	268
query70	1209	1170	1162	1162
query71	413	307	258	258
query72	6101	3926	3838	3838
query73	833	770	362	362
query74	9969	9349	9029	9029
query75	3226	3165	2684	2684
query76	3784	1185	783	783
query77	473	385	308	308
query78	10151	9980	9362	9362
query79	3482	819	586	586
query80	1687	550	442	442
query81	570	273	234	234
query82	357	150	131	131
query83	271	173	150	150
query84	288	93	89	89
query85	809	359	301	301
query86	467	332	302	302
query87	4742	4571	4459	4459
query88	4684	2191	2165	2165
query89	395	325	298	298
query90	1667	190	194	190
query91	134	139	110	110
query92	68	58	54	54
query93	2790	882	521	521
query94	749	405	297	297
query95	349	276	266	266
query96	512	616	286	286
query97	2805	2851	2755	2755
query98	221	202	190	190
query99	1292	1372	1280	1280
Total cold run time: 314540 ms
Total hot run time: 195472 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.52 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 96860e7a6a5b3a9bbb5bf9afdb10e953ab8cc502, data reload: false

query1	0.04	0.03	0.02
query2	0.07	0.03	0.03
query3	0.24	0.06	0.07
query4	1.61	0.11	0.10
query5	0.42	0.43	0.41
query6	1.17	0.67	0.64
query7	0.02	0.02	0.01
query8	0.03	0.03	0.03
query9	0.57	0.50	0.50
query10	0.56	0.57	0.55
query11	0.14	0.10	0.10
query12	0.15	0.11	0.11
query13	0.62	0.61	0.60
query14	2.83	2.84	2.73
query15	0.89	0.83	0.83
query16	0.38	0.38	0.38
query17	1.01	1.06	1.07
query18	0.23	0.22	0.20
query19	1.89	1.88	1.98
query20	0.01	0.01	0.02
query21	15.36	0.93	0.59
query22	0.76	0.92	0.63
query23	15.24	1.45	0.63
query24	4.04	1.21	0.65
query25	0.18	0.23	0.14
query26	0.23	0.15	0.14
query27	0.05	0.04	0.04
query28	13.42	1.00	0.43
query29	12.55	3.94	3.26
query30	0.25	0.09	0.07
query31	2.82	0.59	0.38
query32	3.23	0.55	0.45
query33	2.98	3.02	3.09
query34	16.59	5.16	4.54
query35	4.56	4.51	4.51
query36	0.66	0.50	0.49
query37	0.09	0.05	0.06
query38	0.04	0.04	0.03
query39	0.03	0.02	0.02
query40	0.17	0.13	0.12
query41	0.08	0.03	0.03
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.28 s
Total hot run time: 30.52 s

@suxiaogang223
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32424 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3b83637ceaaff22e01f5b62bf5e7de8346caba9d, data reload: false

------ Round 1 ----------------------------------
q1	17613	5556	5370	5370
q2	2053	320	190	190
q3	10376	1285	744	744
q4	10211	980	538	538
q5	7565	2420	2165	2165
q6	197	168	134	134
q7	907	765	605	605
q8	9277	1387	1176	1176
q9	5629	5012	4926	4926
q10	6925	2335	1874	1874
q11	483	287	253	253
q12	358	361	225	225
q13	17799	3682	3124	3124
q14	232	234	210	210
q15	509	471	479	471
q16	632	620	593	593
q17	599	869	333	333
q18	6999	6400	6430	6400
q19	1200	972	540	540
q20	323	337	199	199
q21	2928	2193	2039	2039
q22	370	338	315	315
Total cold run time: 103185 ms
Total hot run time: 32424 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5668	5488	5483	5483
q2	243	341	246	246
q3	2287	2663	2332	2332
q4	1406	1778	1359	1359
q5	4355	4766	4628	4628
q6	163	157	125	125
q7	1978	1917	1742	1742
q8	2540	2696	2610	2610
q9	7032	6945	6925	6925
q10	2900	3174	2647	2647
q11	564	509	481	481
q12	658	695	572	572
q13	3294	3717	3067	3067
q14	290	282	261	261
q15	515	450	465	450
q16	640	692	632	632
q17	1185	1690	1218	1218
q18	7221	7248	7221	7221
q19	790	968	1084	968
q20	1904	1974	1787	1787
q21	5522	5056	4851	4851
q22	620	567	581	567
Total cold run time: 51775 ms
Total hot run time: 50172 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 188758 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3b83637ceaaff22e01f5b62bf5e7de8346caba9d, data reload: false

query1	968	403	365	365
query2	6328	2134	2117	2117
query3	6501	215	217	215
query4	32950	23234	23124	23124
query5	4353	653	476	476
query6	287	191	179	179
query7	4549	538	313	313
query8	297	251	241	241
query9	8802	2613	2582	2582
query10	484	333	258	258
query11	17838	15300	14863	14863
query12	158	109	107	107
query13	1385	538	420	420
query14	11063	7034	7207	7034
query15	262	196	189	189
query16	7273	646	450	450
query17	1649	733	541	541
query18	1300	359	297	297
query19	220	202	156	156
query20	119	114	110	110
query21	210	127	99	99
query22	4270	4350	4257	4257
query23	33913	32941	33763	32941
query24	7341	2261	2218	2218
query25	459	448	389	389
query26	1076	258	160	160
query27	2034	486	335	335
query28	5029	2406	2398	2398
query29	567	535	453	453
query30	237	192	154	154
query31	948	890	816	816
query32	85	60	65	60
query33	509	378	306	306
query34	760	901	501	501
query35	814	829	732	732
query36	1024	1035	936	936
query37	125	108	82	82
query38	4387	4276	4172	4172
query39	1477	1441	1401	1401
query40	208	123	106	106
query41	54	52	54	52
query42	126	102	109	102
query43	523	526	505	505
query44	1333	798	791	791
query45	180	174	169	169
query46	930	1094	664	664
query47	1827	1859	1767	1767
query48	377	401	305	305
query49	796	516	419	419
query50	677	684	403	403
query51	6859	6997	6868	6868
query52	108	106	95	95
query53	233	278	193	193
query54	521	552	424	424
query55	80	78	81	78
query56	272	271	255	255
query57	1192	1177	1064	1064
query58	251	231	235	231
query59	3021	3172	3125	3125
query60	287	271	267	267
query61	125	117	112	112
query62	820	719	653	653
query63	229	197	192	192
query64	4497	1111	758	758
query65	3255	3202	3136	3136
query66	1070	442	331	331
query67	15948	15854	15431	15431
query68	2783	878	550	550
query69	464	327	291	291
query70	1206	1223	1172	1172
query71	382	297	268	268
query72	5861	3901	3882	3882
query73	649	793	368	368
query74	9764	9177	9139	9139
query75	3218	3209	2668	2668
query76	3151	1261	802	802
query77	467	380	297	297
query78	10171	10012	9353	9353
query79	3001	833	598	598
query80	1556	582	446	446
query81	558	280	238	238
query82	691	157	126	126
query83	202	175	150	150
query84	247	96	72	72
query85	844	357	318	318
query86	480	322	300	300
query87	4471	4611	4421	4421
query88	4865	2148	2106	2106
query89	407	324	299	299
query90	1876	195	192	192
query91	136	142	111	111
query92	72	60	55	55
query93	2719	886	534	534
query94	772	418	304	304
query95	350	275	255	255
query96	493	665	283	283
query97	2789	2878	2759	2759
query98	233	202	199	199
query99	1302	1431	1259	1259
Total cold run time: 284030 ms
Total hot run time: 188758 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.84 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3b83637ceaaff22e01f5b62bf5e7de8346caba9d, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.04
query3	0.23	0.07	0.07
query4	1.62	0.11	0.10
query5	0.41	0.42	0.42
query6	1.18	0.65	0.64
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.60	0.51	0.52
query10	0.56	0.57	0.55
query11	0.14	0.10	0.10
query12	0.14	0.11	0.12
query13	0.61	0.61	0.60
query14	2.72	2.78	2.89
query15	0.90	0.83	0.82
query16	0.38	0.40	0.38
query17	0.98	1.00	1.01
query18	0.22	0.21	0.22
query19	1.97	1.75	2.01
query20	0.01	0.01	0.02
query21	15.36	0.88	0.57
query22	0.76	1.02	0.60
query23	15.16	1.43	0.55
query24	3.13	1.34	1.22
query25	0.14	0.22	0.11
query26	0.20	0.14	0.14
query27	0.05	0.05	0.06
query28	13.94	1.03	0.44
query29	12.64	3.97	3.27
query30	0.25	0.08	0.06
query31	2.82	0.60	0.40
query32	3.22	0.55	0.46
query33	2.98	3.05	3.07
query34	16.58	5.17	4.48
query35	4.53	4.52	4.50
query36	0.68	0.49	0.49
query37	0.10	0.05	0.06
query38	0.05	0.04	0.04
query39	0.04	0.02	0.02
query40	0.16	0.13	0.12
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 105.78 s
Total hot run time: 30.84 s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 41.52% (10822/26062)
Line Coverage: 31.92% (91461/286538)
Region Coverage: 31.07% (46856/150829)
Branch Coverage: 27.16% (23729/87376)
Coverage Report: http://coverage.selectdb-in.cc/coverage/3b83637ceaaff22e01f5b62bf5e7de8346caba9d_3b83637ceaaff22e01f5b62bf5e7de8346caba9d/report/index.html

@suxiaogang223
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32468 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 07514d47377cc1cf0a86fc4cb8290a281c011f66, data reload: false

------ Round 1 ----------------------------------
q1	17599	5512	5404	5404
q2	2050	292	171	171
q3	10432	1241	739	739
q4	10215	978	526	526
q5	7726	2400	2145	2145
q6	191	167	138	138
q7	916	747	608	608
q8	9227	1356	1234	1234
q9	5014	4860	4904	4860
q10	6871	2348	1884	1884
q11	478	276	249	249
q12	333	364	227	227
q13	17788	3669	3121	3121
q14	223	238	210	210
q15	503	483	467	467
q16	619	616	592	592
q17	580	860	336	336
q18	6876	6485	6495	6485
q19	2823	944	538	538
q20	314	319	195	195
q21	2773	2210	2036	2036
q22	354	326	303	303
Total cold run time: 103905 ms
Total hot run time: 32468 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5660	5509	5566	5509
q2	255	343	253	253
q3	2332	2781	2473	2473
q4	1500	1895	1465	1465
q5	4442	4759	4769	4759
q6	196	165	130	130
q7	2086	1962	1849	1849
q8	2615	2848	2755	2755
q9	7346	7281	7291	7281
q10	3007	3260	2744	2744
q11	571	524	478	478
q12	650	762	625	625
q13	3574	3941	3305	3305
q14	287	315	267	267
q15	515	469	474	469
q16	665	689	617	617
q17	1224	1752	1261	1261
q18	7787	7590	7215	7215
q19	840	1097	1110	1097
q20	2017	2047	1882	1882
q21	5733	5254	4975	4975
q22	656	642	616	616
Total cold run time: 53958 ms
Total hot run time: 52025 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 194362 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 07514d47377cc1cf0a86fc4cb8290a281c011f66, data reload: false

query1	1297	948	938	938
query2	6421	2031	1987	1987
query3	10921	4697	4480	4480
query4	60780	32261	23438	23438
query5	4878	630	467	467
query6	436	199	210	199
query7	5567	524	301	301
query8	346	252	231	231
query9	8309	2605	2573	2573
query10	429	305	257	257
query11	16195	14997	15991	14997
query12	159	109	106	106
query13	1455	517	421	421
query14	11693	7027	7280	7027
query15	226	218	193	193
query16	6837	634	442	442
query17	1117	714	583	583
query18	1115	376	306	306
query19	195	190	189	189
query20	119	112	110	110
query21	225	133	112	112
query22	4314	4597	4274	4274
query23	33730	33319	33490	33319
query24	5787	2410	2350	2350
query25	460	471	412	412
query26	650	284	152	152
query27	1677	498	343	343
query28	3993	2510	2417	2417
query29	575	593	466	466
query30	217	193	159	159
query31	979	893	826	826
query32	78	64	60	60
query33	449	360	313	313
query34	744	875	519	519
query35	822	862	775	775
query36	1019	1046	953	953
query37	124	114	75	75
query38	4203	4473	4354	4354
query39	1499	1437	1436	1436
query40	212	141	104	104
query41	54	48	49	48
query42	122	103	103	103
query43	520	539	506	506
query44	1343	824	805	805
query45	188	177	178	177
query46	879	1054	658	658
query47	1925	1896	1877	1877
query48	400	416	323	323
query49	700	489	407	407
query50	703	676	406	406
query51	6906	6923	7028	6923
query52	104	105	92	92
query53	235	265	190	190
query54	500	519	435	435
query55	90	93	80	80
query56	268	264	264	264
query57	1188	1205	1132	1132
query58	251	264	262	262
query59	3115	3013	2815	2815
query60	289	296	271	271
query61	145	143	115	115
query62	720	696	648	648
query63	216	190	183	183
query64	1244	1033	663	663
query65	3291	3145	3166	3145
query66	678	396	302	302
query67	15810	15610	15365	15365
query68	4972	847	531	531
query69	505	293	259	259
query70	1251	1160	1148	1148
query71	408	292	255	255
query72	6120	3935	3983	3935
query73	790	752	353	353
query74	10165	8928	9040	8928
query75	3183	3107	2663	2663
query76	3683	1170	776	776
query77	475	408	288	288
query78	10381	10118	9285	9285
query79	3386	790	583	583
query80	1654	536	449	449
query81	559	284	243	243
query82	359	153	120	120
query83	280	180	150	150
query84	288	97	80	80
query85	922	352	325	325
query86	456	312	279	279
query87	4605	4566	4448	4448
query88	4513	2156	2123	2123
query89	415	330	298	298
query90	1601	192	205	192
query91	142	136	113	113
query92	77	56	55	55
query93	2663	846	532	532
query94	763	400	315	315
query95	330	274	263	263
query96	489	624	284	284
query97	2854	2806	2777	2777
query98	222	200	197	197
query99	1268	1370	1256	1256
Total cold run time: 311915 ms
Total hot run time: 194362 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.88 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 07514d47377cc1cf0a86fc4cb8290a281c011f66, data reload: false

query1	0.04	0.04	0.04
query2	0.07	0.04	0.03
query3	0.24	0.06	0.07
query4	1.62	0.10	0.10
query5	0.41	0.43	0.41
query6	1.18	0.64	0.65
query7	0.03	0.02	0.01
query8	0.04	0.04	0.03
query9	0.60	0.50	0.51
query10	0.56	0.57	0.55
query11	0.14	0.11	0.11
query12	0.14	0.10	0.10
query13	0.62	0.60	0.61
query14	2.69	2.84	2.73
query15	0.89	0.82	0.82
query16	0.37	0.37	0.38
query17	1.07	1.05	0.98
query18	0.23	0.20	0.21
query19	1.96	1.81	1.93
query20	0.02	0.01	0.01
query21	15.39	0.94	0.58
query22	0.76	0.99	0.77
query23	14.98	1.40	0.57
query24	3.14	1.14	1.73
query25	0.23	0.18	0.14
query26	0.32	0.14	0.15
query27	0.07	0.06	0.05
query28	14.50	0.98	0.43
query29	12.56	3.98	3.31
query30	0.25	0.09	0.06
query31	2.83	0.60	0.39
query32	3.22	0.54	0.45
query33	3.06	3.02	3.03
query34	16.50	5.15	4.43
query35	4.52	4.52	4.54
query36	0.67	0.50	0.47
query37	0.10	0.06	0.06
query38	0.05	0.04	0.03
query39	0.04	0.02	0.03
query40	0.15	0.13	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.04	0.03
Total cold run time: 106.4 s
Total hot run time: 30.88 s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 41.52% (10822/26064)
Line Coverage: 31.91% (91421/286529)
Region Coverage: 31.06% (46845/150809)
Branch Coverage: 27.15% (23716/87362)
Coverage Report: http://coverage.selectdb-in.cc/coverage/07514d47377cc1cf0a86fc4cb8290a281c011f66_07514d47377cc1cf0a86fc4cb8290a281c011f66/report/index.html

@suxiaogang223
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32077 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2b35cbecb7230f2fb1c66105c3342ed3679ac4c2, data reload: false

------ Round 1 ----------------------------------
q1	17580	5460	5379	5379
q2	2047	322	169	169
q3	10727	1243	726	726
q4	10231	972	538	538
q5	7668	2397	2143	2143
q6	193	166	134	134
q7	902	761	601	601
q8	9234	1323	1108	1108
q9	5140	4882	4856	4856
q10	6871	2342	1878	1878
q11	472	272	249	249
q12	342	347	220	220
q13	17787	3722	3094	3094
q14	231	231	209	209
q15	509	460	478	460
q16	634	609	589	589
q17	569	855	327	327
q18	7022	6558	6402	6402
q19	1784	947	524	524
q20	312	322	198	198
q21	2742	2167	1958	1958
q22	361	344	315	315
Total cold run time: 103358 ms
Total hot run time: 32077 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5518	5528	5475	5475
q2	242	339	239	239
q3	2264	2640	2336	2336
q4	1444	1818	1375	1375
q5	4342	4768	4650	4650
q6	165	165	127	127
q7	2019	1929	1813	1813
q8	2613	2789	2713	2713
q9	7360	7244	7279	7244
q10	3046	3281	2795	2795
q11	580	502	477	477
q12	680	721	600	600
q13	3587	3888	3257	3257
q14	292	289	274	274
q15	504	488	465	465
q16	640	704	658	658
q17	1219	1734	1278	1278
q18	7641	7388	7343	7343
q19	784	1139	1103	1103
q20	2009	2045	1879	1879
q21	5787	5224	4964	4964
q22	609	610	591	591
Total cold run time: 53345 ms
Total hot run time: 51656 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 193645 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2b35cbecb7230f2fb1c66105c3342ed3679ac4c2, data reload: false

query1	1319	999	915	915
query2	6296	2089	2060	2060
query3	11076	4478	4473	4473
query4	61070	34294	23024	23024
query5	4938	626	465	465
query6	467	200	190	190
query7	5608	495	293	293
query8	336	252	243	243
query9	8409	2582	2575	2575
query10	438	308	270	270
query11	16087	15066	15678	15066
query12	161	109	110	109
query13	1435	525	412	412
query14	11302	6458	6846	6458
query15	208	218	203	203
query16	7081	640	444	444
query17	1066	729	564	564
query18	1535	405	309	309
query19	195	187	159	159
query20	111	113	113	113
query21	208	120	101	101
query22	4404	4996	4469	4469
query23	33828	33650	33244	33244
query24	5619	2404	2258	2258
query25	459	449	411	411
query26	625	275	149	149
query27	1876	474	326	326
query28	4216	2450	2416	2416
query29	568	566	473	473
query30	214	197	161	161
query31	899	857	824	824
query32	71	61	61	61
query33	437	372	297	297
query34	738	901	524	524
query35	791	849	753	753
query36	1013	1078	973	973
query37	133	108	81	81
query38	4295	4377	4339	4339
query39	1482	1516	1450	1450
query40	202	115	99	99
query41	50	47	51	47
query42	118	106	105	105
query43	538	523	496	496
query44	1311	821	824	821
query45	181	177	165	165
query46	880	1066	645	645
query47	1955	1922	1850	1850
query48	389	422	321	321
query49	708	482	427	427
query50	653	672	410	410
query51	7186	7196	7085	7085
query52	114	98	99	98
query53	220	255	183	183
query54	467	492	419	419
query55	81	75	76	75
query56	252	272	250	250
query57	1189	1157	1077	1077
query58	229	245	226	226
query59	3004	3103	3000	3000
query60	280	270	249	249
query61	123	140	112	112
query62	691	714	632	632
query63	238	194	182	182
query64	1179	1013	676	676
query65	3253	3150	3125	3125
query66	800	392	294	294
query67	15958	15985	15393	15393
query68	6648	831	522	522
query69	499	305	264	264
query70	1200	1140	1148	1140
query71	406	282	248	248
query72	6088	3859	3793	3793
query73	796	738	353	353
query74	10122	9337	9036	9036
query75	3204	3141	2701	2701
query76	3831	1180	782	782
query77	509	372	271	271
query78	10029	10020	9397	9397
query79	2952	792	594	594
query80	682	522	451	451
query81	499	280	240	240
query82	281	149	122	122
query83	174	166	149	149
query84	281	89	75	75
query85	748	374	308	308
query86	366	335	308	308
query87	4621	4506	4464	4464
query88	4487	2146	2120	2120
query89	397	333	287	287
query90	1729	191	187	187
query91	131	135	111	111
query92	64	58	55	55
query93	2888	891	535	535
query94	686	420	301	301
query95	340	263	260	260
query96	478	606	282	282
query97	2784	2847	2739	2739
query98	225	201	192	192
query99	1277	1370	1241	1241
Total cold run time: 312685 ms
Total hot run time: 193645 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 41.52% (10822/26064)
Line Coverage: 31.91% (91434/286534)
Region Coverage: 31.07% (46859/150819)
Branch Coverage: 27.15% (23724/87372)
Coverage Report: http://coverage.selectdb-in.cc/coverage/2b35cbecb7230f2fb1c66105c3342ed3679ac4c2_2b35cbecb7230f2fb1c66105c3342ed3679ac4c2/report/index.html

@doris-robot
Copy link

ClickBench: Total hot run time: 31.03 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2b35cbecb7230f2fb1c66105c3342ed3679ac4c2, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.03
query3	0.25	0.07	0.07
query4	1.62	0.11	0.11
query5	0.44	0.42	0.40
query6	1.19	0.66	0.66
query7	0.02	0.01	0.02
query8	0.04	0.03	0.03
query9	0.57	0.51	0.50
query10	0.58	0.56	0.56
query11	0.15	0.10	0.11
query12	0.14	0.12	0.11
query13	0.61	0.60	0.60
query14	2.70	2.74	2.72
query15	0.92	0.83	0.84
query16	0.36	0.38	0.39
query17	1.05	1.04	1.07
query18	0.23	0.21	0.21
query19	1.98	1.80	2.01
query20	0.02	0.01	0.01
query21	15.36	0.94	0.56
query22	0.76	0.90	0.92
query23	14.97	1.42	0.50
query24	2.70	1.07	1.27
query25	0.31	0.10	0.05
query26	0.21	0.15	0.13
query27	0.06	0.04	0.04
query28	13.60	1.00	0.43
query29	12.60	3.98	3.30
query30	0.25	0.11	0.08
query31	2.81	0.61	0.40
query32	3.23	0.56	0.46
query33	3.00	3.03	3.02
query34	16.64	5.14	4.51
query35	4.58	4.60	4.58
query36	0.66	0.50	0.49
query37	0.10	0.07	0.06
query38	0.04	0.04	0.03
query39	0.04	0.03	0.02
query40	0.16	0.13	0.13
query41	0.08	0.03	0.02
query42	0.04	0.02	0.03
query43	0.04	0.03	0.03
Total cold run time: 105.23 s
Total hot run time: 31.03 s

@suxiaogang223
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32312 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 59bac57d84f60152b7536e75176337c186f6f214, data reload: false

------ Round 1 ----------------------------------
q1	17583	5491	5363	5363
q2	2043	317	165	165
q3	10418	1274	713	713
q4	10203	973	532	532
q5	7538	2412	2173	2173
q6	188	169	134	134
q7	920	777	613	613
q8	9227	1382	1216	1216
q9	5257	4869	4858	4858
q10	6830	2344	1883	1883
q11	463	272	261	261
q12	344	359	224	224
q13	17753	3684	3136	3136
q14	229	220	211	211
q15	503	476	454	454
q16	642	633	584	584
q17	561	919	324	324
q18	6918	6484	6439	6439
q19	1223	958	541	541
q20	335	327	206	206
q21	2896	2215	1970	1970
q22	377	334	312	312
Total cold run time: 102451 ms
Total hot run time: 32312 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5446	5467	5460	5460
q2	240	328	231	231
q3	2265	2672	2328	2328
q4	1430	1825	1397	1397
q5	4332	4772	4643	4643
q6	164	154	121	121
q7	1981	2033	1804	1804
q8	2690	2814	2709	2709
q9	7252	7170	7193	7170
q10	3004	3271	2767	2767
q11	578	506	492	492
q12	671	781	617	617
q13	3582	3907	3272	3272
q14	279	314	275	275
q15	529	459	464	459
q16	639	679	654	654
q17	1221	1712	1258	1258
q18	7643	7562	7434	7434
q19	833	992	1125	992
q20	2008	2037	1868	1868
q21	5770	5156	5149	5149
q22	602	621	558	558
Total cold run time: 53159 ms
Total hot run time: 51658 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 184063 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 59bac57d84f60152b7536e75176337c186f6f214, data reload: false

query1	984	403	381	381
query2	6525	2016	2077	2016
query3	6787	218	214	214
query4	33342	23194	22834	22834
query5	4438	641	497	497
query6	333	201	178	178
query7	4592	488	310	310
query8	293	240	250	240
query9	9598	2674	2669	2669
query10	449	329	258	258
query11	17926	15171	14907	14907
query12	151	119	103	103
query13	1646	534	389	389
query14	9746	6336	7367	6336
query15	214	198	194	194
query16	7947	592	482	482
query17	1596	716	544	544
query18	2053	393	298	298
query19	202	183	156	156
query20	119	117	114	114
query21	203	120	99	99
query22	4175	4289	4260	4260
query23	33973	33370	32860	32860
query24	6543	2316	2294	2294
query25	493	482	414	414
query26	1212	274	156	156
query27	2001	477	330	330
query28	5316	2442	2435	2435
query29	661	529	463	463
query30	235	181	157	157
query31	983	852	779	779
query32	74	59	60	59
query33	503	358	283	283
query34	744	866	530	530
query35	796	811	722	722
query36	996	1035	927	927
query37	117	109	79	79
query38	4238	4179	4048	4048
query39	1452	1386	1404	1386
query40	214	118	103	103
query41	56	57	53	53
query42	125	111	107	107
query43	515	537	494	494
query44	1316	815	829	815
query45	177	177	175	175
query46	868	1058	644	644
query47	1821	1845	1786	1786
query48	383	410	319	319
query49	790	495	375	375
query50	609	664	380	380
query51	4201	4187	4105	4105
query52	107	104	96	96
query53	230	255	189	189
query54	497	507	416	416
query55	87	83	80	80
query56	264	255	263	255
query57	1130	1157	1082	1082
query58	251	225	228	225
query59	2955	3104	2976	2976
query60	282	264	259	259
query61	118	113	120	113
query62	842	709	661	661
query63	219	196	191	191
query64	4144	1068	688	688
query65	3241	3208	3208	3208
query66	1069	412	308	308
query67	15879	15613	15317	15317
query68	2357	829	560	560
query69	397	309	267	267
query70	1198	1109	1151	1109
query71	325	306	254	254
query72	5029	3758	3816	3758
query73	653	754	352	352
query74	9661	9136	8809	8809
query75	3178	3141	2675	2675
query76	2163	1157	757	757
query77	338	347	272	272
query78	10099	10270	9313	9313
query79	1008	899	604	604
query80	642	543	439	439
query81	488	277	234	234
query82	1159	225	121	121
query83	283	171	158	158
query84	232	87	76	76
query85	745	358	310	310
query86	356	327	293	293
query87	4476	4493	4442	4442
query88	3080	2213	2192	2192
query89	388	330	303	303
query90	1702	190	191	190
query91	138	145	110	110
query92	56	58	55	55
query93	950	868	540	540
query94	478	404	284	284
query95	342	268	258	258
query96	490	613	277	277
query97	2761	2892	2794	2794
query98	219	202	195	195
query99	1254	1364	1241	1241
Total cold run time: 272691 ms
Total hot run time: 184063 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.99 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 59bac57d84f60152b7536e75176337c186f6f214, data reload: false

query1	0.04	0.03	0.03
query2	0.06	0.04	0.03
query3	0.25	0.07	0.06
query4	1.61	0.10	0.10
query5	0.41	0.42	0.42
query6	1.17	0.66	0.65
query7	0.02	0.02	0.02
query8	0.03	0.03	0.03
query9	0.59	0.51	0.50
query10	0.54	0.56	0.57
query11	0.15	0.11	0.10
query12	0.14	0.12	0.11
query13	0.61	0.59	0.60
query14	2.73	2.71	2.76
query15	0.90	0.84	0.82
query16	0.39	0.38	0.38
query17	1.03	1.06	1.05
query18	0.25	0.20	0.20
query19	1.93	1.92	1.92
query20	0.01	0.01	0.01
query21	15.36	0.90	0.58
query22	0.76	0.73	0.61
query23	15.39	1.46	0.51
query24	2.91	0.94	0.44
query25	0.21	0.09	0.14
query26	0.40	0.15	0.14
query27	0.06	0.04	0.04
query28	13.50	1.13	0.44
query29	12.59	4.02	3.24
query30	0.25	0.08	0.06
query31	2.84	0.62	0.39
query32	3.22	0.56	0.45
query33	2.98	2.99	3.04
query34	16.47	5.21	4.44
query35	4.51	4.51	4.48
query36	0.66	0.49	0.50
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.04	0.03	0.02
query40	0.17	0.14	0.13
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.02
Total cold run time: 105.47 s
Total hot run time: 29.99 s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 41.94% (10938/26079)
Line Coverage: 32.20% (92320/286706)
Region Coverage: 31.33% (47287/150954)
Branch Coverage: 27.37% (23926/87430)
Coverage Report: http://coverage.selectdb-in.cc/coverage/59bac57d84f60152b7536e75176337c186f6f214_59bac57d84f60152b7536e75176337c186f6f214/report/index.html

@doris-robot
Copy link

ClickBench: Total hot run time: 31.22 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1a5ddb19e473910ee5f8c663de1d8cbc1e050c9f, data reload: false

query1	0.04	0.05	0.03
query2	0.07	0.03	0.03
query3	0.24	0.07	0.07
query4	1.62	0.11	0.11
query5	0.44	0.40	0.39
query6	1.16	0.66	0.68
query7	0.03	0.02	0.01
query8	0.04	0.03	0.03
query9	0.60	0.53	0.52
query10	0.60	0.59	0.57
query11	0.15	0.10	0.10
query12	0.15	0.12	0.11
query13	0.62	0.61	0.60
query14	2.84	2.69	2.71
query15	0.92	0.86	0.87
query16	0.38	0.38	0.38
query17	1.04	1.01	1.02
query18	0.21	0.19	0.19
query19	1.97	1.84	2.02
query20	0.01	0.01	0.01
query21	15.73	0.94	0.54
query22	0.91	1.24	0.67
query23	15.02	1.37	0.60
query24	7.09	1.32	1.52
query25	0.52	0.25	0.09
query26	0.59	0.15	0.15
query27	0.05	0.05	0.05
query28	10.00	0.87	0.44
query29	12.56	4.05	3.39
query30	0.27	0.09	0.07
query31	2.80	0.58	0.39
query32	3.22	0.54	0.46
query33	2.99	2.97	3.08
query34	15.80	5.12	4.57
query35	4.51	4.51	4.58
query36	0.67	0.49	0.48
query37	0.09	0.06	0.06
query38	0.06	0.04	0.04
query39	0.03	0.02	0.03
query40	0.16	0.13	0.13
query41	0.09	0.02	0.03
query42	0.03	0.03	0.02
query43	0.03	0.02	0.02
Total cold run time: 106.35 s
Total hot run time: 31.22 s

@suxiaogang223 suxiaogang223 force-pushed the partition_prun_runtime_filter branch from 1e8e7af to e19475c Compare February 8, 2025 07:03
@suxiaogang223
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity cloud ut coverage result:
Function Coverage: 82.38% (1061/1288)
Line Coverage: 65.94% (17564/26635)
Region Coverage: 65.46% (8665/13237)
Branch Coverage: 55.37% (4674/8442)
Coverage Report: http://coverage.selectdb-in.cc/coverage/e19475c6dddb805c9f5ac93101656c81ba02f020_e19475c6dddb805c9f5ac93101656c81ba02f020_cloud/report/index.html

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 42.64% (11173/26202)
Line Coverage: 32.64% (93959/287852)
Region Coverage: 31.81% (48188/151509)
Branch Coverage: 27.72% (24327/87760)
Coverage Report: http://coverage.selectdb-in.cc/coverage/e19475c6dddb805c9f5ac93101656c81ba02f020_e19475c6dddb805c9f5ac93101656c81ba02f020/report/index.html

@suxiaogang223
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity cloud ut coverage result:
Function Coverage: 82.38% (1061/1288)
Line Coverage: 65.88% (17547/26635)
Region Coverage: 65.42% (8659/13237)
Branch Coverage: 55.29% (4668/8442)
Coverage Report: http://coverage.selectdb-in.cc/coverage/2689d78d92619eb18a6a2a6d6d84d05f3c978770_2689d78d92619eb18a6a2a6d6d84d05f3c978770_cloud/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 31560 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2689d78d92619eb18a6a2a6d6d84d05f3c978770, data reload: false

------ Round 1 ----------------------------------
q1	17597	5164	5084	5084
q2	2050	309	176	176
q3	10459	1283	752	752
q4	10229	1025	537	537
q5	7584	2469	2396	2396
q6	185	170	139	139
q7	943	751	599	599
q8	9307	1408	1122	1122
q9	4968	4574	4682	4574
q10	6846	2315	1887	1887
q11	516	266	255	255
q12	355	363	223	223
q13	17788	3680	3108	3108
q14	238	252	206	206
q15	516	447	465	447
q16	619	615	591	591
q17	602	893	345	345
q18	6660	6332	6167	6167
q19	1857	985	556	556
q20	312	318	189	189
q21	2814	2156	1904	1904
q22	370	332	303	303
Total cold run time: 102815 ms
Total hot run time: 31560 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5532	5167	5174	5167
q2	239	334	231	231
q3	2183	2733	2308	2308
q4	1408	1813	1321	1321
q5	4247	4186	4356	4186
q6	211	166	126	126
q7	1888	1828	1729	1729
q8	2670	2693	2581	2581
q9	7254	7222	7191	7191
q10	3030	3213	2807	2807
q11	586	532	513	513
q12	720	775	633	633
q13	3404	3849	3342	3342
q14	294	304	275	275
q15	529	470	464	464
q16	634	681	649	649
q17	1166	1594	1411	1411
q18	7689	7415	7340	7340
q19	818	841	830	830
q20	1968	2066	1898	1898
q21	5508	5058	4894	4894
q22	661	602	549	549
Total cold run time: 52639 ms
Total hot run time: 50445 ms

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants