Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[feature](Nereids): InferPredicates support In #29458

Merged
merged 3 commits into from
Jan 5, 2024
Merged

Conversation

jackwener
Copy link
Member

@jackwener jackwener commented Jan 3, 2024

Proposed changes

InferPredicates support In (all children is literal)

such as a = b & a in [1, 2] -> b in [1, 2]

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

@jackwener
Copy link
Member Author

run buildall

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.78 seconds
stream load tsv: 564 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.1 seconds inserted 10000000 Rows, about 355K ops/s
storage size: 17183905444 Bytes

@jackwener
Copy link
Member Author

run buildall

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.84 seconds
stream load tsv: 566 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 27.9 seconds inserted 10000000 Rows, about 358K ops/s
storage size: 17183527088 Bytes

@xiaokang xiaokang added usercase Important user case type label dev/2.0.4 labels Jan 3, 2024
@jackwener
Copy link
Member Author

run buildall

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpch-tools

Tpch sf100 test result on commit 1d698fa93ce8f76ac1b645c1772af0f8fe60567c, data reload: false

run tpch-sf100 query with default conf and session variables
q1	5558	5096	5118	5096
q2	399	155	158	155
q3	1457	1188	1168	1168
q4	1092	851	787	787
q5	3158	3033	3052	3033
q6	221	144	134	134
q7	987	552	518	518
q8	2165	2236	2169	2169
q9	6696	6657	6663	6657
q10	3188	3134	3106	3106
q11	342	215	220	215
q12	380	237	239	237
q13	4385	3665	3629	3629
q14	256	223	228	223
q15	577	535	526	526
q16	463	398	399	398
q17	1046	514	524	514
q18	7113	6774	6735	6735
q19	1640	1567	1508	1508
q20	589	330	335	330
q21	2860	2419	2455	2419
q22	394	320	336	320
Total cold run time: 44966 ms
Total hot run time: 39877 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	5071	5039	4974	4974
q2	343	239	235	235
q3	3361	3339	3298	3298
q4	2160	2026	2042	2026
q5	5967	5905	5931	5905
q6	226	126	126	126
q7	2368	1937	1920	1920
q8	3587	3645	3665	3645
q9	9002	8979	8975	8975
q10	3868	3902	3903	3902
q11	561	487	488	487
q12	819	636	648	636
q13	3888	3193	3168	3168
q14	298	268	268	268
q15	588	542	535	535
q16	553	511	510	510
q17	2036	1829	1824	1824
q18	8677	8367	8361	8361
q19	1742	1700	1675	1675
q20	2253	1983	1971	1971
q21	5749	5439	5247	5247
q22	577	505	492	492
Total cold run time: 63694 ms
Total hot run time: 60180 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.74 seconds
stream load tsv: 564 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 28.1 seconds inserted 10000000 Rows, about 355K ops/s
storage size: 17187310826 Bytes

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpch-tools

Tpch sf100 test result on commit 1d698fa93ce8f76ac1b645c1772af0f8fe60567c, data reload: false

------ Round 1 ----------------------------------
q1	18329	5821	5092	5092
q2	2023	155	147	147
q3	10607	1079	1104	1079
q4	10235	711	828	711
q5	7787	2956	2959	2956
q6	215	135	140	135
q7	922	523	529	523
q8	9269	1983	2019	1983
q9	6852	6427	6416	6416
q10	8262	3099	3070	3070
q11	435	206	213	206
q12	379	230	224	224
q13	18008	3613	3611	3611
q14	241	211	211	211
q15	547	544	513	513
q16	461	400	392	392
q17	959	489	484	484
q18	7546	6780	6747	6747
q19	1580	1320	1428	1320
q20	726	360	316	316
q21	2773	2374	2440	2374
q22	368	312	335	312
Total cold run time: 108524 ms
Total hot run time: 38822 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5169	5096	5141	5096
q2	343	233	250	233
q3	3301	3280	3269	3269
q4	2102	2020	1978	1978
q5	5802	5808	5786	5786
q6	211	125	127	125
q7	2301	1900	1912	1900
q8	3344	3434	3458	3434
q9	8812	8712	8735	8712
q10	3793	3863	3848	3848
q11	591	473	475	473
q12	822	666	657	657
q13	8745	3193	3223	3193
q14	299	274	272	272
q15	578	517	518	517
q16	567	504	498	498
q17	1978	1785	1792	1785
q18	8689	8307	8282	8282
q19	1617	1566	1587	1566
q20	2189	1973	1961	1961
q21	5707	5320	5382	5320
q22	539	482	482	482
Total cold run time: 67499 ms
Total hot run time: 59387 ms

@doris-robot
Copy link

TPC-DS test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpcds-tools

TPC-DS sf100 test result on commit 1d698fa93ce8f76ac1b645c1772af0f8fe60567c, data reload: false

run tpcds-sf100 query with default conf and session variables
query1	997	348	335	335
query2	4019	1947	1926	1926
query3	6647	215	205	205
query4	26348	22343	22437	22343
query5	3532	512	488	488
query6	263	171	179	171
query7	4590	293	257	257
query8	217	197	192	192
query9	8202	2536	2466	2466
query10	440	230	239	230
query11	16083	15439	15555	15439
query12	132	74	75	74
query13	1642	315	319	315
query14	10887	7021	7196	7021
query15	212	186	193	186
query16	5444	262	260	260
query17	1858	489	479	479
query18	1748	269	275	269
query19	190	131	135	131
query20	79	75	77	75
query21	171	104	95	95
query22	4741	4796	4669	4669
query23	32181	31146	31204	31146
query24	12105	2800	2835	2800
query25	608	341	328	328
query26	1427	141	143	141
query27	2915	271	278	271
query28	7063	1899	1901	1899
query29	1687	390	376	376
query30	294	139	146	139
query31	982	781	755	755
query32	95	59	54	54
query33	737	246	265	246
query34	916	436	441	436
query35	873	795	793	793
query36	1310	1247	1251	1247
query37	136	75	69	69
query38	3453	3332	3247	3247
query39	1308	1277	1275	1275
query40	246	90	88	88
query41	38	34	36	34
query42	84	88	86	86
query43	507	478	463	463
query44	1065	694	701	694
query45	192	185	182	182
query46	1067	645	635	635
query47	1693	1606	1548	1548
query48	329	254	258	254
query49	1151	307	304	304
query50	733	357	355	355
query51	5420	5313	5217	5217
query52	92	84	94	84
query53	210	146	146	146
query54	1425	548	560	548
query55	102	90	76	76
query56	181	188	191	188
query57	1008	933	906	906
query58	218	203	195	195
query59	2755	2503	2558	2503
query60	238	214	208	208
query61	88	82	83	82
query62	657	438	443	438
query63	158	146	151	146
query64	5098	1687	1688	1687
query65	3319	3222	3264	3222
query66	1333	331	330	330
query67	15747	15479	15256	15256
query68	11360	516	533	516
query69	468	264	255	255
query70	1744	1561	1487	1487
query71	474	214	198	198
query72	5903	3503	3556	3503
query73	2359	314	307	307
query74	7034	6533	6506	6506
query75	5152	2271	2297	2271
query76	6115	1080	1114	1080
query77	671	246	249	246
query78	9142	8731	8555	8555
query79	1014	509	486	486
query80	524	338	350	338
query81	439	212	206	206
query82	190	97	96	96
query83	164	137	143	137
query84	246	52	53	52
query85	928	303	266	266
query86	379	406	403	403
query87	3558	3410	3413	3410
query88	2901	2271	2274	2271
query89	325	268	259	259
query90	1945	199	197	197
query91	123	89	93	89
query92	62	55	53	53
query93	2021	520	506	506
query94	1664	181	177	177
query95	462	418	406	406
query96	651	332	336	332
query97	4407	4246	4212	4212
query98	211	203	196	196
query99	1071	844	835	835
Total cold run time: 285393 ms
Total hot run time: 178574 ms

@wm1581066 wm1581066 removed the usercase Important user case type label label Jan 4, 2024
@jackwener
Copy link
Member Author

run buildall

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpch-tools

Tpch sf100 test result on commit 90a9712a6f5178dae5040a33aafc7e3e3b92d0dc, data reload: false

run tpch-sf100 query with default conf and session variables
q1	5526	5148	5146	5146
q2	389	156	148	148
q3	1461	1214	1158	1158
q4	1083	791	802	791
q5	3084	3044	3132	3044
q6	222	140	139	139
q7	987	547	526	526
q8	2155	2282	2247	2247
q9	6726	6681	6680	6680
q10	3206	3126	3152	3126
q11	351	218	212	212
q12	392	232	235	232
q13	4420	3665	3679	3665
q14	242	225	216	216
q15	583	528	525	525
q16	470	397	401	397
q17	1039	475	524	475
q18	7109	6708	6847	6708
q19	1636	1484	1603	1484
q20	613	329	322	322
q21	2893	2443	2542	2443
q22	392	327	359	327
Total cold run time: 44979 ms
Total hot run time: 40011 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	5157	5045	5127	5045
q2	339	237	236	236
q3	3393	3363	3345	3345
q4	2159	2019	2023	2019
q5	5928	5944	5937	5937
q6	226	129	131	129
q7	2377	1950	1939	1939
q8	3552	3701	3661	3661
q9	9086	8994	9043	8994
q10	3881	3909	3896	3896
q11	584	470	481	470
q12	809	636	669	636
q13	3882	3244	3184	3184
q14	297	260	270	260
q15	595	534	521	521
q16	552	517	512	512
q17	2051	1856	1824	1824
q18	8776	8309	8574	8309
q19	1742	1663	1655	1655
q20	2251	1982	1953	1953
q21	5797	5344	5365	5344
q22	557	539	506	506
Total cold run time: 63991 ms
Total hot run time: 60375 ms

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpch-tools

Tpch sf100 test result on commit 90a9712a6f5178dae5040a33aafc7e3e3b92d0dc, data reload: false

------ Round 1 ----------------------------------
q1	17682	5180	5100	5100
q2	2010	150	135	135
q3	10541	1085	1078	1078
q4	10186	786	836	786
q5	7808	3019	2925	2925
q6	207	136	136	136
q7	920	507	522	507
q8	9293	1996	1992	1992
q9	6829	6378	6333	6333
q10	8207	3029	3023	3023
q11	455	222	196	196
q12	382	222	223	222
q13	18009	3620	3655	3620
q14	236	213	213	213
q15	553	520	511	511
q16	464	408	402	402
q17	943	450	464	450
q18	7254	6692	6669	6669
q19	1582	1260	1440	1260
q20	689	346	322	322
q21	2786	2336	2383	2336
q22	383	334	340	334
Total cold run time: 107419 ms
Total hot run time: 38550 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5140	5068	5056	5056
q2	339	233	241	233
q3	3286	3277	3252	3252
q4	2120	2024	1995	1995
q5	5748	5742	5743	5742
q6	211	124	123	123
q7	2332	1894	1906	1894
q8	3341	3395	3409	3395
q9	8796	8745	8719	8719
q10	3748	3813	3825	3813
q11	572	486	480	480
q12	784	639	620	620
q13	6945	3179	3167	3167
q14	291	272	269	269
q15	583	512	521	512
q16	556	514	500	500
q17	1920	1752	1796	1752
q18	8632	8356	8224	8224
q19	1613	1574	1528	1528
q20	2207	1955	1924	1924
q21	5589	5308	5244	5244
q22	501	507	448	448
Total cold run time: 65254 ms
Total hot run time: 58890 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.71 seconds
stream load tsv: 569 seconds loaded 74807831229 Bytes, about 125 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.4 seconds inserted 10000000 Rows, about 352K ops/s
storage size: 17184093132 Bytes

@doris-robot
Copy link

TPC-DS test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpcds-tools

TPC-DS sf100 test result on commit 90a9712a6f5178dae5040a33aafc7e3e3b92d0dc, data reload: false

run tpcds-sf100 query with default conf and session variables
query1	930	337	328	328
query2	6426	1984	1989	1984
query3	6659	208	200	200
query4	29395	22272	22361	22272
query5	4308	501	488	488
query6	267	175	176	175
query7	4574	284	258	258
query8	231	200	198	198
query9	8345	2632	2555	2555
query10	415	243	252	243
query11	16163	15524	15533	15524
query12	138	79	76	76
query13	1635	322	329	322
query14	11909	7029	6997	6997
query15	221	181	192	181
query16	6398	264	254	254
query17	1862	486	480	480
query18	1944	270	262	262
query19	188	140	133	133
query20	77	76	74	74
query21	192	98	95	95
query22	5086	4589	4782	4589
query23	32094	30975	31089	30975
query24	11030	2815	2758	2758
query25	596	351	331	331
query26	1563	138	146	138
query27	2871	279	275	275
query28	7061	1899	1893	1893
query29	1372	380	377	377
query30	285	149	145	145
query31	987	767	780	767
query32	80	58	57	57
query33	725	248	249	248
query34	914	451	436	436
query35	884	750	747	747
query36	1275	1221	1235	1221
query37	101	65	69	65
query38	3386	3281	3317	3281
query39	1340	1288	1271	1271
query40	187	88	87	87
query41	39	35	34	34
query42	86	91	82	82
query43	531	509	482	482
query44	1058	702	699	699
query45	192	188	184	184
query46	1067	636	638	636
query47	1644	1576	1583	1576
query48	332	266	255	255
query49	1079	317	313	313
query50	801	318	325	318
query51	5344	5283	5241	5241
query52	89	87	87	87
query53	215	155	138	138
query54	1359	551	568	551
query55	89	89	75	75
query56	185	188	187	187
query57	1013	962	954	954
query58	221	189	197	189
query59	2821	2572	2557	2557
query60	228	234	226	226
query61	86	86	86	86
query62	649	454	454	454
query63	158	145	140	140
query64	4949	1736	1736	1736
query65	3330	3256	3218	3218
query66	1168	337	360	337
query67	15549	15679	15299	15299
query68	12316	530	540	530
query69	511	254	250	250
query70	1690	1542	1517	1517
query71	495	217	242	217
query72	5716	3545	3528	3528
query73	2845	312	309	309
query74	7027	6357	6504	6357
query75	4990	2298	2245	2245
query76	6266	1114	1121	1114
query77	691	251	253	251
query78	9088	8610	8868	8610
query79	1028	516	502	502
query80	537	358	344	344
query81	466	209	208	208
query82	205	96	92	92
query83	162	137	138	137
query84	248	51	52	51
query85	914	278	282	278
query86	393	392	393	392
query87	3513	3404	3356	3356
query88	2931	2250	2259	2250
query89	352	264	257	257
query90	1914	194	196	194
query91	122	90	95	90
query92	53	52	51	51
query93	1293	478	443	443
query94	794	182	181	181
query95	471	398	400	398
query96	615	316	328	316
query97	4267	4144	4152	4144
query98	207	193	194	193
query99	1111	860	827	827
Total cold run time: 291627 ms
Total hot run time: 178446 ms

@jackwener
Copy link
Member Author

run buildall

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpch-tools

Tpch sf100 test result on commit 5ff80604802e32473771e59962077e825654abc6, data reload: false

------ Round 1 ----------------------------------
q1	17645	5307	5063	5063
q2	2047	157	142	142
q3	10532	1120	1098	1098
q4	10176	762	779	762
q5	7789	3024	2960	2960
q6	214	135	138	135
q7	904	510	551	510
q8	9286	2023	2030	2023
q9	6937	6422	6378	6378
q10	8246	3098	3004	3004
q11	436	230	216	216
q12	382	234	228	228
q13	17999	3644	3661	3644
q14	246	212	224	212
q15	570	528	517	517
q16	459	396	392	392
q17	960	468	460	460
q18	7308	6713	6730	6713
q19	1584	1404	1413	1404
q20	708	353	306	306
q21	2833	2367	2430	2367
q22	370	320	334	320
Total cold run time: 107631 ms
Total hot run time: 38854 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5180	5119	5150	5119
q2	342	235	226	226
q3	3326	3302	3279	3279
q4	2120	2041	1981	1981
q5	5811	5821	5815	5815
q6	213	125	125	125
q7	2376	1905	1913	1905
q8	3356	3427	3417	3417
q9	8875	8854	8802	8802
q10	3770	3850	3838	3838
q11	589	483	496	483
q12	807	629	633	629
q13	6955	3222	3168	3168
q14	295	285	257	257
q15	574	525	513	513
q16	549	490	493	490
q17	1951	1787	1788	1787
q18	8702	8345	8344	8344
q19	1629	1612	1608	1608
q20	2208	1955	1947	1947
q21	5713	5265	5398	5265
q22	532	503	515	503
Total cold run time: 65873 ms
Total hot run time: 59501 ms

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpch-tools

Tpch sf100 test result on commit 5ff80604802e32473771e59962077e825654abc6, data reload: false

run tpch-sf100 query with default conf and session variables
q1	5471	5159	5100	5100
q2	386	167	147	147
q3	1458	1219	1160	1160
q4	1101	786	732	732
q5	3157	3096	3127	3096
q6	224	137	137	137
q7	982	560	513	513
q8	2144	2213	2188	2188
q9	6741	6657	6642	6642
q10	3168	3117	3110	3110
q11	356	225	225	225
q12	371	237	236	236
q13	4384	3658	3603	3603
q14	244	215	210	210
q15	578	535	526	526
q16	469	423	394	394
q17	1035	553	538	538
q18	7113	6698	6739	6698
q19	1639	1476	1463	1463
q20	553	345	323	323
q21	2871	2529	2479	2479
q22	402	315	329	315
Total cold run time: 44847 ms
Total hot run time: 39835 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	5100	5105	5062	5062
q2	337	237	238	237
q3	3358	3329	3302	3302
q4	2158	2075	2068	2068
q5	5954	5914	5927	5914
q6	223	128	132	128
q7	2395	1925	1928	1925
q8	3560	3641	3691	3641
q9	9065	9024	8992	8992
q10	3853	3908	3914	3908
q11	588	478	496	478
q12	803	633	640	633
q13	3934	3178	3148	3148
q14	306	275	273	273
q15	591	523	524	523
q16	551	495	495	495
q17	2034	1822	1793	1793
q18	8756	8337	8441	8337
q19	1751	1674	1691	1674
q20	2283	1999	1968	1968
q21	5850	5345	5374	5345
q22	572	481	534	481
Total cold run time: 64022 ms
Total hot run time: 60325 ms

@doris-robot
Copy link

TPC-DS test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpcds-tools

TPC-DS sf100 test result on commit 5ff80604802e32473771e59962077e825654abc6, data reload: false

run tpcds-sf100 query with default conf and session variables
query1	936	343	331	331
query2	6438	1943	1927	1927
query3	6647	209	203	203
query4	27160	22713	22496	22496
query5	3886	507	558	507
query6	260	183	178	178
query7	4601	268	269	268
query8	229	216	195	195
query9	8249	2523	2516	2516
query10	434	251	244	244
query11	16186	15469	15559	15469
query12	131	75	76	75
query13	1617	317	318	317
query14	11529	7132	6980	6980
query15	216	183	190	183
query16	6114	270	259	259
query17	1827	496	477	477
query18	1801	267	259	259
query19	182	142	133	133
query20	81	75	74	74
query21	187	96	99	96
query22	4821	4771	4830	4771
query23	32167	31192	31062	31062
query24	12089	2824	2757	2757
query25	610	345	341	341
query26	1792	147	143	143
query27	2887	268	273	268
query28	6999	1913	1896	1896
query29	2192	399	391	391
query30	287	147	148	147
query31	967	757	776	757
query32	87	57	51	51
query33	725	265	272	265
query34	875	449	445	445
query35	872	765	801	765
query36	1270	1225	1260	1225
query37	187	69	69	69
query38	3391	3335	3259	3259
query39	1325	1275	1280	1275
query40	302	87	88	87
query41	37	37	35	35
query42	90	82	78	78
query43	513	516	485	485
query44	1065	701	696	696
query45	195	185	187	185
query46	1079	637	646	637
query47	1702	1513	1575	1513
query48	345	257	259	257
query49	1220	328	306	306
query50	791	367	355	355
query51	5371	5243	5269	5243
query52	89	88	85	85
query53	211	150	145	145
query54	1418	554	561	554
query55	97	91	79	79
query56	197	190	187	187
query57	1085	976	1002	976
query58	225	198	197	197
query59	2827	2671	2703	2671
query60	224	222	220	220
query61	88	86	88	86
query62	650	454	464	454
query63	161	152	153	152
query64	5891	1686	1675	1675
query65	3307	3247	3254	3247
query66	1383	348	329	329
query67	15902	15653	15228	15228
query68	11994	527	529	527
query69	505	243	263	243
query70	1689	1583	1470	1470
query71	490	217	213	213
query72	5657	3614	3629	3614
query73	2779	314	348	314
query74	6946	6539	6589	6539
query75	5028	2272	2302	2272
query76	6307	1107	1103	1103
query77	667	274	277	274
query78	9045	8830	8580	8580
query79	1033	508	514	508
query80	557	351	340	340
query81	451	210	209	209
query82	205	96	93	93
query83	157	137	141	137
query84	251	54	55	54
query85	957	292	280	280
query86	385	379	376	376
query87	3553	3378	3388	3378
query88	2977	2266	2267	2266
query89	331	251	253	251
query90	1942	193	183	183
query91	120	93	94	93
query92	58	50	51	50
query93	1764	519	432	432
query94	791	177	173	173
query95	450	426	402	402
query96	633	337	318	318
query97	4280	4178	4188	4178
query98	205	200	185	185
query99	1107	809	828	809
Total cold run time: 292033 ms
Total hot run time: 179100 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 49.28 seconds
stream load tsv: 570 seconds loaded 74807831229 Bytes, about 125 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.3 seconds inserted 10000000 Rows, about 353K ops/s
storage size: 17183739503 Bytes

@jackwener
Copy link
Member Author

run buildall

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpch-tools

Tpch sf100 test result on commit 5ff80604802e32473771e59962077e825654abc6, data reload: false

------ Round 1 ----------------------------------
q1	18664	5287	5043	5043
q2	2030	153	141	141
q3	10614	1114	1077	1077
q4	10234	800	776	776
q5	7835	2969	2908	2908
q6	214	133	135	133
q7	928	506	540	506
q8	9331	2042	1996	1996
q9	6857	6457	6387	6387
q10	8229	3080	3021	3021
q11	433	210	209	209
q12	392	233	237	233
q13	18024	3631	3639	3631
q14	249	215	208	208
q15	563	528	515	515
q16	465	386	397	386
q17	970	468	452	452
q18	7497	6694	6673	6673
q19	1599	1337	1287	1287
q20	681	324	330	324
q21	2810	2349	2424	2349
q22	359	312	343	312
Total cold run time: 108978 ms
Total hot run time: 38567 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5131	5099	5042	5042
q2	343	238	227	227
q3	3303	3305	3281	3281
q4	2128	1997	2014	1997
q5	5824	5815	5776	5776
q6	218	125	129	125
q7	2323	1899	1918	1899
q8	3410	3442	3450	3442
q9	8814	8766	8781	8766
q10	3810	3836	3833	3833
q11	585	464	479	464
q12	790	641	658	641
q13	7467	3213	3155	3155
q14	290	266	264	264
q15	569	522	520	520
q16	569	503	508	503
q17	1933	1789	1766	1766
q18	8780	8552	8335	8335
q19	1633	1601	1553	1553
q20	2160	1960	1948	1948
q21	5605	5192	5278	5192
q22	535	469	486	469
Total cold run time: 66220 ms
Total hot run time: 59198 ms

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpch-tools

Tpch sf100 test result on commit 5ff80604802e32473771e59962077e825654abc6, data reload: false

run tpch-sf100 query with default conf and session variables
q1	5434	5208	5102	5102
q2	386	166	158	158
q3	1461	1123	1131	1123
q4	1091	847	792	792
q5	3114	3105	3128	3105
q6	222	132	136	132
q7	968	554	524	524
q8	2160	2284	2251	2251
q9	6672	6668	6680	6668
q10	3199	3110	3095	3095
q11	352	218	207	207
q12	377	234	235	234
q13	4403	3667	3602	3602
q14	253	219	224	219
q15	573	530	517	517
q16	462	416	426	416
q17	1058	528	549	528
q18	7077	6742	6819	6742
q19	1639	1524	1595	1524
q20	551	331	315	315
q21	2912	2536	2465	2465
q22	383	310	337	310
Total cold run time: 44747 ms
Total hot run time: 40029 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	5128	5081	5045	5045
q2	337	242	231	231
q3	3370	3349	3324	3324
q4	2170	2061	2083	2061
q5	5956	5928	5937	5928
q6	224	125	131	125
q7	2404	1913	1911	1911
q8	3564	3684	3667	3667
q9	9034	8996	9036	8996
q10	3875	3902	3891	3891
q11	574	481	487	481
q12	804	651	630	630
q13	3890	3221	3160	3160
q14	295	277	274	274
q15	590	522	528	522
q16	554	507	498	498
q17	2044	1821	1818	1818
q18	8749	8431	8343	8343
q19	1757	1679	1701	1679
q20	2287	1986	1976	1976
q21	5804	5397	5286	5286
q22	558	486	504	486
Total cold run time: 63968 ms
Total hot run time: 60332 ms

@doris-robot
Copy link

TPC-DS test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpcds-tools

TPC-DS sf100 test result on commit 5ff80604802e32473771e59962077e825654abc6, data reload: false

run tpcds-sf100 query with default conf and session variables
query1	921	345	335	335
query2	4936	2024	1808	1808
query3	6649	209	206	206
query4	26949	22386	22311	22311
query5	3378	515	501	501
query6	265	180	177	177
query7	4572	265	262	262
query8	223	200	197	197
query9	8267	2573	2524	2524
query10	506	237	234	234
query11	16097	15404	15429	15404
query12	125	76	76	76
query13	1623	312	311	311
query14	10288	7165	7088	7088
query15	211	186	186	186
query16	5675	264	269	264
query17	1820	492	480	480
query18	1754	272	267	267
query19	176	128	132	128
query20	79	74	74	74
query21	184	104	96	96
query22	5236	5046	4973	4973
query23	32120	31391	31158	31158
query24	12123	2800	2820	2800
query25	594	341	342	341
query26	1699	141	143	141
query27	2905	278	279	278
query28	6990	1875	1870	1870
query29	1867	379	375	375
query30	283	142	142	142
query31	980	761	753	753
query32	83	60	57	57
query33	737	253	256	253
query34	941	441	435	435
query35	878	774	766	766
query36	1294	1174	1242	1174
query37	173	69	72	69
query38	3406	3271	3246	3246
query39	1312	1269	1278	1269
query40	290	86	90	86
query41	38	35	36	35
query42	84	97	80	80
query43	539	479	506	479
query44	1053	703	710	703
query45	194	185	176	176
query46	1054	634	646	634
query47	1672	1562	1495	1495
query48	338	252	247	247
query49	1175	302	314	302
query50	780	376	374	374
query51	5404	5359	5181	5181
query52	89	80	81	80
query53	208	144	146	144
query54	1426	542	554	542
query55	95	84	81	81
query56	176	188	189	188
query57	1005	981	974	974
query58	224	187	200	187
query59	2770	2724	2619	2619
query60	226	218	223	218
query61	82	82	83	82
query62	723	455	458	455
query63	169	141	145	141
query64	5644	1705	1726	1705
query65	3328	3246	3279	3246
query66	1384	343	334	334
query67	15804	15208	15337	15208
query68	12269	498	495	495
query69	481	249	261	249
query70	1677	1460	1527	1460
query71	478	214	200	200
query72	5519	3487	3477	3477
query73	2781	311	308	308
query74	6940	6346	6320	6320
query75	4952	2256	2247	2247
query76	6052	1115	1100	1100
query77	659	239	252	239
query78	9161	8605	8546	8546
query79	1181	507	488	488
query80	636	346	355	346
query81	456	212	204	204
query82	191	95	92	92
query83	160	134	136	134
query84	246	52	51	51
query85	962	267	261	261
query86	385	373	369	369
query87	3558	3364	3338	3338
query88	3260	2249	2255	2249
query89	341	256	260	256
query90	1899	192	189	189
query91	117	87	87	87
query92	60	52	58	52
query93	2183	459	419	419
query94	892	177	174	174
query95	447	407	404	404
query96	631	319	320	319
query97	4252	4135	4167	4135
query98	201	203	199	199
query99	1064	835	829	829
Total cold run time: 288384 ms
Total hot run time: 178261 ms

Copy link
Contributor

github-actions bot commented Jan 5, 2024

PR approved by anyone and no changes requested.

Copy link
Contributor

github-actions bot commented Jan 5, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 5, 2024
@englefly englefly merged commit 7a0734d into apache:master Jan 5, 2024
27 of 28 checks passed
@apache apache deleted a comment from doris-robot Jan 9, 2024
@apache apache deleted a comment from doris-robot Jan 9, 2024
@apache apache deleted a comment from doris-robot Jan 9, 2024
@apache apache deleted a comment from doris-robot Jan 9, 2024
@apache apache deleted a comment from doris-robot Jan 9, 2024
@apache apache deleted a comment from doris-robot Jan 9, 2024
@wm1581066 wm1581066 added the usercase Important user case type label label Jan 12, 2024
HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Jan 12, 2024
jackwener added a commit to jackwener/doris that referenced this pull request Jan 16, 2024
jackwener added a commit to jackwener/doris that referenced this pull request Jan 16, 2024
jackwener added a commit to jackwener/doris that referenced this pull request Jan 16, 2024
@jackwener jackwener deleted the in branch January 16, 2024 08:12
jackwener added a commit that referenced this pull request Jan 16, 2024
jackwener added a commit to jackwener/doris that referenced this pull request Jan 29, 2024
jackwener added a commit that referenced this pull request Jan 30, 2024
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.5-merged dev/3.0.0-merged meta-change reviewed usercase Important user case type label
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants