Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[feature](Nereids): inferPredicate support InPredicate into Branch 2.0 #30007

Merged
merged 1 commit into from
Jan 16, 2024
Merged

[feature](Nereids): inferPredicate support InPredicate into Branch 2.0 #30007

merged 1 commit into from
Jan 16, 2024

Conversation

jackwener
Copy link
Member

@jackwener jackwener commented Jan 16, 2024 •
edited
Loading

Proposed changes

(cherry picked from commit 7a0734d)

#29458

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

@jackwener
Copy link
Member Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 50016 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f1f51b9503d44327118106530987d6e63b6cceef, data reload: false

------ Round 1 ----------------------------------
q1	17924	4381	4348	4348
q2	2042	157	146	146
q3	10332	1949	1906	1906
q4	10100	1240	1345	1240
q5	8493	4044	3976	3976
q6	226	125	126	125
q7	2051	1615	1617	1615
q8	9274	2726	2724	2724
q9	10790	10631	10531	10531
q10	8678	3521	3447	3447
q11	422	235	247	235
q12	472	298	299	298
q13	18363	3989	4058	3989
q14	358	322	320	320
q15	516	458	456	456
q16	711	597	590	590
q17	1130	961	968	961
q18	7277	6777	7051	6777
q19	1697	1561	1504	1504
q20	562	309	315	309
q21	4488	4135	4152	4135
q22	510	405	384	384
Total cold run time: 116416 ms
Total hot run time: 50016 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4345	4297	4314	4297
q2	323	220	220	220
q3	4137	4188	4154	4154
q4	2736	2756	2731	2731
q5	7352	7195	7293	7195
q6	235	122	121	121
q7	3270	2831	2828	2828
q8	4429	4482	4493	4482
q9	17123	16867	16918	16867
q10	4281	4300	4287	4287
q11	789	697	670	670
q12	1018	864	854	854
q13	6847	3779	3741	3741
q14	447	425	431	425
q15	498	455	461	455
q16	778	718	712	712
q17	3808	3836	3865	3836
q18	8855	8771	8813	8771
q19	1713	1717	1643	1643
q20	2401	2154	2141	2141
q21	8433	8438	8727	8438
q22	1015	956	972	956
Total cold run time: 84833 ms
Total hot run time: 79824 ms

@jackwener
Copy link
Member Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 49541 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3bbb5587e8dca1edee21ab9019746dfff7602649, data reload: false

------ Round 1 ----------------------------------
q1	17821	4379	4349	4349
q2	2045	146	136	136
q3	10466	1836	1872	1836
q4	10335	1224	1313	1224
q5	8590	3947	3951	3947
q6	225	123	121	121
q7	2013	1594	1602	1594
q8	9285	2698	2707	2698
q9	10815	10602	10324	10324
q10	8671	3545	3454	3454
q11	423	258	233	233
q12	470	301	305	301
q13	18354	3962	4002	3962
q14	345	337	324	324
q15	496	458	460	458
q16	707	599	595	595
q17	1120	959	937	937
q18	7220	6782	6939	6782
q19	1678	1539	1515	1515
q20	526	301	307	301
q21	4471	4102	4052	4052
q22	487	399	398	398
Total cold run time: 116563 ms
Total hot run time: 49541 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4302	4301	4294	4294
q2	318	223	217	217
q3	4128	4098	4117	4098
q4	2738	2730	2746	2730
q5	7319	7289	7293	7289
q6	236	118	115	115
q7	3306	2814	2894	2814
q8	4331	4461	4530	4461
q9	17078	17033	16864	16864
q10	4275	4213	4267	4213
q11	810	691	678	678
q12	1015	859	828	828
q13	6805	3733	3744	3733
q14	442	417	419	417
q15	504	457	446	446
q16	756	698	710	698
q17	3902	3891	3902	3891
q18	8844	8880	8740	8740
q19	1710	1696	1637	1637
q20	2383	2156	2094	2094
q21	8611	8444	8426	8426
q22	1023	1003	915	915
Total cold run time: 84836 ms
Total hot run time: 79598 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 234388 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3bbb5587e8dca1edee21ab9019746dfff7602649, data reload: false

query1	910	392	385	385
query2	6537	2157	2300	2157
query3	6920	203	196	196
query4	20977	17981	18051	17981
query5	19708	6239	6221	6221
query6	292	214	226	214
query7	4144	298	286	286
query8	259	250	228	228
query9	3086	2666	2600	2600
query10	407	284	279	279
query11	11370	10724	10700	10700
query12	119	76	70	70
query13	5643	642	625	625
query14	17313	13210	13254	13210
query15	365	239	228	228
query16	6463	262	265	262
query17	3189	865	854	854
query18	2260	399	402	399
query19	217	140	140	140
query20	70	74	76	74
query21	187	97	91	91
query22	5262	5160	5085	5085
query23	32557	31847	31843	31843
query24	7038	6548	6453	6453
query25	536	411	410	410
query26	622	157	155	155
query27	2041	299	293	293
query28	6103	2222	2214	2214
query29	2867	2860	2713	2713
query30	243	160	162	160
query31	887	740	737	737
query32	65	56	55	55
query33	398	254	254	254
query34	842	455	474	455
query35	1115	912	968	912
query36	1541	1436	1576	1436
query37	86	59	61	59
query38	3088	2963	2880	2880
query39	1349	1318	1321	1318
query40	198	94	96	94
query41	35	30	31	30
query42	87	80	96	80
query43	638	569	584	569
query44	1118	707	718	707
query45	239	229	229	229
query46	1231	976	979	976
query47	1782	1666	1675	1666
query48	992	675	651	651
query49	622	366	364	364
query50	851	593	554	554
query51	5717	5473	5525	5473
query52	83	73	80	73
query53	432	322	308	308
query54	2417	2290	2233	2233
query55	79	79	81	79
query56	223	192	184	184
query57	1274	1191	1118	1118
query58	209	188	193	188
query59	3381	3296	3177	3177
query60	201	212	186	186
query61	87	86	85	85
query62	800	488	555	488
query63	486	329	322	322
query64	2044	1312	1296	1296
query65	3689	3669	3605	3605
query66	845	360	368	360
query67	16115	16860	15184	15184
query68	8105	656	656	656
query69	560	329	338	329
query70	2111	1774	1636	1636
query71	356	287	319	287
query72	4316	3345	3375	3345
query73	717	313	309	309
query74	6362	5860	5820	5820
query75	4243	3462	3471	3462
query76	4731	1193	1220	1193
query77	589	238	245	238
query78	31926	49576	44533	44533
query79	12654	647	646	646
query80	3818	391	381	381
query81	566	230	228	228
query82	386	99	95	95
query83	333	139	140	139
query84	257	69	71	69
query85	1778	280	274	274
query86	447	379	428	379
query87	3287	3019	2975	2975
query88	5578	2302	2313	2302
query89	345	295	284	284
query90	2503	214	217	214
query91	165	126	118	118
query92	63	51	52	51
query93	1053	585	581	581
query94	1660	210	206	206
query95	1112	1054	1041	1041
query96	632	336	327	327
query97	6563	6416	6290	6290
query98	188	179	176	176
query99	3538	893	1063	893
Total cold run time: 333893 ms
Total hot run time: 234388 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.41 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3bbb5587e8dca1edee21ab9019746dfff7602649, data reload: false

query1	0.02	0.01	0.02
query2	0.06	0.02	0.02
query3	0.25	0.05	0.05
query4	1.84	0.07	0.07
query5	0.53	0.52	0.52
query6	1.23	0.62	0.61
query7	0.02	0.02	0.01
query8	0.03	0.02	0.02
query9	0.54	0.50	0.48
query10	0.53	0.53	0.54
query11	0.12	0.09	0.08
query12	0.12	0.09	0.09
query13	0.62	0.61	0.60
query14	0.78	0.77	0.78
query15	0.80	0.77	0.78
query16	0.36	0.36	0.37
query17	1.01	1.02	1.02
query18	0.23	0.24	0.24
query19	1.93	1.81	1.84
query20	0.01	0.01	0.01
query21	15.49	0.56	0.55
query22	1.93	2.30	1.21
query23	17.01	0.96	0.87
query24	7.14	1.36	1.12
query25	2.22	0.16	0.14
query26	0.14	0.15	0.16
query27	0.15	0.16	0.15
query28	6.35	0.74	0.75
query29	12.61	2.36	2.10
query30	0.55	0.55	0.51
query31	2.81	0.40	0.38
query32	3.39	0.50	0.49
query33	3.09	3.07	3.07
query34	15.26	4.81	4.78
query35	4.86	4.84	4.83
query36	1.05	1.00	1.02
query37	0.06	0.04	0.05
query38	0.04	0.02	0.02
query39	0.02	0.01	0.01
query40	0.16	0.14	0.14
query41	0.07	0.01	0.01
query42	0.02	0.02	0.02
query43	0.02	0.01	0.02
Total cold run time: 105.47 s
Total hot run time: 30.41 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 3bbb5587e8dca1edee21ab9019746dfff7602649 with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.4 seconds inserted 10000000 Rows, about 467K ops/s

@jackwener jackwener requested a review from xiaokang January 16, 2024 04:47
@jackwener jackwener merged commit a24c035 into apache:branch-2.0 Jan 16, 2024
25 of 27 checks passed
@jackwener jackwener deleted the branch-2.0_limit branch January 16, 2024 08:21
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@jackwener
[feature](Nereids): InferPredicates support In (apache#29458) (apache…
7027430 
…#30007)

(cherry picked from commit 7a0734d)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@xiaokang xiaokang Awaiting requested review from xiaokang

Assignees
No one assigned
Labels
area/nereids kind/test
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@jackwener @doris-robot