Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](auditloader) support audit table millisecond and fix stmt truncated by '\r' #29479

Merged
merged 1 commit into from
Jan 3, 2024
Merged

[fix](auditloader) support audit table millisecond and fix stmt truncated by '\r' #29479

merged 1 commit into from
Jan 3, 2024

Conversation

nextdreamblue
Copy link
Contributor

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

@nextdreamblue
Copy link
Contributor Author

run buildall

lide-reed
lide-reed approved these changes Jan 3, 2024
View reviewed changes
Copy link
Contributor

@lide-reed lide-reed left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 3, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jan 3, 2024

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jan 3, 2024

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpch-tools

Tpch sf100 test result on commit 2edc59f1c174d2814e5addb954c49d7a03397683, data reload: false

------ Round 1 ----------------------------------
q1	18761	5445	5046	5046
q2	2035	157	148	148
q3	10605	1159	1093	1093
q4	10204	755	900	755
q5	7785	2946	2977	2946
q6	209	137	131	131
q7	949	517	534	517
q8	9345	2018	2007	2007
q9	6838	6411	6400	6400
q10	8246	3015	3042	3015
q11	430	222	221	221
q12	389	236	231	231
q13	18000	3614	3601	3601
q14	244	222	213	213
q15	574	551	534	534
q16	459	399	401	399
q17	953	526	579	526
q18	7404	6847	6774	6774
q19	1583	1297	1313	1297
q20	744	334	356	334
q21	2863	2345	2346	2345
q22	368	306	331	306
Total cold run time: 108988 ms
Total hot run time: 38839 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5143	5102	5109	5102
q2	338	236	242	236
q3	3324	3292	3265	3265
q4	2162	2045	2022	2022
q5	5826	5815	5779	5779
q6	211	124	121	121
q7	2311	1945	1903	1903
q8	3377	3417	3456	3417
q9	8843	8755	8773	8755
q10	3798	3821	3872	3821
q11	577	481	495	481
q12	823	635	616	616
q13	10699	3196	3190	3190
q14	280	272	267	267
q15	588	531	535	531
q16	580	480	517	480
q17	1941	1777	1762	1762
q18	8759	8449	8255	8255
q19	1617	1621	1618	1618
q20	2254	1987	1970	1970
q21	5502	5286	5242	5242
q22	587	521	492	492
Total cold run time: 69540 ms
Total hot run time: 59325 ms

@nextdreamblue nextdreamblue mentioned this pull request Jan 3, 2024
Merged
cambyzju
cambyzju approved these changes Jan 3, 2024
View reviewed changes
Copy link
Contributor

@cambyzju cambyzju left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@doris-robot
Copy link

TPC-DS test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpcds-tools

TPC-DS sf100 test result on commit 2edc59f1c174d2814e5addb954c49d7a03397683, data reload: false

run tpcds-sf100 query with default conf and session variables
query1	889	345	336	336
query2	5122	1907	1873	1873
query3	5185	205	197	197
query4	24009	22452	22288	22288
query5	3235	508	505	505
query6	254	193	188	188
query7	4091	289	294	289
query8	251	223	216	216
query9	7365	2742	2657	2657
query10	337	237	237	237
query11	16026	15362	15731	15362
query12	129	76	73	73
query13	1507	329	319	319
query14	10512	7411	7340	7340
query15	309	218	225	218
query16	5673	269	271	269
query17	1613	523	530	523
query18	1782	289	263	263
query19	180	155	153	153
query20	83	76	82	76
query21	159	101	101	101
query22	5081	4820	4507	4507
query23	32243	31643	31202	31202
query24	11710	3457	3423	3423
query25	624	374	374	374
query26	1603	158	157	157
query27	2709	288	291	288
query28	6277	1966	1942	1942
query29	1810	422	407	407
query30	305	149	156	149
query31	1118	777	820	777
query32	86	59	54	54
query33	673	292	270	270
query34	841	444	435	435
query35	968	783	809	783
query36	1350	1282	1167	1167
query37	175	74	75	74
query38	3456	3306	3274	3274
query39	1341	1307	1285	1285
query40	286	92	93	92
query41	38	36	35	35
query42	102	89	93	89
query43	519	500	479	479
query44	1035	706	717	706
query45	232	224	223	223
query46	1167	781	748	748
query47	1652	1574	1581	1574
query48	370	277	281	277
query49	1139	331	329	329
query50	1077	877	831	831
query51	5361	5261	5418	5261
query52	91	91	90	90
query53	217	185	179	179
query54	1581	607	599	599
query55	95	86	85	85
query56	209	204	192	192
query57	1049	965	996	965
query58	228	216	207	207
query59	2703	2617	2480	2480
query60	244	238	242	238
query61	84	86	87	86
query62	1457	509	507	507
query63	188	173	165	165
query64	5775	1755	1790	1755
query65	3376	3283	3263	3263
query66	1154	344	324	324
query67	15898	15189	15232	15189
query68	11800	539	561	539
query69	498	256	243	243
query70	1695	1531	1546	1531
query71	485	226	224	224
query72	6468	4094	4067	4067
query73	2778	311	322	311
query74	6857	6489	6505	6489
query75	5263	2257	2312	2257
query76	6091	1190	1127	1127
query77	667	293	310	293
query78	9127	8864	8553	8553
query79	5500	536	535	535
query80	747	391	393	391
query81	483	215	215	215
query82	196	103	103	103
query83	185	142	140	140
query84	244	53	52	52
query85	879	281	283	281
query86	397	412	429	412
query87	3546	3522	3394	3394
query88	3713	2292	2289	2289
query89	358	295	293	293
query90	1798	216	210	210
query91	126	93	94	93
query92	59	60	51	51
query93	3396	442	489	442
query94	761	189	188	188
query95	472	405	394	394
query96	636	321	319	319
query97	4262	4179	4150	4150
query98	215	202	190	190
query99	1579	936	889	889
Total cold run time: 289689 ms
Total hot run time: 181684 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.11 seconds
stream load tsv: 561 seconds loaded 74807831229 Bytes, about 127 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 27.2 seconds inserted 10000000 Rows, about 367K ops/s
storage size: 17183276427 Bytes

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpch-tools

Tpch sf100 test result on commit 2edc59f1c174d2814e5addb954c49d7a03397683, data reload: false

run tpch-sf100 query with default conf and session variables
q1	5491	5128	5157	5128
q2	390	156	158	156
q3	1459	1186	1199	1186
q4	1096	870	793	793
q5	3115	3086	3116	3086
q6	225	137	132	132
q7	996	551	518	518
q8	2158	2216	2207	2207
q9	6708	6634	6669	6634
q10	3183	3176	3094	3094
q11	325	218	214	214
q12	384	229	243	229
q13	4397	3600	3603	3600
q14	243	225	212	212
q15	600	556	557	556
q16	452	410	391	391
q17	1049	542	487	487
q18	7109	6706	6769	6706
q19	1659	1491	1605	1491
q20	578	342	355	342
q21	2838	2499	2445	2445
q22	399	339	317	317
Total cold run time: 44854 ms
Total hot run time: 39924 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	5199	5119	5094	5094
q2	336	240	261	240
q3	3351	3331	3353	3331
q4	2145	2003	1996	1996
q5	5943	5920	5950	5920
q6	232	130	126	126
q7	2358	1909	1868	1868
q8	3582	3650	3630	3630
q9	9052	9015	8980	8980
q10	3878	3936	3939	3936
q11	567	480	481	480
q12	792	646	651	646
q13	3918	3191	3185	3185
q14	313	279	295	279
q15	615	545	528	528
q16	589	531	528	528
q17	2008	1831	1810	1810
q18	8800	8345	8376	8345
q19	1754	1651	1680	1651
q20	2285	2002	1981	1981
q21	5673	5409	5337	5337
q22	582	522	490	490
Total cold run time: 63972 ms
Total hot run time: 60381 ms

@yiguolei yiguolei merged commit d93812d into apache:master Jan 3, 2024
27 of 30 checks passed
nextdreamblue added a commit to nextdreamblue/doris that referenced this pull request Jan 3, 2024
@nextdreamblue
[fix](auditloader) support audit table millisecond and fix stmt trunc…
5d3087e 
…ated by '\r' (apache#29479)

Signed-off-by: nextdreamblue <[email protected]>
@nextdreamblue nextdreamblue mentioned this pull request Jan 3, 2024
Merged
xiaokang pushed a commit that referenced this pull request Jan 7, 2024
@nextdreamblue
[fix](auditloader) support audit table millisecond and fix stmt trunc…
2d0dc8a 
…ated by '\r' (#29479) (#29486)
HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Jan 12, 2024
@nextdreamblue @HappenLee
[fix](auditloader) support audit table millisecond and fix stmt trunc…
070721e 
…ated by '\r' (apache#29479)

Signed-off-by: nextdreamblue <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@lide-reed lide-reed lide-reed approved these changes

@cambyzju cambyzju cambyzju approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/1.2.8-merged dev/2.0.4-merged dev/3.0.0-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
8 participants
@nextdreamblue @doris-robot @lide-reed @cambyzju @xiaokang @morningman @yiguolei @dataroaring