Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

branch-3.0: [opt](trino-connector) close scanner after finished #47857 #47873

Open
wants to merge 1 commit into
base: branch-3.0
Choose a base branch
from

Conversation

github-actions[bot]
Copy link
Contributor

Cherry-picked from #47857

### What problem does this PR solve?

The source should be closed after scanner is done to save memory.
Or there will be some objects remaining like:

```
  13:        192980        9263040  org.apache.kudu.client.RowwiseRowResult
```

Although these objects can be gc, eventually.
@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring reopened this Feb 13, 2025
@hello-stephen
Copy link
Contributor

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40653 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2dabb5415349be6a04b18f33544fc8e0768b31fe, data reload: false

------ Round 1 ----------------------------------
q1	17587	7453	7227	7227
q2	2068	176	171	171
q3	10594	1079	1184	1079
q4	10560	748	732	732
q5	7765	2835	2804	2804
q6	242	148	142	142
q7	969	626	622	622
q8	9356	1927	1978	1927
q9	6541	6397	6367	6367
q10	7085	2346	2353	2346
q11	477	274	272	272
q12	403	217	210	210
q13	18750	2967	2983	2967
q14	246	206	223	206
q15	563	530	516	516
q16	658	588	601	588
q17	964	527	595	527
q18	7235	6702	6472	6472
q19	1397	1104	1132	1104
q20	470	208	200	200
q21	4001	3181	3187	3181
q22	1098	1004	993	993
Total cold run time: 109029 ms
Total hot run time: 40653 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7192	7217	7223	7217
q2	331	232	230	230
q3	2911	2949	2907	2907
q4	2013	1770	1774	1770
q5	5653	5712	5722	5712
q6	223	143	141	141
q7	2286	1799	1839	1799
q8	3320	3477	3501	3477
q9	8737	8856	8818	8818
q10	3593	3578	3560	3560
q11	611	514	501	501
q12	818	618	600	600
q13	9789	3165	3143	3143
q14	306	276	275	275
q15	596	542	525	525
q16	708	649	668	649
q17	1847	1612	1600	1600
q18	8218	7696	7656	7656
q19	1646	1626	1515	1515
q20	2203	1843	1867	1843
q21	5396	5403	5412	5403
q22	1155	1045	1063	1045
Total cold run time: 69552 ms
Total hot run time: 60386 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 198439 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2dabb5415349be6a04b18f33544fc8e0768b31fe, data reload: false

query1	1301	940	907	907
query2	6249	2162	2107	2107
query3	10809	4306	4484	4306
query4	66378	28305	23481	23481
query5	4940	450	446	446
query6	403	184	174	174
query7	5556	305	299	299
query8	303	229	224	224
query9	8834	2710	2700	2700
query10	445	274	275	274
query11	17166	15454	16050	15454
query12	155	101	109	101
query13	1505	452	432	432
query14	10861	7746	7208	7208
query15	197	184	175	175
query16	7184	481	507	481
query17	1073	603	595	595
query18	1823	331	316	316
query19	215	167	169	167
query20	123	112	122	112
query21	226	115	117	115
query22	5036	4485	4409	4409
query23	34478	34352	34146	34146
query24	6179	2921	2894	2894
query25	552	420	429	420
query26	667	167	173	167
query27	1721	374	365	365
query28	4550	2472	2440	2440
query29	720	470	474	470
query30	243	161	165	161
query31	994	806	838	806
query32	66	57	60	57
query33	406	289	282	282
query34	914	511	527	511
query35	870	754	728	728
query36	1103	955	953	953
query37	120	66	75	66
query38	4075	3910	3976	3910
query39	1549	1475	1491	1475
query40	208	103	99	99
query41	51	49	49	49
query42	115	100	104	100
query43	556	507	499	499
query44	1199	838	846	838
query45	184	173	169	169
query46	1152	730	708	708
query47	2061	1979	1961	1961
query48	482	401	390	390
query49	742	395	405	395
query50	845	443	432	432
query51	7358	7185	7175	7175
query52	100	89	84	84
query53	259	185	182	182
query54	545	442	456	442
query55	78	74	78	74
query56	269	249	240	240
query57	1225	1127	1105	1105
query58	218	204	214	204
query59	3279	3120	3231	3120
query60	268	246	253	246
query61	118	138	108	108
query62	841	732	707	707
query63	221	185	186	185
query64	1477	679	648	648
query65	3260	3185	3193	3185
query66	734	302	302	302
query67	16036	15693	15632	15632
query68	3703	613	572	572
query69	439	290	261	261
query70	1183	1085	1147	1085
query71	362	250	261	250
query72	6401	4113	4037	4037
query73	748	348	353	348
query74	10375	9169	9051	9051
query75	3340	2606	2657	2606
query76	1775	1114	1168	1114
query77	488	274	292	274
query78	10569	9682	9523	9523
query79	1771	599	599	599
query80	1376	438	435	435
query81	534	243	231	231
query82	1287	117	114	114
query83	274	145	140	140
query84	280	80	78	78
query85	1034	319	303	303
query86	420	261	280	261
query87	4441	4333	4381	4333
query88	3897	2375	2344	2344
query89	426	292	291	291
query90	1943	183	187	183
query91	183	150	148	148
query92	68	48	51	48
query93	2395	548	561	548
query94	865	290	294	290
query95	375	254	251	251
query96	608	289	292	289
query97	3325	3215	3244	3215
query98	210	216	195	195
query99	1679	1409	1425	1409
Total cold run time: 320644 ms
Total hot run time: 198439 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.99 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2dabb5415349be6a04b18f33544fc8e0768b31fe, data reload: false

query1	0.03	0.03	0.03
query2	0.06	0.03	0.02
query3	0.23	0.06	0.06
query4	1.65	0.11	0.11
query5	0.51	0.49	0.54
query6	1.12	0.74	0.72
query7	0.02	0.02	0.01
query8	0.04	0.04	0.03
query9	0.56	0.52	0.48
query10	0.54	0.56	0.56
query11	0.14	0.11	0.10
query12	0.15	0.11	0.11
query13	0.60	0.59	0.59
query14	2.84	2.83	2.82
query15	0.91	0.83	0.84
query16	0.38	0.37	0.36
query17	1.02	1.01	1.05
query18	0.23	0.23	0.22
query19	1.92	1.78	2.14
query20	0.02	0.01	0.01
query21	15.35	0.58	0.57
query22	2.67	2.14	2.47
query23	17.01	1.03	0.89
query24	3.26	0.90	1.58
query25	0.24	0.07	0.08
query26	0.58	0.13	0.13
query27	0.04	0.04	0.05
query28	10.24	1.11	1.07
query29	12.58	3.26	3.25
query30	0.24	0.05	0.06
query31	2.87	0.38	0.38
query32	3.24	0.45	0.46
query33	3.01	3.00	3.01
query34	17.04	4.50	4.43
query35	4.56	4.49	4.52
query36	0.68	0.48	0.52
query37	0.09	0.06	0.06
query38	0.04	0.04	0.04
query39	0.03	0.03	0.02
query40	0.15	0.12	0.12
query41	0.08	0.03	0.02
query42	0.04	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 107.05 s
Total hot run time: 32.99 s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants