Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](mv) Fix use sync mv wrongly when use rbo materialized view rewrite rule #47650

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

seawinde
Copy link
Contributor

@seawinde seawinde commented Feb 8, 2025

What problem does this PR solve?

if mv def sql is as following:

create matrialized view test_mv as 
select k1, k3, sum(k2), count(k4) from ${tblName} group by k1, k3;

if query are as following, should not use test_mv above

select count(k1) from agg_use_key_direct;

select sum(k1) from agg_use_key_direct

select avg(k1) from agg_use_key_direct

the wrong case would happen when in branch-2.1 or set enable_sync_mv_cost_based_rewrite = false; on branch-3.0 or master, the pr fix this

if query are as following with distinct or is max or min, can use test_mv above

select count(distinct k1) from agg_use_key_direct;

select sum(distinct k1) from agg_use_key_direct

select avg(distinct k1) from agg_use_key_direct

select max(k1) from agg_use_key_direct

select min(k1) from agg_use_key_direct

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

Fix use sync mv wrongly when use rbo materialized view rewrite rule

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@seawinde
Copy link
Contributor Author

seawinde commented Feb 8, 2025

run buildall

Copy link
Contributor

github-actions bot commented Feb 8, 2025

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Feb 8, 2025
Copy link
Contributor

github-actions bot commented Feb 8, 2025

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 31877 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d5a5290beab4e8322333c2b59f146562aba92d94, data reload: false

------ Round 1 ----------------------------------
q1	17609	5177	5254	5177
q2	2046	306	164	164
q3	10406	1406	728	728
q4	10218	1026	542	542
q5	7537	2387	2378	2378
q6	182	168	133	133
q7	902	778	618	618
q8	9297	1325	1134	1134
q9	4784	4601	4664	4601
q10	6830	2310	1910	1910
q11	466	280	259	259
q12	340	350	226	226
q13	17770	3666	3172	3172
q14	227	245	203	203
q15	511	463	465	463
q16	628	626	609	609
q17	558	868	334	334
q18	6647	6283	6144	6144
q19	1071	957	539	539
q20	309	322	187	187
q21	2888	2299	2041	2041
q22	366	341	315	315
Total cold run time: 101592 ms
Total hot run time: 31877 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5175	5134	5140	5134
q2	233	328	234	234
q3	2154	2695	2341	2341
q4	1490	1897	1425	1425
q5	4266	4154	4223	4154
q6	207	166	123	123
q7	1881	1836	1687	1687
q8	2636	2722	2630	2630
q9	7260	7124	7088	7088
q10	2996	3168	2753	2753
q11	573	504	479	479
q12	683	759	614	614
q13	3569	3968	3323	3323
q14	283	297	288	288
q15	520	472	467	467
q16	636	653	615	615
q17	1135	1593	1357	1357
q18	7522	7312	7348	7312
q19	802	832	833	832
q20	1996	2006	1858	1858
q21	5371	5087	4850	4850
q22	629	596	537	537
Total cold run time: 52017 ms
Total hot run time: 50101 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 183510 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d5a5290beab4e8322333c2b59f146562aba92d94, data reload: false

query1	971	375	378	375
query2	6525	1931	1891	1891
query3	6804	210	216	210
query4	26190	23516	23460	23460
query5	4296	678	489	489
query6	272	184	204	184
query7	4606	492	296	296
query8	297	229	223	223
query9	8608	2573	2567	2567
query10	477	321	258	258
query11	15789	15103	14918	14918
query12	155	110	107	107
query13	1650	522	394	394
query14	9928	6916	6217	6217
query15	204	194	179	179
query16	7440	657	462	462
query17	1164	695	540	540
query18	1957	394	295	295
query19	190	179	149	149
query20	129	121	116	116
query21	209	122	103	103
query22	4464	4406	4519	4406
query23	34391	33365	32792	32792
query24	7743	2327	2351	2327
query25	557	445	372	372
query26	1229	267	149	149
query27	2136	475	324	324
query28	3911	2398	2369	2369
query29	715	538	413	413
query30	231	181	149	149
query31	947	886	783	783
query32	68	60	62	60
query33	557	342	294	294
query34	781	845	504	504
query35	778	812	728	728
query36	975	990	914	914
query37	119	101	78	78
query38	4168	4201	4058	4058
query39	1461	1368	1415	1368
query40	209	118	112	112
query41	58	57	59	57
query42	117	101	103	101
query43	510	505	490	490
query44	1299	781	789	781
query45	181	165	157	157
query46	855	1051	636	636
query47	1785	1791	1733	1733
query48	391	422	310	310
query49	773	514	408	408
query50	674	747	406	406
query51	4124	4157	4090	4090
query52	102	108	92	92
query53	232	258	183	183
query54	469	482	401	401
query55	79	76	83	76
query56	249	268	244	244
query57	1111	1143	1064	1064
query58	241	225	292	225
query59	2710	2728	2574	2574
query60	271	274	260	260
query61	111	113	107	107
query62	794	740	674	674
query63	221	178	183	178
query64	4342	977	683	683
query65	3208	3158	3130	3130
query66	1074	397	299	299
query67	15770	15471	15467	15467
query68	2220	768	524	524
query69	418	299	270	270
query70	1193	1145	1037	1037
query71	335	324	248	248
query72	5502	3573	3740	3573
query73	637	738	351	351
query74	8867	9227	9008	9008
query75	3100	3177	2692	2692
query76	2299	1159	767	767
query77	344	365	289	289
query78	9887	10053	9273	9273
query79	1299	878	597	597
query80	1405	563	468	468
query81	533	282	307	282
query82	417	157	120	120
query83	257	164	156	156
query84	229	104	71	71
query85	789	353	289	289
query86	375	316	295	295
query87	4452	4454	4353	4353
query88	2865	2211	2185	2185
query89	377	317	274	274
query90	1732	198	193	193
query91	133	136	107	107
query92	130	60	56	56
query93	1148	975	577	577
query94	642	383	291	291
query95	347	260	250	250
query96	488	549	268	268
query97	2737	2848	2717	2717
query98	217	211	204	204
query99	1298	1463	1266	1266
Total cold run time: 261724 ms
Total hot run time: 183510 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.94 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d5a5290beab4e8322333c2b59f146562aba92d94, data reload: false

query1	0.04	0.04	0.04
query2	0.07	0.03	0.04
query3	0.24	0.06	0.06
query4	1.62	0.10	0.10
query5	0.43	0.42	0.39
query6	1.14	0.67	0.67
query7	0.03	0.02	0.01
query8	0.04	0.02	0.03
query9	0.60	0.52	0.53
query10	0.57	0.58	0.57
query11	0.15	0.10	0.10
query12	0.15	0.11	0.11
query13	0.62	0.61	0.60
query14	2.72	2.81	2.83
query15	0.93	0.85	0.85
query16	0.38	0.37	0.38
query17	1.04	1.03	1.02
query18	0.21	0.19	0.19
query19	1.94	1.88	2.07
query20	0.01	0.01	0.01
query21	15.35	0.88	0.55
query22	0.78	1.12	0.84
query23	14.77	1.41	0.62
query24	6.75	1.59	1.90
query25	0.56	0.20	0.15
query26	0.54	0.16	0.14
query27	0.05	0.05	0.04
query28	9.94	0.81	0.43
query29	12.54	3.93	3.30
query30	0.25	0.09	0.07
query31	2.82	0.57	0.38
query32	3.22	0.55	0.46
query33	2.98	3.09	3.11
query34	15.82	5.16	4.60
query35	4.54	4.53	4.56
query36	0.66	0.50	0.48
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.02
query40	0.17	0.14	0.12
query41	0.08	0.03	0.03
query42	0.04	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 105 s
Total hot run time: 31.94 s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.x dev/3.0.x p0_w reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants