Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[improve](move-memtable) improve error log and message for "not enough streams" #47470

Merged
merged 2 commits into from
Feb 10, 2025
Merged

[improve](move-memtable) improve error log and message for "not enough streams" #47470

merged 2 commits into from
Feb 10, 2025

Conversation

kaijchen
Copy link
Contributor

What problem does this PR solve?

Problem Summary:
improve error log and message for "not enough streams", make it easier to understand.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Jan 27, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@kaijchen
Copy link
Contributor Author

run buildall

@kaijchen
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32484 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b7ac63f18ceb591507c56c01c059dc1c38ce41ce, data reload: false

------ Round 1 ----------------------------------
q1	17585	5491	5379	5379
q2	2054	298	167	167
q3	10564	1280	744	744
q4	10283	959	542	542
q5	8908	2417	2128	2128
q6	199	168	138	138
q7	897	778	606	606
q8	9238	1381	1216	1216
q9	5356	4864	4920	4864
q10	6781	2327	1885	1885
q11	511	274	270	270
q12	341	352	227	227
q13	17762	3731	3172	3172
q14	230	226	206	206
q15	517	475	464	464
q16	637	632	581	581
q17	570	862	325	325
q18	7121	6484	6478	6478
q19	1536	943	553	553
q20	316	323	190	190
q21	2921	2260	2029	2029
q22	383	326	320	320
Total cold run time: 104710 ms
Total hot run time: 32484 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5499	5452	5437	5437
q2	238	332	230	230
q3	2231	2578	2330	2330
q4	1409	1839	1335	1335
q5	4325	4695	4684	4684
q6	181	165	129	129
q7	2073	1967	1850	1850
q8	2605	2763	2654	2654
q9	7236	7162	7314	7162
q10	2989	3250	2784	2784
q11	573	520	507	507
q12	706	773	633	633
q13	3588	3941	3350	3350
q14	278	308	264	264
q15	510	477	455	455
q16	664	684	662	662
q17	1220	1721	1241	1241
q18	7695	7322	7232	7232
q19	785	764	1154	764
q20	2015	2047	1853	1853
q21	5718	5178	4960	4960
q22	595	605	544	544
Total cold run time: 53133 ms
Total hot run time: 51060 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 185530 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b7ac63f18ceb591507c56c01c059dc1c38ce41ce, data reload: false

query1	1005	379	388	379
query2	6535	2036	2000	2000
query3	6792	222	216	216
query4	33405	23349	23031	23031
query5	4330	626	475	475
query6	311	206	195	195
query7	4608	514	317	317
query8	297	243	233	233
query9	9414	2710	2705	2705
query10	488	326	256	256
query11	18406	15197	15064	15064
query12	158	106	110	106
query13	1674	537	421	421
query14	10370	6664	6955	6664
query15	252	191	188	188
query16	8080	627	479	479
query17	1602	770	576	576
query18	2122	414	305	305
query19	209	189	161	161
query20	117	112	111	111
query21	211	126	106	106
query22	4215	4152	4316	4152
query23	34145	33407	33000	33000
query24	6502	2311	2316	2311
query25	498	462	391	391
query26	1205	276	151	151
query27	2007	484	346	346
query28	5410	2474	2456	2456
query29	722	594	420	420
query30	229	189	159	159
query31	931	884	800	800
query32	106	67	62	62
query33	549	359	297	297
query34	766	849	511	511
query35	813	830	764	764
query36	1004	1017	959	959
query37	119	103	86	86
query38	4056	4190	4025	4025
query39	1433	1432	1404	1404
query40	207	116	103	103
query41	53	51	49	49
query42	120	101	101	101
query43	506	529	487	487
query44	1326	816	811	811
query45	182	169	160	160
query46	883	1063	661	661
query47	1781	1821	1744	1744
query48	389	416	320	320
query49	784	481	411	411
query50	654	674	410	410
query51	4231	4168	4231	4168
query52	110	117	94	94
query53	236	267	192	192
query54	480	505	421	421
query55	86	82	84	82
query56	275	258	257	257
query57	1149	1156	1075	1075
query58	241	243	247	243
query59	3055	3098	2944	2944
query60	278	267	255	255
query61	125	114	124	114
query62	798	714	646	646
query63	226	198	194	194
query64	4508	996	701	701
query65	3261	3172	3118	3118
query66	913	400	310	310
query67	16222	15642	15432	15432
query68	4269	837	544	544
query69	450	310	262	262
query70	1231	1125	1092	1092
query71	371	294	255	255
query72	5925	3865	3834	3834
query73	663	769	359	359
query74	10069	9271	9178	9178
query75	3194	3180	2695	2695
query76	3126	1181	778	778
query77	486	382	288	288
query78	10022	9936	9283	9283
query79	3153	860	608	608
query80	1483	558	476	476
query81	567	277	240	240
query82	618	164	121	121
query83	182	172	163	163
query84	241	107	72	72
query85	843	359	316	316
query86	434	320	299	299
query87	4329	4457	4556	4457
query88	5069	2187	2147	2147
query89	389	337	301	301
query90	1741	195	194	194
query91	133	142	109	109
query92	66	59	55	55
query93	2714	862	546	546
query94	745	419	284	284
query95	331	272	261	261
query96	491	631	291	291
query97	2807	2880	2751	2751
query98	223	204	199	199
query99	1297	1451	1287	1287
Total cold run time: 285837 ms
Total hot run time: 185530 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.1 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b7ac63f18ceb591507c56c01c059dc1c38ce41ce, data reload: false

query1	0.04	0.03	0.04
query2	0.07	0.04	0.03
query3	0.24	0.08	0.07
query4	1.61	0.10	0.10
query5	0.41	0.42	0.41
query6	1.18	0.68	0.65
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.59	0.50	0.50
query10	0.56	0.57	0.55
query11	0.14	0.11	0.10
query12	0.14	0.10	0.11
query13	0.61	0.60	0.60
query14	2.81	2.73	2.84
query15	0.89	0.82	0.82
query16	0.38	0.39	0.36
query17	1.00	1.04	0.99
query18	0.23	0.21	0.20
query19	1.91	1.84	1.97
query20	0.01	0.01	0.01
query21	15.36	0.91	0.59
query22	0.75	0.78	0.71
query23	15.31	1.45	0.54
query24	2.86	1.49	1.93
query25	0.26	0.18	0.05
query26	0.17	0.16	0.13
query27	0.04	0.04	0.04
query28	14.40	0.96	0.43
query29	12.59	3.98	3.24
query30	0.25	0.08	0.06
query31	2.83	0.58	0.40
query32	3.23	0.55	0.47
query33	2.93	3.09	3.02
query34	16.72	5.16	4.52
query35	4.52	4.50	4.54
query36	0.65	0.50	0.48
query37	0.09	0.06	0.05
query38	0.04	0.04	0.03
query39	0.03	0.03	0.02
query40	0.17	0.13	0.12
query41	0.07	0.03	0.02
query42	0.04	0.02	0.02
query43	0.04	0.04	0.03
Total cold run time: 106.23 s
Total hot run time: 31.1 s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 42.06% (10971/26082)
Line Coverage: 32.34% (92740/286805)
Region Coverage: 31.48% (47547/151025)
Branch Coverage: 27.53% (24083/87492)
Coverage Report: http://coverage.selectdb-in.cc/coverage/b7ac63f18ceb591507c56c01c059dc1c38ce41ce_b7ac63f18ceb591507c56c01c059dc1c38ce41ce/report/index.html

airborne12
airborne12 approved these changes Jan 29, 2025
View reviewed changes
Copy link
Member

@airborne12 airborne12 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 29, 2025
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

dataroaring
dataroaring approved these changes Feb 8, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@liaoxin01 liaoxin01 merged commit 2d8a862 into apache:master Feb 10, 2025
26 of 28 checks passed
github-actions bot pushed a commit that referenced this pull request Feb 10, 2025
@kaijchen
[improve](move-memtable) improve error log and message for "not enoug…
90dbcf5 
…h streams" (#47470)

improve error log and message for "not enough streams", make it easier
to understand.
@github-actions github-actions bot mentioned this pull request Feb 10, 2025
Merged
github-actions bot pushed a commit that referenced this pull request Feb 10, 2025
@kaijchen
[improve](move-memtable) improve error log and message for "not enoug…
69560ea 
…h streams" (#47470)

improve error log and message for "not enough streams", make it easier
to understand.
@github-actions github-actions bot mentioned this pull request Feb 10, 2025
Merged
yiguolei pushed a commit that referenced this pull request Feb 12, 2025
@github-actions @kaijchen
branch-2.1: [improve](move-memtable) improve error log and message fo…
f85176b 
…r "not enough streams" #47470 (#47722)

Cherry-picked from #47470

Co-authored-by: Kaijie Chen <[email protected]>
lzyy2024 pushed a commit to lzyy2024/doris that referenced this pull request Feb 21, 2025
@kaijchen @lzyy2024
[improve](move-memtable) improve error log and message for "not enoug…
c6d9759 
…h streams" (apache#47470)

improve error log and message for "not enough streams", make it easier
to understand.
dataroaring pushed a commit that referenced this pull request Feb 24, 2025
@github-actions @kaijchen
branch-3.0: [improve](move-memtable) improve error log and message fo…
cc91ebb 
…r "not enough streams" #47470 (#47721)

Cherry-picked from #47470

Co-authored-by: Kaijie Chen <[email protected]>
@yiguolei yiguolei mentioned this pull request Mar 25, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@airborne12 airborne12 airborne12 approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.9-merged dev/3.0.5-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
7 participants
@kaijchen @Thearas @doris-robot @airborne12 @dataroaring @yiguolei @liaoxin01