-
Notifications
You must be signed in to change notification settings - Fork 3.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[feat](spill) spill and reserve #47462
Conversation
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
run buildall |
TPC-H: Total hot run time: 32548 ms
|
TPC-DS: Total hot run time: 193144 ms
|
ClickBench: Total hot run time: 31.37 s
|
9dce485
to
ccc257c
Compare
run buildall |
ccc257c
to
6613edc
Compare
run buildall |
TeamCity cloud ut coverage result: |
TPC-H: Total hot run time: 31551 ms
|
TPC-DS: Total hot run time: 190786 ms
|
ClickBench: Total hot run time: 31.46 s
|
6613edc
to
85b6160
Compare
run buildall |
TeamCity cloud ut coverage result: |
TPC-H: Total hot run time: 31894 ms
|
TPC-DS: Total hot run time: 190712 ms
|
ClickBench: Total hot run time: 31.48 s
|
85b6160
to
2b1cf11
Compare
run buildall |
TeamCity cloud ut coverage result: |
TPC-H: Total hot run time: 31885 ms
|
TPC-DS: Total hot run time: 190515 ms
|
2b1cf11
to
1187a61
Compare
run buildall |
run buildall |
TPC-H: Total hot run time: 31797 ms
|
TPC-DS: Total hot run time: 184125 ms
|
ClickBench: Total hot run time: 30.72 s
|
TeamCity be ut coverage result: |
run cloudut |
TeamCity cloud ut coverage result: |
PR approved by at least one committer and no changes requested. |
PR approved by anyone and no changes requested. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
And also fix the WorkloadGroupTest::testCreateNormal After merged the spill and reserve code(apache#47462), the properties of `WorkloadGroup` was changed.
…pache#48300) ### What problem does this PR solve? And also fix the WorkloadGroupTest::testCreateNormal After merged the spill and reserve code(apache#47462), the properties of `WorkloadGroup` was changed.
…k_ptr" thrown" in DataStreamRecvrTest (apache#48319) There are conflicts between the following two PRs. One PR uses unique_ptr to construct query_ctx, while the other PR uses shared_from_this apache#48188 apache#47462
### What problem does this PR solve? Problem Summary: A brand-new spilling triggering strategy: 1. Use workload groups to control/manage the memory usage of queries. 2. Trigger spilling when the memory reservation attempt fails.
…pache#48300) ### What problem does this PR solve? And also fix the WorkloadGroupTest::testCreateNormal After merged the spill and reserve code(apache#47462), the properties of `WorkloadGroup` was changed.
…k_ptr" thrown" in DataStreamRecvrTest (apache#48319) There are conflicts between the following two PRs. One PR uses unique_ptr to construct query_ctx, while the other PR uses shared_from_this apache#48188 apache#47462
### What problem does this PR solve? Fix: #47462 `SwitchResourceContext` no longer checks whether `SCOPED_ATTACH_TASK` has been called before, which makes `SwitchResourceContext` more flexible. For example, when calling the `MemTableWriter::flush_async` function, it is correct whether the upper code attaches the task or not. Fix: ``` F 2025-02-27 02:20:55,511 11069 thread_context.h:201] Check failed: is_attach_task() *** Check failure stack trace: *** @ 0x558647d61466 google::LogMessage::SendToLog() @ 0x558647d5deb0 google::LogMessage::Flush() @ 0x558647d61ca9 google::LogMessageFatal::~LogMessageFatal() @ 0x55863da71801 doris::SwitchResourceContext::SwitchResourceContext() @ 0x55863d3b90f9 doris::MemTableWriter::flush_async() @ 0x55863d3b0d60 doris::MemTableMemoryLimiter::_flush_active_memtables() @ 0x55863d3b1217 doris::MemTableMemoryLimiter::flush_workload_group_memtables() @ 0x55863da900dc doris::WorkloadGroupMgr::flush_memtable_from_current_group_() @ 0x55863da8d1be doris::WorkloadGroupMgr::handle_paused_queries() @ 0x55863cb74f65 doris::Daemon::memory_maintenance_thread() @ 0x55863dc1c631 doris::Thread::supervise_thread() @ 0x7f60f7117ac3 (unknown) @ 0x7f60f71a9850 (unknown) @ (nil) (unknown) *** Query id: 0-0 *** *** is nereids: 0 *** *** tablet id: 0 *** *** Aborted at 1740594055 (unix time) try "date -d @1740594055" if you are using GNU date *** *** Current BE git commitID: e0d8588 *** *** SIGABRT unknown detail explain (@0x22fc) received by PID 8956 (TID 11069 OR 0x7f5d09c04640) from PID 8956; stack trace: *** 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /home/zcp/repo_center/doris_master/doris/be/src/common/signal_handler.h:421 1# 0x00007F60F70C5520 in /lib/x86_64-linux-gnu/libc.so.6 2# pthread_kill at ./nptl/pthread_kill.c:89 3# raise at ../sysdeps/posix/raise.c:27 4# abort at ./stdlib/abort.c:81 5# 0x0000558647D6BD3D in /mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be 6# 0x0000558647D5E37A in /mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be 7# google::LogMessage::SendToLog() in /mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be 8# google::LogMessage::Flush() in /mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be 9# google::LogMessageFatal::~LogMessageFatal() in /mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be 10# doris::SwitchResourceContext::SwitchResourceContext(std::shared_ptr<doris::ResourceContext> const&) in /mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be 11# doris::MemTableWriter::flush_async() at /home/zcp/repo_center/doris_master/doris/be/src/olap/memtable_writer.cpp:159 12# doris::MemTableMemoryLimiter::_flush_active_memtables(unsigned long, long) at /home/zcp/repo_center/doris_master/doris/be/src/olap/memtable_memory_limiter.cpp:262 13# doris::MemTableMemoryLimiter::flush_workload_group_memtables(unsigned long, long) in /mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be 14# doris::WorkloadGroupMgr::flush_memtable_from_current_group_(std::shared_ptr<doris::WorkloadGroup>, long) at /home/zcp/repo_center/doris_master/doris/be/src/runtime/workload_group/workload_group_manager.cpp:568 15# doris::WorkloadGroupMgr::handle_paused_queries() in /mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be 16# doris::Daemon::memory_maintenance_thread() at /home/zcp/repo_center/doris_master/doris/be/src/common/daemon.cpp:329 17# doris::Thread::supervise_thread(void*) at /home/zcp/repo_center/doris_master/doris/be/src/util/thread.cpp:499 18# start_thread at ./nptl/pthread_create.c:442 19# 0x00007F60F71A9850 at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:83 ```
What problem does this PR solve?
Problem Summary:
A brand-new spilling triggering strategy:
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)