-
Notifications
You must be signed in to change notification settings - Fork 3.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
util/mon: reduce monitor-related allocations #143305
base: master
Are you sure you want to change the base?
Conversation
It's a very minor reduction in allocations. But it is nice to have structured names instead of strings.
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewed 2 of 2 files at r1, 1 of 1 files at r2, 2 of 2 files at r3, 1 of 1 files at r4, 33 of 33 files at r5, 8 of 8 files at r6, 3 of 3 files at r7, 77 of 77 files at r8, 17 of 17 files at r9, 14 of 14 files at r10, 1 of 1 files at r11, all commit messages.
Reviewable status:complete! 1 of 0 LGTMs obtained (waiting on @asg0451, @jbowens, @kev-cao, and @xinhaoz)
pkg/util/mon/bytes_usage.go
line 281 at r8 (raw file):
} // Name is used to identify monitors in logging messages. It consists of:
nit: let's also mention crdb_internal.node_memory_monitors
virtual table where the monitor name shows up.
pkg/sql/rowflow/row_based_flow.go
line 464 at r9 (raw file):
diskMonitors := make([]*mon.BytesMonitor, len(spec.Streams)) for i := range spec.Streams { // TODO(mgartner): Is a StreamID ever greater than math.MaxInt32? If
I don't think it can ever exceed max int32 in practice. StreamID
s are indexes into PhysicalInfrastructure.Streams
slice, which contains all pair-wise "links" between two processors within the same plan, so I'd expect that we'd OOM-crash if stream ID reaches 1 million, way before max int 32.
Release note: None
Release note: None
The `processorID`, `suffix`, and `i` fields will be used in future commits to reduce allocations related to MonitorNames. Release note: None
Release note: None
Using structured `mon.MonitorName` objects reduces allocations of strings in some cases. This commit doesn't replace all usages of string monitor names - there are myriad such usages. We can iteratively chip away at them in future commits. Release note: None
The size of `MonitorName` has been reduced by replacing the string suffix with a suffix enum. Release note: None
The size of `MonitorName` has been reduced by using the same 4 bytes to store either the `int32` processor ID or the `uuid.Short`, and by reducing the optional integer suffix from an `int32` to `uint16`, which should be sufficient in all cases. Release note: None
Release note: None
All usages of the deprecated NewMonitorWithStringName function have been replaced with NewMonitor, and the former has been removed. Release note: None
All usages of the deprecated NewMonitorInheritWithLimitAndStringName function have been replaced with NewMonitorInheritWithLimit, and the former has been removed. Release note: None
Allocations of `BoundAccount`s in `createUnlimitedMemAccountsLocked` are now batched into a single allocation. Release note: None
The monitor names created in `(*rowBasedFlow).setupRouter` now use `(*mon.Name).WithID` to add the stream ID to the name instead of allocating a new string. Release note: None
92c6177
to
7d4c6ac
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewable status:
complete! 0 of 0 LGTMs obtained (and 1 stale) (waiting on @asg0451, @jbowens, @kev-cao, @xinhaoz, and @yuzefovich)
pkg/sql/rowflow/row_based_flow.go
line 464 at r9 (raw file):
Previously, yuzefovich (Yahor Yuzefovich) wrote…
I don't think it can ever exceed max int32 in practice.
StreamID
s are indexes intoPhysicalInfrastructure.Streams
slice, which contains all pair-wise "links" between two processors within the same plan, so I'd expect that we'd OOM-crash if stream ID reaches 1 million, way before max int 32.
Makes sense, I'll add another commit to address this TODO then. Thanks!
pkg/util/mon/bytes_usage.go
line 281 at r8 (raw file):
Previously, yuzefovich (Yahor Yuzefovich) wrote…
nit: let's also mention
crdb_internal.node_memory_monitors
virtual table where the monitor name shows up.
done.
util/mon: rename fields of MonitorName
Release note: None
util/uuid: add comment to uuid.Short
Release note: None
util/mon: add fields to MonitorName
The
processorID
,suffix
, andi
fields will be used in futurecommits to reduce allocations related to MonitorNames.
Release note: None
util/mon: compile-time checks for size of BytesMonitor and BoundAccount
Release note: None
sql: use mon.MonitorName in more places
Using structured
mon.MonitorName
objects reduces allocations ofstrings in some cases. This commit doesn't replace all usages of string
monitor names - there are myriad such usages. We can iteratively chip
away at them in future commits.
Release note: None
util/mon: reduce size of MonitorName
The size of
MonitorName
has been reduced by replacing the stringsuffix with a suffix enum.
Release note: None
util/mon: further reduce the size of MonitorName
The size of
MonitorName
has been reduced by using the same 4 bytes tostore either the
int32
processor ID or theuuid.Short
, and byreducing the optional integer suffix from an
int32
touint16
, whichshould be sufficient in all cases.
Release note: None
util/mon: rename MonitorName to Name
Release note: None
util/mon: remove deprecated NewMonitorWithStringName
All usages of the deprecated NewMonitorWithStringName function have been
replaced with NewMonitor, and the former has been removed.
Release note: None
util/mon: remove deprecated NewMonitorInheritWithLimitAndStringName
All usages of the deprecated NewMonitorInheritWithLimitAndStringName
function have been replaced with NewMonitorInheritWithLimit, and the
former has been removed.
Release note: None
colexec: batch allocations of BoundAccount
Allocations of
BoundAccount
s increateUnlimitedMemAccountsLocked
arenow batched into a single allocation.
Release note: None
sql/rowflow: refactor monitor name
The monitor names created in
(*rowBasedFlow).setupRouter
now use(*mon.Name).WithID
to add the stream ID to the name instead ofallocating a new string.
Epic: None
Release note: None