Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: Upgrade dskit, use LimitsMap from flagext #10768

Merged
merged 10 commits into from
Feb 28, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 50 additions & 0 deletions cmd/mimir/config-descriptor.json
Original file line number Diff line number Diff line change
Expand Up @@ -715,6 +715,56 @@
"fieldFlag": "server.path-prefix",
"fieldType": "string",
"fieldCategory": "advanced"
},
{
"kind": "block",
"name": "cluster_validation",
"required": false,
"desc": "",
"blockEntries": [
{
"kind": "field",
"name": "label",
"required": false,
"desc": "Optionally define server's cluster validation label.",
"fieldValue": null,
"fieldDefaultValue": "",
"fieldFlag": "server.cluster-validation.label",
"fieldType": "string"
},
{
"kind": "block",
"name": "grpc",
"required": false,
"desc": "",
"blockEntries": [
{
"kind": "field",
"name": "enabled",
"required": false,
"desc": "When enabled, cluster label validation will be executed.",
"fieldValue": null,
"fieldDefaultValue": false,
"fieldFlag": "server.cluster-validation.grpc.enabled",
"fieldType": "boolean"
},
{
"kind": "field",
"name": "softvalidation",
"required": false,
"desc": "When enabled, soft cluster label validation will be executed. Can be enabled only together with server.cluster-validation.grpc.enabled",
"fieldValue": null,
"fieldDefaultValue": false,
"fieldFlag": "server.cluster-validation.grpc.soft-validation",
"fieldType": "boolean"
}
],
"fieldValue": null,
"fieldDefaultValue": null
}
],
"fieldValue": null,
"fieldDefaultValue": null
}
],
"fieldValue": null,
Expand Down
6 changes: 6 additions & 0 deletions cmd/mimir/help-all.txt.tmpl
Original file line number Diff line number Diff line change
Expand Up @@ -3161,6 +3161,12 @@ Usage of ./cmd/mimir/mimir:
Comma separated list of yaml files with the configuration that can be updated at runtime. Runtime config files will be merged from left to right.
-runtime-config.reload-period duration
How often to check runtime config files. (default 10s)
-server.cluster-validation.grpc.enabled
When enabled, cluster label validation will be executed.
-server.cluster-validation.grpc.soft-validation
When enabled, soft cluster label validation will be executed. Can be enabled only together with server.cluster-validation.grpc.enabled
-server.cluster-validation.label string
Optionally define server's cluster validation label.
-server.graceful-shutdown-timeout duration
Timeout for graceful shutdowns (default 30s)
-server.grpc-conn-limit int
Expand Down
6 changes: 6 additions & 0 deletions cmd/mimir/help.txt.tmpl
Original file line number Diff line number Diff line change
Expand Up @@ -771,6 +771,12 @@ Usage of ./cmd/mimir/mimir:
The tenant's shard size when sharding is used by ruler. Value of 0 disables shuffle sharding for the tenant, and tenant rules will be sharded across all ruler replicas.
-runtime-config.file comma-separated-list-of-strings
Comma separated list of yaml files with the configuration that can be updated at runtime. Runtime config files will be merged from left to right.
-server.cluster-validation.grpc.enabled
When enabled, cluster label validation will be executed.
-server.cluster-validation.grpc.soft-validation
When enabled, soft cluster label validation will be executed. Can be enabled only together with server.cluster-validation.grpc.enabled
-server.cluster-validation.label string
Optionally define server's cluster validation label.
-server.grpc-listen-address string
gRPC server listen address.
-server.grpc-listen-port int
Expand Down
15 changes: 15 additions & 0 deletions docs/sources/mimir/configure/configuration-parameters/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -755,6 +755,21 @@ grpc_tls_config:
# (advanced) Base path to serve all API routes from (e.g. /v1/)
# CLI flag: -server.path-prefix
[http_path_prefix: <string> | default = ""]

cluster_validation:
# Optionally define server's cluster validation label.
# CLI flag: -server.cluster-validation.label
[label: <string> | default = ""]

grpc:
# When enabled, cluster label validation will be executed.
# CLI flag: -server.cluster-validation.grpc.enabled
[enabled: <boolean> | default = false]

# When enabled, soft cluster label validation will be executed. Can be
# enabled only together with server.cluster-validation.grpc.enabled
# CLI flag: -server.cluster-validation.grpc.soft-validation
[softvalidation: <boolean> | default = false]
```

### distributor
Expand Down
4 changes: 2 additions & 2 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ require (
github.com/golang/snappy v0.0.4
github.com/google/gopacket v1.1.19
github.com/gorilla/mux v1.8.1
github.com/grafana/dskit v0.0.0-20250213152722-e83d24ebed15
github.com/grafana/dskit v0.0.0-20250228172433-c35b68b89416
github.com/grafana/e2e v0.1.2-0.20240118170847-db90b84177fc
github.com/hashicorp/golang-lru v1.0.2 // indirect
github.com/influxdata/influxdb/v2 v2.7.11
Expand Down Expand Up @@ -221,7 +221,7 @@ require (
github.com/googleapis/enterprise-certificate-proxy v0.3.4 // indirect
github.com/googleapis/gax-go/v2 v2.14.1 // indirect
github.com/gosimple/slug v1.1.1 // indirect
github.com/grafana/gomemcache v0.0.0-20241016125027-0a5bcc5aef40
github.com/grafana/gomemcache v0.0.0-20250228145437-da7b95fd2ac1
github.com/hashicorp/consul/api v1.31.0 // indirect
github.com/hashicorp/errwrap v1.1.0 // indirect
github.com/hashicorp/go-cleanhttp v0.5.2 // indirect
Expand Down
8 changes: 4 additions & 4 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -1272,16 +1272,16 @@ github.com/grafana-tools/sdk v0.0.0-20220919052116-6562121319fc h1:PXZQA2WCxe85T
github.com/grafana-tools/sdk v0.0.0-20220919052116-6562121319fc/go.mod h1:AHHlOEv1+GGQ3ktHMlhuTUwo3zljV3QJbC0+8o2kn+4=
github.com/grafana/alerting v0.0.0-20250225150117-15e285d78df2 h1:kESrzm0FcRVLmGIQCgl1MCwDGLH4sLzWphr7mcFdbfI=
github.com/grafana/alerting v0.0.0-20250225150117-15e285d78df2/go.mod h1:hdGB3dSl8Ma9Rjo2YiAEAjMkZ5HiNJbNDqRKDefRZrM=
github.com/grafana/dskit v0.0.0-20250213152722-e83d24ebed15 h1:5Mdedurc8b1Cc3RW9qRHNtxekGN1PEOYTymq18A0Ddc=
github.com/grafana/dskit v0.0.0-20250213152722-e83d24ebed15/go.mod h1:SPLNCARd4xdjCkue0O6hvuoveuS1dGJjDnfxYe405YQ=
github.com/grafana/dskit v0.0.0-20250228172433-c35b68b89416 h1:Of6D3MxfXQ5tCaAYT/BtcKU5oGsp4YHDGeNF4Kc/wzI=
github.com/grafana/dskit v0.0.0-20250228172433-c35b68b89416/go.mod h1:bs/kOQFsasdmaU2gT1xcRJN4Em5zg0BX5G8O9Pt3XJY=
github.com/grafana/e2e v0.1.2-0.20240118170847-db90b84177fc h1:BW+LjKJDz0So5LI8UZfW5neWeKpSkWqhmGjQFzcFfLM=
github.com/grafana/e2e v0.1.2-0.20240118170847-db90b84177fc/go.mod h1:JVmqPBe8A/pZWwRoJW5ZjyALeY5OXMzPl7LrVXOdZAI=
github.com/grafana/franz-go v0.0.0-20241009100846-782ba1442937 h1:fwwnG/NcygoS6XbAaEyK2QzMXI/BZIEJvQ3CD+7XZm8=
github.com/grafana/franz-go v0.0.0-20241009100846-782ba1442937/go.mod h1:NreRdJ2F7dziDY/m6VyspWd6sNxHKXdMZI42UfQ3GXM=
github.com/grafana/goautoneg v0.0.0-20240607115440-f335c04c58ce h1:WI1olbgS+sEl77qxEYbmt9TgRUz7iLqmjh8lYPpGlKQ=
github.com/grafana/goautoneg v0.0.0-20240607115440-f335c04c58ce/go.mod h1:GFAN9Jn9t1cX7sNfc6ZoFyc4f7i8jtm3SajrWdZM2EE=
github.com/grafana/gomemcache v0.0.0-20241016125027-0a5bcc5aef40 h1:1TeKhyS+pvzOeyLV1XPZsiqebnKky/AKS3pJNNbHVPo=
github.com/grafana/gomemcache v0.0.0-20241016125027-0a5bcc5aef40/go.mod h1:IGRj8oOoxwJbHBYl1+OhS9UjQR0dv6SQOep7HqmtyFU=
github.com/grafana/gomemcache v0.0.0-20250228145437-da7b95fd2ac1 h1:vR5nELq+KtGO+IiGW+AclWeQ7uhLHCEz/zyQwbQVNnQ=
github.com/grafana/gomemcache v0.0.0-20250228145437-da7b95fd2ac1/go.mod h1:j/s0jkda4UXTemDs7Pgw/vMT06alWc42CHisvYac0qw=
github.com/grafana/memberlist v0.3.1-0.20220714140823-09ffed8adbbe h1:yIXAAbLswn7VNWBIvM71O2QsgfgW9fRXZNR0DXe6pDU=
github.com/grafana/memberlist v0.3.1-0.20220714140823-09ffed8adbbe/go.mod h1:MS2lj3INKhZjWNqd3N0m3J+Jxf3DAOnAH9VT3Sh9MUE=
github.com/grafana/mimir-prometheus v0.0.0-20250227233049-0c8e28d13db8 h1:HMH5igbkbJvH01RV8bnjAsFUBLLyoEK8ab6Rb+/p+js=
Expand Down
13 changes: 6 additions & 7 deletions pkg/ruler/notifier.go
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,6 @@ import (

"github.com/grafana/mimir/pkg/util"
util_log "github.com/grafana/mimir/pkg/util/log"
"github.com/grafana/mimir/pkg/util/validation"
)

var (
Expand All @@ -52,11 +51,11 @@ func (cfg *NotifierConfig) RegisterFlags(f *flag.FlagSet) {
}

type OAuth2Config struct {
ClientID string `yaml:"client_id"`
ClientSecret flagext.Secret `yaml:"client_secret"`
TokenURL string `yaml:"token_url"`
Scopes flagext.StringSliceCSV `yaml:"scopes,omitempty"`
EndpointParams validation.LimitsMap[string] `yaml:"endpoint_params" category:"advanced"`
ClientID string `yaml:"client_id"`
ClientSecret flagext.Secret `yaml:"client_secret"`
TokenURL string `yaml:"token_url"`
Scopes flagext.StringSliceCSV `yaml:"scopes,omitempty"`
EndpointParams flagext.LimitsMap[string] `yaml:"endpoint_params" category:"advanced"`
}

func (cfg *OAuth2Config) RegisterFlagsWithPrefix(prefix string, f *flag.FlagSet) {
Expand All @@ -65,7 +64,7 @@ func (cfg *OAuth2Config) RegisterFlagsWithPrefix(prefix string, f *flag.FlagSet)
f.StringVar(&cfg.TokenURL, prefix+"token_url", "", "Endpoint used to fetch access token.")
f.Var(&cfg.Scopes, prefix+"scopes", "Optional scopes to include with the token request.")
if !cfg.EndpointParams.IsInitialized() {
cfg.EndpointParams = validation.NewLimitsMap[string](nil)
cfg.EndpointParams = flagext.NewLimitsMap[string](nil)
}
f.Var(&cfg.EndpointParams, prefix+"endpoint-params", "Optional additional URL parameters to send to the token URL.")
}
Expand Down
9 changes: 4 additions & 5 deletions pkg/ruler/notifier_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,6 @@ import (
"github.com/stretchr/testify/require"

"github.com/grafana/mimir/pkg/util"
"github.com/grafana/mimir/pkg/util/validation"
)

func TestBuildNotifierConfig(t *testing.T) {
Expand Down Expand Up @@ -387,7 +386,7 @@ func TestBuildNotifierConfig(t *testing.T) {
ClientID: "oauth2-client-id",
ClientSecret: flagext.SecretWithValue("test"),
TokenURL: "https://oauth2-token-endpoint.local/token",
EndpointParams: validation.NewLimitsMapWithData[string](
EndpointParams: flagext.NewLimitsMapWithData[string](
map[string]string{
"param1": "value1",
"param2": "value2",
Expand Down Expand Up @@ -551,12 +550,12 @@ func TestBuildNotifierConfig(t *testing.T) {
func TestOAuth2Config_ValidateEndpointParams(t *testing.T) {
for name, tc := range map[string]struct {
args []string
expected validation.LimitsMap[string]
expected flagext.LimitsMap[string]
error string
}{
"basic test": {
args: []string{"-map-flag", "{\"param1\": \"value1\" }"},
expected: validation.NewLimitsMapWithData(map[string]string{
expected: flagext.NewLimitsMapWithData(map[string]string{
"param1": "value1",
}, nil),
},
Expand All @@ -566,7 +565,7 @@ func TestOAuth2Config_ValidateEndpointParams(t *testing.T) {
},
} {
t.Run(name, func(t *testing.T) {
v := validation.NewLimitsMap[string](nil)
v := flagext.NewLimitsMap[string](nil)

fs := flag.NewFlagSet("test", flag.ContinueOnError)
fs.SetOutput(&bytes.Buffer{}) // otherwise errors would go to stderr.
Expand Down
12 changes: 6 additions & 6 deletions pkg/util/validation/limits.go
Original file line number Diff line number Diff line change
Expand Up @@ -216,8 +216,8 @@ type Limits struct {
RulerRecordingRulesEvaluationEnabled bool `yaml:"ruler_recording_rules_evaluation_enabled" json:"ruler_recording_rules_evaluation_enabled"`
RulerAlertingRulesEvaluationEnabled bool `yaml:"ruler_alerting_rules_evaluation_enabled" json:"ruler_alerting_rules_evaluation_enabled"`
RulerSyncRulesOnChangesEnabled bool `yaml:"ruler_sync_rules_on_changes_enabled" json:"ruler_sync_rules_on_changes_enabled" category:"advanced"`
RulerMaxRulesPerRuleGroupByNamespace LimitsMap[int] `yaml:"ruler_max_rules_per_rule_group_by_namespace" json:"ruler_max_rules_per_rule_group_by_namespace" category:"experimental"`
RulerMaxRuleGroupsPerTenantByNamespace LimitsMap[int] `yaml:"ruler_max_rule_groups_per_tenant_by_namespace" json:"ruler_max_rule_groups_per_tenant_by_namespace" category:"experimental"`
RulerMaxRulesPerRuleGroupByNamespace flagext.LimitsMap[int] `yaml:"ruler_max_rules_per_rule_group_by_namespace" json:"ruler_max_rules_per_rule_group_by_namespace" category:"experimental"`
RulerMaxRuleGroupsPerTenantByNamespace flagext.LimitsMap[int] `yaml:"ruler_max_rule_groups_per_tenant_by_namespace" json:"ruler_max_rule_groups_per_tenant_by_namespace" category:"experimental"`
RulerProtectedNamespaces flagext.StringSliceCSV `yaml:"ruler_protected_namespaces" json:"ruler_protected_namespaces" category:"experimental"`
RulerMaxIndependentRuleEvaluationConcurrencyPerTenant int64 `yaml:"ruler_max_independent_rule_evaluation_concurrency_per_tenant" json:"ruler_max_independent_rule_evaluation_concurrency_per_tenant" category:"experimental"`

Expand Down Expand Up @@ -246,8 +246,8 @@ type Limits struct {
AlertmanagerReceiversBlockCIDRNetworks flagext.CIDRSliceCSV `yaml:"alertmanager_receivers_firewall_block_cidr_networks" json:"alertmanager_receivers_firewall_block_cidr_networks"`
AlertmanagerReceiversBlockPrivateAddresses bool `yaml:"alertmanager_receivers_firewall_block_private_addresses" json:"alertmanager_receivers_firewall_block_private_addresses"`

NotificationRateLimit float64 `yaml:"alertmanager_notification_rate_limit" json:"alertmanager_notification_rate_limit"`
NotificationRateLimitPerIntegration LimitsMap[float64] `yaml:"alertmanager_notification_rate_limit_per_integration" json:"alertmanager_notification_rate_limit_per_integration"`
NotificationRateLimit float64 `yaml:"alertmanager_notification_rate_limit" json:"alertmanager_notification_rate_limit"`
NotificationRateLimitPerIntegration flagext.LimitsMap[float64] `yaml:"alertmanager_notification_rate_limit_per_integration" json:"alertmanager_notification_rate_limit_per_integration"`

AlertmanagerMaxGrafanaConfigSizeBytes flagext.Bytes `yaml:"alertmanager_max_grafana_config_size_bytes" json:"alertmanager_max_grafana_config_size_bytes"`
AlertmanagerMaxConfigSizeBytes int `yaml:"alertmanager_max_config_size_bytes" json:"alertmanager_max_config_size_bytes"`
Expand Down Expand Up @@ -362,12 +362,12 @@ func (l *Limits) RegisterFlags(f *flag.FlagSet) {
f.BoolVar(&l.RulerSyncRulesOnChangesEnabled, "ruler.sync-rules-on-changes-enabled", true, "True to enable a re-sync of the configured rule groups as soon as they're changed via ruler's config API. This re-sync is in addition of the periodic syncing. When enabled, it may take up to few tens of seconds before a configuration change triggers the re-sync.")
// Needs to be initialised to a value so that the documentation can pick up the default value of `{}` because this is set as JSON from the command-line.
if !l.RulerMaxRulesPerRuleGroupByNamespace.IsInitialized() {
l.RulerMaxRulesPerRuleGroupByNamespace = NewLimitsMap[int](nil)
l.RulerMaxRulesPerRuleGroupByNamespace = flagext.NewLimitsMap[int](nil)
}
f.Var(&l.RulerMaxRulesPerRuleGroupByNamespace, "ruler.max-rules-per-rule-group-by-namespace", "Maximum number of rules per rule group by namespace. Value is a map, where each key is the namespace and value is the number of rules allowed in the namespace (int). On the command line, this map is given in a JSON format. The number of rules specified has the same meaning as -ruler.max-rules-per-rule-group, but only applies for the specific namespace. If specified, it supersedes -ruler.max-rules-per-rule-group.")

if !l.RulerMaxRuleGroupsPerTenantByNamespace.IsInitialized() {
l.RulerMaxRuleGroupsPerTenantByNamespace = NewLimitsMap[int](nil)
l.RulerMaxRuleGroupsPerTenantByNamespace = flagext.NewLimitsMap[int](nil)
}
f.Var(&l.RulerMaxRuleGroupsPerTenantByNamespace, "ruler.max-rule-groups-per-tenant-by-namespace", "Maximum number of rule groups per tenant by namespace. Value is a map, where each key is the namespace and value is the number of rule groups allowed in the namespace (int). On the command line, this map is given in a JSON format. The number of rule groups specified has the same meaning as -ruler.max-rule-groups-per-tenant, but only applies for the specific namespace. If specified, it supersedes -ruler.max-rule-groups-per-tenant.")
f.Var(&l.RulerProtectedNamespaces, "ruler.protected-namespaces", "List of namespaces that are protected from modification unless a special HTTP header is used. If a namespace is protected, it can only be read, not modified via the ruler's configuration API. The value is a list of strings, where each string is a namespace name. On the command line, this list is given as a comma-separated list.")
Expand Down
Loading
Loading