Add a flag to adjust queries to some max future time

[chencs]

What this PR does

This PR introduces an experimental flag, query-frontend.adjust-to-max-future-query-window. When set with a duration, any range queries querying farther than now + maxFutureWindow into the future will be adjusted to query only up to now + maxFutureWindow.

Which issue(s) this PR fixes or relates to

Fixes #

Checklist

• Tests updated.
• Documentation added.
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX].
• about-versioning.md updated with experimental features.

[github-actions]

💻 Deploy preview deleted.

[chencs]

I think, in the current form, the new limit won't touch queries that were modified by the max-lookback limit — because the latter adjusts the query and moves on, before the new one takes place. Is this what we want for this PR?

I'm not sure what you mean by this. IIUC queries which fully query outside of the allowed max-lookback will not be executed, but requests which are only modified by max-lookback will continue on to be checked against other limits:

mimir/pkg/frontend/querymiddleware/limits.go

Line 180 in bfde5db

r, err = r.WithStartEnd(minStartTime, r.GetEnd())

[tacole02]

Thank you, @chencs !

[tacole02]

Could we use a word other than mutate? Update maybe?

[chencs]

I was following the pattern of the query-frontend.align-queries-with-step description. Is "mutate" an ineffective word choice? This does automatically change the actual query being executed, as opposed to, e.g., being "updated" at the behest of the user executing the query.

[tacole02]

If mutate is the technically accurate word choice, we can leave it. It just sounds somewhat awkward/jarring to me.

[narqo]

queries which fully query outside of the allowed max-lookback will not be executed, but requests which are only modified by max-lookback will continue on to be checked against other limits

You are right (I read the surrounding code wrong).

🔥 The changes looks good to me (except the comments about the changelog)

[dimitarvdimitrov]

We just removed basically the same option in 2.14 😄 See #7496 and 286c23c

The assumption was that these queries are very cheap to execute. Even hundreds of queries should clear very quickly. The reason why we removed them was that it prevents people from running predict_linear queries for the future.

If you really want to keep it, then can we at least use the previous name of the flag?

[chencs]

We just removed basically the same option in 2.14

I didn't realize! Thank you for pointing that out.

The assumption was that these queries are very cheap to execute.

This is true, but if a tenant tries to execute a future-looking query over many days, they can also quickly hit query-scheduler.max-outstanding-requests-per-tenant, and it can take a while for those "cheap" queries to clear in an environment with many tenants. I thought about checking the query for a predict_linear and not applying the limit if found. I ended up not doing it for clarity/simplicity. Do you think this would be a reasonable workaround, or would it make this flag too confusing?

I'm also happy to just revert this PR if it's not the right way to go about fixing this.