Exercise Remote Write as part of Prombench

[csmarchbanks]

It would be valuable to be able to benchmark proposed remote write changes in a reproducible manner. Right now we are dependent on having a few people deploy branches to their instances to see if improvements work for them.

Perhaps have this optional so that remote write is not always exercised?

[krasi-georgiev]

Shouldn't be difficult to implement.

Are you interested in working on this?

[geekodour]

We can enable prombench to setup prometheus in custom ways, if we allow specifying the features to add to both of the test prometheus instances.

Example comments:
/benchmark [remote_write,]
/benchmark master [remote_write,]
/benchmark v2.11.1 [remote_write, remote_read, something_else]

A regex like this would work:

(?mi)^/benchmark\s*(master|[0-9]+\.[0-9]+\.[0-9]+\S*)?\s*(?:\[(.*).*\])?\s*$

https://regex101.com/r/8rF6xa/2/

It will give a captured group with remote_write,remote_read,something_else
we can then pass it as an env var (eg.PROMETHEUS_OPTIONS) to the prombench docker image (which uses make) and we can have a Makefile target which deploys custom manifest files based on PROMETHEUS_OPTIONS

cc @csmarchbanks @krasi-georgiev what do you think?

[krasi-georgiev]

yep that sound like the way to go, but unless someone has the time to work on this the scalability tests are much higher priority

[csmarchbanks]

I have interest in working on this, but am not sure when I will have time. If someone else gets to it first I would not be upset.

As far as as implementation goes, that seems reasonable for how to trigger it.

[krasi-georgiev]

ok ping me or @geekodour for directions if you find the time to work on this.

[geekodour]

@csmarchbanks taking a second look on this issue, will there be need of additional queries /tests to view the results better? or just having the prometheus instances use remote write will be good enough?

[csmarchbanks]

It would be nice to have a couple more queries, specifically for some of the remote-write specific data including number of shards, and if a queue is backed up at all. Some good queries exist in the prometheus mixin remote write dashboard: https://github.com/prometheus/prometheus/blob/master/documentation/prometheus-mixin/dashboards.libsonnet#L101

[roidelapluie]

Do we have a remote-write 'black-hole' that would accept all samples (and randomly fail)?

[geekodour]

@roidelapluie I'll just mention @csmarchbanks riot response here:

https://github.com/prometheus/prometheus/blob/master/documentation/examples/remote_storage/example_write_adapter/server.go

Yes, will add something like that only, thanks for the links :)

[LeviHarrison]

Is anyone working on this? I've recently come across the need to do some remote write benchmarks for Prometheus and I recognize the benefit an easy setup like Prombench could provide. I'd be happy to implement it.

[LeviHarrison]

I'm working on this.

[bwplotka]

Related #766