In this project, We will demo how to expose the http response time in [Prometheus] format through Envoy.
This demo has three components: App, Envoy and statsd_exporter. These three components will deployed in three Docker containers.
How to deploy this demo in Kubernetes can be found here.
This App will simulate a http service, which will have a latency of specified milliseconds for each http request.
Start this App in Docker container:
cd components/app
sh run.sh
If the app works, it can be accessed via http://localhost:8080/workload.php/?value=100
.
Envoy will work as a http proxy for the App: All http requests for the App will go through Envoy
, and Envoy
will monitor the http response time for the requests.
Before start the Envoy
, provide a correct configuration file for Envoy
image:
cd components/envoy
vim ./conf/envoy.json ## modify this file according to your host IP address
sh run.sh
If everything works fine, the app can be accessed via http://localhost:9090/workload.php/?value=100
.
In addition, some metrics (not including the response time) can be accessed via http://localhost:8001/stats
.
The timer metris of Envoy
are only forwarded to statsd
server (in this demo, it is statsd_exporter
).
Two functions of statsd_exporter
in this demo:
- Receive all the metrics (including the response time);
- Convert the metrics from
statsd
format intoPrometheus
format.
Start statsd_exporter
in Docker container:
cd components/statsd_exporter
sh run.sh
If everything works fine, the metrics can be accessed via http://localhost:9102/metrics
.
Make sure envoy_cluster related metrics can be found in the results of http://localhost:9102/metrics
. If not, check the two settings in Envoy
configuration file:
Replace the two IP addresses to a non-loopback address.
...
envoy_cluster_manager_total_clusters 1
# HELP envoy_cluster_video_service_external_upstream_rq_200 Metric autogenerated by statsd_exporter.
# TYPE envoy_cluster_video_service_external_upstream_rq_200 counter
envoy_cluster_video_service_external_upstream_rq_200 1672
# HELP envoy_cluster_video_service_external_upstream_rq_2xx Metric autogenerated by statsd_exporter.
# TYPE envoy_cluster_video_service_external_upstream_rq_2xx counter
envoy_cluster_video_service_external_upstream_rq_2xx 1672
# HELP envoy_cluster_video_service_external_upstream_rq_time Metric autogenerated by statsd_exporter.
# TYPE envoy_cluster_video_service_external_upstream_rq_time summary
envoy_cluster_video_service_external_upstream_rq_time{quantile="0.5"} 104
envoy_cluster_video_service_external_upstream_rq_time{quantile="0.9"} 106
envoy_cluster_video_service_external_upstream_rq_time{quantile="0.99"} 109
envoy_cluster_video_service_external_upstream_rq_time_sum 173175
envoy_cluster_video_service_external_upstream_rq_time_count 1673
...