Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🌵 [SPIKE] Investigate feasibility of using prom-client for Ghost metrics #20981

Draft
wants to merge 6 commits into
base: main
Choose a base branch
from

Conversation

cmraible
Copy link
Contributor

@cmraible cmraible commented Sep 11, 2024

ref https://linear.app/tryghost/issue/ENG-1505/start-monitoring-event-loop-utilization-in-production-with
ref https://linear.app/tryghost/issue/PRO-197/node-runtime-performance-prometheus-exporter

  • This spike adds a /metrics endpoint to Ghost that exposes all the default metrics included in prom-client. The list includes some basics, like CPU and memory usage, metrics for Node's event loop, and metrics for Node's garbage collection. For many of these metrics, we currently have no visibility into them.
  • It also includes some custom metrics like Ghost's connection pool utilization, request queue depth, active workers for the jobs manager, and some basic HTTP response time percentile data.
  • It also adds two new services to our docker-compose.yml for development: prometheus, to scrape the metrics from Ghost, and Grafana to expose the metrics in a nice browser-based dashboard so you can view these metrics locally:

250rps-main

  • With this foundation, we can also add our own custom metrics to our code, that will automatically be reported to prometheus and can be visualized in Grafana.

Todo

  • Decouple from existing express apps and expose this on its own port, to avoid exposing metrics publicly and so Ghost will (hopefully) respond to scraping requests even if the main express apps are falling over
  • Pull the prometheus (and any custom metrics) into their own package?
  • Figure out why request queue unit tests are failing
  • Investigate using docker compose profiles to only boot prometheus and grafana if opted in via environment variable

@cmraible cmraible changed the title Added prometheus metrics endpoint using prom-client 🌵 [SPIKE] Investigate feasibility of using prom-client for Ghost metrics Sep 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant