Merge pull request #1 from mojaloop/feat/performance-maintenance-characterisation-als

feat(mojaloop/#3400): benchmarking performance for ALS - mojaloop/project#3400
- created documentation, and scenarios for characterizing ALS
- documented performance data from ALS characterization runs

feat(mojaloop/#3424): analyse als perf results - mojaloop/project#3424
- Added analysis for scenarios 1-16
- re-factored structure into domains (i.e. fspiop-discovery, etc)
- added status/summary to the top of domain readmes

Author: mdebarros

README.md

@@ -0,0 +1,68 @@
+# ML Performance Characterization Repository
+
+Here you will find Performance Characterizations for the Mojaloop Services.
+
+## 1. High-Level Characterization Scenarios
+
+| High-level Scenario  | Description  | Documentation | Notes |
+|---|---|---|---|
+| 1. | FSPIOP Discovery | [./fspiop-discovery/README](./fspiop-discovery/README.md) | Done |
+| 2. | ~~FSPIOP Agreement~~ |   | To Be Done |
+| 3. | ~~FSPIOP Transfers~~ | [./fspiop-transfers/README](./fspiop-transfers/README.md) | To Be Done |
+| 4. | ~~FSPIOP Discovery + Agreement + Transfers~~ |   | To Be Done |
+
+### 2. Capturing End-to-end Metrics
+
+We have two approaches to capture the End-to-end metrics of a transaction.
+
+#### 2.1 Tracestate Headers
+
+The [Tracestate](https://github.com/mojaloop/mojaloop-specification/blob/master/fspiop-api/documents/Tracing%20v1.0.md#table-4--data-model-for-tracestate-list-member-values) header is part of the [Mojaloop Specification](https://github.com/mojaloop/mojaloop-specification/blob/master/fspiop-api/documents/Tracing%20v1.0.md) which conforms to the [W3C](https://github.com/mojaloop/mojaloop-specification/blob/master/fspiop-api/documents/Tracing%20v1.0.md#5-references) Tracing standards.
+
+As such we are able to take advantage of this header by propogating the following key-value pairs during the End-to-end transaction:
+
+| tracestate-key | tracestate-value | Notes |
+|---|---|---|
+| tx_end2end_start_ts | [timestamp](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Date/now) | Generated by the Test-runner (i.e. K6) |
+| tx_callback_start_ts | [timestamp](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Date/now) | Generated by the Payee Participant Simupator (e.g. when reciving the FSPIOP GET /parties Request) |
+
+Example header: `tracestate=tx_end2end_start_ts={{TIMESTAMP}}, tx_callback_start_ts={{TIMESTAMP}}`
+
+#### 2.2 WebScoket Subscriptions
+
+The Simulators (i.e. "Callback Handler Service") have been developed to support a simple WebSocket (WS) mechanism that allows the Test Executer (i.e. K6) to subscribe for Callback events.
+
+For example, let's take the FSPIOP GET /parties use-case. Here we have K6 subscribe to a Callback via a WS on the Payer Participant Simulator based on the following properties:
+
+1. The **TraceID**
+2. The HTTP **Operation** (i.e. PUT)
+3. The Party **ID** (i.e. MSDISN Number)
+
+This ensure that the K6 subscription-notification will be unique for each test.
+
+We gain two benefits by using this approach:
+
+1. The K6 Runner will only iterate once the current request is completed End-to-end which means that our execution strategy is closer to a real-work scenario.
+2. The K6 Runner will be able to report on the End-to-end duration and operations per second.
+
+The down-sides of this approach, is that it only works well when we have a single Payer Participant Simulator. Its possible that we can support scaling the Payer Participant Simulator by having the K6 Runners subscribe to multiple instances, but that is currently not supported.
+
+## 3. Types of tests
+
+| Test Type  | Description  |
+|---|---|
+| **Smoke** | Validates scripts works and that our target env/system performs adequately under minimal load. |
+| **Average-load** | Assess how the system performs under expected normal conditions. |
+| **Stress** | Assess how the system performs at its limits when load exceeds the expected average. |
+| **Spike** | Validates the behavior and survival of the system in cases of sudden, short, and massive increases in activity. |
+| **Breakpoint** | Gradually increase load to identify the capacity limits of the system. |
+
+[Reference](https://k6.io/docs/test-types/load-test-types/#different-tests-for-different-goals).
+
+## 4. Tools Used
+
+| Tool | Description |
+|---|---|
+| **ml-core-test-harness** | The [ml-core-test-harness](https://github.com/mojaloop/ml-core-test-harness) is a light-weight Docker-composed based test harness used by the Mojaloop community to execute Functional, and now Performance-Characterization tests |
+| **K6** | [Grafana k6](https://k6.io/docs/) is an open-source load testing tool. |
+| **Docker Compose**  | [Docker Compose](https://docs.docker.com/compose/) is a tool for defining and running multi-container Docker applications. |

Template-DATE/s0-testId/README.md

@@ -0,0 +1,53 @@
+# Scenario 1 - FSPIOP Discovery GET Parties with Sims-only - ALS:v14.2.2, scale:1, k6vu:1
+
+```yaml
+testid:
+  - 1690367402771
+params: &from=XXXXX&to=XXXXXXXX
+ACCOUNT_LOOKUP_SERVICE_VERSION: v14.2.2
+```
+
+## Environment
+
+- m6i.2xlarge
+  - 8 CPU - 3.5 GHz 3rd Generation Intel Xeon Scalable processors (Ice Lake 8375C)
+  - 32gb RAM
+- https://gist.github.com/mdebarros/6d9ac90f33c96031cbce6b9a3ea8048e
+
+## k6 Test Config
+
+```json
+{
+  "scenarios": { // define scenarios
+    // warm-up
+    "accountLookup": { // original scenario for accountLookup
+      "executor": "ramping-vus",
+      "exec": "accountLookupScenarios",
+      "startVUs": 1,
+      "stages": [
+        { "duration": "2m", "target": 1 },
+        { "duration": "5m", "target": 1 },
+      ]
+    },
+  },
+  "thresholds": {
+    "iteration_duration": [ "p(95)<1000" ],
+    "http_req_failed": [ "rate<0.01" ],
+    "http_req_duration": [ "p(95)<1000" ]
+  }
+}
+```
+
+## Snapshots
+
+- [Docker](INSERT_LINK_HERE)
+- [K6](INSERT_LINK_HERE)
+- [Callback Handler Service](INSERT_LINK_HERE)
+- [Account Lookup Service](INSERT_LINK_HERE)
+- [Nodejs moja_als](INSERT_LINK_HERE)
+- [Nodejs cbs](INSERT_LINK_HERE)
+- [MySQL](INSERT_LINK_HERE)
+
+## Observations
+
+## Recommendations

Template-DATE/s0-testId/images/.gitignore

@@ -0,0 +1,70 @@
+# Scenario 1: FSPIOP Discovery GET Parties with Sims-only - ALS:v14.2.2, scale:1, k6vu:1
+
+The End-to-end operation from the K6 test-runner included the following HTTP operations for each *iteration*:
+
+1. ADMIN GET /participants request to the Central-Ledger to validate payerFspId. <-- sync response
+2. ADMIN GET /participants request to the Central-Ledger to validate payeeFspId. <-- sync response
+3. ORACLE GET /participants request to the Oracle to resolve FSPID for payeeId. <-- sync response
+4. FSPIOP GET /parties request to the ALS <-- async callback response
+5. WS Subscription to the `Callback-Handler` Service for Callback Response notifications
+
+```conf
+var-testid=1690367402771
+params=&var-testid=1690367402771&from=1690367297867&to=1690368635328
+```
+
+## Environment
+
+- m6i.2xlarge
+  - 8 CPU - 3.5 GHz 3rd Generation Intel Xeon Scalable processors (Ice Lake 8375C)
+  - 32gb RAM
+- https://gist.github.com/mdebarros/6d9ac90f33c96031cbce6b9a3ea8048e
+
+## k6 Test Config
+
+```json
+{
+  "scenarios": { // define scenarios
+    // warm-up
+    "accountLookup": { // original scenario for accountLookup
+      "executor": "ramping-vus",
+      "exec": "accountLookupScenarios",
+      "startVUs": 1,
+      "stages": [
+        { "duration": "2m", "target": 1 },
+        { "duration": "15m", "target": 1 },
+        { "duration": "2m", "target": 0 }
+      ]
+    },
+  },
+  "thresholds": {
+    "iteration_duration": [ "p(95)<1000" ],
+    "http_req_failed": [ "rate<0.01" ],
+    "http_req_duration": [ "p(95)<1000" ]
+  }
+}
+```
+
+## Snapshots
+
+- https://snapshots.raintank.io/dashboard/snapshot/ie215NIaFsLwXIzrebhe7Dqy1QCkSFcG
+- https://snapshots.raintank.io/dashboard/snapshot/ysBZLoJedpygVROKbERs287JNLjuz8k5
+- https://snapshots.raintank.io/dashboard/snapshot/Nab8aB5S31oK3ey1hqb3LGq31lGJNokr
+- https://snapshots.raintank.io/dashboard/snapshot/i16hs25XuA5NJ7B2eNHiF1gD0I4XYX3Q
+
+## Observations
+
+- `Callback-Handler` Simulator Service is able to handle `400+ Ops/s` End-to-end, while sustaining an average duration of just over `2ms`. This is shown by the following dashboards/metrics:
+  - [K6](./images/Official%20k6%20Test%20Result.png)
+    - `Iteration Rate` (Mean) = `461 Ops/s`
+    - `Ieration Duration (avg)` (Mean) = `2.22ms`
+  - [Callback Handler Svc](./images/Supporting%20Services%20-%20Callback%20Hander%20Service.png)
+    - `op:fspiop_put_parties_end2end - success:true` - observe the `E2E, Request, Response Calculations Processed Per Second` Graph. Note the Mean includes the pre/post run.
+    - `op:fspiop_put_parties_end2end - success:true` - observe the `E2E, Request, Response Performance Timing Calculations`. Mean is `1.86ms`.
+    - The `op:fspiop_put_parties_request` and `op:fspiop_put_parties_response` fall-inline with the observes `Ops/s` and the `request` where most of the duration is spend due to the Callback-Handler sending out the **Async** `FSPIOP PUT /parties` callback response.
+  - [Docker Node Monitoring](./images/docker-prometheus-monitoring.png)
+    - `Callback-Handler` services show no observable resource constraint from both a memory and cpu usage.
+
+## Recommendations
+
+- Observe `Scenario #2+` and compare the `Callback-Handler`'s metrics against this **baseline** to determine if there are any issues with either the Mocked Simulators (i.e. `Callback-Handlers`) or the **Async** `FSPIOP PUT /parties` callback response.