Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(mojaloop/#3400): benchmarking performance for ALS #1

Merged
merged 39 commits into from
Aug 4, 2023
Merged
Show file tree
Hide file tree
Changes from 34 commits
Commits
Show all changes
39 commits
Select commit Hold shift + click to select a range
19205a4
added initial template, and scneario runs for 20230726
mdebarros Jul 26, 2023
4d8649e
updates
mdebarros Jul 26, 2023
54713b7
added exports for test-scenarios
mdebarros Jul 26, 2023
2ef7855
added environment
mdebarros Jul 26, 2023
35c8fd3
added log files
mdebarros Jul 26, 2023
dda6de3
added test scenario s5-1690447500991
mdebarros Jul 27, 2023
c4fc6c8
added data to s5-1690447500991
mdebarros Jul 27, 2023
89cd047
added 20230727/s7-1690407403663
mdebarros Jul 27, 2023
e6b3edd
moved s7-1690407403663/ to 20230726 folder
mdebarros Jul 27, 2023
5128fb1
feat: added two scenarios
vijayg10 Jul 27, 2023
f8d84c7
updates to main readme, and template
mdebarros Jul 27, 2023
2b37a12
feat: added s9 scenario results
vijayg10 Jul 27, 2023
7fbce24
Merge branch 'feat/performance-maintenance-characterisation-als' of h…
vijayg10 Jul 27, 2023
1d697ba
added more test-cases
mdebarros Jul 27, 2023
5e86192
chore: add test results for #11-13
kleyow Jul 28, 2023
cf88741
chore: remove
kleyow Jul 28, 2023
1d79c2a
fleshed out main readme with approach, and added drawio diagrams
mdebarros Jul 28, 2023
990fe64
fleshed out main readme with approach, and added drawio diagrams
mdebarros Jul 28, 2023
f7f0d20
fleshed out main readme with approach, and added drawio diagrams
mdebarros Jul 28, 2023
9249293
fleshed out main readme with approach, and added drawio diagrams
mdebarros Jul 28, 2023
4e4fe59
feat: captured s14 data
vijayg10 Jul 28, 2023
1daa63d
Merge branch 'feat/performance-maintenance-characterisation-als' of h…
vijayg10 Jul 28, 2023
e5c63d9
added docker version
mdebarros Jul 28, 2023
a03e42b
Merge branch 'feat/performance-maintenance-characterisation-als' of g…
mdebarros Jul 28, 2023
c03cd34
feat: added empty folder structures for s15 and s16 scenarios
vijayg10 Jul 28, 2023
86b3442
chore: add images for s15 and s16
kleyow Jul 28, 2023
6ccdcd6
feat(mojaloop/#3424): analyse als perf results
mdebarros Aug 1, 2023
53b53f7
Merge branch 'feat/performance-maintenance-characterisation-als' of g…
mdebarros Aug 1, 2023
f52e52d
feat(mojaloop/#3424): analyse als perf results
mdebarros Aug 1, 2023
5cf459b
fixes to headers for consistency
mdebarros Aug 1, 2023
cd2f080
fixed git.ignore --> .gitignore
mdebarros Aug 1, 2023
c54bda3
fix: standardised test scenarios names
mdebarros Aug 1, 2023
b66131e
fixed comment
mdebarros Aug 1, 2023
5e9b78c
fixes to scratched out TSs
mdebarros Aug 1, 2023
6d356c3
fixed bad search-replace
mdebarros Aug 1, 2023
1642418
added story links
mdebarros Aug 2, 2023
200ed66
added follow-up story for end-to-end metric instrumentation on the ALS
mdebarros Aug 2, 2023
ce30a8d
Updated documentation based on review comments.
mdebarros Aug 3, 2023
15877d6
added k8s test case and scenario name/config
mdebarros Aug 4, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
70 changes: 70 additions & 0 deletions 20230726/s1-1690367402771/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
# Scenario 1: FSPIOP Discovery GET Parties with Sims-only - ALS:v14.2.2, scale:1, k6vu:1

The End-to-end operation from the K6 test-runner included the following HTTP operations for each *iteration*:

1. ADMIN GET /participants request to the Central-Ledger to validate payerFspId. <-- sync response
2. ADMIN GET /participants request to the Central-Ledger to validate payeeFspId. <-- sync response
3. ORACLE GET /participants request to the Oracle to resolve FSPID for payeeId. <-- sync response
4. FSPIOP GET /parties request to the ALS <-- async callback response
5. WS Subscription to the `Callback-Handler` Service for Callback Response notifications

```conf
var-testid=1690367402771
params=&var-testid=1690367402771&from=1690367297867&to=1690368635328
```

## Environment

- m6i.2xlarge
- 8 CPU - 3.5 GHz 3rd Generation Intel Xeon Scalable processors (Ice Lake 8375C)
- 32gb RAM
- https://gist.github.com/mdebarros/6d9ac90f33c96031cbce6b9a3ea8048e

## k6 Test Config

```json
{
"scenarios": { // define scenarios
// warm-up
"accountLookup": { // original scenario for accountLookup
"executor": "ramping-vus",
"exec": "accountLookupScenarios",
"startVUs": 1,
"stages": [
{ "duration": "2m", "target": 1 },
{ "duration": "15m", "target": 1 },
{ "duration": "2m", "target": 0 }
]
},
},
"thresholds": {
"iteration_duration": [ "p(95)<1000" ],
"http_req_failed": [ "rate<0.01" ],
"http_req_duration": [ "p(95)<1000" ]
}
}
```

## Snapshots

- https://snapshots.raintank.io/dashboard/snapshot/ie215NIaFsLwXIzrebhe7Dqy1QCkSFcG
- https://snapshots.raintank.io/dashboard/snapshot/ysBZLoJedpygVROKbERs287JNLjuz8k5
- https://snapshots.raintank.io/dashboard/snapshot/Nab8aB5S31oK3ey1hqb3LGq31lGJNokr
- https://snapshots.raintank.io/dashboard/snapshot/i16hs25XuA5NJ7B2eNHiF1gD0I4XYX3Q

## Observations

- `Callback-Handler` Simulator Service is able to handle `400+ Ops/s` End-to-end, while sustaining an average duration of just over `2ms`. This is shown by the following dashboards/metrics:
- [K6](./images/Official%20k6%20Test%20Result.png)
- `Iteration Rate` (Mean) = `461 Ops/s`
- `Ieration Duration (avg)` (Mean) = `2.22ms`
- [Callback Handler Svc](./images/Supporting%20Services%20-%20Callback%20Hander%20Service.png)
- `op:fspiop_put_parties_end2end - success:true` - observe the `E2E, Request, Response Calculations Processed Per Second` Graph. Note the Mean includes the pre/post run.
- `op:fspiop_put_parties_end2end - success:true` - observe the `E2E, Request, Response Performance Timing Calculations`. Mean is `1.86ms`.
- The `op:fspiop_put_parties_request` and `op:fspiop_put_parties_response` fall-inline with the observes `Ops/s` and the `request` where most of the duration is spend due to the Callback-Handler sending out the **Async** `FSPIOP PUT /parties` callback response.
- [Docker Node Monitoring](./images/docker-prometheus-monitoring.png)
- `Callback-Handler` services show no observable resource constraint from both a memory and cpu usage.

## Recommendations

- Observe `Scenario #2+` and compare the `Callback-Handler`'s metrics against this **baseline** to determine if there are any issues with either the Mocked Simulators (i.e. `Callback-Handlers`) or the **Async** `FSPIOP PUT /parties` callback response.
Empty file.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Empty file.
Loading