UNL Member Operational Diagnostics

[mtrippled]

This discussion is intended to help resolve a recurring issue in XRP Ledger operation: intermittent UNL node instability. The result of this, if spread beyond quorum thresholds, is network delays in processing transactions. Generally, the most effective way to diagnose this type of issue is through analysis of the debug log.

However, there are several things that make this difficult, including:

1. This log is local to each server. To share, the operator needs to manually log into the server and upload the file to some agreed-upon place for analysis.
2. By default, the log is very verbose and often generates several GB per day. Therefore, many operators reduce the verbosity drastically to save disk space. However, this reduces how useful the log is for diagnostics. For trouble-shooting, this further complicates things if logging verbosity is too low during an event. It means that the verbosity needs to increase and hope for recurrence.

I want to offer a simple solution that should be implementable relatively quickly, as follows:

1. For UNL operators, be willing to share log data with the community if they need assistance with troubleshooting. The log data is scrubbed of things like secrets and should be safe to share.
2. rippled should be enhanced as follows:
   a. Have a distinct log level or log entry type that always logs. This could be a new log level, above FATAL, that cannot be disabled. Or it could be some other mechanism internal to rippled accomplishing the same.
   b. Place in this log level essential information for initial diagnostics of a stability issue pertaining to consensus.

Here's how I picture this being implemented. Namely, there are several steps in consensus that a required to work consistently with each node in close to lock step. The information does not need to be extremely verbose, but should include the following (all having timestamps and some measuring wall clock durations):

• Consensus phase changes and results of previous phases.
• Proposals and validations received from peers on the UNL.
• Calculations for each step, such as when to close ledgers, accept proposals, and so on.

This proposal is not attempting to do the following:

• Build some type of distributed diagnostics infrastructure and tooling, processes.
• Overhaul rippled logging.

Those are big efforts. Instead, this proposal aims to make it relatively easy for us to help each other out with diagnostics that can provide a starting point for trouble-shooting beyond flying blind with speculation. I think the rippled part can be implemented relatively quickly, within a couple of days, and then folded into the next release. Then, there will be no special configuration for node operators. It will slightly increase log verbosity.