Proposal: live preimage collector

[protolambda]

Background

The OP-Stack currently requires an archive node for some special tasks:

• output-root proposals (withdrawal storage root): spanning at least an hour, or as long as the proposer is configured to wait before proposals. This may be multiplied, to not handle proposer downtime.
• fault-proof program execution (historical L2 state): the last week worth of data, assuming a disputed L2 output root is no older than the last finalized L2 output root. This may be extended if there are very infrequent output-roots.

Archive nodes retain all historical versions of the state, and are thus more burdensome to run than full nodes, due to increased resource usage, and difficulty syncing (just snap-syncing the latest state is insufficient).

The archive data that the op-stack relies on however does not span more than the most recent week worth of state. This can help bound the resource usage of the archive data significantly.

Note that go-ethereum may implement a "rolling archive mode" at some point, but this would be single-client, and may not handle intermediate merkle-patricia-trie nodes as necessary.

For reference, OP-Mainnet full-node (pebble DB, path scheme) stats:

| Key-Value store       | Path trie account nodes | 18.98 GiB  |  165149260 |
| Key-Value store       | Path trie storage nodes | 113.25 GiB | 1140900306 |
| Key-Value store       | Account snapshot        | 8.05 GiB   |  124274313 |
| Key-Value store       | Storage snapshot        | 57.15 GiB  |  832582174 |

While large, this is signficiantly less than running an archive node (OP-Mainnet Hash-db, leveldb), which takes just under 5 TB total (most of which is state, since a full-node is ~450 GB total).

Towards a solution

To mitigate this, the OP-Stack could be extended with a special service that collects and maintains just the last week worth of preimage data. The preimage data can be collected when it is still "hot", i.e. from a full node, which retains the most recent history (128 L2 blocks, just over 4 minutes of data).

Since output-proposing and fault-proving is not constrained on performance, the preimage collection does not have to be optimized. If the withdrawal-storage is available within an hour, and arbitrary state is available within hours, the use-case of the data is not really affected.

Possible shape of archive

There are two ways to do this:

• A full copy of all state pre-images that are attached to a recent state-root
• A combination of the latest state, and a running list of recently deleted preimages.

The latter approach may be more efficient, but also relies on the live DB serving the preimages reliably. Which includes intermediate MerklePatriciaTrie nodes and such.

The former approach requires more disk-space, but may be easier to maintain conceptually:

• maintain a set of known state-roots
• every few seconds, poll for new state-roots, add to the collection
• every few seconds, poll for old state-roots, remove from the collection
• garbage-collect any merkle-patricia-trie node that is not reachable from a known state-root. This can be done by implementing reference counting or some sweeping routine.
• trace the L2 blocks of state-roots with unknown preimage data, to collect the list of changed accounts and storage values, and then collect the missing trie branches for the changes with batched eth_getProof calls.

The latter approach may be implemented by attaching a log to the execution-layer directly: whenever any preimage gets replaced/deleted, write it to the log. Then prune the log to not contain anything older than a week. The latest state, combined with recent changes, should cover the full set of preimages necessary for the OP-Stack.

Possible variants

By re-using the op-program, the preimages can be generated locally, by processing blocks on top of the existing pre-state. This removes the need for state-fetching through eth_getProof or similar, and also test-runs the op-program consistently; state-roots can be compared against the actual block state-roots.

API

This service should expose a simple key-value getter API, for the op-program (in host mode) to collect state preimages from.

[tynes]

With the recent addition of the db in op-node for safe heads, could this make sense to live as a subsystem directly in op-node? It would then enable things like a p2p network protocol for requesting output preimages