You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardexpand all lines: docs/adr/2023-11-07_029-streaming-persistence.md
+70-24
Original file line number
Diff line number
Diff line change
@@ -2,7 +2,7 @@
2
2
slug: 29
3
3
title: |
4
4
29. EventServer abstraction for event persistence
5
-
authors: []
5
+
authors: [@cardenaso11]
6
6
tags: []
7
7
---
8
8
@@ -14,10 +14,10 @@ N/A
14
14
The current Hydra API is materialized as a firehose WebSocket connection: upon connecting, a client receives a deluge of information, such as all past transactions. Additionally, there is no way to pick and choose which messages to receive, and not all events are exposed via the API.
15
15
16
16
This forces some awkward interactions when integrating with a Hydra node externally, such as when using it as a component in other protocols, or trying to operationalize the Hydra server for monitoring and persistence. Here's just a few of the things that prove to be awkward:
17
-
- An application would need to store it's own notion of progress through the event log, so it knew which messages to "ignore" on reconnection
17
+
- An application that only cared about certain Hydra resources (like say, the current UTxO set), would be unable to subscribe to just that resource. The application would be made to either parse the Hydra incremental persistence log directly (which may change), or subscribe to the full firehose of Websocket API events.
18
18
- An application that wanted to write these hydra events to a message broker like RabbitMQ or Kinesis would need it's own custom integration layer
19
-
- An application that only cared about a limited set of message types may be overwhelmed (and slowed down) by the barage of messages on connection
20
-
- An application that wanted a deeper view into the workings of the hydra protocol would have no access to several of the internal event types
19
+
- An application that wishes to build on Hydra would need to fork the Hydra project in order to change how file persistence worked.
20
+
21
21
22
22
Additionally, much of the changes that would need to happen to solve any of these would apply equally well to the on-disk persistence the Hydra node currently provides.
23
23
@@ -27,45 +27,91 @@ Additionally, much of the changes that would need to happen to solve any of thes
27
27
28
28
- Client must be flexible and ready to handle many different events
29
29
30
-
- There is no way to simply hand off transactions to the hyrda-node currently, a full connection must be initiated before observed transactions can be applied, and bulky JSON objects can slow down "bulk" operations
30
+
- There is no way to simply hand off transactions to the hyrda-node currently, a full connection must be initiated before observed transactions can be applied.
31
31
32
-
-Many applications, like SundaeSwap's Gummiworm Protocol, do not need need the full general API, and benefit from any performance improvement
32
+
-Using history=0 in the query string allows ignoring prior events, but it's not possible to ignore events going forward, only disabling UTxO on Snapshot events.
33
33
34
-
- The challenge of subscribing to event types is complex to handle, but would be applicable to many parts of Hydra, not just for subscribing to new transactions. It's a good fit for passing stuff off to a message queue (MQTT, Kafka, whatever), probably from a dedicated process (see "Chain Server" below)
35
-
- Could also just directly use a "compatible" spec like STOMP or MQTT, but that would lock in implementation details
34
+
- Many applications, like SundaeSwap's Gummiworm Protocol, would benefit from using the API in ways which are currently not supported
36
35
36
+
- Custom event management is intended to be for a minority of Hydra users, that intend to integrate heavily with Hydra. It's a good fit for passing stuff off to a message queue (MQTT, Kafka, whatever) for further processing, probably from a dedicated process (see "Chain Server" below)
37
+
- The event management is intended to modularize and unify the Websocket API, and (incremental, full) persistence
38
+
- The default event management is intended to be transparent to most users
39
+
- There exist "highly compatible" specifications like STOMP or MQTT, but supporting these directly as a substitute for API or persistence, would lock in a significant amount of implementation details
40
+
41
+
42
+
<!-- The following section can be removed from the ADR, but mostly just recorded background, motivation, prior art -->
37
43
- Previous mock chain using ZeroMQ was [removed][zmq] as part of [#119][#119], due to complexity and feeling of ZeroMQ being unmaintained (primary author no longer contributing, new tag release versions not being published)
38
44
- This mock chain was used to mock the layer L1 chain, not the L2 ledger
39
45
- Attempt in February 2023 to externalize chainsync server as part of [#230][#230]
40
46
- Similar to [#119][#119], but the "Chain Server" component in charge of translating Hydra Websocket messages into direct chain, mock chain, or any hypothetical message queue, not necessarily just ZeroMQ
41
47
- Deemed low priority due to ambiguous use-case at the time. SundaeSwap's Gummiworm Protocol would benefit from the additional control enabled by the Event Server
42
48
43
-
# Decision
49
+
- Offline mode intended to persist UTxO state to a file for simplified offline-mode persistence. As a standalone feature, the interface would be too ad-hoc. A less ad-hoc way to keep a single updated UTxO state file, would instead allow for keeping an updated file for any Hydra resource.
44
50
45
-
A new abstraction, the EventServer, will be introduced. Each internal hydra event will be sent to the event server, which will be responsible for persisting that event and returning a durable monotonically increasing global event ID. Different implementations of the event server can handle this event differently. Initially, we will implement the following:
46
-
- MockEventServer, which increments a counter for the ID, and discards the event
47
-
- FileEventServer, which increments a counter for the ID, and encapsulates the existing file persistence mechanism
48
-
- SocketEventServer, which increments a counter for the ID, and writes the event to an arbitrary unix socket
49
-
- WebsocketBroadcastEventServer, which broadcasts the publicly visible events over the websocket API
50
-
- UTxOStateServer, which increments a counter for the ID, and updates a file with the latest UTxO after the event
51
-
- Generalizes the UTxO persistence mechanism introduced in [offline mode][offline-mode]
52
-
- May be configured to only persist UTxO state periodically and/or on SIGTERM, allowing for performant in-memory Hydra usage. This is intended for offline mode usecases where the transactions ingested are already persisted elsewhere.
53
-
- One configuration which we expect will be common and useful, is the usage of a MultiplexingEventServer configured with a primary MockEventServer, and a secondary UTxOStateServer, which will persist the UTxO state to disk.
54
-
- MultiplexingEventServer, which has a primary event server (which it writes to first, and uses its ID) and a list of secondary event servers (which it writes to in sequence, but discards the ID)
55
-
56
-
New configuration options will be introduced for choosing between and configuring these options for the event server. The default will be a multiplexing event server, using the file event server as its primary event server, and a websocket broadcast event server as its secondary event server.
51
+
# Decision
52
+
Each internal hydra event will have a durable, monotonically increasing event ID, ordering all the internal events in the persistence log.
53
+
54
+
A new abstraction, the EventSink, will be introduced. The node's state will contain a non-empty list of EventSinks. Each internal hydra event will be sent to a non-empty list of event sinks, which will be responsible for persisting or serving that event in a specific manner. When multiple event sinks are specified, they run in order. Active event can change at runtime. Initially, we will implement the following:
55
+
- MockEventSink, which discards the event. This is exclusive to offline mode, since this would change the semantics of the Hydra node in online mode. This is the only way to have an "empty" EventSink list
56
+
- APIBroadcastEventSink, which broadcasts the publicly visible resource events over the websocket API.
57
+
- Subsumes the existing Websocket API.
58
+
- Can be created via CLI subcommand if Websocket client IP is known. Can be created at runtime by the top-level API (APIServerEventSource)
59
+
- Two modes:
60
+
- Full event log mode. Similar to existing Websocket API, broadcasts all events.
61
+
- Single-resource event log mode. Broadcasts the state changes of a single resource.
62
+
- One APIBroadcastEventSink per listener. A Hydra node running with no one listening would have 0 APIBroadcastEventSink's in the EventSink list
63
+
- Establishing a websocket connection will add a new event sink to handle broadcasting messages
64
+
- Resources should all support JSON content type. UTXO resource, Tx resource, should support CBOR.
65
+
- APIBroadcastResourceSink, which broadcasts the latest resource after a StateChanged event
66
+
- Runs in single-resource event log mode, broadcasting the current state of a single resource.
67
+
- EventFileSink, which updates a file with the state changed by a StateChanged event
68
+
- Two modes:
69
+
- Full event log mode. Encapsulates the existing incremental file persistence. Appends all server events incrementally to a file.
70
+
- One of these in the EventSink list is required in Online mode, for proper Hydra semantics
71
+
- Single-resource event log mode. Incrementally appends an event log file for a single resource
72
+
- Persists StateChanged changes
73
+
- ResourceFileSink, which updates a file with the latest resource after a StateChanged event
74
+
- Two modes:
75
+
- Full event log mode. Encapsulates the existing non-incremental full file persistence mechanism. Appends all server events incrementally to a file.
76
+
- Single-resource event log mode. Maintains an up-to-date file for a single resource
77
+
- Consuming an up-to-date single resource will no longer be coupled with overall Hydra state format, only the encoding schema of that particular resource
78
+
- Generalizes the UTxO persistence mechanism previously discussed in [offline mode][offline-mode]
79
+
- May be configured to only persist resource state:
80
+
- Periodically, if the last write was more than a certain interval ago
81
+
- On graceful shutdown/SIGTERM
82
+
- Allows for performant in-memory Hydra usage, for offline mode usecases where the transactions ingested are already persisted elsewhere, and only certain resources are considered important
83
+
84
+
- One configuration which we expect will be common and useful, is the usage of a ResourceFileSink on the UTxO resource in tandem with a MockEventServer in offline mode.
85
+
86
+
The event server will be configured via a new subcommand ("initial-sinks"), which takes an unbounded list of positional arguments. Each positional argument adds a sink to the initial event sink list. There is one argument constructor per EventSink type. Arguments are added in-order to the initial EventSink list. The default parameters configure an EventFileSink for full incremental persistence.
87
+
88
+
The top-level API will change to implement the API changes described in [ADR 25][adr-25]
89
+
- Top level Websocket subscription adds a Full event log EventFileSink
90
+
- API on /vN/ (for some N) will feature endpoints different resources
91
+
- POST verbs emit state change events to modify the resource in question
92
+
- Websocket upgrades on GET emit state change events for the EventSink list (itself a resource) to establish new ongoing Websocket client subscriptions
93
+
- This will expose the new single-resource APIBroadcastEventSink and APIBroadcastResourceSink
94
+
95
+
<!-- Full EventSource implementation probably should be in its own ADR. This is included here, for now, to give an idea of what the bigger picture is-->
96
+
A new abstraction, the EventSource, will be introduced.
97
+
- APIServerEventSource
98
+
- Top level API
99
+
- Top level Websocket API adds a Full event log EventFileSink
100
+
- Handles non-websocket-upgraded single-shot HTTP API verbs [ADR 25][adr-25]
101
+
- POST verbs emit state change events to modify the resource in question
102
+
- Websocket upgraded verbs modify the EventSink list (itself a resource) to establish new ongoing subscribers
57
103
58
104
## Consequences
59
105
60
106
The primary consequence of this is to enable deeper integration and better operationalization of the Hydra node. For example:
61
-
- Users may now use the SocketEventServer to implement custom integrations with existing ecosystem tools
62
-
-To avoid the overhead of a unix socket, they may submit pull requests to add integrations with the most popular tools
107
+
- Users may now use the new sinks to implement custom integrations with existing ecosystem tools
108
+
-Users may use the file sinks to reduce overhead significantly in Offline mode
63
109
- Developers may more easily maintain downstream forks with custom implementations that aren't appropriate for community-wide adoption, such as the Gummiworm Protocol
64
-
- Developers can get exposure to events that aren't normally surfaced in the websocket API
65
110
- Logging, metrics, and durability can be improved or tailored to the application through such integrations
66
111
67
112
Note that while a future goal of this work is to improve the websocket API, making it more stateless and "subscription" based, this ADR does not seek to make those changes, only make them easier to implement in the future.
0 commit comments