|
| 1 | +# Cardano => Kafka Example |
| 2 | + |
| 3 | +This example shows how to leverage _Oura_ to stream data from a Cardano node into a _Kafka_ topic. |
| 4 | + |
| 5 | +## About Kafka |
| 6 | + |
| 7 | +> Apache Kafka is a framework implementation of a software bus using stream-processing. It is an open-source software platform developed by the Apache Software Foundation written in Scala and Java. |
| 8 | +
|
| 9 | +Find [more info](https://en.wikipedia.org/wiki/Apache_Kafka) about _Kafka_ in wikipedia. |
| 10 | + |
| 11 | +## Prerequisites |
| 12 | + |
| 13 | +This examples assumes the following prerequisites: |
| 14 | + |
| 15 | +- A running Cardano node locally accesible via a unix socket. |
| 16 | +- A Kafka cluster accesible through the network. |
| 17 | +- An already existing Kafka topic where to output events |
| 18 | +- _Oura_ binary release installed in local system |
| 19 | + |
| 20 | +## Instructions |
| 21 | + |
| 22 | +### 1. Create an Oura configuration file `caradno2kafka.toml` |
| 23 | + |
| 24 | +```toml |
| 25 | +[source] |
| 26 | +type = "N2C" |
| 27 | +address = ["Unix", "/opt/cardano/cnode/sockets/node0.socket"] |
| 28 | +magic = "testnet" |
| 29 | + |
| 30 | +[sink] |
| 31 | +type = "Kafka" |
| 32 | +brokers = ["kafka-broker-0:9092"] |
| 33 | +topic = "cardano-events" |
| 34 | +``` |
| 35 | + |
| 36 | +Some comments regarding the above configuration: |
| 37 | + |
| 38 | +- the `[source]` section indicates _Oura_ from where to pull chain data. |
| 39 | + - the `N2C` source type value tells _Oura_ to get the data from a Cardano node using Node-to-Client mini-protocols (chain-sync instantiated to full blocks). |
| 40 | + - the `address` field indicates that we should connect via `Unix` socket at the specified path. This value should match the location of your local node socket path. |
| 41 | + - the `magic` field indicates that our node is running in the `testnet` network. Change this value to `mainnet` if appropriate. |
| 42 | +- the `[sink]` section tells _Oura_ where to send the information it gathered. |
| 43 | + - the `Kafka` sink type value indicates that _Oura_ should use a _Kafka_ producer client as the output |
| 44 | + - the `brokers` field indicates the location of the _Kafka_ brokers within the network. Several hostname:port pairs can be added to the array for a "cluster" scenario. |
| 45 | + - the `topic` fields indicates which _Kafka_ topic to used for the outbound messages. |
| 46 | + |
| 47 | +### 2. Run _Oura_ in `daemon` mode |
| 48 | + |
| 49 | +Run the following command from your terminal to start the daemon process: |
| 50 | + |
| 51 | +```sh |
| 52 | +RUST_LOG=info oura daemon --config cardano2kafka.toml |
| 53 | +``` |
| 54 | + |
| 55 | +You should see an output similar to the following: |
| 56 | + |
| 57 | +``` |
| 58 | +[2021-12-13T22:16:43Z INFO oura::sources::n2n::setup] handshake output: Accepted(7, VersionData { network_magic: 764824073, initiator_and_responder_diffusion_mode: false }) |
| 59 | +[2021-12-13T22:16:43Z INFO oura::sources::n2n::setup] chain point query output: Some(Tip(Point(47867448, "f170baa5702c91b23580291c3a184195df7c77d3e1a03b3d6424793aacc850d6"), 6624258)) |
| 60 | +[2021-12-13T22:16:43Z INFO oura::sources::n2n::setup] node tip: Point(47867448,"f170baa5702c91b23580291c3a184195df7c77d3e1a03b3d6424793aacc850d6") |
| 61 | +[2021-12-13T22:16:44Z INFO oura::sources::n2n] rolling block to point Point(47867448, "f170baa5702c91b23580291c3a184195df7c77d3e1a03b3d6424793aacc850d6") |
| 62 | +[2021-12-13T22:16:52Z INFO oura::sources::n2n] requesting block fetch for point Some(Point(47867448, "f170baa5702c91b23580291c3a184195df7c77d3e1a03b3d6424793aacc850d6")) |
| 63 | +[2021-12-13T22:17:15Z INFO oura::sources::n2n] requesting block fetch for point Some(Point(47867448, "f170baa5702c91b23580291c3a184195df7c77d3e1a03b3d6424793aacc850d6")) |
| 64 | +[2021-12-13T22:17:20Z INFO oura::sources::n2n] requesting block fetch for point Some(Point(47867448, "f170baa5702c91b23580291c3a184195df7c77d3e1a03b3d6424793aacc850d6")) |
| 65 | +``` |
| 66 | + |
| 67 | + |
0 commit comments