As an Open Data Hub maintainer i want a precise message specification on top of which build and broadcast all messages

[Luscha]

As an Open Data Hub maintainer i want a precise message specification on top of which build and broadcast all messages.

Questions to be defined:

• do we want to assign an ID to each message?
• which fields do we want to have in each message (e.g. origin, data collector, timestamp, history, etc.)?

This is mainly a documentation issue.

[sseppi]

@Luscha can we discuss this issue a bit more into detail on Monday, in order to define the priority and the right Milestone?

[clezag]

The current format (by convention - as defined in WriterRoute) is

{
string provider
string timestamp
string rawdata
}

• provider is in format provider/dataset
• timestamp is in format ISO 8601 (not sure how timezones are handled - has to be tested)
• rawdata is a string, could be a raw json or yaml, or base64 encoded whatever. max 16MB due to mongodb limitations

Reasonable candidates to be included are:

• id (a unique ID propagated as correlation ID in rabbitmq, so that we can track a datapoint from start to finish)
• content type to enable handling of specific data formats, and document what blobs actually are