Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

As an Open Data Hub maintainer i want a precise message specification on top of which build and broadcast all messages #42

Open
Luscha opened this issue Oct 26, 2023 · 2 comments
Labels
documentation Improvements or additions to documentation priority/high

Comments

@Luscha
Copy link
Contributor

Luscha commented Oct 26, 2023

As an Open Data Hub maintainer i want a precise message specification on top of which build and broadcast all messages.

Questions to be defined:

  • do we want to assign an ID to each message?
  • which fields do we want to have in each message (e.g. origin, data collector, timestamp, history, etc.)?

This is mainly a documentation issue.

@Luscha Luscha added documentation Improvements or additions to documentation priority/high labels Oct 26, 2023
@sseppi
Copy link

sseppi commented Oct 30, 2023

@Luscha can we discuss this issue a bit more into detail on Monday, in order to define the priority and the right Milestone?

@sseppi sseppi added question Further information is requested and removed question Further information is requested labels Oct 30, 2023
@clezag
Copy link
Member

clezag commented Jul 16, 2024

The current format (by convention - as defined in WriterRoute) is

{
string provider
string timestamp
string rawdata
}
  • provider is in format provider/dataset
  • timestamp is in format ISO 8601 (not sure how timezones are handled - has to be tested)
  • rawdata is a string, could be a raw json or yaml, or base64 encoded whatever. max 16MB due to mongodb limitations

Reasonable candidates to be included are:

  • id (a unique ID propagated as correlation ID in rabbitmq, so that we can track a datapoint from start to finish)
  • content type to enable handling of specific data formats, and document what blobs actually are

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation priority/high
Projects
None yet
Development

No branches or pull requests

3 participants