Improve accessing nested elements in JSON payloads

[jpmens]

I think we should add JSONPath support (perhaps JSONPath RW to mqttwarn; it would make people's lives easier when accessing elements in JSON payloads.

[amotl]

We have very good experiences with the jsonpointer Python module, please consider this as well. Personally, i would give JSON Pointer (RFC 6901) preference over JSONPath, as the former also sports RFC formalities.

It is really easy to work with and we intensively use it across the board in different projects, e.g. here or here.

[jpmens]

That’s how we’ll do it.

[amotl]

Hi again,

at ¹ ff., @u-l-m-i shared with us how TTN's optional JSON event message filtering capabilities look like: You can use an expression like up.uplink_message.decoded_payload, in order to focus on specific elements in a nested data structure, and leave all others on the transformation process behind. Effectively, it looks like a jq expression, and could be implemented using the Python bindings for jq in form of the Python jq package.

Other than this, I recently discovered the Morph! package, which also looks like an excellent candidate to spice up the transformation subsystem in one way or another, maybe only in the form of proper user guidance, maybe more.

With kind regards,
Andreas.

Footnotes

1. https://community.hiveeyes.org/t/tts-ttn-daten-an-kotori-weiterleiten/1422/41 ↩

[amotl]

I just discovered the excellent json-joy package by @streamich through ¹. Together with GH-667, this could be an excellent option for JavaScript folks.

Footnotes

1. https://news.ycombinator.com/item?id=36030866 ↩

[amotl]

On this topic, I just rediscovered the JMESPath query language and the corresponding Python implementation jmespath.py by @jamesls and contributors. Apparently, it is involved in the Mars Helicopter mission in some way or another 💯. Other than this, it seems to be well engineered (citing some highlights below), and supporting multiple languages would blend well with ideas like GH-667 and GH-669.

A Complete Specification

The JMESPath language is described in an ABNF grammar with a complete specification. This ensures that the language syntax is precisely defined.

A compliance test suite

JMESPath has a full suite of data driven testcases. This ensures parity for multiple libraries, and makes it easy for developers to implement JMESPath in their language of choice.

Libraries in Multiple Languages

Each JMESPath library passes a complete suite of compliance tests to ensure they work as intended. There are libraries in multiple languages including python, php, javascript and lua.

On Mars

[amotl]

Hi again,

at ¹², we shared a recipe including a JMESPath expression to extract the most recent scalar measurement value from a list of measurement objects. The data is coming from Paxcounter devices. In this case, with respect to the example data shared below, the answer to the question in form of the JMESPath expression is 42.

With kind regards,
Andreas.

Expression

reverse(sort_by([], &time)) | [0].pax

Data Sample

paxcounter_demo_data = [
    {
        "ble": 0.0,
        "pax": 2.0,
        "time": "2023-06-08T18:52:39.502Z",
        "timesync_seqno": None,
        "wifi": 2.0
    },
    {
        "ble": 0.0,
        "pax": 42.0,
        "time": "2023-06-08T18:53:39.501Z",
        "timesync_seqno": None,
        "wifi": 2.0
    }
]

Footnotes

1. Den aktuellsten publizierten Datensatz oder Einzelwert eines bestimmten Datenkanals abrufen ↩

2. https://gist.github.com/amotl/21d7f8647920388358fc504ee819ce58 ↩