Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[mqtt.homeassistant][JINJA] Transformation problem with value template key #15145

Closed
apella12 opened this issue Jun 27, 2023 · 15 comments · Fixed by #17412
Closed

[mqtt.homeassistant][JINJA] Transformation problem with value template key #15145

apella12 opened this issue Jun 27, 2023 · 15 comments · Fixed by #17412
Labels
bug An unexpected problem or unintended behavior of an add-on

Comments

@apella12
Copy link
Contributor

I did read this is potentially the wrong place (just need some feedback at this point). I'm not fluent in Java but can read a little bit of code. The actual log message is from the Addons repository, but when I looked into that class it appears to call on the transformation services in the core. However, from within the core I could not find the source.

Background: I'm experimenting with zwave-js-ui for OH. On a test system I can discover all my test zwave devices in OH4M3. Test setup with openhabian (on WSL - Debian 11 with MQTT, Json Path & Jinja transform installed) and all the channels/topics appear. It is early in testing so far, but most channels/topics can be linked to items and display the state (ON/OFF or Number).

The one type of channel /topic with a problem so far;
{"value_template":"{{ {22: \"Window/door is open\",23: \"Window/door is closed\",5632: \"Window/door is open in regular position\",5633: \"Window/door is open in tilt position\"}[value_json.value] | default(value_json.value) }}","icon":"mdi:alarm-light","state_topic":"zwave/nodeID_10/113/0/Access_Control/Door_state","json_attributes_topic":"zwave/nodeID_10/113/0/Access_Control/Door_state","device":{"identifiers":["zwavejs2mqtt_0xe2bd1ff1_node10"],"manufacturer":"Zooz","model":"Tilt Shock XS Sensor (ZSE43)","name":"nodeID_10","sw_version":"1.20.1"},"name":"nodeID_10_notification_access_control_door_state","unique_id":"zwavejs2mqtt_0xe2bd1ff1_10-113-0-Access_Control-Door_state"}

Expected Behavior

I would hope this should return the string corresponding with the number. I don't know if this is a bug or just not allowed by the Jinja /JSON specification.

Current Behavior

2023-06-27 14:32:56.534 [WARN ] [t.generic.ChannelStateTransformation] - Executing the JINJA-transformation failed: An error occurred while transformation. InterpretException: Error resolving expression [{22: "Window/door is open",23: "Window/door is closed",5632: "Window/door is open in regular position",5633: "Window/door is open in tilt position"}[value_json.value] | default(value_json.value)]: TemplateStateException: Dict key must be a string or identifier, was: 22

and another;
2023-06-27 14:32:56.174 [WARN ] [t.generic.ChannelStateTransformation] - Executing the JINJA-transformation failed: An error occurred while transformation. InterpretException: Error resolving expression [{0: "off", 1: "heat", 2: "cool"}[value_json.value] | default('off')]: TemplateStateException: Dict key must be a string or identifier, was: 0

Possible Solution

Allow a number to be the index to get the string. If it helps, all the numbers will be a member of integers.

Context

zwave-js-ui includes several features not currently available with the OH Zwave binding including S2 security, OTA firmware updates, 700/800 series controllers and has applied for certification with the Zwave Alliance standards body. I do not know if this is going to work. There could be more issues (possibly impossible to resolve) as I add different devices.

@apella12 apella12 added the bug An unexpected problem or unintended behavior of an add-on label Jun 27, 2023
@J-N-K J-N-K transferred this issue from openhab/openhab-core Jun 27, 2023
@apella12
Copy link
Contributor Author

Thanks for moving. Sorry about that.

Today I captured some Debug level logs around the expression in question.
debug-jinja.txt

@apella12 apella12 changed the title [transform] Jinja transformation problem with home assistant channel [MQTT][Home Assistant][JINJA] Transformation problem with value template key Jun 30, 2023
@gargamelonly
Copy link

Could also be related:
Koenkk/zigbee2mqtt#17395

@apella12
Copy link
Contributor Author

I believe this will need to be addressed upstream first. I opened an issue on HubSpot/jinjava and it appears there is an open PR to address. []HubSpot/jinjava#934. After that OH will need to update to the version that contains that PR.

@openhab-bot
Copy link
Collaborator

This issue has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/modify-jinja-transform-to-handle-a-number-key-in-json/148484/1

@openhab-bot
Copy link
Collaborator

This issue has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/test-zwave-js-ui-in-place-of-zwave-binding/150007/3

@fcaputo
Copy link

fcaputo commented Nov 13, 2023

Wouldn't it be possible to iterate over the keys and simply call toString() on them before passing the value to jinjava?

@apella12
Copy link
Contributor Author

@fcaputo I did try unsuccessfully for some time to modify the HA configs to get jinjava to recognize the keys as strings before reverting to using the generic mqtt thing(s) to set up 135 channels. I'm fully on zwave-js-ui at this point. Modifying the HA configs channels is challenging as they are fixed in OH. Since zwave-js-ui is the main Zwave provider for HA, modifying the configs in the application is not an option. I used Mqtt Explorer to publish test config topics (like adding the toString()) and see how they were handled by OH. I also created and updated jinjava .jars with increased logging and other ideas to get this to work (at least at my level of knowledge).

Regardless, my "meta" idea was to promote the zwave-js-ui 3rd party app as a more modern and more widely used "binding" for OH zwave devices, similar to zigbee to Mqtt. That will not be possible for many OH users without a working discovery component. Even if I missed a way to get the keys recognized as strings, I think it would be too much to ask a casual OH user to modify the HA configs. It is actually more straightforward, if not repetitive to just use the generic mqtt thing. The number key needs to be addressed at the source (IMO). As noted above there is an open PR on the git Hubspot/jinjava, so there is some hope. However, I'd also note that other HA configs don't always work perfectly in OH. At best I think there will need to be a mix of discovered topics/channels and manually created ones. Numbered keys will help tilt the balance to discovery, so I left this open for the "meta" objective.

@fcaputo
Copy link

fcaputo commented Nov 15, 2023

Hi Robert,

let's start a discussion in the openhab forum. I totally agree to your vision of making zwave-js-ui the guy, who connects OH to the devices. But we definitely need autodiscovery. IMO all devices listed in https://devices.zwave-js.io should work out of the box. So I guess we need a bridge. zwave-js-ui has a REST and a WebSocket API, which might be useful. As far as I understood it's providing a "Home Assistant Backend API".

I'm willing to implement such a bridge though I'm completely new to OH. But I need it. I bought a 700 stick and want it to work in OH.

@apella12
Copy link
Contributor Author

apella12 commented Nov 16, 2023

The HA discovery does have some OH limitations (see Docs) and they also only cover certain device types
hass device component.txt. Probably the most impactful is the climate (thermostats). I haven't been able to test if the "keys as numbers" would resolve that. (If you have some maven build knowledge I did compile a jinjava with the PR fix noted above for OH testing, but could not get it compiled into OH transform). I was going to try again once winter sets in and my outdoor activities are restricted. My opinion is that full discovery will take some time and always require some generic (manual) mqtt channels.

I'm not a zigbee user, but there seems to be a lot of zigbee2mqtt OH users. I have been wondering how many use discovery or the manual method or both. That would be a good forum question.

An alternative approach that will take some work and time is to create new Mqtt discovery service in OH (easy) and create a complete alternative set of "configs" in zwave-js-ui (very hard-at least for me). The HA discovery templates have been around a long time and are complex. Some could be reused, but some would have to be written (to avoid the "number as key issue" and other quirks and integrated into the ZUI structure. There would probably be other issues. Getting the "number as key" to work will likely be easier and quicker.

Lastly, I wrote a 700 controller PR and compiled it into a test "Release" binding on my github website. Which one to use depends on your OH version. That is going to be the quickest (to use your 700 controller) for now if you want to try.

@fcaputo
Copy link

fcaputo commented Nov 16, 2023

On which branch is it based?
I tried to run on OH 4.0.4 with the following exception:

2023-11-16 19:02:45.147 [WARN ] [org.apache.felix.fileinstall        ] - Error while starting bundle: file:/usr/share/openhab/addons/org.openhab.binding.zwave-4.1.0-SNAPSHOT.jar
org.osgi.framework.BundleException: Could not resolve module: org.openhab.binding.zwave [240]
  Unresolved requirement: Import-Package: org.openhab.core.io.transport.serial

        at org.eclipse.osgi.container.Module.start(Module.java:463) ~[org.eclipse.osgi-3.18.0.jar:?]
        at org.eclipse.osgi.internal.framework.EquinoxBundle.start(EquinoxBundle.java:445) ~[org.eclipse.osgi-3.18.0.jar:?]
        at org.apache.felix.fileinstall.internal.DirectoryWatcher.startBundle(DirectoryWatcher.java:1260) ~[?:?]
        at org.apache.felix.fileinstall.internal.DirectoryWatcher.startBundles(DirectoryWatcher.java:1233) ~[?:?]
        at org.apache.felix.fileinstall.internal.DirectoryWatcher.startAllBundles(DirectoryWatcher.java:1221) ~[?:?]
        at org.apache.felix.fileinstall.internal.DirectoryWatcher.doProcess(DirectoryWatcher.java:515) ~[?:?]
        at org.apache.felix.fileinstall.internal.DirectoryWatcher.process(DirectoryWatcher.java:365) ~[?:?]
        at org.apache.felix.fileinstall.internal.DirectoryWatcher.run(DirectoryWatcher.java:316) ~[?:?]

@apella12
Copy link
Contributor Author

apella12 commented Nov 16, 2023

Go into the karaf console (sudo openhab-cli console), your sudo password, then habopen. Then type feature:install openhab-transport-serial Note: this is normally installed with the UI installation of the zwave binding to allow communication with the Zstick. It is not a 700 issue.

They have made some changes in the build since 4.0, but I think 4.0.4 will work with the 10-18-23 version. If a problem (some sort of karaf message) try the July 27 version. Since I converted over to ZUI, I'm not able to test with a 700 controller, but it had been working for about 9 months through all the 4.0 milestones

@Skinah Skinah changed the title [MQTT][Home Assistant][JINJA] Transformation problem with value template key [mqtt.homeassistant][JINJA] Transformation problem with value template key Dec 10, 2023
@ccutrer
Copy link
Contributor

ccutrer commented Sep 5, 2024

This has been fixed in upstream Jinjava; we're just waiting on them to make a release so we can update.

@apella12
Copy link
Contributor Author

apella12 commented Sep 6, 2024

@ccutrer I was aware you fixed this. Thanks for your efforts. However, the jinjava code owners do not seem to be in any hurry as their last release was Feb 15. IMO - It is too bad OH has to rely on this code for the HA configs. I'll close this

@apella12 apella12 closed this as completed Sep 6, 2024
@ccutrer
Copy link
Contributor

ccutrer commented Sep 6, 2024

👍

Yeah, it's a little frustrating. I opened a new issue with them today asking for a release.

And heh, I got you confused since I mostly see the GitHub notifications which shows your full name, but then going through the issues in your username, and I didn't connect them together!

@apella12
Copy link
Contributor Author

I did see your PR. WTG! I've had a test ready to go for months.

No worries on the confusion on username. It's kinda meant that way ;-)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug An unexpected problem or unintended behavior of an add-on
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants