Plugin: Lox2MQTT

[poklutom]

Operating System: Debian GNU/Linux 11 (bullseye)
Kernel: Linux 6.1.21-v8+
Architecture: arm64

DietPi v9.0.2 : 10:12 - Tue 02/20/24
────────────────────────────────────────────────── ───
- Device model : RPi 4 Model B (aarch64)

[nufke]

Just released v0.8.0 which should fix the missing node package dependencies and also does restart the plugin after reboot

[poklutom]

Installed & tested. It is working perfectly. Thank you for fast and great support.

[poklutom]

The plugin works without any problem, curretly I am storing all values into my test InfluxDB2 for testing purpose (Lox2mqtt --> Telegraf --> InfluxDB2)

Additional question - the topic name is currently defined as <mqttTopic>/<serialnr>/<uuid>. For some cases it could be fast/good enough but since my main purpose is to persist data into InfluxDB it can be quite tricky to have Telegraf renaming configuration for each topic separately and also I don't have currently any possibility to group topics based on MQTT topic wildcards (I have currently arround 300 topics).

Didn't you plan to add possibility to change topic name to more friendly name if needed (similliar like Stats4Lox doing)?

Like to add mapping list: <serialnr>/<uuid> --> /xxxx/yyyyy/zzzz/....
It can also help if the the block will be replaced in Loxone config and the value will get the new uuid.

[nufke]

If I understand you correctly, you'd like to see (sub)topic showing the actual control and/or location, and not just the uuid for the specific event?
At the moment the plugin is just publishing the change event using the UUID, without tracking what the value actually represents and to which control, room or category it belongs. In this way, I can keep the plugin and forwarding very simple.

Of course we could consider changing the (sub)topic hierarchy, but it requires that I need to parse the entire loxapp3.json structure to build some sort of look-up table for the remapping. And actually I think this remapping needs to happen at the subscriber/client/App side, not at the server/publisher side, because different people/apps have a different expectation how this topic hierarchy should look like.

For example, should the room or category name added as a subtopic, or just the name of the control? And what should I do with an array of different UUIDs as part of a temperature range in an IRoomController? Therefore I did not go this route, because of the risk of having too many subtopic flavors or configurations. Furthermore, I need to 'slug'/flatten all subtopic names, because some characters are problematic in a MQTTT topic. So this requires additional management of the look-up table.

Could you give a concrete example how the topic hierarchy should look like? In the end, I'd like to keep things very simple.

[poklutom]

I have the same opinion that is good to keep the plugin simple as possible. Therefore the current aproach should be as a default.

But lets imagine my case, I have currently arround 300 topics. 10 of them are states of motion PIR sensors. If I would like to store them directly into InfluxDB, then I have to make separate configuration for each individual topic and set measurement and other tags one by one.

But if I will have possibility to change topic name to eg: loxone1/motion/PIR_kitchen, loxone1/motion/PIR_living_room, .... then I will need just one configuration and configure extraction of measurement and other tags just from topic name:

example part of telegraf.conf:

[[inputs.mqtt_consumer.topic_parsing]]
topic = "loxone1/motion/+"
measurement = "_/measurement/_"​

To keep it simple and most flexble, I would just need to have configuration file for your plugin with following pairs:

loxone/5041234567/1913afaf-0312-0450-ffff6a82f4e34704 loxone1/motion/PIR_kitchen
loxone/5041234567/1913afaf-0312-0451-ffff6a82f4e34704 loxone1/motion/PIR_living_room


This means, the configuration file will be read into the memory just once by plugin startup and then used for replacing of the topic name. If your plugin will process topic which is not in the list, then it will publish topic name in default mode like now. Therefore there is no need parse entire loxapp3.json structure from your side.


Like I mentioned, just for beginning would be enough to have the configuration file maintained directly by SSH and if we will see it as a good aproach, then you can add modification of the file directly from Loxberry GUI but in full manual mode.

[nufke]

adding such lookup/mapping table is technically not a real issue, my main concern is how to support and maintain it for a bigger user base.

I can imagine that people can make a such mapping file, but this file needs to be updated when controls are changed or added, and this might be forgotten. I'd like to avoid 1001 questions on unexplained behavior of controls and states not propagating correctly, because the lookup table seems out-of-sync with the Miniserver JSON structure file. Also building this mapping table for a few hundred items may sound easy, a mistake (typo) is quickly made, and difficult to trace back, since I do not see this local mapping file when issues are reported.

Therefore it would be my preference that this topic mapping is using the information from the Miniserver JSON structure itself. I will study a bit if this can be done and added easily.

[poklutom]

ok, looking forward to see the progress. I could imagine that you can automatically generate topic like "LoxSN_number/category/room/name". There might be issue with special characters and blanks in the names. Keep in mind there might be the same name for 2 different uuids for the same category and room. Therefore probably anyway some optional configuration will be needed.
Please keep update me about the progress.

[nufke]

I've created a development branch in my Lox2MQTT repo which maps each uuid to a topic with name <category>/<room>/<control name>/<state>
So download the zip file using the "Code" button in GitHub and install the zip file using the regular Loxberry plugin install procedure.

In order to enable the topic names, you should enable the option "Use control category and room as MQTT subtopic names" at the plugin configuration page.

Feedback welcome.

[poklutom]

I tried to install it immediately. The plugin is working without any problems as expected. Well done!
I will now try to test some different telegraf configurations for storinng the data into influxDB.

One remark - if it is needed to rename the object in the Loxone app to have a different label, then the topic name is changed as well and therefore the value will be stored as a new measurement. But now idea how to avoid it except fixed mapping: uuid --> control_name

[nufke]

I made an update to the development branch (latest commit 338e1c2), please install & test again. I made the following changes:

• the subtopic string 'states' is removed from the topic names, so the full topic name now becomes <category>/<room>/<control>/<state> <value>
• there is an undocumented (!) feature in the config file which is called "publish_mapping" which will publish the mapping table between uuids and topic names. You need to enable this (false -> true) via ssh, since I will not add this option to the GUI config. You then can grab this mapping table using an MQTT subscriber listening to <mqtt_topic>/<serialnr>/mapping.
• another undocumented (!) feature is to override the internal mapping table by a table published to <mqtt_topic>/<serialnr>/mapping/cmd. For this, the config option "publish_mapping" should also be set to true. Note that the format of this mapping table needs to be exactly the same as the published mapping table. Therefore it is recommended to make required edits to the published mapping table and send this back to the plugin. If the override of the mapping table is successful, you see this as an info message in the log file: INFO: MQTT Adaptor - Topic mapping table read.

[poklutom]

Cool, I had to test it ASAP .
I tried both undocumented features and I can confirm that they are working as you described and it is really close to my requirement.

The only problem I found is that "custom mapping table" is erased when plugin is restarted therefore is needed to publish mapping table again. Will not be possible to read custom mapping table optionally from the file from the config location?

[nufke]

Just check the latest development branch. When a new mapping table is received via MQTT, I store this mapping table in the plugin data directory. When you restart or update the plugin, it will read this mapping table from the data directory first. In this way, the mapping table is persistent.
Note that I do not consider publishing and/or changing the mapping table a default capability of the plugin, because it is too error prone and can result in very unexpected behavior. Therefore the setup and its use will not be accessible via plugin GUI and/or documentation.

[poklutom]

Cool, thanks a lot. I can see the file and can even update it directly. But it is a good feature to have the possibility to save custom mapping just by publishing the message. Well done!

Two minor details
1. when you publish mapping as a message, then only mapping is valid and the rest is sent as standard (lox/sr_nr/uu_id), when you restart the plugin, then plugin is reading the custom mapping from file (as expected) but the rest is sent as (lox/sr_nr/category/room/name) - it means different behaviour before and after first reset. But it is a really minor detail which I don't need to be fixed.


2. When the plugin is reinstalled then the custom mapping file is deleted. But again, nothing important for now.

You will see in the feature if more people will be interested in such custom mapping to do it more officially if needed.

But I am personally very happy and going to play&test the custom mapping. Thank you for all this effort and really fast implementation.

By the way if you are interesting - during the weekend I was testing your auto generated topics <category>/<room>/<control name>/<state>
and found following list of loxone blocks which was still sent as uu_id:

modes
globalStates
LightControllerV2 - states activeMoods
LightControllerV2 - states activeMoodsNum
LightControllerV2 - subcontrols states active
Jalousie - states position
messageCenter - states changed
Meter - status total
Irrigation - states (all)
Pushbutton - states active
Switch - states (all)
Radio - states activeOutput
Daytimes - states (all)


To be honest, not all I would use them but for example "Meter(Energy consumption ).Total consumption" was important for me. But this I have solved by custom mapping.

[nufke]

check the latest dev tree. Item 1 and 2 should be resolved.

I am not able to reproduce the issue with the missing states in the listed controls. At my end these controls and states are mapped. Perhaps there is a naming convention issue or incorrrect duplicate found, we need to check that.
I can confirm that globalStates and modes are not yet parsed, I will check this in the weekend.
Please enable debug of the plugin and see if it traverses over the controls and states.

Alternatively, send me your lox3app.json in a PM and I will study it further. Also send the log file if possible in a PM.