Announcement

Collapse
No announcement yet.

Scripting assistance - Parse/interpret MQTT data from temperature and humidity sensor

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

    Scripting assistance - Parse/interpret MQTT data from temperature and humidity sensor

    I have a few battery operated temperature and humidity sensors (DIGOO DG-R8S 433MHZ) returning raw data to MCSMQTT via a Sonoff 433Mhz bridge.
    Dev: |TASMOTA_RFBRIDGE|RESULT:RfRawata
    Sub: tele/RFBRIDGE1/RESULT:RfRawata
    Pub: the following Topic on Device command
    AA B1 04 02C6 0FD2 07D0 2314 010202010102010102010101020202020202020201010102010102010202 0202010201020203 55
    I found the logic behind it to decode/extract the temp/humidity values (https://github.com/Portisch/RF-Bridg...BB1/issues/119 - last post)
    but Iooking at a way to end up with usable temp/humidity values in MCSMQTT or native HS3 devices.

    Someone also came up with a script on a different platform to parse the data (https://github.com/dgomes/homeGW/blob/master/digoo.cpp)

    Code:
     [TABLE]
    [TR]
    [TD]*/[/TD]
     		[/TR]
    [TR]
    [TD] [/TD]
     			[TD]#include <digoo.h>[/TD]
     		[/TR]
    [TR]
    [TD] [/TD]
     			[TD] [/TD]
     		[/TR]
    [TR]
    [TD] [/TD]
     			[TD]digoo::digoo() {[/TD]
     		[/TR]
    [TR]
    [TD] [/TD]
     			[TD]packet_size = 37;[/TD]
     		[/TR]
    [TR]
    [TD] [/TD]
     			[TD]END_PACKET = 3000;[/TD]
     		[/TR]
    [TR]
    [TD] [/TD]
     			[TD]MIN_PACKET = 650;[/TD]
     		[/TR]
    [TR]
    [TD] [/TD]
     			[TD]}[/TD]
     		[/TR]
    [TR]
    [TD] [/TD]
     			[TD] [/TD]
     		[/TR]
    [TR]
    [TD] [/TD]
     			[TD] [/TD]
     		[/TR]
    [TR]
    [TD] [/TD]
     			[TD]uint8_t digoo::getId(uint64_t packet) {[/TD]
     		[/TR]
    [TR]
    [TD] [/TD]
     			[TD]uint8_t id = (packet >> 28) & 0xFF;[/TD]
     		[/TR]
    [TR]
    [TD] [/TD]
     			[TD]return id;[/TD]
     		[/TR]
    [TR]
    [TD] [/TD]
     			[TD]}[/TD]
     		[/TR]
    [TR]
    [TD] [/TD]
     			[TD] [/TD]
     		[/TR]
    [TR]
    [TD] [/TD]
     			[TD]uint8_t digoo::getBattery(uint64_t packet) {[/TD]
     		[/TR]
    [TR]
    [TD] [/TD]
     			[TD]uint8_t batt = (packet >> 24) & 0x8;[/TD]
     		[/TR]
    [TR]
    [TD] [/TD]
     			[TD]return batt ? 1 : 0;[/TD]
     		[/TR]
    [TR]
    [TD] [/TD]
     			[TD]}[/TD]
     		[/TR]
    [TR]
    [TD] [/TD]
     			[TD] [/TD]
     		[/TR]
    [TR]
    [TD] [/TD]
     			[TD]uint8_t digoo::getChannel(uint64_t packet) {[/TD]
     		[/TR]
    [TR]
    [TD] [/TD]
     			[TD]uint8_t channel = (packet >> 24) & 0x3;[/TD]
     		[/TR]
    [TR]
    [TD] [/TD]
     			[TD]return channel+1;[/TD]
     		[/TR]
    [TR]
    [TD] [/TD]
     			[TD]}[/TD]
     		[/TR]
    [TR]
    [TD] [/TD]
     			[TD] [/TD]
     		[/TR]
    [TR]
    [TD] [/TD]
     			[TD]float digoo::getTemperature(uint64_t packet) {[/TD]
     		[/TR]
    [TR]
    [TD] [/TD]
     			[TD]int16_t t = packet >> 12 & 0x0FFF;[/TD]
     		[/TR]
    [TR]
    [TD] [/TD]
     			[TD]t = 0x0800 & t ? 0xF000 | t : t;[/TD]
     		[/TR]
    [TR]
    [TD] [/TD]
     			[TD]float temperature = float(t) / 10;[/TD]
     		[/TR]
    [TR]
    [TD] [/TD]
     			[TD]return temperature;[/TD]
     		[/TR]
    [TR]
    [TD] [/TD]
     			[TD]}[/TD]
     		[/TR]
    [TR]
    [TD] [/TD]
     			[TD] [/TD]
     		[/TR]
    [TR]
    [TD] [/TD]
     			[TD]uint8_t digoo::getHumidity(uint64_t packet) {[/TD]
     		[/TR]
    [TR]
    [TD] [/TD]
     			[TD]uint8_t humidity = packet & 0xFF;[/TD]
     		[/TR]
    [TR]
    [TD] [/TD]
     			[TD]return humidity;[/TD]
     		[/TR]
    [TR]
    [TD] [/TD]
     			[TD]}[/TD]
     		[/TR]
    [TR]
    [TD] [/TD]
     			[TD] [/TD]
     		[/TR]
    [TR]
    [TD] [/TD]
     			[TD]uint8_t digoo::isValidWeather(uint64_t ppacket) {[/TD]
     		[/TR]
    [TR]
    [TD] [/TD]
     			[TD]uint8_t humidity = getHumidity(ppacket);[/TD]
     		[/TR]
    [TR]
    [TD] [/TD]
     			[TD]//Specs http://www.amazon.co.uk/gp/product/B00327G0MA/ref=oh_details_o00_s00_i00[/TD]
     		[/TR]
    [TR]
    [TD] [/TD]
     			[TD]if (humidity > 100) { //sanity check according to specs[/TD]
     		[/TR]
    [TR]
    [TD] [/TD]
     			[TD]return INVALID_HUMIDITY;[/TD]
     		[/TR]
    [TR]
    [TD] [/TD]
     			[TD]}[/TD]
     		[/TR]
    [TR]
    [TD] [/TD]
     			[TD]float temperature = getTemperature(ppacket);[/TD]
     		[/TR]
    [TR]
    [TD] [/TD]
     			[TD]if (temperature < -20.0 || temperature > 50.0) { //sanity check according to specs[/TD]
     		[/TR]
    [TR]
    [TD] [/TD]
     			[TD]return INVALID_TEMPERATURE;[/TD]
     		[/TR]
    [TR]
    [TD] [/TD]
     			[TD]}[/TD]
     		[/TR]
    [TR]
    [TD] [/TD]
     			[TD]return OK;[/TD]
     		[/TR]
    [TR]
    [TD] [/TD]
     			[TD]}[/TD]
     		[/TR]
    [TR]
    [TD] [/TD]
     			[TD] [/TD]
     		[/TR]
    [TR]
    [TD] [/TD]
     			[TD]void digoo::processPacket() {[/TD]
     		[/TR]
    [TR]
    [TD] [/TD]
     			[TD]packet = 0;[/TD]
     		[/TR]
    [TR]
    [TD] [/TD]
     			[TD]for(unsigned i=1; i< bitsRead; i++) {[/TD]
     		[/TR]
    [TR]
    [TD] [/TD]
     			[TD]unsigned duration = timings[i];[/TD]
     		[/TR]
    [TR]
    [TD] [/TD]
     			[TD]if(duration > digoo::ONE) {[/TD]
     		[/TR]
    [TR]
    [TD] [/TD]
     			[TD]packet = packet << 1;[/TD]
     		[/TR]
    [TR]
    [TD] [/TD]
     			[TD]bitSet(packet, 0);[/TD]
     		[/TR]
    [TR]
    [TD] [/TD]
     			[TD]} else if(duration > digoo::ZERO) {[/TD]
     		[/TR]
    [TR]
    [TD] [/TD]
     			[TD]packet = packet << 1;[/TD]
     		[/TR]
    [TR]
    [TD] [/TD]
     			[TD]bitClear(packet, 0);[/TD]
     		[/TR]
    [TR]
    [TD] [/TD]
     			[TD]}[/TD]
     		[/TR]
    [TR]
    [TD] [/TD]
     			[TD] [/TD]
     		[/TR]
    [TR]
    [TD] [/TD]
     			[TD]}[/TD]
     		[/TR]
    [TR]
    [TD] [/TD]
     			[TD]#ifdef DEBUG[/TD]
     		[/TR]
    [TR]
    [TD] [/TD]
     			[TD]Serial.print("~0x");[/TD]
     		[/TR]
    [TR]
    [TD] [/TD]
     			[TD]Serial.println((unsigned long) packet, HEX);[/TD]
     		[/TR]
    [TR]
    [TD] [/TD]
     			[TD]if (packet == 0) {[/TD]
     		[/TR]
    [TR]
    [TD] [/TD]
     			[TD]for(unsigned i=0; i < bitsRead; i++)[/TD]
     		[/TR]
    [TR]
    [TD] [/TD]
     			[TD]Serial.println(timings[i]);[/TD]
     		[/TR]
    [TR]
    [TD] [/TD]
     			[TD]}[/TD]
     		[/TR]
    [TR]
    [TD] [/TD]
     			[TD]#endif[/TD]
     		[/TR]
    [TR]
    [TD] [/TD]
     			[TD]}[/TD]
     		[/TR]
    [/TABLE]
    Based on feedback from Micheal (MCSMQTT), the best approach would be to register the topics with a script and mcsMQTT would call the script when one of those topics has been received.


    6.8 Scripting Callback
    If a user desires to handle the raw MQTT received payload in a script then a callback can be used to receive MQTT topics with their payload. This is setup with a call to the plug-in to register the script. The RegisterTopicReceivedScript callback expects an array of three parameters. The first is the filename that will be located in the HS scripts folder. The second is the name of the procedure in this file. The third is the Topic that will result in a callback. Multiple Topics can be requested with multiple calls to this function. MQTT Topic wildcards can also be used.

    hs.PluginFunction("mcsMQTT", "", "RegisterTopicReceivedScript",{"MyScriptFileName.vb", "MyFunctionName", "MyMQTTTopic"})

    As an example, the following two scripts are used in the file TestCallback.vb. The first (Main) was invoked by event with script action. The second (TheCallback) was invoked by mcsMQTT when it received the MQTT Topic “test/topic”.
    sub Main(parm as object)
    hs.WriteLog("TestCallback", "Registering Callback for Script")
    Dim callbackParameters = New String {"TestCallback", "TheCallback", "test/topic"}
    hs.PluginFunction(
    "mcsMQTT",
    "",
    "RegisterTopicReceivedScript",
    callbackParameters
    )
    End Sub
    sub TheCallback(parm as object)
    hs.Writelog("TestCallback", "Received Topic " & parm(0) & " with Payload " & parm(1))
    End Sub

    However, I don't have enough experience to translate the above script and have it interact with MCSMQTT;
    If anyone could volunteer some help to complete the script, I'll gladly do the testing.
    I guess it could help other users and/or provide basis for other 433Mhz raw data device integration.

    Thanks!

    see also: https://github.com/arendst/Tasmota/w...-RF-Bridge-433 "raw sniffing"

    #2
    Every manufacturer has a different encoding mechanism for their devices. This example appears to be somebody's temp/humidity sensor reporting. Bert with the RFXCOM plugin has many of these decoded. I have a subset decoded in the mcsXapW800 xAP node.

    What I see are two levels of issues with the decoding. The first is how to interpret the raw data into engineering units. The second is to know the device encoding (e.g. is it humidity and where is in the humidity value)

    The SDR receiver is a valuable resource for decoding known 433 MHz transmissions. https://www.rtl-sdr.com/tag/433-mhz/ and in particular RTL_433. This part tells you what you are receiving so you have a chance of developing the script for the set of devices that you have on your RF that you want into HS. Overall this is not trivial.

    Comment


      #3
      Thanks Michael, actually the decoding "recipe" has already been identified for this specific device:

      https://github.com/Portisch/RF-Bridg...BB1/issues/119 - see last post

      The format is constant and temp/humidity values can be extracted manually from the raw data. The objective is to "automate" the extraction.

      --------------------

      As a plan B, I'll purchase a second Sonoff bridge and hack it to use a different platform that seems to decode this natively. (VS Tasmota+Portisch)

      https://1technophile.blogspot.com/20...or-how-to.html
      https://github.com/xoseperez/espurna...---Direct-Hack

      Will report back later. Thx

      Comment


        #4
        So your only need is for decoding this one specific device?

        Comment


          #5
          Yes. All other devices I want to use are already tested and working pefectly with the SonoffRFbridge/Tasmota/Portisch combo:

          https://github.com/arendst/Tasmota/w...-RF-Bridge-433

          Easy hack. Works perfectly/tested for : Motion sensors, Door sensors, Water leak sensors, Smoke detectors. (All from Aliexpress, 5-10$ each)
          All of these devices typically send just a few simple codes (on, off, etc) so the MQTT/HS integration is quite easy.
          But for the Temp/Humidity sensor, the range of data is much larger and requires conversion/parsing of the raw data to extract Temp/Humidity values.

          I also deployed DS18x20 and Si7021 wired sensors with Sonoff Basic and TH10/16. great solution.
          But I like the idea of having a few wireless sensors I can move around as needed.

          I am currently flashing the stock OMG software to my Sonoff Bridge, will see how it behaves... https://docs.openmqttgateway.com/

          Worst case, I'll do the physical hack and use this solution instead which is documented as working:

          https://1technophile.blogspot.com/20...or-how-to.html
          https://github.com/xoseperez/espurna...---Direct-Hack

          hopefully it decodes both the simple and more advanced codes so I don't have to deploy 2 bridges (which wouldn't be a major problem either)

          Will report back later.

          Cheers,

          Comment


            #6
            I looked at one of your references and now understand the raw format of the 0, 1, 2 stream. The 0 is the separator. The 1 is binary 0. The 2 is binary 1. For humidity use the last 8 binary bits. For temperature throw away the rightmost 12 and then the new rightmost 12 is the temp scaled by 10. This specific decode is not hard in script and is just a variant of the one you posted.

            Comment


              #7
              Done!
              Hacking completed and OpenMQTTgateway uploaded successfully.
              Not an easy hack but works a charm.
              Since only one SonoffRF unit seems enough to cover a regular house it's worth the effort.
              Everything centralized & processed via the same unit.

              Receiving all MQTT codes correctly including the decoded/parsed Temperature and Humidity (and battery status) data from the Digoo DG-R8S.

              The only thing is that the unit sends two distinct MQTT messages simultaneously (ID 63 & ID 147.0) but only one is valid: ID63 "TFA protocol"
              How would you suggest I manage the data in order to create 2 separate HS devices for :

              ID63 "TFA protocol" : Temperature
              ID63 "TFA protocol" : Humidity

              since they share the same SUB?
              let me know if I should move this to your Forum/Thread.


              10 S 2020-01-21 12:51:46 home/OpenMQTTGateway_SRFB/PilighttoMQTT {"message":{"id":63,"temperature":24.30,"humidity":10.00,"ba ttery":1,"channel":1},"protocol":"tfa","length":"63","repeat s":2,"status":2}
              11 S 2020-01-21 12:51:46 home/OpenMQTTGateway_SRFB/PilighttoMQTT {"message":{"id":147.0,"temperature":1.5,"humidity":24.0,"ba ttery":1.0},"protocol":"teknihall","length":"147","repeats": 2,"status":2}
              12 S 2020-01-21 12:52:36 home/OpenMQTTGateway_SRFB/PilighttoMQTT {"message":{"id":63,"temperature":24.30,"humidity":10.00,"ba ttery":1,"channel":1},"protocol":"tfa","length":"63","repeat s":2,"status":2}
              13 S 2020-01-21 12:52:36 home/OpenMQTTGateway_SRFB/PilighttoMQTT {"message":{"id":147.0,"temperature":1.5,"humidity":24.0,"ba ttery":1.0},"protocol":"teknihall","length":"147","repeats": 2,"status":2}
              14 S 2020-01-21 12:53:26 home/OpenMQTTGateway_SRFB/PilighttoMQTT {"message":{"id":63,"temperature":24.30,"humidity":10.00,"ba ttery":1,"channel":1},"protocol":"tfa","length":"63","repeat s":2,"status":2}
              15 S 2020-01-21 12:53:26 home/OpenMQTTGateway_SRFB/PilighttoMQTT {"message":{"id":147.0,"temperature":1.5,"humidity":24.0,"ba ttery":1.0},"protocol":"teknihall","length":"147","repeats": 2,"status":2}
              16 S 2020-01-21 12:54:16 home/OpenMQTTGateway_SRFB/PilighttoMQTT {"message":{"id":63,"temperature":24.30,"humidity":10.00,"ba ttery":1,"channel":1},"protocol":"tfa","length":"63","repeat s":2,"status":2}
              17 S 2020-01-21 12:54:16 home/OpenMQTTGateway_SRFB/PilighttoMQTT {"message":{"id":147.0,"temperature":1.5,"humidity":24.0,"ba ttery":1.0},"protocol":"teknihall","length":"147","repeats": 2,"status":2}
              18 S 2020-01-21 12:55:06 home/OpenMQTTGateway_SRFB/PilighttoMQTT {"message":{"id":63,"temperature":24.30,"humidity":10.00,"ba ttery":1,"channel":1},"protocol":"tfa","length":"63","repeat s":2,"status":2}
              19 S 2020-01-21 12:55:06 home/OpenMQTTGateway_SRFB/PilighttoMQTT {"message":{"id":147.0,"temperature":1.5,"humidity":24.0,"ba ttery":1.0},"protocol":"teknihall","length":"147","repeats": 2,"status":2}

              Click image for larger version

Name:	Capture.PNG
Views:	796
Size:	30.3 KB
ID:	1355750

              Comment


                #8
                I did a decode of your packet and the values do not look reasonable based upon the algorithm I saw at https://github.com/Portisch/RF-Bridg...BB1/issues/119 -

                In any case the attached is the script with instructions for use at the top. It goes in the \scripts folder. I tested on Windows. The event I used to register the callback is
                Click image for larger version

Name:	Capture.PNG
Views:	715
Size:	22.9 KB
ID:	1355752

                Comment


                  #9
                  Thanks for looking into it Michael, much appreciated;
                  The algorithm was based on the DG-R8H, I assumed it would be similar but doesn's seem to be the case.
                  Could still be useful if someone doesn't want to hack the Sonoff Bridge.

                  The above (my previous post - after hacking device and getting decoded MQTT data) should be simpler to tackle I guess.
                  Can you provide your comments on the approach to to create separate HS devices for each device/sub ?

                  I did some further testing and highlighted below the data that is either unique to each device (message:unit, message:ID) or can help distinguish one from another (protocol, message:channel)

                  Thanks!

                  Click image for larger version  Name:	Capture.JPG Views:	0 Size:	76.9 KB ID:	1355765

                  Comment


                    #10
                    [qoute]Can you provide your comments on the approach to to create separate HS devices for each device/sub ?[/quote]
                    I am not understanding your question. Each line that you highlighted, as well the others, will go into separate HS devices.

                    Comment


                      #11
                      Here are some examples for more clarity:

                      The following 3 devices (Door sensor, motion sensor, water leak) share the same protocol but have a different "unit" code, however their "state" is published under the same topic.

                      Sub: home/OpenMQTTGateway_SRFB/PilighttoMQTT:message:state

                      Check time stamps for Topic changes.


                      So I would need to combine two topics to make it unique


                      Otherwise if I create a HS device for Sub:

                      home/OpenMQTTGateway_SRFB/PilighttoMQTT:message:state,

                      I will get status from all devices....


                      DOOR SENSOR "CLOSED"
                      Click image for larger version  Name:	door sensor closed.JPG Views:	0 Size:	68.5 KB ID:	1355797


                      DOOR SENSOR "OPENED"

                      Click image for larger version  Name:	door sensor opened.JPG Views:	0 Size:	76.9 KB ID:	1355798

                      MOTION SENSOR "ON"
                      Click image for larger version  Name:	Motion sensor on.JPG Views:	0 Size:	75.8 KB ID:	1355799

                      WATER LEAK SENSOR "ON"
                      Click image for larger version  Name:	Water leak detector.JPG Views:	0 Size:	74.2 KB ID:	1355800


                      For Temperature/Humidity, the "id" differentiates the correctly decoded signal (id 63, protocol tfa) vs the incorrect "noise" signal which is of no use (id 147.0, protocol teknihall)

                      Again here if I created a single device with Sub: home/OpenMQTTGateway_SRFB/PilighttoMQTT:message:temperature
                      Temperature value will cycle between "id 63" and "id 147"

                      Click image for larger version  Name:	Temperature-Humidity incorrect protocol.JPG Views:	0 Size:	73.9 KB ID:	1355801Click image for larger version  Name:	Temperature-Humidity correct protocol.JPG Views:	0 Size:	73.4 KB ID:	1355802

                      I need to have a way to "group" things so that it points to unique devices.

                      Let me know what you think,

                      Thx.,


                      Click image for larger version

Name:	image_85486.jpg
Views:	731
Size:	66.8 KB
ID:	1355803

                      Comment


                        #12
                        It is not an uncommon situation where the publisher has designed messages where part of the topic is put into the payload. In you examples where does the unit code show up? I’m guessing it is the id key. Shelly also uses id in its announce topic to identify which unit is making the announcement.

                        The problem dealing with this is not making the topic include the id so it becomes unique, but to know when id key is used for this purpose and then when other keys need to used in a similar way. Global settings are easy, topic-specific settings can only be established after a topic has been seen. Let me see what I can do to handle the general case.

                        Comment


                          #13
                          Depending on the devices, the unit code is either


                          message:unit (almost all devices - first three above + everything else I have tested so far)

                          or

                          message:id (Temp/humidity sensor - probably when it's a "raw" decoded signal)

                          Thx

                          Comment


                            #14
                            Would it be feasible to add a form of secondary topic "dependency" in the Edit page?

                            IE: Create the device as usual (ex: based on topic home/OpenMQTTGateway_SRFB/PilighttoMQTT:message:temperature)

                            and an optional "dependency" field in the edit page which would require to identify a secondary topic
                            Ex: home/OpenMQTTGateway_SRFB/PilighttoMQTT:message:id + associated "trigger" value (ex: 63)

                            Which would mean :

                            IF message:id = 63 THEN register temperature value to the associated HS device.
                            ELSE, ignore.

                            not sure if it's worth having more than one level of logic but I guess it should cover most cases.


                            This would be targeted at single messages containing multiple payloads. Like the Temp/humidity example
                            10 S 2020-01-21 12:51:46 home/OpenMQTTGateway_SRFB/PilighttoMQTT {"message":{"id":63,"temperature":24.30,"humidity":10.00, "ba ttery":1,"channel":1},"protocol":"tfa","length":"63","repeat s":2,"status":2}
                            However, you would need the ability to create multiple HS devices over the same topic.

                            There's surely other options, let me know what you think,

                            Thx

                            Comment


                              #15
                              What I am thinking about doing on the Edit tab is a checkbox to identify the item (e.g home/OpenMQTTGateway_SRFB/PilighttoMQTT:message:id) to be elevated as a topic component for all other members of the json group. Continuing your example further it would become

                              home/OpenMQTTGateway_SRFB/PilighttoMQTT:message:id (no change from is currently visible and remains as only a placeholder and cannot be associated with HS device)
                              home/OpenMQTTGateway_SRFB/PilighttoMQTT:message:63:temperature
                              home/OpenMQTTGateway_SRFB/PilighttoMQTT:message:63:humidity
                              :

                              home/OpenMQTTGateway_SRFB/PilighttoMQTT:message:75:temperature
                              etc

                              There may be implications that I don't remember right now, but seems like a reasonable approach. It is not something, however, that can be done in a couple hours.

                              Comment

                              Working...
                              X