Announcement

Collapse
No announcement yet.

Interfacing IOT devices (Shelly, Tuya, SmartLife, Sonoff et.al)

Collapse
This is a sticky topic.
X
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

    Interfacing IOT devices (Shelly, Tuya, SmartLife, Sonoff et.al)

    Today's intelligent products that address the Internet Of Things (IOT) marketplace are usually based upon the Espressif ESP8266 or ESP8285 microcontrollers. These device come out of China at a very low price point that enable manufacturers to offer products at under $10 price points that have WiFi assess and computational power equivalent to that of what was needed for the first Moon Landing.

    To exploit these the manufactures have generally followed the roll-out plan of developing a smartphone app that is able to communicate with these products. Rather than development of software themselves, these companies have most often elected to contract Tuya to provide the firmware and software App. This App can be found in the marketplace as either Tuya or SmartLife and variants such as JinVoo.

    An exception is Alterco which is a Bulgaria company that has elected to keep the product software/firmware under their control. They sell a product under the Shelly brand. They have been recognized as being responsive to customer feedback. They release their software with a good cloud-oriented App, but also try to address the DIY market by including REST and MQTT protocols in their firmware to allow non-cloud use of their hardware.

    An early entry in the marketplace is Sonoff that offers a wide variety of products that are generally based upon the Espressif micocontrollers. These device have been adopted by the DIY marketplace because of the ease at which custom firmware can be installed and the extremely low price point of under $5 for an intelligent device that has a power supply, relay, and external input capability.

    A third party market place has developed to replace the manufactures's firmware so gain control of the hardware capability and avoid the cloud access that the manufactures found to be the easiest way to sell their products. Tasmota, Espurna, and ESPEasy are three major players in this alternate firmware marketplace. All are open source and vibrant user communities that are growing the capability well beyond what was originally released by the manufacturer.

    For Tuya/SmartLife devices there are two options to avoid the cloud for their use. One is to change the firmware using software that runs on a RPi called Tuya Convert. The second is to reverse-engineer the Tuya protocol and then emulate it. In recent times Tuya used a protocol that was only marginally sensitive to security. This is identified as Version 3.1. In 2019 their devices from the factory was replaced with Version 3.3. Both of these protocols have been decoded by the user community. Alternates are available to use the hardware without the cloud using the 3rd party implementations of the Tuya emulations. Undoubtedly Tuya will again change the protocol based upon what has been exploited for new products.

    Tuya and many other IOT device manufactures employ encryption on the communications with their devices. Generally the hurdle to locally communicate with these devices is to capture the encryption key so that the communication can be decrypted. Most common technique with this is to snoop on the communication of the device with the smartphone App.

    Sonoff uses cloud software under the Ewelink name. They have also started to introduce factory firmware for DIY uses that provide a way around cloud access. The products with this DIY firmware are limited and the DIY capability is wanting. The future holds hope for a better implementation and a greater breadth of products that are oriented to the DIY marketplace.



    Up to this point has been a discussion is the way that a company is able to sell their product in the marketplace. They need a piece of hardware and a smartphone App that can control it. Unfortunately what is best for the manufacturer is not what is best for the HA user.

    The cloud will use a HTTP-based protocol where the assumption is that the device being controlled is totally available and a point to point communication channel will exist between the cloud server and the local device. During the setup process this is generally an acceptable assumption. If it does not work then try again. It is not a good assumption in the general case after then install is complete.

    IOT devices are by their very nature may not be online and often will be be asleep when power by batteries. Wren an application uses HTTP, including the REST variant, is used to communicate with an IOT device and the IOT device does not happen to be online then the communication does not occur.

    When the IOT device uses MQTT as the communication protocol then a Quality Of Service (QOS) is guaranteed so that when the IOT device does come back online it will receive the communication if the QOS is set to guarantee the communication. There are case, such as a periodic sensor update reporting where the QOS is not significant so a lost communication does not need to be guaranteed.

    MQTT was initially defined by IBM before the plethora of IOT devices hit the marketplace. It is now a standard and has been updated just like the WiFi standards have been updated to address the expanding needs of the marketplace. Most implementations now use Version 3.1 of the standard while Version 5.0 has been release. Much like Wifi IP6 vs IP4 where IP4 is in common use but IP6 is available for extended needs.



    What does this all mean to the typical HS user?

    IOT devices from Amazon, Ebay, Aliexpress and other marketing sites are cheaply available to augment ones HA implementation. These are usually WiFi devices. The user can continue to use the sales-oriented capability provided by the manufacture. In most case this will be via a smartphone.

    In the Shelly case the manufacture otters three options. One is like others where a cloud service coordinates the device. The second is with REST which is a command/response protocol variant of HTTP per HTML5 to bypass the cloud. The third is MQTT which is a protocol designed for IOT-type devices.

    When REST is used then the client (HS Plugin) is responsible to make a request for status of the IOT device. This is done periodically at some polling rate that is a compromise between minimum latency and CPU/Network use. REST was designed to support Servers where availability is extremely high. An IOT device, by its very nature of a minimalist implementation for connection to a WiFi network, will at some time not be able to communicate. Either the communication is lost or the application/plugin will need to implement failure management logic to know what to do when the communication fails.

    When MQTT is used the protocol guarantees the communication to the level of service that was requested. MQTT is the desired HA protocol vs. REST which is the desired protocol for server/cloud oriented communications.

    For Shelly devices use of MQTT is a simple configuration setting. Selecting MQTT disconnects the cloud and makes all communications local. Shelly also makes it easy to install 3rd party firmware as they openly expose the pins to make a serial connection to install alternate firmware.

    For Sonoff devices there is no provision provided by ITEAD (the Sonoff manufacturer) to use MQTT. The only option is to install 3rd part firmware. This can be done without opening the case using SonOTA.exe. It can also be done after opening the case and using a USB/Serial adapter connected to the exposed pin-out of the Sonoff device. There is much on the internet on changing the firmware of Sonoff devices. Sonoff is also trying to address the DIY marketplace with a new DIY product line. It is now lacking in both capability and breath and still does not yet include native MQTT capability.

    For Tuly/SmartHome/JinVoo and others the only easy option to use MQTT is to use Tuya Convert to install 3rd party firmware. At this time there are two version of Tuya Convert that correspond to the two version of the Tuya firmware that have been released. For these devices is it also possible to use a local emulation of the Tuya protocol. Since the Tuya protocol has changed over time it should be expected that it will continue to change and firware update or new products will need new Tuya emulation plugins.

    mcsMQTT is a HS3 plugin that provides the glue to interface HS with IOT devices that employ the MQTT protocol. The plan with HS4 to migrate this to mcsIOT and expand the protocol support beyond MQTT to include that which is in common use of IOT devices. Other MQTT plugins are also available for HS3.

    For those that elect to not use MQTT protocol there are two other HS plugins available to interface IOT devices. These are:

    For Tuya devices there is a HS plugin that emulates the Tuya protocol for a selective set of devices that have Tuya-based firmware installed.

    For Shelly and Sonoff DIY there is a HS plugin that uses REST to communicate with the devices.

    #2
    Nice write-up Michael !! I try to use MQTT as much as possible since it is one the most flexible and capable protocols - especially when used with 3rd party firmware like Tasmota. mcsMQTT provides an awesome capability to integrate MQTT with HomeSeer. Just curious as to what the additional protocols are that mcsIOT will include?

    Comment


      #3
      My intention is that the plugin be user-need driven. An example is the 8 channel relay/input that interfaces using a custom TCP protocol. It was integrated into mcsMQTT in a manner that makes the device look to the user like a MQTT device.

      I am on the fence for Tuya/SmartLife devices. I don't especially like chasing proprietary software that needs reverse engineering. HS4 also will have what is likely a native cloud-based interface that will be good enough for most casual users. A released CPS plugin is also promised. I have prototype code that talks Tuya-speak, but I do not have an automatic way yet to get the encryption key.

      As products surface that are IP based then I am open to adding them. It really does not matter what the specific protocol may be. I'm leaning toward splitting out the BLE/Location tracking application and keep the plugin more product vs. application focused.

      Comment


        #4
        Loving this so much!!

        Comment


          #5
          I am finding it a challenge to select the right technology mix that will support a 'production' system -my home device control - whilst at the same time exploiting new technologies. I have to balance my interest (hobby I suppose) with maintaining something that is truly useful (not all Home Automation stuff is) and which my wife and family can reliably use whether I am there or not.
          Since I added a Node-red server (on a Raspberry pi) to my set up and added the mcsMQTT plugin to my HS I have been able to easily add a lot more functionality and at least for now I think I have a flexible and reasonably future proof set up. The number of modules now available in Node-Red are growing rapidly (in response to Michael's comments above I checked and see that N-R modules for Shelly and Tuya are already in place or in development).

          Comment


            #6
            Originally posted by SteveH View Post
            I am finding it a challenge to select the right technology mix that will support a 'production' system -my home device control - whilst at the same time exploiting new technologies. I have to balance my interest (hobby I suppose) with maintaining something that is truly useful (not all Home Automation stuff is) and which my wife and family can reliably use whether I am there or not.
            Since I added a Node-red server (on a Raspberry pi) to my set up and added the mcsMQTT plugin to my HS I have been able to easily add a lot more functionality and at least for now I think I have a flexible and reasonably future proof set up. The number of modules now available in Node-Red are growing rapidly (in response to Michael's comments above I checked and see that N-R modules for Shelly and Tuya are already in place or in development).
            For my interest.

            What are N-R modules?

            —-
            John

            Comment


              #7
              Thank you Michael for a nice overview!
              - Pete

              Auto mator
              Homeseer 3 Pro - 3.0.0.548 (Linux) - Ubuntu 18.04/W7e 64 bit Intel Haswell CPU 16Gb
              Homeseer Zee2 (Lite) - 3.0.0.548 (Linux) - Ubuntu 18.04/W7e - CherryTrail x5-Z8350 BeeLink 4Gb BT3 Pro
              HS4 Lite - Ubuntu 22.04 / Lenovo Tiny M900 / 32Gb Ram

              HS4 Pro - V4.1.18.1 - Ubuntu 22.04 / Lenova Tiny M900 / 32Gb Ram
              HSTouch on Intel tabletop tablets (Jogglers) - Asus AIO - Windows 11

              X10, UPB, Zigbee, ZWave and Wifi MQTT automation-Tasmota-Espurna. OmniPro 2, Russound zoned audio, Alexa, Cheaper RFID, W800 and Home Assistant

              Comment


                #8
                What are N-R modules?
                Node Red is a graphical means using a browser to interconnect widgets. It was originally an IBM development and now has much interest in the automation market. https://nodered.org/

                Think of it as a souped up event engine where boxes and lines are drawn rather than something like the arcane trigger/action text in the HS browser. It is kind of like a picture being worth a thousand words as it is very easy to visualize the logic connections and one can view the logic execution every step of the way. A N-R module is one of the widgets. It could be a vacuum, an Influx database, a light bulb, etc. Node Red makes combining all these things into a powerful and easy to understand system a reality for the average maker.

                Comment


                  #9
                  Really getting the feeling that MQTT could be a better platform than Z-Wave if you already have a mesh “all of home” WiFi network in place....
                  The end point device cost if MQTT devices are leagues cheaper - probably related to the z-wave licence and chip cost.
                  Great write up and tempting intro to the tech for ones that currently don’t use MQTT.
                  Thanks Michael


                  Sent from my iPhone using Tapatalk

                  Comment


                    #10
                    Great write up...much appreciated.

                    I've been in HA for some time now, but for the most part have elected to stay with zwave for my lighting, etc, albeit at a higher price.

                    my question is... if moving a significant amount of nodes to WiFi , how does that effect the latency of one's Wi-Fi network if at all?

                    Comment


                      #11
                      There should be no issue with latency for the general case as IOT devices usually have minimal bandwidth needs. The consideration with WiFi is the ability of the Router/AP to handle a large number of clients. Consumer grade routers are made to handle the typical household of a some phones, laptops and appliances. IOT was not on their radar screen. New WiFi standards are addressing the need for the expanding number of simultaneous WiFi connections. I have not experienced any issues with my single Ubiquiti LR AP with 30 or 40 WiFi devices.

                      Comment


                        #12
                        Here switched from Ubiquiti to Ruckus and have a dedicated SSID for the new WiFi devices. Put Alexa devices on another dedicated SSID.

                        I am also monitoring them now with Jon00's monitoring utility which works well. Have 9 devices up right now.

                        That said this is the first time I have gone to using Wireless devices for automation and typically have not utilize WLAN too much here.

                        Response times are excellent. I use wired to alarm panel sensors and compare them the the combo WiFi Tasmota GDO / Sensors set up here.

                        Click image for larger version  Name:	WiFi-iOT.jpg Views:	0 Size:	92.5 KB ID:	1350379
                        Click image for larger version  Name:	traffic-iOT.jpg Views:	0 Size:	30.4 KB ID:	1350380

                        Been also using Linux MQTT Explorer to look see at the traffic.

                        Click image for larger version  Name:	mqtt explorer.jpg Views:	0 Size:	83.4 KB ID:	1350381

                        Migrating my old TempXX (serial) stuff over to MQTT. Multiple DS18X wired temperature sensors to one SonOff basic device. Works fine here.

                        Click image for larger version

Name:	MQTT-1Wire.jpg
Views:	4964
Size:	117.7 KB
ID:	1350401

                        - Pete

                        Auto mator
                        Homeseer 3 Pro - 3.0.0.548 (Linux) - Ubuntu 18.04/W7e 64 bit Intel Haswell CPU 16Gb
                        Homeseer Zee2 (Lite) - 3.0.0.548 (Linux) - Ubuntu 18.04/W7e - CherryTrail x5-Z8350 BeeLink 4Gb BT3 Pro
                        HS4 Lite - Ubuntu 22.04 / Lenovo Tiny M900 / 32Gb Ram

                        HS4 Pro - V4.1.18.1 - Ubuntu 22.04 / Lenova Tiny M900 / 32Gb Ram
                        HSTouch on Intel tabletop tablets (Jogglers) - Asus AIO - Windows 11

                        X10, UPB, Zigbee, ZWave and Wifi MQTT automation-Tasmota-Espurna. OmniPro 2, Russound zoned audio, Alexa, Cheaper RFID, W800 and Home Assistant

                        Comment


                          #13
                          Hi Pete, I think I remember some post where you were describing how to daisy chain temperature sensors with the Sonoff Basic ("Multiple DS18X wired temperature sensors to one SonOff basic device"), would you mind pointing out to the right post and/or confirm setup details?
                          I'd like to reproduce the same thing here.

                          Thanks!

                          Comment


                            #14
                            I would like to configure my Sonoff TH to output the temperature in Celsius instead of Fahrenheit. I have the Sonoff flashed with Tasmota software and I have tried to reconfigure the Domoticz parameters but nothing seems to work.
                            HomeSeer Version: HS3 Pro Edition 3.0.0.368, Operating System: Microsoft Windows 10 - Home, Number of Devices: 373, Number of Events: 666, Enabled Plug-Ins
                            2.0.83.0: BLRF, 2.0.10.0: BLUSBUIRT, 3.0.0.75: HSTouch Server, 3.0.0.58: mcsXap, 3.0.0.11: NetCAM, 3.0.0.36: X10, 3.0.1.25: Z-Wave,Alexa,HomeKit

                            Comment


                              #15
                              Take a look at SetOption8 https://github.com/arendst/Tasmota/w...nds#SetOption8
                              You could also do it in mcsMQTT from Edit tab/page with numeric expression of $$PAYLOAD:*1.8+32

                              Comment

                              Working...
                              X