Today's intelligent products that address the Internet Of Things (IOT) marketplace are usually based upon the Espressif ESP8266 or ESP8285 microcontrollers. These device come out of China at a very low price point that enable manufacturers to offer products at under $10 price points that have WiFi assess and computational power equivalent to that of what was needed for the first Moon Landing.
To exploit these the manufactures have generally followed the roll-out plan of developing a smartphone app that is able to communicate with these products. Rather than development of software themselves, these companies have most often elected to contract Tuya to provide the firmware and software App. This App can be found in the marketplace as either Tuya or SmartLife and variants such as JinVoo.
An exception is Alterco which is a Bulgaria company that has elected to keep the product software/firmware under their control. They sell a product under the Shelly brand. They have been recognized as being responsive to customer feedback. They release their software with a good cloud-oriented App, but also try to address the DIY market by including REST and MQTT protocols in their firmware to allow non-cloud use of their hardware.
An early entry in the marketplace is Sonoff that offers a wide variety of products that are generally based upon the Espressif micocontrollers. These device have been adopted by the DIY marketplace because of the ease at which custom firmware can be installed and the extremely low price point of under $5 for an intelligent device that has a power supply, relay, and external input capability.
A third party market place has developed to replace the manufactures's firmware so gain control of the hardware capability and avoid the cloud access that the manufactures found to be the easiest way to sell their products. Tasmota, Espurna, and ESPEasy are three major players in this alternate firmware marketplace. All are open source and vibrant user communities that are growing the capability well beyond what was originally released by the manufacturer.
For Tuya/SmartLife devices there are two options to avoid the cloud for their use. One is to change the firmware using software that runs on a RPi called Tuya Convert. The second is to reverse-engineer the Tuya protocol and then emulate it. In recent times Tuya used a protocol that was only marginally sensitive to security. This is identified as Version 3.1. In 2019 their devices from the factory was replaced with Version 3.3. Both of these protocols have been decoded by the user community. Alternates are available to use the hardware without the cloud using the 3rd party implementations of the Tuya emulations. Undoubtedly Tuya will again change the protocol based upon what has been exploited for new products.
Tuya and many other IOT device manufactures employ encryption on the communications with their devices. Generally the hurdle to locally communicate with these devices is to capture the encryption key so that the communication can be decrypted. Most common technique with this is to snoop on the communication of the device with the smartphone App.
Sonoff uses cloud software under the Ewelink name. They have also started to introduce factory firmware for DIY uses that provide a way around cloud access. The products with this DIY firmware are limited and the DIY capability is wanting. The future holds hope for a better implementation and a greater breadth of products that are oriented to the DIY marketplace.
Up to this point has been a discussion is the way that a company is able to sell their product in the marketplace. They need a piece of hardware and a smartphone App that can control it. Unfortunately what is best for the manufacturer is not what is best for the HA user.
The cloud will use a HTTP-based protocol where the assumption is that the device being controlled is totally available and a point to point communication channel will exist between the cloud server and the local device. During the setup process this is generally an acceptable assumption. If it does not work then try again. It is not a good assumption in the general case after then install is complete.
IOT devices are by their very nature may not be online and often will be be asleep when power by batteries. Wren an application uses HTTP, including the REST variant, is used to communicate with an IOT device and the IOT device does not happen to be online then the communication does not occur.
When the IOT device uses MQTT as the communication protocol then a Quality Of Service (QOS) is guaranteed so that when the IOT device does come back online it will receive the communication if the QOS is set to guarantee the communication. There are case, such as a periodic sensor update reporting where the QOS is not significant so a lost communication does not need to be guaranteed.
MQTT was initially defined by IBM before the plethora of IOT devices hit the marketplace. It is now a standard and has been updated just like the WiFi standards have been updated to address the expanding needs of the marketplace. Most implementations now use Version 3.1 of the standard while Version 5.0 has been release. Much like Wifi IP6 vs IP4 where IP4 is in common use but IP6 is available for extended needs.
What does this all mean to the typical HS user?
IOT devices from Amazon, Ebay, Aliexpress and other marketing sites are cheaply available to augment ones HA implementation. These are usually WiFi devices. The user can continue to use the sales-oriented capability provided by the manufacture. In most case this will be via a smartphone.
In the Shelly case the manufacture otters three options. One is like others where a cloud service coordinates the device. The second is with REST which is a command/response protocol variant of HTTP per HTML5 to bypass the cloud. The third is MQTT which is a protocol designed for IOT-type devices.
When REST is used then the client (HS Plugin) is responsible to make a request for status of the IOT device. This is done periodically at some polling rate that is a compromise between minimum latency and CPU/Network use. REST was designed to support Servers where availability is extremely high. An IOT device, by its very nature of a minimalist implementation for connection to a WiFi network, will at some time not be able to communicate. Either the communication is lost or the application/plugin will need to implement failure management logic to know what to do when the communication fails.
When MQTT is used the protocol guarantees the communication to the level of service that was requested. MQTT is the desired HA protocol vs. REST which is the desired protocol for server/cloud oriented communications.
For Shelly devices use of MQTT is a simple configuration setting. Selecting MQTT disconnects the cloud and makes all communications local. Shelly also makes it easy to install 3rd party firmware as they openly expose the pins to make a serial connection to install alternate firmware.
For Sonoff devices there is no provision provided by ITEAD (the Sonoff manufacturer) to use MQTT. The only option is to install 3rd part firmware. This can be done without opening the case using SonOTA.exe. It can also be done after opening the case and using a USB/Serial adapter connected to the exposed pin-out of the Sonoff device. There is much on the internet on changing the firmware of Sonoff devices. Sonoff is also trying to address the DIY marketplace with a new DIY product line. It is now lacking in both capability and breath and still does not yet include native MQTT capability.
For Tuly/SmartHome/JinVoo and others the only easy option to use MQTT is to use Tuya Convert to install 3rd party firmware. At this time there are two version of Tuya Convert that correspond to the two version of the Tuya firmware that have been released. For these devices is it also possible to use a local emulation of the Tuya protocol. Since the Tuya protocol has changed over time it should be expected that it will continue to change and firware update or new products will need new Tuya emulation plugins.
mcsMQTT is a HS3 plugin that provides the glue to interface HS with IOT devices that employ the MQTT protocol. The plan with HS4 to migrate this to mcsIOT and expand the protocol support beyond MQTT to include that which is in common use of IOT devices. Other MQTT plugins are also available for HS3.
For those that elect to not use MQTT protocol there are two other HS plugins available to interface IOT devices. These are:
For Tuya devices there is a HS plugin that emulates the Tuya protocol for a selective set of devices that have Tuya-based firmware installed.
For Shelly and Sonoff DIY there is a HS plugin that uses REST to communicate with the devices.
To exploit these the manufactures have generally followed the roll-out plan of developing a smartphone app that is able to communicate with these products. Rather than development of software themselves, these companies have most often elected to contract Tuya to provide the firmware and software App. This App can be found in the marketplace as either Tuya or SmartLife and variants such as JinVoo.
An exception is Alterco which is a Bulgaria company that has elected to keep the product software/firmware under their control. They sell a product under the Shelly brand. They have been recognized as being responsive to customer feedback. They release their software with a good cloud-oriented App, but also try to address the DIY market by including REST and MQTT protocols in their firmware to allow non-cloud use of their hardware.
An early entry in the marketplace is Sonoff that offers a wide variety of products that are generally based upon the Espressif micocontrollers. These device have been adopted by the DIY marketplace because of the ease at which custom firmware can be installed and the extremely low price point of under $5 for an intelligent device that has a power supply, relay, and external input capability.
A third party market place has developed to replace the manufactures's firmware so gain control of the hardware capability and avoid the cloud access that the manufactures found to be the easiest way to sell their products. Tasmota, Espurna, and ESPEasy are three major players in this alternate firmware marketplace. All are open source and vibrant user communities that are growing the capability well beyond what was originally released by the manufacturer.
For Tuya/SmartLife devices there are two options to avoid the cloud for their use. One is to change the firmware using software that runs on a RPi called Tuya Convert. The second is to reverse-engineer the Tuya protocol and then emulate it. In recent times Tuya used a protocol that was only marginally sensitive to security. This is identified as Version 3.1. In 2019 their devices from the factory was replaced with Version 3.3. Both of these protocols have been decoded by the user community. Alternates are available to use the hardware without the cloud using the 3rd party implementations of the Tuya emulations. Undoubtedly Tuya will again change the protocol based upon what has been exploited for new products.
Tuya and many other IOT device manufactures employ encryption on the communications with their devices. Generally the hurdle to locally communicate with these devices is to capture the encryption key so that the communication can be decrypted. Most common technique with this is to snoop on the communication of the device with the smartphone App.
Sonoff uses cloud software under the Ewelink name. They have also started to introduce factory firmware for DIY uses that provide a way around cloud access. The products with this DIY firmware are limited and the DIY capability is wanting. The future holds hope for a better implementation and a greater breadth of products that are oriented to the DIY marketplace.
Up to this point has been a discussion is the way that a company is able to sell their product in the marketplace. They need a piece of hardware and a smartphone App that can control it. Unfortunately what is best for the manufacturer is not what is best for the HA user.
The cloud will use a HTTP-based protocol where the assumption is that the device being controlled is totally available and a point to point communication channel will exist between the cloud server and the local device. During the setup process this is generally an acceptable assumption. If it does not work then try again. It is not a good assumption in the general case after then install is complete.
IOT devices are by their very nature may not be online and often will be be asleep when power by batteries. Wren an application uses HTTP, including the REST variant, is used to communicate with an IOT device and the IOT device does not happen to be online then the communication does not occur.
When the IOT device uses MQTT as the communication protocol then a Quality Of Service (QOS) is guaranteed so that when the IOT device does come back online it will receive the communication if the QOS is set to guarantee the communication. There are case, such as a periodic sensor update reporting where the QOS is not significant so a lost communication does not need to be guaranteed.
MQTT was initially defined by IBM before the plethora of IOT devices hit the marketplace. It is now a standard and has been updated just like the WiFi standards have been updated to address the expanding needs of the marketplace. Most implementations now use Version 3.1 of the standard while Version 5.0 has been release. Much like Wifi IP6 vs IP4 where IP4 is in common use but IP6 is available for extended needs.
What does this all mean to the typical HS user?
IOT devices from Amazon, Ebay, Aliexpress and other marketing sites are cheaply available to augment ones HA implementation. These are usually WiFi devices. The user can continue to use the sales-oriented capability provided by the manufacture. In most case this will be via a smartphone.
In the Shelly case the manufacture otters three options. One is like others where a cloud service coordinates the device. The second is with REST which is a command/response protocol variant of HTTP per HTML5 to bypass the cloud. The third is MQTT which is a protocol designed for IOT-type devices.
When REST is used then the client (HS Plugin) is responsible to make a request for status of the IOT device. This is done periodically at some polling rate that is a compromise between minimum latency and CPU/Network use. REST was designed to support Servers where availability is extremely high. An IOT device, by its very nature of a minimalist implementation for connection to a WiFi network, will at some time not be able to communicate. Either the communication is lost or the application/plugin will need to implement failure management logic to know what to do when the communication fails.
When MQTT is used the protocol guarantees the communication to the level of service that was requested. MQTT is the desired HA protocol vs. REST which is the desired protocol for server/cloud oriented communications.
For Shelly devices use of MQTT is a simple configuration setting. Selecting MQTT disconnects the cloud and makes all communications local. Shelly also makes it easy to install 3rd party firmware as they openly expose the pins to make a serial connection to install alternate firmware.
For Sonoff devices there is no provision provided by ITEAD (the Sonoff manufacturer) to use MQTT. The only option is to install 3rd part firmware. This can be done without opening the case using SonOTA.exe. It can also be done after opening the case and using a USB/Serial adapter connected to the exposed pin-out of the Sonoff device. There is much on the internet on changing the firmware of Sonoff devices. Sonoff is also trying to address the DIY marketplace with a new DIY product line. It is now lacking in both capability and breath and still does not yet include native MQTT capability.
For Tuly/SmartHome/JinVoo and others the only easy option to use MQTT is to use Tuya Convert to install 3rd party firmware. At this time there are two version of Tuya Convert that correspond to the two version of the Tuya firmware that have been released. For these devices is it also possible to use a local emulation of the Tuya protocol. Since the Tuya protocol has changed over time it should be expected that it will continue to change and firware update or new products will need new Tuya emulation plugins.
mcsMQTT is a HS3 plugin that provides the glue to interface HS with IOT devices that employ the MQTT protocol. The plan with HS4 to migrate this to mcsIOT and expand the protocol support beyond MQTT to include that which is in common use of IOT devices. Other MQTT plugins are also available for HS3.
For those that elect to not use MQTT protocol there are two other HS plugins available to interface IOT devices. These are:
For Tuya devices there is a HS plugin that emulates the Tuya protocol for a selective set of devices that have Tuya-based firmware installed.
For Shelly and Sonoff DIY there is a HS plugin that uses REST to communicate with the devices.
Comment