Announcement

Collapse
No announcement yet.

Monitoring some power/gas/water meters via an RTL-SDR

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

    #16
    Originally posted by aldo View Post
    I'm impressed by your code, I'm not as near as you in regards of this. I have to do lot of homework to get close to your level. So far I did not use any MQTT in any of my projects, it is time to start.

    In the meantime, I have a small issue that I can not figure out. I was lucky to get a virtualbox machine working with the meter program. I tried again on a production machine and I really do not understand the verbiage of the author. What does he mean when he says "This will produce the binary*$GOPATH/bin/rtlamr. For convenience it's common to add*$GOPATH/bin*to the path." I'm having hard time understanding it, no matter what I do it does not let me run the installation program rtlamr.

    This is what I learned so far and fails on the last part.
    - Run Sudo apt install golang-go
    - Run Sudo apt instal git
    - Run sudo apt install rtl-sdr
    • Install Goland. When executing go get github.com/bemasher/rtlamr it will ask if you want to install the goLand. Install it if it has not been installed yet.
    • Install git. If this program is not installed, the program will tell you to install it.
    • Run this from the command prompt - go get github.com/bemasher/rtlamr. It installs the rtlamr utility.

    Thanks,
    Aldo
    I can't take full credit for the code; Kyle Gordon wrote the core of the mqtt, but I updated some of the paho mqtt calls for supporting mqtt username and password. At some point I'll probably enable tls certs so I can go encrypted...

    So here's what I did, set a GOPATH env variable to a base directory where you want it (either in that terminal session, or in /etc/profile.d, or /etc/profile, etc.) It's kind of annoying tbh, but looks like something GoLang may have standardized on. I set it to me home/USERNAME/src directory, and it nested everything under there, however you can put it into any directory your username has access too. Once you do that, when you do the go get command, it will put it in the $GOPATH/bin/rtlamr path. Once done, to make debugging easier, copy that rtlamr file to the same directory as these scripts and config file and you can run them all from there.

    I've run it from the HS3 box for testing, however long term will run it for a dedicated Pi on the network. For now it could be run via cron. It can also be run as a service if you're interested. I hadn't done this yet as it was logging some useless header info every time it ran, so figured at some point I'd clean it up rather than clog the log.

    If you want to use MQTT, then yes, you'll need an mqtt server running on that box or somewhere on the network. However if you end up using mysql commands, then it'll make the MQTT pieces OBE.

    Slight tangent: I switched to using MQTT as got tired of having lots of different protocols and standards for doing communication. I had 1 wire devices, an unsupported TED1000 power monitor, ESP8266, NodeMCU, etc. With MQTT it's a fairly simple and well documented messaging protocol, so lends it's self ideally to a common devices messaging ability. It's not perfect, but fit my needs quite happily

    Long term I'll probably get this up on to Github, here's the 1wire to MQTT forked version I did a year or so ago: https://github.com/mloebl/mqtt-owfs-temp

    Comment


      #17
      I try to write a program that would take the JSON file and move it into mysql. I would love a suggestion on how to figure out the kwh. Right now every time the consumption value change it would write to sql database but it will only give me the actual meter reading like 19253, then 19254 after few minutes and etc. etc. How should I get the kWH from it? I guess everytime it changes is 1 KWH consumed, what should I write in the database? Do you see what I'm trying to say.

      Aldo

      Comment


        #18
        Slight tangent: I switched to using MQTT as got tired of having lots of different protocols and standards for doing communication. I had 1 wire devices, an unsupported TED1000 power monitor, ESP8266, NodeMCU, etc. With MQTT it's a fairly simple and well documented messaging protocol, so lends it's self ideally to a common devices messaging ability. It's not perfect, but fit my needs quite happily
        What did you do for MQTT in/out of HS? For your 1-wire do you have more than temperature via MQTT?

        Comment


          #19
          Originally posted by Michael McSharry View Post
          What did you do for MQTT in/out of HS? For your 1-wire do you have more than temperature via MQTT?
          There is an MQTT plugin for HS3 that's in the updater; there are a couple threads on it here on the forum:
          https://forums.homeseer.com/showthread.php?t=180804
          https://forums.homeseer.com/showthread.php?t=185697

          I broke up my large 1wire network up into a couple of segments using either a BeagleBone or a Raspberry Pi. In a nutshell I then go 1wire devices->owfs->pythonscript as a service->mqtt.

          I have 1wire temperature, humidity, pressure and light sensors. The initial script on Github was temperature I think only with maybe humidity support? When I forked it, I added add'l devices that I had (basically anything owfs supports that I was using). In that python script, I ingest a csv of the different devices:
          https://github.com/mloebl/mqtt-owfs-...ces.csv.sample

          I then read that in the owsensortype variable where depending on the device type from that csv, I process it as needed.

          TBH I used 1wire and xAP for a very very long time and your plugins and apps worked great. However I wanted to reuse the existing CAT5e I had from the basement to the attic for 1wire as a dedicated ethernet/wifi IoT network. I had a hard time finding a simple light weight xAP server for Linux to load on a Pi.

          Another example is the TED1000 that's no longer supported by HS3. I took a script someone wrong for decoding TED1000 messages, and plumbed it thru the same base MQTT python script. https://github.com/mloebl/mqtt-ted1000 It has no problem updating device values every second very reliably with current power usage. Only issue now is I have a bug if the TED1000 loses power, it doesn't reconnect the USB device

          WeeWX I've been running for a long time with my Davis weather station, and it also natively supports MQTT messages.

          MQTT also lets you (reasonably) easily bridge MQTT servers together. I've been playing with LoRA, and using TTN (The Things Network). I can bridge my MQTT server to their MQTT services, and get my LoRA transmitted devices that way as well.

          I spent a couple hours monkeying with the JSON calls for HS3 until I changed my approach and tweaked my python MQTT script. Within less than an hour was pumping messages out to MQTT. As I mentioned above, I really want to learn Go so I can do it natively there, especially as not everything beacons every second.

          Comment


            #20
            Originally posted by aldo View Post
            I try to write a program that would take the JSON file and move it into mysql. I would love a suggestion on how to figure out the kwh. Right now every time the consumption value change it would write to sql database but it will only give me the actual meter reading like 19253, then 19254 after few minutes and etc. etc. How should I get the kWH from it? I guess everytime it changes is 1 KWH consumed, what should I write in the database? Do you see what I'm trying to say.

            Aldo
            So I haven't tackled the current KWH yet myself, but I think he did it here:
            https://blog.kroy.io/monitoring-home...-less-than-25/
            His scripts use a SQLite database, and I believe he's selecting the old values out of the database to calculate values based on current value.

            Code:
              private function getLastReading($table) {
                         $ld_res = $this->db->query("select * from {$table} order by id desc limit 1");
                         if ($ld_row = $ld_res->fetchArray()) {
                             return $this->reading-$ld_row['reading'];
                         }
                         return 0;
            
                     }
            I haven't time yet to sit down and tackle the calculations yet. I'd consider this a project I'm about 2/3 of the way thru, unfortunately. I'll keep posting here over the next few weeks as I get more time to thinker.

            Comment


              #21
              I have 1wire temperature, humidity, pressure and light sensors. The initial script on Github was temperature I think only with maybe humidity support? When I forked it, I added add'l devices that I had (basically anything owfs supports that I was using). In that python script, I ingest a csv of the different devices:
              https://github.com/mloebl/mqtt-owfs-...ces.csv.sample

              I then read that in the owsensortype variable where depending on the device type from that csv, I process it as needed.

              TBH I used 1wire and xAP for a very very long time and your plugins and apps worked great. However I wanted to reuse the existing CAT5e I had from the basement to the attic for 1wire as a dedicated ethernet/wifi IoT network. I had a hard time finding a simple light weight xAP server for Linux to load on a Pi.
              It is nice that somebody has gone beyond the basic temperature with small-device 1-wire. Pete asked a question about recommendations about doing something very similar to what you did. I did not have a good answer for him so hopefully he is looking at this thread.

              I also have a much 1-wire infrastructure that is being fed with old technology that just keeps working, but some day it will give out and the parts not replaceable so I am preparing for alternatives. I have a water meter counter that has been problematic. Each irrigation season I need to go figure out why it stopped working with the DS2423 1-wire. I'm now trying a Sonoff (<$5) configured as a counter and doing MQTT WiFi to replace the 1-wire at that location. Time will tell if the reliability will span multiple seasons. I would really like to interface 1-wire with the Sonoff. It's Tasmota software has DS18B20 library. I would like to extend it to cover the devices that I use. Likely will replace the 1-wire hubs with multiple Sonoff as the hubs are not replaceable. No urgency, but something to tinker with to be ready.

              If you used the HS xAP plugin then you will be familiar with mcsMQTT plugin.
              https://forums.homeseer.com/showthread.php?t=192675. I have also taken xapmcsDatabase that I use to maintain all my sensor history in a SQL Server database and added MQTT to it so it talks both xAP and MQTT. Simple way to migrate toward IOT incrementally and leverage my current technology.

              Comment


                #22
                Originally posted by Michael McSharry View Post
                If you used the HS xAP plugin then you will be familiar with mcsMQTT plugin.
                https://forums.homeseer.com/showthread.php?t=192675. I have also taken xapmcsDatabase that I use to maintain all my sensor history in a SQL Server database and added MQTT to it so it talks both xAP and MQTT. Simple way to migrate toward IOT incrementally and leverage my current technology.
                I had *not* seen this, will check this out, thank you!!!

                Comment


                  #23
                  Here doing a bit similar to Mike going baby steps...

                  Migrating Davis weather station over to the Meteostick. The console will continue to work but will not get serial data from it anymore...

                  Davis Console ==> buffered serial cable ==> RPi running Cumulus

                  Davis and other wireless weather stuff ==> Meteostick ==> WeeWx.

                  Over the last few months did this configuration for another Homeseer user and it's been working fine.

                  Also adding an RFXCom RX/TX box to the mix.

                  Cumulus never was configured to work with 1-Wire sensors and the author is retiring and WeeWx allows for more types of sensors including 1-Wire.

                  Over on the Domoticz forum there has been implementation using the SDR radio for similiar stuff that Mike is doing. Author there appears to have written the plugin there in LUA.

                  Here my 1-wire stuff was divided into multiple networks using Michael's xAP and multiple Temp0X devices and a couple of 9097's inside and outside. Still have two AAG original weather stations here from the late 1990's (plus Davis and a fine offset).

                  So here re purposed the 1-wire catXX cable which used to go down to the basement such that it goes to an RPi (ZNet like device) in the attic.

                  Recently have added the SDR radio to the RPi and it is connected to a small V-Dipole horizontally mounted for NOAA satellites. I do have another SDR radio...not sure if the RPi2 will work with two SDR radios.

                  I have Digitemp and OWFS running on the RPi (and MQTT from a long time ago)...just recently though drifted over to the NOAA satellite stuff.

                  Thank you Michael and Mike for all of this information!!
                  - Pete

                  Auto mator
                  Homeseer 3 Pro - 3.0.0.548 (Linux) - Ubuntu 18.04/W7e 64 bit Intel Haswell CPU 16Gb
                  Homeseer Zee2 (Lite) - 3.0.0.548 (Linux) - Ubuntu 18.04/W7e - CherryTrail x5-Z8350 BeeLink 4Gb BT3 Pro
                  HS4 Lite - Ubuntu 22.04 / Lenovo Tiny M900 / 32Gb Ram

                  HS4 Pro - V4.1.18.1 - Ubuntu 22.04 / Lenova Tiny M900 / 32Gb Ram
                  HSTouch on Intel tabletop tablets (Jogglers) - Asus AIO - Windows 11

                  X10, UPB, Zigbee, ZWave and Wifi MQTT automation-Tasmota-Espurna. OmniPro 2, Russound zoned audio, Alexa, Cheaper RFID, W800 and Home Assistant

                  Comment


                    #24
                    Kind of amazed this stuff is transmitted in the clear. Seems like someone could bypass their meter and transmit in lieu of the meter a much lower power consumption.
                    HS3 Pro Edition 3.0.0.435 (Windows Server 8.1 on ESXi box)

                    Plug-Ins Enabled:
                    Z-Wave:,RaspberryIO:,AirplaySpeak:,Ecobee:,
                    weatherXML:,JowiHue:,APCUPSD:,PHLocation:,Chromecast:,EasyTr igger:

                    Comment


                      #25
                      Typically by the time this stuff is approved it is already legacy.

                      IE: Smart Cards / Smart chip / encryption technologies were reverse engineered in the 1990's using software.

                      It is better though than a magnetic strip.
                      - Pete

                      Auto mator
                      Homeseer 3 Pro - 3.0.0.548 (Linux) - Ubuntu 18.04/W7e 64 bit Intel Haswell CPU 16Gb
                      Homeseer Zee2 (Lite) - 3.0.0.548 (Linux) - Ubuntu 18.04/W7e - CherryTrail x5-Z8350 BeeLink 4Gb BT3 Pro
                      HS4 Lite - Ubuntu 22.04 / Lenovo Tiny M900 / 32Gb Ram

                      HS4 Pro - V4.1.18.1 - Ubuntu 22.04 / Lenova Tiny M900 / 32Gb Ram
                      HSTouch on Intel tabletop tablets (Jogglers) - Asus AIO - Windows 11

                      X10, UPB, Zigbee, ZWave and Wifi MQTT automation-Tasmota-Espurna. OmniPro 2, Russound zoned audio, Alexa, Cheaper RFID, W800 and Home Assistant

                      Comment


                        #26
                        Some utilities are starting to move to cellular and other means that are a bit more "secure". Plus, at least for my power meter, it beacons almost every second. The power our's is putting out is enough that with an antenna in my office, and my neighbor's power meter a few hundred feet away comes in pretty clear (also helps 900mhz penetrates structures reasonably well). I think you'd have to over power all those, and that would look suspicious unless you reproduced your neighborhood. They would probably catch on, and with triangulation, would be pretty easy to find and get a knock on the door about I guess not impossible, but probably improbable (not disagreeing the whole thing is insecure.)

                        I'm kind of disappointed, I can see my neighbor's gas meter beacon, still yet to see mine. Also unfortunately been too busy to do any work on this. Hopefully in a few weeks I can pick this up again.

                        Comment


                          #27
                          Question, how does 1-wire work? Do you run wire (type? e.g. 22/4 work?) to every room in your house or only certain rooms? Does it end in an electrical box or low voltage box or somewhere else? I've started noticing about this reference (1-wire), but not sure what it really does or if needed generally...

                          Thanks, Dave

                          Comment


                            #28
                            Originally posted by NetworkGuy View Post
                            Question, how does 1-wire work? Do you run wire (type? e.g. 22/4 work?) to every room in your house or only certain rooms? Does it end in an electrical box or low voltage box or somewhere else? I've started noticing about this reference (1-wire), but not sure what it really does or if needed generally...

                            Thanks, Dave
                            Kind of opening a can of worms question

                            Here's the formal articles on 1-wire:
                            https://www.maximintegrated.com/en/a...ex.mvp/id/1796
                            https://www.maximintegrated.com/en/a...dex.mvp/id/148

                            It can be run over RJ11, but many people (like myself) use CAT5 or better. I ran normal CAT5e/CAT6 to my jacks in different rooms, punched them down as normal ethernet, but used the jack for 1wire. Many of the devices use either RJ11 or RJ45 connectors as well. For the most part, there are 2 wires used (though I have seem some proprietary variations where more than 2 are used.) The longer the wire, the more devices, etc it may require a power injector or it's unreliable. You can T tap off and add devices anywhere off the bus as well, and they even sell/sold hubs that combined power injecting and switching to enable more complex setups. Each device has it's own unique assigned address which is why this works.

                            In my opinion, it's kind of a dying technology (others may disagree here) as unless looking for a temperature sensor (Amazon/Ebay has them), I find it's getting harder (and more expensive) to get replacement devices for other sensor types, hubs and injectors. I am slowly phasing it out.

                            There are a few ways you can interface to it; USB, Serial interfaces. Also many devices like Raspberry Pis and Arduinos can directly connect, however, may run into distance limitations and need a script or some code to read:
                            https://pinout.xyz/pinout/1_wire
                            https://playground.arduino.cc/Learning/OneWire

                            Pete may have a better answer, but that's my .02

                            Comment


                              #29
                              Thanks mloebl, I may just ignore the tech, but will read up a bit around it.

                              Comment


                                #30
                                I agree with Mike as it is a dying technology today.

                                My term for using it is the "long math" way (serial).

                                Many Homeseer 1-2 users used 1-Wire technology back when.

                                Today it has gone a bit to a wireless Arduino / RPi environmental type devices.

                                Here recently built an RPi 1-wire hub connected to 2nd floor temperature and humidity sensors from the attic down reusing what I had in place. I have though been collecting these sensors since the 1990's and they never fail.

                                BTW the default drivers for 1-wire these days are part of the Linux Debian operating system and reintroduced to first RPi's running Linux Wheezy, then Jessie and today Stretch.
                                - Pete

                                Auto mator
                                Homeseer 3 Pro - 3.0.0.548 (Linux) - Ubuntu 18.04/W7e 64 bit Intel Haswell CPU 16Gb
                                Homeseer Zee2 (Lite) - 3.0.0.548 (Linux) - Ubuntu 18.04/W7e - CherryTrail x5-Z8350 BeeLink 4Gb BT3 Pro
                                HS4 Lite - Ubuntu 22.04 / Lenovo Tiny M900 / 32Gb Ram

                                HS4 Pro - V4.1.18.1 - Ubuntu 22.04 / Lenova Tiny M900 / 32Gb Ram
                                HSTouch on Intel tabletop tablets (Jogglers) - Asus AIO - Windows 11

                                X10, UPB, Zigbee, ZWave and Wifi MQTT automation-Tasmota-Espurna. OmniPro 2, Russound zoned audio, Alexa, Cheaper RFID, W800 and Home Assistant

                                Comment

                                Working...
                                X