www.homeseer.com    
 

Go Back   HomeSeer Message Board > HomeSeer Products & Services > HomeSeer General Discussion Area

HomeSeer General Discussion Area General discussion about HomeSeer that does not fall into any other category or are not specific to 1.x or 2.x versions of HomeSeer.

Reply
 
Thread Tools Display Modes
  #1  
Old December 30th, 2017, 04:02 PM
mloebl's Avatar
mloebl mloebl is offline
Seer Master
 
Join Date: Apr 2002
Location: Northern, MA
Posts: 767
Monitoring some power/gas/water meters via an RTL-SDR

I stumbled on to this today:
https://blog.kroy.io/monitoring-home...-less-than-25/

Looks to be written in Go, and supports native CSV and JSON outputting. Took me literally 5mins on a Linux box to get running with an RTL-SDR I had sitting around. Not quite as detailed as I'd want with my two meters (just basic SCM with consumption), but pretty interesting to see the usage stats. My gas meter I haven't figured out yet (pretty certain it's mine as it's the only one it's seeing), as the dial doesn't line up to the numbers I'm seeing unlike the electric meter. Meter is in CCF, but the value I'm seeing doesn't seem to be CCF, Therms or kwh.
Reply With Quote
  #2  
Old December 30th, 2017, 04:12 PM
mrhappy mrhappy is offline
OverSeer
 
Join Date: Nov 2007
Location: W.Mids, UK
Posts: 7,120
This is exciting, I have a RTL-SDR module that is surplus to use and a radio water meter (I have no idea where it is, long story but it is somewhere buried in the road) that is read remotely. I'm going to give it a go thanks for the link.
Reply With Quote
  #3  
Old December 30th, 2017, 07:39 PM
mloebl's Avatar
mloebl mloebl is offline
Seer Master
 
Join Date: Apr 2002
Location: Northern, MA
Posts: 767
Quote:
Originally Posted by mrhappy View Post
This is exciting, I have a RTL-SDR module that is surplus to use and a radio water meter (I have no idea where it is, long story but it is somewhere buried in the road) that is read remotely. I'm going to give it a go thanks for the link.
I'm not sure how it works in the UK with non-900mhz devices, but worth a shot.

I find it interesting as run it this am, was seeing the gas meter. Now this evening only seeing the electricity meter.
Reply With Quote
  #4  
Old December 31st, 2017, 01:06 PM
aldo's Avatar
aldo aldo is offline
Super Seer
 
Join Date: Feb 2002
Location: Massachusetts
Posts: 1,560
There are several models at Amazon, could you please send me a link of the one that you have?

Thanks,
Aldo

Quote:
Originally Posted by mloebl View Post
I stumbled on to this today:
https://blog.kroy.io/monitoring-home...-less-than-25/

Looks to be written in Go, and supports native CSV and JSON outputting. Took me literally 5mins on a Linux box to get running with an RTL-SDR I had sitting around. Not quite as detailed as I'd want with my two meters (just basic SCM with consumption), but pretty interesting to see the usage stats. My gas meter I haven't figured out yet (pretty certain it's mine as it's the only one it's seeing), as the dial doesn't line up to the numbers I'm seeing unlike the electric meter. Meter is in CCF, but the value I'm seeing doesn't seem to be CCF, Therms or kwh.
Reply With Quote
  #5  
Old December 31st, 2017, 01:42 PM
Pete's Avatar
Pete Pete is offline
OverSeer
 
Join Date: Jan 2001
Location: House
Posts: 14,479
There are automation plugins today using RTL-SDR for the Domoticz open source automation program.

Here mini sizing my NOAA satellite weather maps downloading to an RPI2 / antenna that today is doing GPIO ZWave, 1-wire stuff and running Domoticz on the side. For text data and jpg images do a share drive to the data directory on HS3.

Many or most of the plugins are written in LUA. Very tight and easy peasy stuff that runs on OpenWRT OS...well like the Almond + or Vera or Wheezy or Stretch (linux in general).

Here only the water meter uses a tiny transmitter. Updates now to it allow for remote views from the entrance of the subdivision.

Just received this gift to tinker with from Ebay...(had one SDR radio from a few years back here). Note that you can get the USB radios for around $10 these days.

It's the little SMA connectors that you want with antennas. Earlier testing all of my test SMA connectors purchased cost more than the radio.

TCXO SMA Software Defined Radio Dongle With 2x Telescopic Antennas

This "kit" with antennas, sma connectors and radio was around $28 USD. I got mine from China in less than a week.

Lately Amazon Prime 2 day for this kind of stuff doesn't work for me and have noticed that the cost of the devices are more.

Name:  rtl.jpg
Views: 111
Size:  29.0 KB
__________________
- Pete

Automator


Hardware | HS3 Pro - Haswell iSeries 3 - 16Gb | HS3 Lite Pine64 2Gb computers

HS3 Pro & Lite
Edition Beta 3.0.0.398 | Ubuntu 16.04 64 bit | Oracle Windows Virtual Box ==> for Wintel only SAPI and HS3 plugins

HS3 Plugins - a bunch
Light switches - X10,UPB, ZWave and Zigbee
Speech - Microsoft SAPI - Amazon Echo
Security - Leviton Omni Pro 2
Weather - Davis Vantage Vue - MeteoStick
Firewall - PFSense
CCTV - Zoneminder



Last edited by Pete; December 31st, 2017 at 02:12 PM.
Reply With Quote
  #6  
Old December 31st, 2017, 01:48 PM
mloebl's Avatar
mloebl mloebl is offline
Seer Master
 
Join Date: Apr 2002
Location: Northern, MA
Posts: 767
Quote:
Originally Posted by aldo View Post
There are several models at Amazon, could you please send me a link of the one that you have?

Thanks,
Aldo
I got mine as part of a kit awhile back when I was experimenting with Outernet (the satellite service, not sure why as kinda useless for me ) It's essentially this:
https://www.amazon.com/NooElec-NESDR.../dp/B06Y1HKLHY

But in theory any of the RTL SDR sticks that do 900mhz should work.
Reply With Quote
  #7  
Old December 31st, 2017, 02:21 PM
aldo's Avatar
aldo aldo is offline
Super Seer
 
Join Date: Feb 2002
Location: Massachusetts
Posts: 1,560
Thank you

Quote:
Originally Posted by mloebl View Post
I got mine as part of a kit awhile back when I was experimenting with Outernet (the satellite service, not sure why as kinda useless for me ) It's essentially this:
https://www.amazon.com/NooElec-NESDR.../dp/B06Y1HKLHY

But in theory any of the RTL SDR sticks that do 900mhz should work.
Reply With Quote
  #8  
Old January 12th, 2018, 03:15 PM
aldo's Avatar
aldo aldo is offline
Super Seer
 
Join Date: Feb 2002
Location: Massachusetts
Posts: 1,560
All, thank you for this post. I was able to read my meter. My next step would be to parse the json file in mysql. Did anyone do it already? If not, could you put me in the right direction?

Thanks,
Aldo
Reply With Quote
  #9  
Old January 12th, 2018, 03:30 PM
mloebl's Avatar
mloebl mloebl is offline
Seer Master
 
Join Date: Apr 2002
Location: Northern, MA
Posts: 767
So here's the flow I did... I started by trying to do it via a JSON update to HS3, but was having issues posting the update of just the status as wanted a status only device. I ended up writing a python script that runs the rtlamr utility once, waits for the ID(s) to come thru that you specify, decodes the JSON, then posts it to MQTT so I can get it from there. This is very low CPU usage as not running all the time.

The python script could be easily adapted to change the mqtt submits to whatever people want to do.

That being said, if I have some time, I'd love to learn GoLang a bit to try to add an MQTT module to the existing tool, so no script is necessary, it just posts constantly from a Pi dedicated to this task.

I'll try to clean up the script tonight if I have some time and will post it here. It's enough for sure to get someone going. FWIW I went this route as didn't want to run it from my HS box as it's an arm board, so wanted to take the CPU load off by running it elsewhere off the network.
Reply With Quote
  #10  
Old January 12th, 2018, 04:21 PM
aldo's Avatar
aldo aldo is offline
Super Seer
 
Join Date: Feb 2002
Location: Massachusetts
Posts: 1,560
I'm enjoying this, you guys are good!!!! I like your idea, I was also thinking to get these values in mysql database, then from there the sky has no limit. I could use Jon00 or grafana to display those values. Unfortunately I'm really bad with scripts and json. I will look forward to your script.

Thanks,
Aldo
Reply With Quote
  #11  
Old January 12th, 2018, 04:52 PM
mloebl's Avatar
mloebl mloebl is offline
Seer Master
 
Join Date: Apr 2002
Location: Northern, MA
Posts: 767
Quote:
Originally Posted by aldo View Post
I'm enjoying this, you guys are good!!!! I like your idea, I was also thinking to get these values in mysql database, then from there the sky has no limit. I could use Jon00 or grafana to display those values. Unfortunately I'm really bad with scripts and json. I will look forward to your script.

Thanks,
Aldo
No worrries, I've got a busy weekend, but will see what I can do I've *really* wanted to get started with grafana myself, but had issues awhile ago getting it going.

In the mean time, not sure if you saw this from the rtlamr guy:
https://github.com/bemasher/rtlamr-collect
Reply With Quote
  #12  
Old January 12th, 2018, 04:59 PM
aldo's Avatar
aldo aldo is offline
Super Seer
 
Join Date: Feb 2002
Location: Massachusetts
Posts: 1,560
Thanks that will get me started. I really like Grafana especially now that they have integrated MySQL.
Reply With Quote
  #13  
Old January 12th, 2018, 07:43 PM
mloebl's Avatar
mloebl mloebl is offline
Seer Master
 
Join Date: Apr 2002
Location: Northern, MA
Posts: 767
I forked this off an existng script I use to push TED1000 or 1wire messages to MQTT via a service, so it's not perfect, but worked enough for my testing. I take no responsibility for the damage it may cause The script itself is based on an original MQTT publishing script I found here on github, https://github.com/kylegordon/mqtt-owfs-temp.

Attached is the python script, and a sample config file. The python script reads that config sample config file where you can set path values, credentials, etc. For my testing I had rtlamr, the log, and config file all in the same directory. If you want to use this script, rename the SAMPLE-mqtt-rtlamr.cfg to mqtt-rtlamr.cfg and set values accordingly. It's not perfect, but something I threw together in an evening to test how it may work. It's definitely a good starting point for someone looking to use a python script to process the data.

SAMPLE-mqtt-rtlamr.cfg
Code:
[global]
DEBUG = False
LOGFILE = mqtt-rtlamr.log
RTLAMR_PATH =./rtlamr
MQTT_HOST = CHANGEME-MQTTHOST
MQTT_PORT = CHANGEME-MQTTPORT
MQTT_TOPIC = /#
MQTT_SUBTOPIC = /rtlamr/
MQTT_USERNAME = CHANGEME-MQTTUSERNAME
MQTT_PASSWORD = CHANGEME-MQTTPASSWORD
POLLINTERVAL = 1
METRICUNITS = 0
UNITID = CHANGEME-UNITID
MSGTYPE = scm
Code:
#!/usr/bin/env python
# -*- coding: iso-8859-1 -*-

# Script based on mqtt-owfs-temp written by Kyle Gordon and converted for use with script
# Source: https://github.com/kylegordon/mqtt-owfs-temp

import os
import logging
import signal
import socket
import time
import sys
import ConfigParser
import json

import subprocess
import paho.mqtt.client as paho

import setproctitle
# from datetime import datetime, timedelta

# Read the config file
config = ConfigParser.RawConfigParser()
config.read("mqtt-rtlamr.cfg")

# Use ConfigParser to pick out the settings
DEBUG = config.getboolean("global", "debug")
LOGFILE = config.get("global", "logfile")
MQTT_HOST = config.get("global", "mqtt_host")
MQTT_PORT = config.getint("global", "mqtt_port")
MQTT_SUBTOPIC = config.get("global", "MQTT_SUBTOPIC")
MQTT_TOPIC = "/raw/" + socket.getfqdn() + MQTT_SUBTOPIC
MQTT_USERNAME = config.get("global", "MQTT_USERNAME")
MQTT_PASSWORD = config.get("global", "MQTT_PASSWORD")
METRICUNITS = config.get("global", "METRICUNITS")

POLLINTERVAL = config.getint("global", "pollinterval")

RTLAMR_PATH = config.get("global", "RTLAMR_PATH")
UNITID = config.get("global", "UNITID")
MSGTYPE = config.get("global", "MSGTYPE")


APPNAME = "rtlamr"

PRESENCETOPIC = "clients/" + socket.getfqdn() + "/" + APPNAME + "/state"
setproctitle.setproctitle(APPNAME)
client_id = APPNAME + "_%d" % os.getpid()

mqttc = paho.Client()

LOGFORMAT = '%(asctime)-15s %(message)s'

if DEBUG:
    logging.basicConfig(filename=LOGFILE,
                        level=logging.DEBUG,
                        format=LOGFORMAT)
else:
    logging.basicConfig(filename=LOGFILE,
                        level=logging.INFO,
                        format=LOGFORMAT)

logging.info("Starting " + APPNAME)
logging.info("INFO MODE")
logging.debug("DEBUG MODE")

# All the MQTT callbacks start here


def on_publish(mosq, obj, mid):
    """
    What to do when a message is published
    """
    logging.debug("MID " + str(mid) + " published.")


def on_subscribe(mosq, obj, mid, qos_list):
    """
    What to do in the event of subscribing to a topic"
    """
    logging.debug("Subscribe with mid " + str(mid) + " received.")


def on_unsubscribe(mosq, obj, mid):
    """
    What to do in the event of unsubscribing from a topic
    """
    logging.debug("Unsubscribe with mid " + str(mid) + " received.")


def on_connect(mosq, obj, result_code):
    """
    Handle connections (or failures) to the broker.
    This is called after the client has received a CONNACK message
    from the broker in response to calling connect().
    The parameter rc is an integer giving the return code:
    0: Success
    1: Refused – unacceptable protocol version
    2: Refused – identifier rejected
    3: Refused – server unavailable
    4: Refused – bad user name or password (MQTT v3.1 broker only)
    5: Refused – not authorised (MQTT v3.1 broker only)
    """
    logging.debug("on_connect RC: " + str(result_code))
    if result_code == 0:
        logging.info("Connected to %s:%s", MQTT_HOST, MQTT_PORT)
        # Publish retained LWT as per
        # http://stackoverflow.com/q/97694
        # See also the will_set function in connect() below
        mqttc.publish(PRESENCETOPIC, "1", retain=True)
        process_connection()
    elif result_code == 1:
        logging.info("Connection refused - unacceptable protocol version")
        cleanup()
    elif result_code == 2:
        logging.info("Connection refused - identifier rejected")
        cleanup()
    elif result_code == 3:
        logging.info("Connection refused - server unavailable")
        logging.info("Retrying in 30 seconds")
        time.sleep(30)
    elif result_code == 4:
        logging.info("Connection refused - bad user name or password")
        cleanup()
    elif result_code == 5:
        logging.info("Connection refused - not authorised")
        cleanup()
    else:
        logging.warning("Something went wrong. RC:" + str(result_code))
        cleanup()


def on_disconnect(mosq, obj, result_code):
    """
    Handle disconnections from the broker
    """
    if result_code == 0:
        logging.info("Clean disconnection")
    else:
        logging.info("Unexpected disconnection! Reconnecting in 5 seconds")
        logging.debug("Result code: %s", result_code)
        time.sleep(5)


def on_message(mosq, obj, msg):
    """
    What to do when the client recieves a message from the broker
    """
    logging.debug("Received: " + msg.payload +
                  " received on topic " + msg.topic +
                  " with QoS " + str(msg.qos))
    process_message(msg)


def on_log(mosq, obj, level, string):
    """
    What to do with debug log output from the MQTT library
    """
    logging.debug(string)

# End of MQTT callbacks


def cleanup(signum, frame):
    """
    Signal handler to ensure we disconnect cleanly
    in the event of a SIGTERM or SIGINT.
    """
    logging.info("Disconnecting from broker")
    # Publish a retained message to state that this client is offline
    mqttc.publish(PRESENCETOPIC, "0", retain=True)
    mqttc.disconnect()
    mqttc.loop_stop()
    logging.info("Exiting on signal %d", signum)
    sys.exit(signum)

def connect():
    """
    Connect to the broker, define the callbacks, and subscribe
    This will also set the Last Will and Testament (LWT)
    The LWT will be published in the event of an unclean or
    unexpected disconnection.
    """
    logging.info("Connecting to %s:%s", MQTT_HOST, MQTT_PORT)

    if MQTT_USERNAME:
        logging.info("Found username %s", MQTT_USERNAME)
        mqttc.username_pw_set(MQTT_USERNAME, MQTT_PASSWORD)

    # Set the Last Will and Testament (LWT) *before* connecting
    mqttc.will_set(PRESENCETOPIC, "0", qos=0, retain=True)
    result = mqttc.connect(MQTT_HOST, MQTT_PORT, 60)
    if result != 0:
        logging.info("Connection failed with error code %s. Retrying", result)
        
        time.sleep(10)
        connect()

    # Define the callbacks
    mqttc.on_connect = on_connect
    mqttc.on_disconnect = on_disconnect
    mqttc.on_publish = on_publish
    mqttc.on_subscribe = on_subscribe
    mqttc.on_unsubscribe = on_unsubscribe
    mqttc.on_message = on_message
    if DEBUG:
        mqttc.on_log = on_log

    mqttc.loop_start()

def process_connection():
    """
    What to do when a new connection is established
    """
    logging.debug("Processing connection")


def process_message(mosq, obj, msg):
    """
    What to do with the message that's arrived
    """
    logging.debug("Received: %s", msg.topic)


def find_in_sublists(lst, value):
    for sub_i, sublist in enumerate(lst):
        try:
            return (sub_i, sublist.index(value))
        except ValueError:
            pass

    raise ValueError("%s is not in lists" % value)

def slicer(my_str,sub):
    index=my_str.find(sub)
    if index !=-1 :
        return my_str[index:] 
    else :
        raise Exception('Sub string not found!')

def decodeJSON(jsonString):
    logging.debug('Processing JSON: ', jsonString)
    return(json.loads(jsonString))

def getjsonvalue(jsonData, jsonmessage):
    return(jsonData["Message"][jsonmessage])

def getMeterType(metertype):
    meter = "unknown"
    if (metertype == 4) or (metertype == 5) or (metertype == 7) or (metertype == 8):  
        meter = "electric"
    if (metertype == 2) or (metertype == 9) or (metertype == 12): 
        meter = "gas"
    if (metertype == 11) or (metertype == 13):
        meter = "water"
    return meter


def main_loop():
    """
    The main loop in which we stay connected to the broker
    """
    

    while True:
        rawjson = subprocess.check_output([RTLAMR_PATH, '-filterid='+UNITID, '-msgtype='+MSGTYPE,'--unique=true' ,'--format=json','--single=true'])
        logging.debug(slicer(rawjson, "{"))
        jsondata = decodeJSON(rawjson)

        json_time=jsondata["Time"]
        json_id = getjsonvalue(jsondata, "ID")
        json_type = getjsonvalue(jsondata, "Type")
        json_tamper = getjsonvalue(jsondata, "TamperPhy")
        json_consumption = getjsonvalue(jsondata, "Consumption")

        logging.debug("Current consumption: %.2f kw/h" % (json_consumption / 100))

        string_consumption = "%0.2f" % (float(json_consumption) / 100)  

        mqttc.publish(MQTT_TOPIC + str(json_id) + "/time", json_time)
        mqttc.publish(MQTT_TOPIC + str(json_id) + "/unitid", json_id)
        mqttc.publish(MQTT_TOPIC + str(json_id) + "/msgtype", MSGTYPE)
        mqttc.publish(MQTT_TOPIC + str(json_id) + "/metertype", getMeterType(json_type))
        mqttc.publish(MQTT_TOPIC + str(json_id) + "/consumption_raw", json_consumption)
        mqttc.publish(MQTT_TOPIC + str(json_id) + "/consumption_kwh", string_consumption)
        mqttc.publish(MQTT_TOPIC + str(json_id) + "/tamperphy", json_tamper)
        
        time.sleep(POLLINTERVAL*60)

# Use the signal module to handle signals
signal.signal(signal.SIGTERM, cleanup)
signal.signal(signal.SIGINT, cleanup)

# Connect to the broker and enter the main loop
connect()

# Try to start the main loop
try:
    main_loop()
except KeyboardInterrupt:
    logging.info("Interrupted by keypress")
    sys.exit(0)

Last edited by mloebl; January 12th, 2018 at 08:01 PM. Reason: Forum decided not to attach my scripts...
Reply With Quote
  #14  
Old January 13th, 2018, 10:13 AM
aldo's Avatar
aldo aldo is offline
Super Seer
 
Join Date: Feb 2002
Location: Massachusetts
Posts: 1,560
I'm impressed by your code, I'm not as near as you in regards of this. I have to do lot of homework to get close to your level. So far I did not use any MQTT in any of my projects, it is time to start.

In the meantime, I have a small issue that I can not figure out. I was lucky to get a virtualbox machine working with the meter program. I tried again on a production machine and I really do not understand the verbiage of the author. What does he mean when he says "This will produce the binary*$GOPATH/bin/rtlamr. For convenience it's common to add*$GOPATH/bin*to the path." I'm having hard time understanding it, no matter what I do it does not let me run the installation program rtlamr.

This is what I learned so far and fails on the last part.
- Run Sudo apt install golang-go
- Run Sudo apt instal git
- Run sudo apt install rtl-sdr
• Install Goland. When executing go get github.com/bemasher/rtlamr it will ask if you want to install the goLand. Install it if it has not been installed yet.
• Install git. If this program is not installed, the program will tell you to install it.
• Run this from the command prompt - go get github.com/bemasher/rtlamr. It installs the rtlamr utility.

Thanks,
Aldo
Reply With Quote
  #15  
Old January 13th, 2018, 03:09 PM
aldo's Avatar
aldo aldo is offline
Super Seer
 
Join Date: Feb 2002
Location: Massachusetts
Posts: 1,560
Sorry if this is a stupid question, do you run it from HS3? It looks like is just what I need. Right now I have the meter sensor on a virtual machine. Do I need to install MQTT on it?

Thanks

Quote:
Originally Posted by mloebl View Post
I forked this off an existng script I use to push TED1000 or 1wire messages to MQTT via a service, so it's not perfect, but worked enough for my testing. I take no responsibility for the damage it may cause The script itself is based on an original MQTT publishing script I found here on github, https://github.com/kylegordon/mqtt-owfs-temp.

Attached is the python script, and a sample config file. The python script reads that config sample config file where you can set path values, credentials, etc. For my testing I had rtlamr, the log, and config file all in the same directory. If you want to use this script, rename the SAMPLE-mqtt-rtlamr.cfg to mqtt-rtlamr.cfg and set values accordingly. It's not perfect, but something I threw together in an evening to test how it may work. It's definitely a good starting point for someone looking to use a python script to process the data.

SAMPLE-mqtt-rtlamr.cfg
Code:
[global]
DEBUG = False
LOGFILE = mqtt-rtlamr.log
RTLAMR_PATH =./rtlamr
MQTT_HOST = CHANGEME-MQTTHOST
MQTT_PORT = CHANGEME-MQTTPORT
MQTT_TOPIC = /#
MQTT_SUBTOPIC = /rtlamr/
MQTT_USERNAME = CHANGEME-MQTTUSERNAME
MQTT_PASSWORD = CHANGEME-MQTTPASSWORD
POLLINTERVAL = 1
METRICUNITS = 0
UNITID = CHANGEME-UNITID
MSGTYPE = scm
Code:
#!/usr/bin/env python
# -*- coding: iso-8859-1 -*-

# Script based on mqtt-owfs-temp written by Kyle Gordon and converted for use with script
# Source: https://github.com/kylegordon/mqtt-owfs-temp

import os
import logging
import signal
import socket
import time
import sys
import ConfigParser
import json

import subprocess
import paho.mqtt.client as paho

import setproctitle
# from datetime import datetime, timedelta

# Read the config file
config = ConfigParser.RawConfigParser()
config.read("mqtt-rtlamr.cfg")

# Use ConfigParser to pick out the settings
DEBUG = config.getboolean("global", "debug")
LOGFILE = config.get("global", "logfile")
MQTT_HOST = config.get("global", "mqtt_host")
MQTT_PORT = config.getint("global", "mqtt_port")
MQTT_SUBTOPIC = config.get("global", "MQTT_SUBTOPIC")
MQTT_TOPIC = "/raw/" + socket.getfqdn() + MQTT_SUBTOPIC
MQTT_USERNAME = config.get("global", "MQTT_USERNAME")
MQTT_PASSWORD = config.get("global", "MQTT_PASSWORD")
METRICUNITS = config.get("global", "METRICUNITS")

POLLINTERVAL = config.getint("global", "pollinterval")

RTLAMR_PATH = config.get("global", "RTLAMR_PATH")
UNITID = config.get("global", "UNITID")
MSGTYPE = config.get("global", "MSGTYPE")


APPNAME = "rtlamr"

PRESENCETOPIC = "clients/" + socket.getfqdn() + "/" + APPNAME + "/state"
setproctitle.setproctitle(APPNAME)
client_id = APPNAME + "_%d" % os.getpid()

mqttc = paho.Client()

LOGFORMAT = '%(asctime)-15s %(message)s'

if DEBUG:
    logging.basicConfig(filename=LOGFILE,
                        level=logging.DEBUG,
                        format=LOGFORMAT)
else:
    logging.basicConfig(filename=LOGFILE,
                        level=logging.INFO,
                        format=LOGFORMAT)

logging.info("Starting " + APPNAME)
logging.info("INFO MODE")
logging.debug("DEBUG MODE")

# All the MQTT callbacks start here


def on_publish(mosq, obj, mid):
    """
    What to do when a message is published
    """
    logging.debug("MID " + str(mid) + " published.")


def on_subscribe(mosq, obj, mid, qos_list):
    """
    What to do in the event of subscribing to a topic"
    """
    logging.debug("Subscribe with mid " + str(mid) + " received.")


def on_unsubscribe(mosq, obj, mid):
    """
    What to do in the event of unsubscribing from a topic
    """
    logging.debug("Unsubscribe with mid " + str(mid) + " received.")


def on_connect(mosq, obj, result_code):
    """
    Handle connections (or failures) to the broker.
    This is called after the client has received a CONNACK message
    from the broker in response to calling connect().
    The parameter rc is an integer giving the return code:
    0: Success
    1: Refused – unacceptable protocol version
    2: Refused – identifier rejected
    3: Refused – server unavailable
    4: Refused – bad user name or password (MQTT v3.1 broker only)
    5: Refused – not authorised (MQTT v3.1 broker only)
    """
    logging.debug("on_connect RC: " + str(result_code))
    if result_code == 0:
        logging.info("Connected to %s:%s", MQTT_HOST, MQTT_PORT)
        # Publish retained LWT as per
        # http://stackoverflow.com/q/97694
        # See also the will_set function in connect() below
        mqttc.publish(PRESENCETOPIC, "1", retain=True)
        process_connection()
    elif result_code == 1:
        logging.info("Connection refused - unacceptable protocol version")
        cleanup()
    elif result_code == 2:
        logging.info("Connection refused - identifier rejected")
        cleanup()
    elif result_code == 3:
        logging.info("Connection refused - server unavailable")
        logging.info("Retrying in 30 seconds")
        time.sleep(30)
    elif result_code == 4:
        logging.info("Connection refused - bad user name or password")
        cleanup()
    elif result_code == 5:
        logging.info("Connection refused - not authorised")
        cleanup()
    else:
        logging.warning("Something went wrong. RC:" + str(result_code))
        cleanup()


def on_disconnect(mosq, obj, result_code):
    """
    Handle disconnections from the broker
    """
    if result_code == 0:
        logging.info("Clean disconnection")
    else:
        logging.info("Unexpected disconnection! Reconnecting in 5 seconds")
        logging.debug("Result code: %s", result_code)
        time.sleep(5)


def on_message(mosq, obj, msg):
    """
    What to do when the client recieves a message from the broker
    """
    logging.debug("Received: " + msg.payload +
                  " received on topic " + msg.topic +
                  " with QoS " + str(msg.qos))
    process_message(msg)


def on_log(mosq, obj, level, string):
    """
    What to do with debug log output from the MQTT library
    """
    logging.debug(string)

# End of MQTT callbacks


def cleanup(signum, frame):
    """
    Signal handler to ensure we disconnect cleanly
    in the event of a SIGTERM or SIGINT.
    """
    logging.info("Disconnecting from broker")
    # Publish a retained message to state that this client is offline
    mqttc.publish(PRESENCETOPIC, "0", retain=True)
    mqttc.disconnect()
    mqttc.loop_stop()
    logging.info("Exiting on signal %d", signum)
    sys.exit(signum)

def connect():
    """
    Connect to the broker, define the callbacks, and subscribe
    This will also set the Last Will and Testament (LWT)
    The LWT will be published in the event of an unclean or
    unexpected disconnection.
    """
    logging.info("Connecting to %s:%s", MQTT_HOST, MQTT_PORT)

    if MQTT_USERNAME:
        logging.info("Found username %s", MQTT_USERNAME)
        mqttc.username_pw_set(MQTT_USERNAME, MQTT_PASSWORD)

    # Set the Last Will and Testament (LWT) *before* connecting
    mqttc.will_set(PRESENCETOPIC, "0", qos=0, retain=True)
    result = mqttc.connect(MQTT_HOST, MQTT_PORT, 60)
    if result != 0:
        logging.info("Connection failed with error code %s. Retrying", result)
        
        time.sleep(10)
        connect()

    # Define the callbacks
    mqttc.on_connect = on_connect
    mqttc.on_disconnect = on_disconnect
    mqttc.on_publish = on_publish
    mqttc.on_subscribe = on_subscribe
    mqttc.on_unsubscribe = on_unsubscribe
    mqttc.on_message = on_message
    if DEBUG:
        mqttc.on_log = on_log

    mqttc.loop_start()

def process_connection():
    """
    What to do when a new connection is established
    """
    logging.debug("Processing connection")


def process_message(mosq, obj, msg):
    """
    What to do with the message that's arrived
    """
    logging.debug("Received: %s", msg.topic)


def find_in_sublists(lst, value):
    for sub_i, sublist in enumerate(lst):
        try:
            return (sub_i, sublist.index(value))
        except ValueError:
            pass

    raise ValueError("%s is not in lists" % value)

def slicer(my_str,sub):
    index=my_str.find(sub)
    if index !=-1 :
        return my_str[index:] 
    else :
        raise Exception('Sub string not found!')

def decodeJSON(jsonString):
    logging.debug('Processing JSON: ', jsonString)
    return(json.loads(jsonString))

def getjsonvalue(jsonData, jsonmessage):
    return(jsonData["Message"][jsonmessage])

def getMeterType(metertype):
    meter = "unknown"
    if (metertype == 4) or (metertype == 5) or (metertype == 7) or (metertype == 8):  
        meter = "electric"
    if (metertype == 2) or (metertype == 9) or (metertype == 12): 
        meter = "gas"
    if (metertype == 11) or (metertype == 13):
        meter = "water"
    return meter


def main_loop():
    """
    The main loop in which we stay connected to the broker
    """
    

    while True:
        rawjson = subprocess.check_output([RTLAMR_PATH, '-filterid='+UNITID, '-msgtype='+MSGTYPE,'--unique=true' ,'--format=json','--single=true'])
        logging.debug(slicer(rawjson, "{"))
        jsondata = decodeJSON(rawjson)

        json_time=jsondata["Time"]
        json_id = getjsonvalue(jsondata, "ID")
        json_type = getjsonvalue(jsondata, "Type")
        json_tamper = getjsonvalue(jsondata, "TamperPhy")
        json_consumption = getjsonvalue(jsondata, "Consumption")

        logging.debug("Current consumption: %.2f kw/h" % (json_consumption / 100))

        string_consumption = "%0.2f" % (float(json_consumption) / 100)  

        mqttc.publish(MQTT_TOPIC + str(json_id) + "/time", json_time)
        mqttc.publish(MQTT_TOPIC + str(json_id) + "/unitid", json_id)
        mqttc.publish(MQTT_TOPIC + str(json_id) + "/msgtype", MSGTYPE)
        mqttc.publish(MQTT_TOPIC + str(json_id) + "/metertype", getMeterType(json_type))
        mqttc.publish(MQTT_TOPIC + str(json_id) + "/consumption_raw", json_consumption)
        mqttc.publish(MQTT_TOPIC + str(json_id) + "/consumption_kwh", string_consumption)
        mqttc.publish(MQTT_TOPIC + str(json_id) + "/tamperphy", json_tamper)
        
        time.sleep(POLLINTERVAL*60)

# Use the signal module to handle signals
signal.signal(signal.SIGTERM, cleanup)
signal.signal(signal.SIGINT, cleanup)

# Connect to the broker and enter the main loop
connect()

# Try to start the main loop
try:
    main_loop()
except KeyboardInterrupt:
    logging.info("Interrupted by keypress")
    sys.exit(0)
Reply With Quote
  #16  
Old January 14th, 2018, 01:06 PM
mloebl's Avatar
mloebl mloebl is offline
Seer Master
 
Join Date: Apr 2002
Location: Northern, MA
Posts: 767
Quote:
Originally Posted by aldo View Post
I'm impressed by your code, I'm not as near as you in regards of this. I have to do lot of homework to get close to your level. So far I did not use any MQTT in any of my projects, it is time to start.

In the meantime, I have a small issue that I can not figure out. I was lucky to get a virtualbox machine working with the meter program. I tried again on a production machine and I really do not understand the verbiage of the author. What does he mean when he says "This will produce the binary*$GOPATH/bin/rtlamr. For convenience it's common to add*$GOPATH/bin*to the path." I'm having hard time understanding it, no matter what I do it does not let me run the installation program rtlamr.

This is what I learned so far and fails on the last part.
- Run Sudo apt install golang-go
- Run Sudo apt instal git
- Run sudo apt install rtl-sdr
• Install Goland. When executing go get github.com/bemasher/rtlamr it will ask if you want to install the goLand. Install it if it has not been installed yet.
• Install git. If this program is not installed, the program will tell you to install it.
• Run this from the command prompt - go get github.com/bemasher/rtlamr. It installs the rtlamr utility.

Thanks,
Aldo
I can't take full credit for the code; Kyle Gordon wrote the core of the mqtt, but I updated some of the paho mqtt calls for supporting mqtt username and password. At some point I'll probably enable tls certs so I can go encrypted...

So here's what I did, set a GOPATH env variable to a base directory where you want it (either in that terminal session, or in /etc/profile.d, or /etc/profile, etc.) It's kind of annoying tbh, but looks like something GoLang may have standardized on. I set it to me home/USERNAME/src directory, and it nested everything under there, however you can put it into any directory your username has access too. Once you do that, when you do the go get command, it will put it in the $GOPATH/bin/rtlamr path. Once done, to make debugging easier, copy that rtlamr file to the same directory as these scripts and config file and you can run them all from there.

I've run it from the HS3 box for testing, however long term will run it for a dedicated Pi on the network. For now it could be run via cron. It can also be run as a service if you're interested. I hadn't done this yet as it was logging some useless header info every time it ran, so figured at some point I'd clean it up rather than clog the log.

If you want to use MQTT, then yes, you'll need an mqtt server running on that box or somewhere on the network. However if you end up using mysql commands, then it'll make the MQTT pieces OBE.

Slight tangent: I switched to using MQTT as got tired of having lots of different protocols and standards for doing communication. I had 1 wire devices, an unsupported TED1000 power monitor, ESP8266, NodeMCU, etc. With MQTT it's a fairly simple and well documented messaging protocol, so lends it's self ideally to a common devices messaging ability. It's not perfect, but fit my needs quite happily

Long term I'll probably get this up on to Github, here's the 1wire to MQTT forked version I did a year or so ago: https://github.com/mloebl/mqtt-owfs-temp
Reply With Quote
  #17  
Old January 14th, 2018, 05:46 PM
aldo's Avatar
aldo aldo is offline
Super Seer
 
Join Date: Feb 2002
Location: Massachusetts
Posts: 1,560
I try to write a program that would take the JSON file and move it into mysql. I would love a suggestion on how to figure out the kwh. Right now every time the consumption value change it would write to sql database but it will only give me the actual meter reading like 19253, then 19254 after few minutes and etc. etc. How should I get the kWH from it? I guess everytime it changes is 1 KWH consumed, what should I write in the database? Do you see what I'm trying to say.

Aldo
Reply With Quote
  #18  
Old January 14th, 2018, 07:06 PM
Michael McSharry's Avatar
Michael McSharry Michael McSharry is offline
OverSeer
 
Join Date: Jul 2001
Location: North Bend, WA, USA
Posts: 13,156
Quote:
Slight tangent: I switched to using MQTT as got tired of having lots of different protocols and standards for doing communication. I had 1 wire devices, an unsupported TED1000 power monitor, ESP8266, NodeMCU, etc. With MQTT it's a fairly simple and well documented messaging protocol, so lends it's self ideally to a common devices messaging ability. It's not perfect, but fit my needs quite happily
What did you do for MQTT in/out of HS? For your 1-wire do you have more than temperature via MQTT?
Reply With Quote
  #19  
Old January 14th, 2018, 08:05 PM
mloebl's Avatar
mloebl mloebl is offline
Seer Master
 
Join Date: Apr 2002
Location: Northern, MA
Posts: 767
Quote:
Originally Posted by Michael McSharry View Post
What did you do for MQTT in/out of HS? For your 1-wire do you have more than temperature via MQTT?
There is an MQTT plugin for HS3 that's in the updater; there are a couple threads on it here on the forum:
https://forums.homeseer.com/showthread.php?t=180804
https://forums.homeseer.com/showthread.php?t=185697

I broke up my large 1wire network up into a couple of segments using either a BeagleBone or a Raspberry Pi. In a nutshell I then go 1wire devices->owfs->pythonscript as a service->mqtt.

I have 1wire temperature, humidity, pressure and light sensors. The initial script on Github was temperature I think only with maybe humidity support? When I forked it, I added add'l devices that I had (basically anything owfs supports that I was using). In that python script, I ingest a csv of the different devices:
https://github.com/mloebl/mqtt-owfs-...ces.csv.sample

I then read that in the owsensortype variable where depending on the device type from that csv, I process it as needed.

TBH I used 1wire and xAP for a very very long time and your plugins and apps worked great. However I wanted to reuse the existing CAT5e I had from the basement to the attic for 1wire as a dedicated ethernet/wifi IoT network. I had a hard time finding a simple light weight xAP server for Linux to load on a Pi.

Another example is the TED1000 that's no longer supported by HS3. I took a script someone wrong for decoding TED1000 messages, and plumbed it thru the same base MQTT python script. https://github.com/mloebl/mqtt-ted1000 It has no problem updating device values every second very reliably with current power usage. Only issue now is I have a bug if the TED1000 loses power, it doesn't reconnect the USB device

WeeWX I've been running for a long time with my Davis weather station, and it also natively supports MQTT messages.

MQTT also lets you (reasonably) easily bridge MQTT servers together. I've been playing with LoRA, and using TTN (The Things Network). I can bridge my MQTT server to their MQTT services, and get my LoRA transmitted devices that way as well.

I spent a couple hours monkeying with the JSON calls for HS3 until I changed my approach and tweaked my python MQTT script. Within less than an hour was pumping messages out to MQTT. As I mentioned above, I really want to learn Go so I can do it natively there, especially as not everything beacons every second.
Reply With Quote
  #20  
Old January 14th, 2018, 08:15 PM
mloebl's Avatar
mloebl mloebl is offline
Seer Master
 
Join Date: Apr 2002
Location: Northern, MA
Posts: 767
Quote:
Originally Posted by aldo View Post
I try to write a program that would take the JSON file and move it into mysql. I would love a suggestion on how to figure out the kwh. Right now every time the consumption value change it would write to sql database but it will only give me the actual meter reading like 19253, then 19254 after few minutes and etc. etc. How should I get the kWH from it? I guess everytime it changes is 1 KWH consumed, what should I write in the database? Do you see what I'm trying to say.

Aldo
So I haven't tackled the current KWH yet myself, but I think he did it here:
https://blog.kroy.io/monitoring-home...-less-than-25/
His scripts use a SQLite database, and I believe he's selecting the old values out of the database to calculate values based on current value.

Code:
  private function getLastReading($table) {
             $ld_res = $this->db->query("select * from {$table} order by id desc limit 1");
             if ($ld_row = $ld_res->fetchArray()) {
                 return $this->reading-$ld_row['reading'];
             }
             return 0;

         }
I haven't time yet to sit down and tackle the calculations yet. I'd consider this a project I'm about 2/3 of the way thru, unfortunately. I'll keep posting here over the next few weeks as I get more time to thinker.
Reply With Quote
Reply

Bookmarks

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
UltraFlowMeter3 HSPI - FortrezZ Water Meters Ultrajones UltraFlowMeter3 HSPI 12 December 19th, 2017 04:21 AM
Aeon power meters..... wpiman Lighting & Primary Technology Discussion 8 March 15th, 2017 05:02 PM
water/gas meters TeleFragger General Home Automation Hardware Discussion 2 August 29th, 2010 04:45 PM
2 Water Meters using dual HB Counter Pete MCS Temperature 6 June 4th, 2009 11:00 AM
Heads up! - cheap water meters Bestgear UK/Europe 24 January 22nd, 2008 08:08 PM


All times are GMT -4. The time now is 04:37 AM.


Copyright HomeSeer Technologies, LLC