Announcement

Collapse
No announcement yet.

Feature request: topic names and payloads from templates

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • Feature request: topic names and payloads from templates

    Hi Michael,
    I've been very happy with the mcsMQTT plugin, not to mention fairly impressed with the amount of time you're putting into its development.

    One of the things that I would like to see is greater control over the auto-generated topic names, as well as even the payloads, themselves. While I understand that I can manually define things such as the topic names, I've got close to 240 devices in HS, so tweaking everything becomes tedious.

    Here are my ideas:

    Option to prefer the usage of forward-slashes for device topic separators, rather than vertical pipes.
    The usage of vertical pipes for device name separators hinder MQTT subscribers from leveraging my broker's ability to filter messages for them.

    I'd really like to have the general option within mcsMQTT to specify that auto-generated topic names use forward-slashes (/) for device name separators, rather than vertical pipes.

    For example, I have a nightstand light, and the mcsMQTT generated topic name is "homeseer/mcsMQTT/Master_Bedroom|Lighting|Nightstand_Light".

    However, if the topic name were "homeseer/mcsMQTT/Master_Bedroom/Lighting/Nightstand_Light", a subscriber could filter for all Master Bedroom lights with "homeseer/mcsMQTT/Master_Bedroom/Lighting/+", or even all lights in my home with "homeseer/mcsMQTT/+/Lighting/+". I cannot do the same thing when vertical pipes are used in the topics.

    Here's an area to the section of the MQTT specification regarding topic separators: http://docs.oasis-open.org/mqtt/mqtt...#_Toc398718106

    Template-string control over generated topic names
    While I've got your ear, I'll throw another idea at you: it'd be neat if we had the ability to define a template string (or something similar) that would provide control over the structure of the topic name.

    Imagine, if you will, I were to define a hypothetical template string like "status/${location1}/${location2}/${type}/${name}", the generated topic name for my nightstand light would be "status/Master_Bedroom/Lighting/Insteon/Nightstand_Light".

    You could have a whole slew of replacement tokens for a template, if you wanted:
    1. location1
    2. location2
    3. type - device type
    4. name
    5. value - the current (or just set) value of the device
    6. value_string - the string-mapped value. ex: "ON", "OFF", etc
    7. previous_value - the previous value
    8. previous_value_string - the previous string-mapped value
    9. address - Device address


    Template-string control over payloads
    Basically an extension of my previous idea, but control over the payload string, which would mean easier interoperation with other systems in my MQTT ecosystem.
    It'd be pretty neat if I could define a payload template like:
    Code:
    {
    "room":"${location1}",
    "device":"${name}",
    "new_value":"${value}",
    "old_value":"${previous_value}"
    }

  • #2
    I saw the vertical pipes in the default topic and just left them because my thinking was that the user would not like any any topic syntax I selected so was going to change it anyway. You idea of a topic template is good and should be able to handle easily.

    On the Payloads I do not have a good understanding of the scope. I understand you want to have ability for JSON encoding on the published payload, but not clear what the universe of JSON keys would be. For example you did not show a floor as one of your keys. The key itself is really not well-defined as a German user would want different keys than a British user.

    I think this is a good concept, but before I dive in I think a little more discussion is needed to flush out exactly what would work for the population of users.

    Comment


    • #3
      Originally posted by Michael McSharry View Post
      On the Payloads I do not have a good understanding of the scope. I understand you want to have ability for JSON encoding on the published payload, but not clear what the universe of JSON keys would be. For example you did not show a floor as one of your keys. The key itself is really not well-defined as a German user would want different keys than a British user.
      To paint a clearer picture of the payload templates: I'm not looking for mcsMQTT to be responsible for generating JSON; I was providing an example of a template string that just happened to be JSON. The only concern I would expect mcsMQTT to have is that I should give it a template string, and mcsMQTT would take care of replacing tokens (Ex: "${locaiton1}") with the appropriate values for the device being handled.
      For example, I could just as easily want to do CSV:
      Code:
      ${location1},${name},${value},${previous_value}
      My omission of a "location2" was not to suggest that it should not be made available to a payload template -- rather, my expectation is that the full suite of replacement tokens I listed for topic names also be made available for payloads.

      I'm not 100% certain what you're referring to when you say "keys". Are you talking about the replacement tokens ("location1", "location2", "value", etc), or are you talking about the JSON hash keys? If you're talking about localizing the JSON keys I provided in my example ("room", "new_value", "old_value"), again, my intention is not that mcsMQTT should have to concern itself with the data format I'm trying to represent in the template-string for my payload. I'm just looking for tokens to be replaced. As a result, if someone wanted to create some German-specific JSON templates, they could (ex: "new_value" -> "neuer_wert" maybe).

      Although I don't think this to be the case, but if you happened to be talking about the localizing of the replacement tokens ("location1", etc) in different languages - I would doubt anyone would expect that kind of accommodation. After all, when you write a script for Homeseer, the underlying APIs aren't localized.

      Comment


      • #4
        In the mcsMQTT manual I have documented the set of replacement variables that HS supports to also be supported by mcsMQTT for Device Action Payloads. This is the smaller set than you are inferring. Can you suggest the starting set that should be supported.

        mcsMQTT registers for Device Value and Device String callbacks, as appropriate. This is the trigger to publish a Topic if it is the device that has a MQTT topic. What event are you suggesting be used to publish when the payload has a user-specified template?

        What are you suggesting for the template format that a user would specify for a Payload? The Topic is clear, but the payload is still fuzzy with respect to the user interface?

        As an example of key, consider the payload Wifi:{SSID:xyz, RSSI:80}. I refer to Wifi, SSID and RSSI as keys. They are the items that preceed the ":".

        Comment


        • #5
          I have implemented the templates and substitution variables in 3.2.14.0. The manual has a table with the variables and syntax. Let me know if you want any changes.

          Comment


          • #6
            Originally posted by Michael McSharry View Post
            I have implemented the templates and substitution variables in 3.2.14.0. The manual has a table with the variables and syntax. Let me know if you want any changes.

            Michael, I just got around to upgrading. This is pretty great, thank you!

            Couple things I've noticed:
            - It doesn't appear that the names get sanitized properly - if the value contains a forward-slash, it ends up in the topic. For example, I've got a type = "InsteonDual-BandSwitchLincOn/Off". The forward slash can goof things up. Off-hand, my thought would be to URL-encode substitution variables when being placed into the topic. For example, the above would be encoded as "InsteonDual-BandSwitchLincOn%2FOff". I'd prefer to have the value encoded, than stripped of characters.

            - It looks the beginning of like custom payloads get truncated if up to the first colon (that's not part of a replacement variable). For example, the following:
            Code:
            {"status":"$$status:","timestamp":"$$date: $$time:", "string":"$$string:"}
            should produce
            Code:
            {"status":"On","timestamp":"4/11/2018 8:07 PM", "string":""}
            but instead is sending:
            Code:
            "On","timestamp":"4/11/2018 8:07 PM", "string":""}
            I would guess that maybe it's because of how you're doing the variable substitution. A regular expression like the following should work:
            Code:
            \$\$STATUS:
            And just iterate through each replacement variable.

            Comment


            • #7
              Fixed the initial segment of the substitution. Added HTML encoding for topics where the substituted entity is space, slash, hashtag or plus. Update in 3.2.16.0.

              Comment


              • #8
                Originally posted by Michael McSharry View Post
                Fixed the initial segment of the substitution. Added HTML encoding for topics where the substituted entity is space, slash, hashtag or plus. Update in 3.2.16.0.
                Just got 3.2.16.1 installed, and it looks great! Thanks for all the hard work, Michael!

                Comment

                Working...
                X