No announcement yet.

Encoding substitution variables in publish payloads

  • Filter
  • Time
  • Show
Clear All
new posts

  • Encoding substitution variables in publish payloads

    Hi there,
    I'm using version

    I'm trying to publish JSON payloads as follows:
    This works fine in most cases, however should any of the substitution variables include a double-quote, it breaks the JSON. For example, I've got a rain gauge that measures things in inches, indicated by a double quote:
    {"string":"4 "","value":"4"}
    I can't parse this because it's invalid JSON.

    I've reviewed the documentation, and I don't see anything that would suggest it is possible to do any sort of substitutions in payload variables as we go to publish. If there were a way to either be able to escape double-quotes with a backslash ("4 ""), or maybe do URI/percent encoding ("4%20%22"), that'd be great.

    Perhaps it would be possible to have a set of URI-encoded variables that could optionally be used (ex: '$$STRING_URI_ENCODE:' ) ?

  • #2
    It does does URI for space, slash, hash and plus. I can add the quote and apostrophe, but I suspect that will cause issues with others. The additional substitution variable would seem the safer way. Somewhat confusing because only quote and apostrophe will be URI encoded, but give the attached a try with the new variable $$STRING_URI_ENCODE:.
    Attached Files


    • #3
      I haven't had a chance to try out the build you provided, yet, but I wanted to comment that it has been my experience that only variables in the topic are URI encoded. The payload variables are not URI encoded. Your documentation refers to "HTML-encoding", but I think you mean URI-Encoding (aka URI-Encoding or Percent-Encoding). HTML-encoding is a different thing (ex: spaces can become " ", & -> "&", etc).

      If I could make another suggestion, it would be to use System.Net.WebUtility.UrlEncode(String) for URI encoding the strings. I would prefer that mcsMQTT produce standard URI-encoded strings rather than mcsMQTT-flavored-URI-encoded strings. This would ensure that anyone decoding would be able to do so according to the standards. For example, I just tested creating a device that has %'s in the name, room, and floor, but these "%" characters are not being encoded by mcsMQTT when the device changes. So, the topic is published as "homeseer/custom/My%20Functional%Group/My%20%room/Test%%20Device/670", which can't be decoded.



      • #4
        I believe you are correct about encoding being different between topic and payload. I will look into your suggestion. Likely no value in using what I posted.


        • #5
          I have implemented the suggestion and generalize the implementation. Two things I have run into. The .NET library uses "+" for the space. "+" is a reserved character in MQTT standard so it needs to be replaced by %20. The second is that compound encoding occurs with the current structure when things such as $$STRING_URL_ENCODE: is used for Topics. I can work through both of these which the assumed solution for space is %20 rather than +. I need to think about the best approach to handle the compound URL encoding for substitution variables, or template in general, for the Topic.


          • #6
            I generalized the use of URL encoding for payloads by allowing use of the "_URL_ENCODE" suffix on any of the substitution variables included in the publish payload template. For example "$$ROOM_URL_ENCODE:" in the publish payload template will result in sending "Laundry%20Room" while using "$$ROOM:" will send "Laundry Room" for payload. The .Net library is used for the encoding.

            The update of the plugin and document is in

            If it works for you I will submit to updater.


            • #7
              Mike, Sorry for the delay. When I had recommended System.Net.WebUtility.UrlEncode, I wasn't aware that it would encode space characters as "+". It appears that the correct solution is to use System.Uri.EscapeDataString. There's a post on Stack Overflow here that detail some of the differences between System.Uri.EscapeDataString and it's counterpart, System.Uri.EscapeUriString (which you do not want to use).


              • #8
                I changed it to URI.EscapeDataString, but no matter what is used there will be variance from the standard. URLEncode leaves the apostrophe alone which is likely going to be preferred by most users. Both will percent encode the forward slash which is part of the MQTT standard topic so special provisions are needed to not encode it. The substitution parameter was changed to _URI_ENCODE. The updates are in


                • #9
                  Mike, I just gave a whirl, and it's working great! I see that the topic components are all being URI encoded, now, and my payload parameters are being encoded when I throw _URI_ENCODE onto them. So far, I've tested out $$STATUS_URI_ENCODE: and $$STRING_URI_ENCODE:.

                  Thanks so much for the wonderful work!