Announcement

Collapse
No announcement yet.

Jon00 DataScraper/JSON Parser Script For Homeseer 3 and Homeseer 4

Collapse
This topic is closed.
X
This is a sticky topic.
X
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

    Jon,

    The values don't show up in HS3 when using the following:
    Code:
    Pattern1=(?s)<td width="34%" class="dataName"><span id="langLastBatteryXfer">Last Battery Transfer</span>:</td><td colspan="3">.*?</td>(.*?)</td>
    Pattern2=(?s)<td class="dataName"><span id="langInternalTemp">Internal Temperature</span>:</td><td colspan="3">.*?F</td>(.*?);F</td>
    Pattern3=(?s)<td class="dataName"><span id="langRuntimeRemaining">Runtime Remaining</span>:</td><td colspan="3">.*?</td>(.*?)</td>
    Pattern4=(?s)<td class="dataName"><span id="langInputValtage">Input Voltage</span>:</td><td colspan="2">.*?;<span id="langVAC">VAC</span></td>(.*?);<span id="langVAC">VAC</span></td>
    Pattern5=(?s)<td class="dataName"><span id="langOutputVoltage">Output Voltage</span>:</td><td colspan="2">.*?;<span id="langVAC">VAC</span></td>(.*?);<span id="langVAC">VAC</span></td>

    Comment


      Part two of reply

      The HTML code is
      HTML Code:
      <td width="34%" class="dataName"><span id="langLastBatteryXfer">Last Battery Transfer</span>:</td>  <td colspan="3">Detection of distorted input</td>  </tr> <tr> <td class="dataName"><span id="langInternalTemp">Internal Temperature</span>:</td>  <td colspan="3">82.9&deg;F</td>  </tr> <tr> <td class="dataName"><span id="langRuntimeRemaining">Runtime Remaining</span>:</td>  <td colspan="3"> 2hr 59min</td>  </tr> <tr> <td class="dataSubSection"><span id="langUPSInput">UPS Input</span></td>  <td width="17%">&nbsp;</td>  <td colspan="2">&nbsp;</td>  </tr> <tr> <td class="dataName"><span id="langInputValtage">Input Voltage</span>:</td> <td colspan="2">120.2&nbsp;<span id="langVAC">VAC</span></td>  <td colspan="1">@&nbsp;60.0&nbsp;<span id="langHz">Hz</span> </td>  </tr> <tr> <td colspan="4" class="space">&nbsp;</td>  </tr> <tr> <td class="dataSubSection"><span id="langUPSOutput">UPS Output</span></td>  <td colspan="3">&nbsp;</td>  </tr> <tr> <td class="dataName"><span id="langOutputVoltage">Output Voltage</span>:</td>  <td colspan="2">120.2&nbsp;<span id="langVAC">VAC</span></td>  <td colspan="1">@&nbsp;60.0&nbsp;<span id="langHz">Hz</span></td>  </tr> <tr> <td class="dataName"><span id="langLoadCurrent">Load Current</span>:</td>  <td colspan="3">0.5&nbsp;<span id="langAmps">Amps</span></td>

      Comment


        Please set TextFile=1 in the ini file and run the script.

        Then post or email me the grab1.txt file created (found in the Homeseer HS3\Data\Jon00\datacraper folder).

        I'll use that to get something working for you.
        Jon

        Comment


          Grab Txt File

          <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
          <html xmlns="http://www.w3.org/1999/xhtml">
          <head>
          <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
          <meta http-equiv="Expires" content="-1">
          <meta http-equiv="Pragma" content="no-cache">
          <meta http-equiv="Cache-Control" content="no-cache">
          <link rel="icon" href="images/favicon.ico" type="image/x-icon" />
          <link rel="shortcut icon" href="images/favicon.ico" type="image/x-icon" />
          <link href="core1.css" rel="stylesheet" type="text/css" />
          <title>APC | Log On</title>

          <script type="text/javascript">
          function placeFocus() {
          if (document.forms.length > 0) {
          var field = document.forms[0];
          for (i = 0; i < field.length; i++) {
          if ((field.elements[i].type == "text") || (field.elements[i].type == "textarea")) {
          document.forms[0].elements[i].focus();
          break;
          }
          }
          }
          }
          </script>
          </head>
          <body id="logonbg" onload="placeFocus()">
          <form method="post" action="/NMC/2NGoRUQxIpL7c7U60sN8vQ/Forms/login1" name="HashForm1">
          <table cellpadding="0" cellspacing="0" id="logon">
          <tr>
          <td colspan="2" class="logonTitle" width="100%">Login</td>

          </tr>
          <tr>
          <td class="logonHeaderMessage" width="65%">Main DC UPS</td>
          <td class="logonHeader" width="35%"><select name="prefLanguage" size="1">
          <option value=00000000 selected="true">Use Current Language
          <option value=01000000>English
          </select></td>
          </tr>
          <tr>
          <td colspan="2" class="logonBody" width="100%">&nbsp;</td>

          </tr>
          <tr>
          <td colspan="2">
          <table cellpadding="0" cellspacing="0" width="100%">
          <tr>
          <td width="40%" class="logonIcon"><img src="images/profile.gif" /></td>

          <td width="57%">
          <table cellpadding="0" cellspacing="0" width="100%">
          <tr>
          <td width="42%" class="logonText">User Name:</td>

          <td width="58%" class="logonText"><input type="text" name="login_username" size="23" maxlength="64" value="" autocomplete="off" />
          </td>

          </tr>
          <tr>
          <td class="logonText">Password:</td>

          <td class="logonText"><input type="password" name="login_password" size="23" maxlength="64" value="" />
          </td>

          </tr>
          <tr colspan="2">
          <td class="space">&nbsp;</td>

          </tr>
          <tr>
          <td>&nbsp;</td>

          <td class="logonText"><input type="submit" name="submit" value="Log On" class="btn" />
          &nbsp;<input type="reset" value="Reset" />
          </td>

          </tr>
          </table>
          </td>

          <td width="3%">&nbsp;</td>

          </tr>
          <tr>
          <td colspan="3" class="logonSpace">&nbsp;</td>

          </tr>
          <tr>
          <td>&nbsp;</td>
          <td class="logonLogo"><a href="http://www.apc.com"><img src="images/se_logo.gif" alt="APC Website" border="0" /></a></td>

          <td>&nbsp;</td>

          </tr>
          </table>
          </td>

          </tr>
          </table>

          </form>

          </body>
          </html>

          Comment


            The code you have posted looks like the login page and not the page you want to scrape data from.
            Jon

            Comment


              Jon,
              Here is what I discovered :
              The page is a UPS Status Page :
              Path=http://10.0.1.7/NMC/PeO6rBiT89MOwmw+Mzh9vQ/ulstat.htm
              But when I log off and then log on ( which I tried on 3 different Log off and log on's the following link changes
              http://10.0.1.7/NMC/iIZJolcmDei2TA6cVSUasw/ulstat.htm
              http://10.0.1.7/NMC/tbj2o6AyyZs14Ii4WOTTbQ/ulstat.htm
              http://10.0.1.7/NMC/g1ECrvf710OCanIHKvJBrw/ulstat.htm

              Comment


                How do you link to that URL normally? Is it via another APC web page?
                Jon

                Comment


                  Jon,

                  I usually type in 10.0.1.7 a login page appears which then has a link as follows
                  http://10.0.1.7/NMC/Glq9nrlAIB4QMnCMzYSmGw/logon.htm
                  which coincidentally also changes upon a different logon
                  http://10.0.1.7/NMC/U18zdWCsZ1uW6aqAYqVvAQ/logon.htm

                  So it must be an internal redirection ( I am guessing)

                  Comment


                    I'm not sure I'm going to be able to help with this method. Does the UPS create a data file (CSV) with the info you require?
                    Jon

                    Comment


                      Jon,
                      Well it has snmp but my experience with this in terms of regex is nill.
                      I have connected to it via snmp monitor plugin and have gotten the various OID's using byteview explorer.
                      thus when you see an OID that presents as follows:
                      OID Object Type Value

                      1.3.6.1.4.1.318.1.1.1.4.1.1.0 upsBasicOutputStatus INTEGER onLine(2)
                      1.3.6.1.4.1.318.1.1.1.4.1.2.0 upsBasicOutputPhase INTEGER 1
                      1.3.6.1.4.1.318.1.1.1.4.2.1.0 upsAdvOutputVoltage GAUGE 120

                      Thus using your snmp method how would i configure regex expressions to return onLine(2) to indicate in HS3 that UPS is online rather the just seeing a number (2)

                      Appreciate your help on this
                      BTW this is an APC UPS -- Network Card in a Smart-UPS UPS

                      Thanks
                      RMIES

                      Comment


                        Have you tried my SNMP plugin? You may be able to get the individual metrics out using that.

                        http://board.homeseer.com/showthread.php?t=176282
                        Jon

                        Comment


                          I want to pull out the value "11160 W" from the URL "http://10.0.0.5:8176/variables/smappeeforbruk.html" into a virtual device. Could anybody help me with the ini-file?

                          I only need that value to be updated to the device.

                          The html-file look like this:
                          Code:
                          !DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
                          <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en">
                          	<head>
                          		<title>Indigo Home Control Server</title>
                          		<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
                          	</head>
                          	<body>
                          		<strong>name : </strong>
                          		smappeeforbruk<br />
                          		<strong>displayInUI : </strong>
                          		True<br />
                          		<strong>id : </strong>
                          		1181145578<br />
                          		<strong>isFalse : </strong>
                          		False<br />
                          		<strong>readOnly : </strong>
                          		False<br />
                          		<strong>value : </strong>
                          		11160 W<br />
                          		<a href="/variables/smappeeforbruk.html/edit">Edit Attributes</a>
                          	</body>
                          </html>

                          Comment


                            Try this for [Grab2]

                            PHP Code:
                            [Grab2]
                            Path=http://10.0.0.5:8176/variables/smappeeforbruk.html
                            TextFile=0
                            Encoding
                            =
                            Username=
                            Password=
                            Options=
                            UserAgent=
                            Devicemode=0
                            Pattern1
                            =(?s)<strong>value : </strong>(.*?) W<br />
                            Pattern2=
                            Pattern3=
                            Pattern4=
                            Pattern5=
                            DeviceName1=Power
                            DeviceText1
                            =[0W
                            DeviceValue1
                            =[0]
                            DeviceImage1=
                            Speakbutton1=

                            The device value will be set to the power value as well.
                            Jon

                            Comment


                              Originally posted by jon00 View Post
                              Try this for [Grab2]

                              PHP Code:
                              [Grab2]
                              Path=http://10.0.0.5:8176/variables/smappeeforbruk.html
                              TextFile=0
                              Encoding
                              =
                              Username=
                              Password=
                              Options=
                              UserAgent=
                              Devicemode=0
                              Pattern1
                              =(?s)<strong>value : </strong>(.*?) W<br />
                              Pattern2=
                              Pattern3=
                              Pattern4=
                              Pattern5=
                              DeviceName1=Power
                              DeviceText1
                              =[0W
                              DeviceValue1
                              =[0]
                              DeviceImage1=
                              Speakbutton1=

                              The device value will be set to the power value as well.
                              I get this error in the log, Jon.
                              "Jon00_DataScrape Error - Cannot access path:"

                              des-28 17:47:25 Jon00_DataScrape Error - Cannot access path:
                              des-28 17:47:25 Event Running script in background: C:/Program Files (x86)/HomeSeer HS3/scripts/Jon00DataScraper.vben("Main","2")
                              des-28 17:47:25 Event Event Trigger "System Smappee Grab"
                              des-28 17:46:25 Jon00_DataScrape Error - Cannot access path:
                              des-28 17:46:25 Warning INI setting could not read setting speakbutton1,skipping:Item has already been added. Key in dictionary: 'speakbutton1' Key being added: 'speakbutton1'
                              des-28 17:46:25 Warning INI setting could not read setting deviceimage1,skipping:Item has already been added. Key in dictionary: 'deviceimage1' Key being added: 'deviceimage1'
                              des-28 17:46:25 Warning INI setting could not read setting devicevalue1,skipping:Item has already been added. Key in dictionary: 'devicevalue1' Key being added: 'devicevalue1'
                              des-28 17:46:25 Warning INI setting could not read setting devicetext1,skipping:Item has already been added. Key in dictionary: 'devicetext1' Key being added: 'devicetext1'
                              des-28 17:46:25 Warning INI setting could not read setting devicename1,skipping:Item has already been added. Key in dictionary: 'devicename1' Key being added: 'devicename1'
                              des-28 17:46:25 Warning INI setting could not read setting pattern5,skipping:Item has already been added. Key in dictionary: 'pattern5' Key being added: 'pattern5'
                              des-28 17:46:25 Warning INI setting could not read setting pattern4,skipping:Item has already been added. Key in dictionary: 'pattern4' Key being added: 'pattern4'
                              des-28 17:46:25 Warning INI setting could not read setting pattern3,skipping:Item has already been added. Key in dictionary: 'pattern3' Key being added: 'pattern3'
                              des-28 17:46:25 Warning INI setting could not read setting pattern2,skipping:Item has already been added. Key in dictionary: 'pattern2' Key being added: 'pattern2'
                              des-28 17:46:25 Warning INI setting could not read setting pattern1,skipping:Item has already been added. Key in dictionary: 'pattern1' Key being added: 'pattern1'
                              des-28 17:46:25 Warning INI setting could not read setting devicemode,skipping:Item has already been added. Key in dictionary: 'devicemode' Key being added: 'devicemode'
                              des-28 17:46:25 Warning INI setting could not read setting useragent,skipping:Item has already been added. Key in dictionary: 'useragent' Key being added: 'useragent'
                              des-28 17:46:25 Warning INI setting could not read setting options,skipping:Item has already been added. Key in dictionary: 'options' Key being added: 'options'
                              des-28 17:46:25 Warning INI setting could not read setting password,skipping:Item has already been added. Key in dictionary: 'password' Key being added: 'password'
                              des-28 17:46:25 Warning INI setting could not read setting username,skipping:Item has already been added. Key in dictionary: 'username' Key being added: 'username'
                              des-28 17:46:25 Warning INI setting could not read setting encoding,skipping:Item has already been added. Key in dictionary: 'encoding' Key being added: 'encoding'
                              des-28 17:46:25 Warning INI setting could not read setting textfile,skipping:Item has already been added. Key in dictionary: 'textfile' Key being added: 'textfile'
                              des-28 17:46:25 Warning INI setting could not read setting path,skipping:Item has already been added. Key in dictionary: 'path' Key being added: 'path'
                              des-28 17:46:25 Event Running script in background: C:/Program Files (x86)/HomeSeer HS3/scripts/Jon00DataScraper.vben("Main","2")
                              des-28 17:46:25 Event Event Trigger "System Smappee Grab"

                              Comment


                                Do you have the Jon00DataScraper.vben file in your scripts directory?

                                What OS and version of HS are you running?

                                It looks like you have 2 [Grab2] entries in your ini file. Please post it here.
                                Jon

                                Comment

                                Working...
                                X