Jon00 DataScraper/JSON Parser Script For Homeseer 3 and Homeseer 4

[deanrparry]

I've sent you a PM.

An official thank you again for your time. I've had loads of fun today

[deanrparry]

A quick question and quite an unusual one ;-)

The datascraper works perfectly with my xiaomi robot vacuum and many other devices now including gps trackers etc...

But one small thing for consistency, what is the best practice to keep a simple on/off button or something similar in line with the results scraped?

For example the status of the vacuum = on / off but we have an opportunity to actually switch it on / off via the api

https://home-assistant.io/components...m.xiaomi_miio/

I have got around it by using curl and events for on / off and a virtual device to trigger the event i need. It just feels long winded and was not sure if something simple could be implemented into datascraper to add basic control once the data has been grabbed?

Like i say i expect nothing and it might not fit in with the idea behind the plugin full stop.

Many thanks.

[jon00]

Dean,

It would be out of scope. Datascraper was designed to extract data and has no routines to post/put data back.

[deanrparry]

Dean,

It would be out of scope. Datascraper was designed to extract data and has no routines to post/put data back.

Much appreciated for your honesty i thought as much... i am very surprised nobody has put together a simple plugin to allow this to happen as it would help with ha to ha interaction etc...

The curl setup i have just seems so messy but does work

[fungun]

Don't know how long this has been happening, but I just noticed the Data Scraper for my lakes info is not working.
I checked the website, don't see where anything has changed.
Now I am getting -999 for my values for lake temp and lake stage feet.
Here is the section of the ini file. The data ini file shows no data for these 2 grabs.

Thanks,
Tim

[Grab5]
Path=https://waterservices.usgs.gov/nwis/iv/?format=rdb&sites=06604200&parameterCd=00060,00065,00010&siteType=LK&siteStatus=all
TextFile=0
Encoding=
Username=
Password=
Options=
UserAgent=
Devicemode=2
StripHTML=1

Pattern1=(?s)CST(.*?)P(.*?)P
Pattern2=
Pattern3=
Pattern4=
Pattern5=

DeviceName1=Lake Temp Value
DeviceText1=[0]
DeviceValue1=[0]
DeviceImage1=
Speakbutton1=0
TriggerString1=
SearchMode1=1
TriggerEvent1=

DeviceName2=Lake Stage Value
DeviceText2=[1]
DeviceValue2=[1]
DeviceImage2=
Speakbutton2=0
TriggerString2=
SearchMode2=1
TriggerEvent2=

[Grab6]
Path=https://waterservices.usgs.gov/nwis/iv/?format=rdb&sites=06604000&parameterCd=00060,00065,00010&siteType=LK&siteStatus=all
TextFile=0
Encoding=
Username=
Password=
Options=
UserAgent=
Devicemode=2
StripHTML=1

Pattern1=(?s)CST(.*?)P(.*?)P
Pattern2=
Pattern3=
Pattern4=
Pattern5=

DeviceName1=SL Lake Temp Value
DeviceText1=[0]
DeviceValue1=[0]
DeviceImage1=
Speakbutton1=0
TriggerString1=
SearchMode1=1
TriggerEvent1=

DeviceName2=SL Lake Stage Value
DeviceText2=[1]
DeviceValue2=[1]
DeviceImage2=
Speakbutton2=0
TriggerString2=
SearchMode2=1
TriggerEvent2=

[jon00]

Try changing:

Pattern1=(?s)CST(.*?)P(.*?)P

To:

Pattern1=(?s)CDT(.*?)P(.*?)P

[fungun]

Oh duh

Thanks Jon

Tim

[rmorton27]

Getting started pains

Jon,

My config file looks like this:

[Grab1]
Path=https://www.myinfinitytouch.carrier.com/MyLocations
TextFile=1
Encoding=
Username=[deleted]
Password=[deleted]
Options=
UserAgent=
Devicemode=2
StripHTML=1

Pattern1=(?s)<div class="grid-col col-1-4"><strong class="label">System Mode: </strong><br />(.
*?)</div>

DeviceName1=System Mode:
DeviceText1=
DeviceValue1=[0]
DeviceImage1=
Speakbutton1=
TriggerString1=
SearchMode1=1
TriggerEvent1=

And I run the event manually and it created the device but the data is missing. I can see the file contents in grab1.txt so I know I'm getting to the website and my credentials are being processed.

Here is the page's source that I am working with right now.

<div class="grid system-status">
<div class="grid-col col-1-4">
<strong class="label">System Mode: </strong><br />Heat
</div>
<div class="grid-col col-1-4">
<strong class="label">Current Operating Status: </strong><br />Heatpump Heating
</div>
<div class="grid-col col-1-4">
<strong class="label">Humidity Target: </strong><br />30%
</div>
<div class="grid-col col-1-4">
<strong class="label">Humidity: </strong><br />35%
</div>
</div>

Any help really appreciated.

[jon00]

Try something like:

Pattern1=(?s)System Mode:.*?>(.*?)</div>

[rmorton27]

Jon,

Thanks for the suggestion but my results remain the same. I did notice my ask was poorly framed, let me try again.

I am seeking to scrape the data from the web page that provide me the system setting for my HVAC system.

The variables for the specific field I am searching ( System Mode: ) are heat, cool, auto, off, fan only. In the current example [System Mode: </strong><br />Heat </div>] , the variable I desire is Heat.

The results in the hs device created are:
Device Name = System Mode:
Status = -999
Value = -999
String = -

My understanding is [0] should have the variable data I want. Any further suggestions?

Thanks

[jon00]

Please post the text file which is generated (See TextFile entry in the docs) and I'll take a look when I can.

[rmorton27]

Jon,

Your suggestion about uploading the grab file made me go back through the steps to try and provide an accurate result based the current config. Then when I looked at the contents of the grab file and found the expected stings are not present. My suspicion is the page is not accepting the web page credentials that you warned about in the doc. Is there a method to confirm this suspicion?

Thanks

[jon00]

It sounds like you have already confirmed the issue. If you cannot see a grab of the page in the text file, it has not logged in. Not much I can do to assist unfortunately.

[deanrparry]

I've searched Google until i took it offline!

I am scraping data on gps api and it returns an address BUT on the odd occasions between points/coords the address can return null

How the heck can i keep my existing value eg the previous address record without the device being overwritten by null

I have tried for DAYS i just simply do not understand regexp enough OR even if it is possible?

Surely it has to grab something due to the nature of regexp? or am i missing and !donotmatch secret somewhere?

Many thanks

[jon00]

Regex only does what it is told! First thing would be to see what is downloaded when you get null. Set TextFile=1 and then when it next happens look at the text file generated to see what was returned.