Jon00 DataScraper/JSON Parser Script For Homeseer 3 and Homeseer 4

[joesch]

Hello Jon,
I still didnt find any solution for my problem to display the images grabbed with your script (url in device string) in hs3touch. perhaps you or someone else can help me with that. im not sure if this here is the right area for or should I asked for help in a Hstouch-Thread?
If here is someone can help me I try to explain my problem:

I would like to have a kind of EPG of the current running program of my favourite german tv channels displayed on hstouch. Jon made the script-config for me, im very glad about this, its here:

[Grab2]
Path=http://www.tvtoday.de/programm/tv-programm-jetzt-im-tv/
TextFile=0
Encoding=UTF-8
Username=
Password=
Options=
UserAgent=
StripHTML=0
Pattern1=(?s)<p class="h7">(.*?)</p>
Pattern2=(?s)<p class="lapse">(.*?)</p>
Pattern3=(?s)<div class="cell title">.*?<img src="(.*?)"

DeviceMode=2
DeviceName1=DAS ERSTE
DeviceText1=<div><iframe src='[200]' marginwidth='0' marginheight='0' width='159px' height='109px' scrolling='no' frameborder='no'></iframe></div><div>[100] [0]</div>
DeviceValue1=[hour][minute][second]


DeviceName2=ZDF
DeviceText2=<div><iframe src='[201]' marginwidth='0' marginheight='0' width='159px' height='109px' scrolling='no' frameborder='no'></iframe></div><div>[101] [1]</div>
DeviceValue2=[hour][minute][second]


DeviceName3=RTL
DeviceText3=<div><iframe src='[202]' marginwidth='0' marginheight='0' width='159px' height='109px' scrolling='no' frameborder='no'></iframe></div><div>[102] [2]</div>
DeviceValue3=[hour][minute][second]



It works great in the web view but actually I do not use the Web-Interface but only for configuring. If I try to display the status tracking of the device as picture in hstouch, there is an fault message /http://www...jpg cannot be found. Someone has helpful ideas? I wish there would be an easy way, a way without having to run another script.

regards joesch

[jon00]

Apologies but have been busy. I'll contact you via email as before.

[pagewrite]

Dynamic Path string

Hi Jon

I can't believe I've been struggling on trying to do some web scraping for so long before I discovered this excellent plugin of yours! Thank you so much for writing it!

Anyway, having discovered (and now avidly using) your plugin, I have a problem and wondered whether there was a work-around or some modification you could do.

The problem I have is that the URL I want to put the in Path parameter includes today's date. Naturally, if I hard-code this in the Path parameter, I'm always stuck with a result that quickly becomes out of date.

The URL I'm trying to scrape is:

http://live.myimmersun.com/graph_day...8-06-17&end=31

The end parameter always appears to be 31 (for some reason), but the start parameter naturally needs to change each day in the format:

dd-mm-yy

I've noticed it's very picky and always requires two digits for each field.

Is there a way to dynamically update the Path parameter in your script? Or, can I use a device string and substitute that? (Come to think of it, that would be quite a useful feature).

Many thanks

Ian

[jon00]

No need to change anything:

Just run the following script before you trigger the datascraper script:

    Sub Main(ByVal Parm As Object)

        Dim dt As String = DateTime.Now.ToString("dd-MM-yy")
        Dim Path As String = "http://live.myimmersun.com/graph_day300.htm&start=" & dt & "&end=31"
        hs.SaveIniSetting("[COLOR="Red"]Grab1[/COLOR]", "Path", Path, "Jon00DataScraper.ini")

    End Sub

Remember to change the Grab1 entry (shown in red) to your actual Grab entry.

[pagewrite]

That's great. Thanks Jon. I had no idea about the hs.SaveIniSetting function, so that's useful to store away in the grey matter!

Thank you.

Ian

[Malosa]

Hi Jon ,

a quick question

FOr example :
my device contains this text : this test is 100 percent
how to replace everything and keep the 100 ?


I did try this ,

DeviceValue2=[0][replace "this test is ",""][replace "percent",""]

But it doesnt work, any idea what im doing wrong ?

regards
and have a nice day

[Malosa]

everytime when i put it on this board it solves automatically ...

it works now.. dont know why it did not work before , i tested 2 hours..

regards

[mda]

Looking up text based on scraped values

Hi Jon,

I am scraping values from my pool control system and i need to lookup text based on the scraped value for my HS3 device Status fields.

My grabs are working fine:

[Grab1Data]
0=0
100=2250
200=523
[Grab2Data]
0=9
101=5
102=0
201=102
100=30
200=102

and, for example, the Status of the device i will create based on my Grab2 tag [0] would be:

0 = OK
1 = Communication Error
2 = Check Flow
3 = General Fault
9 = Offline

Can i just do it by setting up Status Text in the Status Graphics tab of the virtual device or do i need to do something in the .ini file (if so, would you mind pointing me in the right direction)?

Thanks!

-mda

[jon00]

If you set DeviceMode=2 and DeviceValue1 to the Grab2 tag [0] and leave the DeviceText1 entry blank, you can set both the Status Text & Status Graphics for the virtual device as you wish.

[mda]

If you set DeviceMode=2 and DeviceValue1 to the Grab2 tag [0] and leave the DeviceText1 entry blank, you can set both the Status Text & Status Graphics for the virtual device as you wish.

Super, thanks!

[Freddan101]

I'm trying to get this script going but seem to fail for some reason. In which folder is the Jon00DataScraperData.ini supposed to be created?

[jon00]

It would be in the config folder.

[Freddan101]

Thank you. I got it working with a more simple test.

I also realized that the Wunderground link in the example ini file is no longer valid why the script failed to fetch any data. No biggie though...

[Freddan101]

I'm struggling with a website where I would like to fetch the current electricity price. I've tested my expression at regex101.com and I get the correct match "38.08". But using the script I don't get anything. I tried UTF-8 as well but with the same result.

Is there something wrong witch my expression?

<div class="elspot-header">
							Aktuellt timpris (13:37)
						</div>
						<div class="elspot-content">
							38,08 öre/kWh*
						</div>

[jon00]

Difficult to assist without looking at the actual web data.

In the ini file, set Textfile=1 and then run the script. In your Homeseer HS3\Data\Jon00\datascraper, please send me the grabX.txt file where X is the grab number you have used.