Web Scraper Plug-in for HS3

[jrhubott]

In the next version I will add a option to not strip HTML content "clean" from the match like it is currently doing.

This will allow you to place anything into a homeseer device.

[jon00]

In the next version I will add a option to not strip HTML content "clean" from the match like it is currently doing.

This will allow you to place anything into a homeseer device.

How would that work when many pages use CSS, JavaScript etc to display content?

[will40]

I will add a

Username=<username>
Password=<password>

option to the next release. These options will work with basic, digest, NTLM, and Kerberos authentication methods. It will not work with proprietary web site logons.

I've tested it and I just need to get it packed up.

Sounds good. Great job.

Thanks

[jrhubott]

How would that work when many pages use CSS, JavaScript etc to display content?

I have a beta plugin that can capture a image off any web page. The web page is rendered by the IE engine and then the plugin can capture a rectangle portion of the screen and save it as a jpg that can be referenced by the HS device.

Depending on the interest I will think about tidying this one up. I currently only use it for one thing and it is pretty messy as it took some COM, pinvoke, and Windows API to make it work.

[jon00]

I have a beta plugin that can capture a image off any web page. The web page is rendered by the IE engine and then the plugin can capture a rectangle portion of the screen and save it as a jpg that can be referenced by the HS device.

Depending on the interest I will think about tidying this one up. I currently only use it for one thing and it is pretty messy as it took some COM, pinvoke, and Windows API to make it work.

Right, that makes more sense. As this is a web scraper plugin, I thought you were going to capture the html code and place that within a device string (with mixed results I expect!). Regarding the capture, I have provided a HTML to Image Utility for many years which does essentially the same thing as your solution.

[jmj09]

Nice add to HS, thanks for your work.
I retrieve very well a flooding level, it would be easier to handle numbers if it was stored in a value instead of a string status. My aim would be to trigger an alarm when exceeding a level.
I do not use HS speaking system I use an other system, I did not succeed in deleting the speak button I think it is created by plugin. Would it be possible to give user a choice about this speak button.
Well as it is this plugin pleases me, keep it simple, forget my wishes if they drive you towards a gaz plant.

Jean Marc

[jrhubott]

Nice add to HS, thanks for your work.
I retrieve very well a flooding level, it would be easier to handle numbers if it was stored in a value instead of a string status. My aim would be to trigger an alarm when exceeding a level.
I do not use HS speaking system I use an other system, I did not succeed in deleting the speak button I think it is created by plugin. Would it be possible to give user a choice about this speak button.
Well as it is this plugin pleases me, keep it simple, forget my wishes if they drive you towards a gaz plant.

Jean Marc

Good ideas. I will add options for both.

[jrhubott]

Right, that makes more sense. As this is a web scraper plugin, I thought you were going to capture the html code and place that within a device string (with mixed results I expect!). Regarding the capture, I have provided a HTML to Image Utility for many years which does essentially the same thing as your solution.

Thanks for letting me know - I will not continue with development of the FrameGrabbing plugin. I will give yours a shot over the next couple of weeks.

Jason

[jrhubott]

I have attached a new version that has the following features. Give them a try and let me know how they work. I will wrap them up into the next full version if they work well.

V1.0.2.0A 8/12/2014
--------------------------------------------
Added Color Console Logging
Add new options
PlaceStatusInValue=true/false Attempt to parse the match into a integer value and place in the Device value
DisableSpeak=true/false Do not create the Speak button/Device Action
Username= These options will work with basic, digest, NTLM, and Kerberos authentication methods. It will not work with proprietary web site logons
Password= These options will work with basic, digest, NTLM, and Kerberos authentication methods. It will not work with proprietary web site logons

[ESSCHENK]

Hello,

The new version is not starting

i saw that there are 2 config files

HSPI_WebScraper

and

HSP_Web Scraper

one with a space

Ed

[jmj09]

No luck either
Jean Marc

[will40]

I admit installing the upgrade was a bit clunky and the lack of a doc doesn't help matters but I replaced the EXEcutable and DLL files from the upgrade ZIP into the HS root directory, added the new option lines to my webscraper.ini file (no space between b and s) and it works like a charm.

Nice plug-in!

[jrhubott]

Sorry about the configuration file name change. I was getting the plugin ready for inclusion in the HS3 updater.

I will have a PDF manual in the next version also.


Sent from my iPad using Tapatalk HD

[ESSCHENK]

Hello,

Can someone help with the next page.
This is where my PV system reports.
I want some info scrape to my homeseer machine

http://www.pvoutput.org/list.jsp?id=31725&sid=29071

Generated, efficiency, peak output, peak time
I have no idea how to do this

Thanks in advanced,

Ed

[jrhubott]

Hello,

Can someone help with the next page.
This is where my PV system reports.
I want some info scrape to my homeseer machine

http://www.pvoutput.org/list.jsp?id=31725&sid=29071

Generated, efficiency, peak output, peak time
I have no idea how to do this

Thanks in advanced,

Ed

The following will extracted the data. Some of the features will require the latest plug-in version (ValueMultiplier)

[pvoutput-generated]
URL=http://www.pvoutput.org/list.jsp?id=31725&sid=29071
RegExSearch=Generation',\n.*\n.*\n.*,(.*?)]
RegExReplace=1
RefreshInterval=60
Enabled=true
Debug=true
PlaceStatusInValue=true
ValueMultiplier=1000

[pvoutput-efficiency]
URL=http://www.pvoutput.org/list.jsp?id=31725&sid=29071
RegExSearch=Efficiency',\n.*\n.*\n.*\n.*\n.*data.*{y:(.*?),
RegExReplace=1
RefreshInterval=60
Enabled=true
Debug=true
PlaceStatusInValue=true
ValueMultiplier=1000

[pvoutput-peakoutput]
URL=http://www.pvoutput.org/list.jsp?id=31725&sid=29071
RegExSearch='padding-right:35px'>(.*?)kW
RegExReplace=1
RefreshInterval=60
Enabled=true
Debug=true
PlaceStatusInValue=true
ValueMultiplier=1000

[pvoutput-peakoutput-time]
URL=http://www.pvoutput.org/list.jsp?id=31725&sid=29071
RegExSearch='padding-right:35px'>.*?<td.*?>(.*?)<
RegExReplace=1
RefreshInterval=60
Enabled=true
Debug=true