Jon00 DataScraper/JSON Parser Script For Homeseer 3 and Homeseer 4

[jpape]

Yes i have. My ini looks like this with my actual ID and Password removed.


[Grab1]
Path=https://monitor.us.sunpower.com/#/login
TextFile=0
Encoding=
Username=xx@xx.com
Password=xxxxxxxxx
Options=
UserAgent=
Devicemode=0
StripHTML=1

Pattern1=(?s)"sp-energy-kws">(.*?)</div>
Pattern2=
Pattern3=
Pattern4=
Pattern5=

DeviceName1=
DeviceText1=
DeviceValue1=
DeviceImage1=
Speakbutton1=1
TriggerString1=
SearchMode1=1
TriggerEvent1=

[jon00]

As per the docs, it may be that web logins will not work. You can see if you have returned data by setting Textfile=1 and seeing if you have a grab1.txt file located in your Homeseer HS3\Data\Jon00\datascraper folder.

[jpape]

Jon,

This is what i get.

[jpape]

Hi Jon,

Did the file I attached show any progress?


Sent from my iPad using Tapatalk

[jon00]

It looks like it logged on correctly, but requires browser interaction to run JavaScript on the page. I'm afraid there is nothing much more that can be done.

[Leylander]

Linux (pi) install

"Install/Update of package Jon00DataScraperHS3 failed."

I copied over the updater_override.txt after I updated the zip file text and name.

What may be wrong?

[jon00]

I'm not sure what you have done. The package does not use the Homeseer installer and does not come with a updater_override.txt file.

Unzip the download file preserving the file structure and then read the PDF file in the docs folder. You can then manually copy the files to the correct HS directories as per the docs.

[Malosa]

Hi Jon,

I was trying to scrape the lightning strikes (distance) from weatherbug.
When i check source in the dom explorer everything is in there, but when i scrape i get a different scrape , is there a workarround to scrape from the dom explorer?

This is the link
https://weather.weatherbug.com/alerts/spark/


Sended from far far away

[jon00]

If you cannot see the data when viewing the page source, then it cannot be scraped with this.

You seem to have a nasty storm over the the Netherlands. It does not want to move...

[Malosa]

When i check the page source its there , but when i scrape and i check the txt file other data is in there.


Yeah the storm in the netherlands was really really bad , but its gone today ,
Finaly we can prepair for the summer

If you cannot see the data when viewing the page source, then it cannot be scraped with this.

You seem to have a nasty storm over the the Netherlands. It does not want to move...

Sended from far far away

[Pete]

Hello Jon,

Been wanting to use your Datascraper script now for many months...

Here right now looking for one UV value...from this text file that exists on one web link....doing similar with other web based text files in the future....

NOAA/EPA ULTRAVIOLET INDEX /UVI/ FORECAST

NOAA/EPA ULTRAVIOLET INDEX /UVI/ FORECAST                        
NWS CLIMATE PREDICTION CENTER CAMP SPRINGS MD                    
136 PM EDT WED JUL 27 2016                                       
                                                                 
VALID JUL 28 2016 AT SOLAR NOON /APPROXIMATELY NOON              
LOCAL STANDARD TIME OR 100 PM LOCAL DAYLIGHT TIME/               
                                                                 
THE UV INDEX IS CATEGORIZED BY THE WORLD HEALTH ORGANIZATION     
AS FOLLOWS:                                                      
           UVI             EXPOSURE LEVEL                        
           0 1 2              LOW                                
           3 4 5              MODERATE                           
           6 7                HIGH                               
           8 9 10             VERY HIGH                          
           11 AND GREATER     EXTREME                            
                                                                 
FOR HEALTH RELATED ISSUES...CONTACT EPA AT 1-800-296-1996        
FOR TECHNICAL INFORMATION ABOUT THE UV INDEX....                 
GO TO THE NATIONAL WEATHER SERVICE UV INDEX WEB PAGE:            
dubdubdub.CPC.NCEP.NOAA.GOV/PRODUCTS/STRATOSPHERE/UV_INDEX            
                                                                 
CITY               STATE  UVI       CITY               STATE  UVI
ALBUQUERQUE          NM    12       LITTLE ROCK          AR     9
ANCHORAGE            AK     4       LOS ANGELES          CA    11
ATLANTIC CITY        NJ     9       LOUISVILLE           KY     5
ATLANTA              GA     9       MEMPHIS              TN     9
BALTIMORE            MD     8       MIAMI                FL    12
BILLINGS             MT     9       MILWAUKEE            WI     8
BISMARCK             ND     8       MINNEAPOLIS          MN     8
BOISE                ID    10       MOBILE               AL    11
BOSTON               MA     9       NEW ORLEANS          LA    11
BUFFALO              NY     7       NEW YORK             NY     9
BURLINGTON           VT     8       NORFOLK              VA    10
CHARLESTON           WV     5       OKLAHOMA CITY        OK    11
CHARLESTON           SC    11       OMAHA                NE     9
CHEYENNE             WY    10       PHILADELPHIA         PA     9
[B]CHICAGO              IL     9[/B]       PHOENIX              AZ    11
CLEVELAND            OH     8       PITTSBURGH           PA     7
CONCORD              NH     9       PORTLAND             ME     9
DALLAS               TX     9       PORTLAND             OR     9
DENVER               CO    11       PROVIDENCE           RI     8
DES MOINES           IA     5       RALEIGH              NC    10
DETROIT              MI     9       SALT LAKE CITY       UT    10
DOVER                DE     9       SAN FRANCISCO        CA    10
HARTFORD             CT     9       SAN JUAN             PU    12
HONOLULU             HI    13       SEATTLE              WA     8
HOUSTON              TX     9       SIOUX FALLS          SD     9
INDIANAPOLIS         IN     9       ST. LOUIS            MO     9
JACKSON              MS     9       TAMPA                FL    11
JACKSONVILLE         FL    11       WASHINGTON           DC     7
LAS VEGAS            NV    10       WICHITA              KS    10

Maybe one or two values but starting out with one...config dot ini file looks like this so far. Am I correct configuring the values? Only looking for one value / one variable for the time bean.

[Grab1]
Path=hxxp://www.cpc.ncep.noaa.gov/products/stratosphere/uv_index/bulletin.txt
TextFile=1
Encoding=
Username=
Password=
Options=
UserAgent=
Devicemode=0
StripHTML=1

[B]Pattern1=(?s)<td class="IL">.*?[/B]
Pattern2=(?s)<td class="text-center even-four">.*?<strong>(.*?)</strong>
Pattern3=(?s)<td class="text-center even-four">.*?<div>(.*?)</div>
Pattern4=(?s)<td class="levels">.*?background-color: #(.*?);"></div>
Pattern5=(?s)<h3><strong>Pollen Type:</strong>(.*?)</h3>

DeviceName1=Chicago UV
DeviceText1=[100] [200]<br>[0]
DeviceValue1=[0]
DeviceImage1=[300].png
Speakbutton1=1
TriggerString1=
SearchMode1=1
TriggerEvent1=

[jon00]

Pete, try this:

[Grab1]
Path=hxxp://www.cpc.ncep.noaa.gov/products/stratosphere/uv_index/bulletin.txt
TextFile=0
Encoding=
Username=
Password=
Options=
UserAgent=
Devicemode=0
StripHTML=0



Pattern1=CHICAGO.*?IL(.*?)PHOENIX
Pattern2=
Pattern3=
Pattern4=
Pattern5=

DeviceName1=Chicago UV
DeviceText1=[0]
DeviceValue1=[0]
DeviceImage1=
Speakbutton1=1
TriggerString1=
SearchMode1=
TriggerEvent1=

[Pete]

Thank you Jon!

Updated INI file and did a test run of the script. It created the device just fine.

See the following in the logs. Looks OK to me.

Guessing then just create a one time daily event and that is it eh?

Jul-28 11:33:31 AM 	  	Event 	Event Jon00UV UV Read triggered by the event page 'Run' button.
Jul-28 11:33:31 AM 	  	Event 	Event Trigger "Jon00UV UV Read"
Jul-28 11:33:31 AM 	  	Event 	Running script in background: /HomeSeer/scripts/Jon00DataScraper.vben
Jul-28 11:33:32 AM 	  	Jon00_DataScrape 	Virtual device created for Address Jon00-DS-001-1 with Reference: 1355

[jon00]

Yes, for that metric, once a day would be fine.

[Thickman]

Pollen scraper

Is everybody's pollen scraper working? It appears to have stopped retrieving data on 12/14.