Jon00 DataScraper/JSON Parser Script For Homeseer 3 and Homeseer 4

[gregoryx]

Thanks SO much!

I went searching for other possible ways to get the MFI info. I may be able to get it by telnet directly to each device and send a command... but yuck, since it is quite a few commands per outlet per device... but it's probably do-able.

Then there's this: https://community.ubnt.com/t5/mFi/mP...76449#U1076449

That makes me think I can grab the data more simply if I can figure out what the heck all that means.

Is that something I might be able to do with this script? or is that more like a telnet thing? I don't get the "CURL" reference. But I'm sure you can see what I'm trying to do here... trying to grab power usage information and stuff it into virtual devices...

[Malosa]

jon can u help me with a scrape ? i was always scraping from facebook messages , now if i scrape it doesnt work , did they change something ?

i always scraped with this <p>(.*?)</p> if i check inspect element , i see the text normal in there ...

[jon00]

If you enable the Text file (Textfile=1), can you see the data in that?

[Malosa]

yes there is , but not the data i want (all other things besides the posts.) ... what i scrape its always the same.

its not protected .. with inspect element i can still see the text i need with no problems

i try with all facebook posts... still the same ,
because i almost use every scrape with a facebook page.. so i hope it can be fixed

If you enable the Text file (Textfile=1), can you see the data in that?

[jon00]

If you cannot see the data in the text file, it cannot be scraped. It sounds like they have converted the page to use in-line elements which can only be decoded by a web browser.

[Malosa]

ow thats really bad .... now many of my scrapes dont work .
and they even killed the rss pages also ..

thanks for the reply jon

kind regards

If you cannot see the data in the text file, it cannot be scraped. It sounds like they have converted the page to use in-line elements which can only be decoded by a web browser.

[Malosa]

jon ,

i fixed it , i use a third party website what embeds the posts so i scrape it from there, works perfect .

[ldragon99]

A Little Help

I am trying to scrap the Bus cancellations from the upper right corner of this page:

https://transinfobhn.ca/Default.aspx.

Looking at the code from the source the cancellations look like:
<code>

<table>
<tr>
<td><input type="image" name="Btn_Cancellations" id="Btn_Cancellations" src="images/Cancellations_40.jpg"onclick="javascript:WebForm_DoPostBackW ithOptions(new WebForm_PostBackOptions(&quot;Btn_Cancellations&quot;, &quot;&quot;, false, &quot;&quot;, &quot;Cancellations.aspx&quot;, false, false))" style="border-width:0px;" /></td>
<td> <a id="Link_Cancellations" class="Delays" href="Cancellations.aspx">Delays &amp; Cancellations</a></td>
</tr>
</table>

</code>

the line that changes is:

<code>

<td> <a id="Link_Cancellations" class="Delays" href="Cancellations.aspx">Delays &amp; Cancellations</a></td>

</code>

[jon00]

I'm not sure what your question is?

I see they provide a RSS feed. It may be better to use that. If you download my RSS reader, you can add filtering to only display feeds that contain certain keywords.

[ldragon99]

I cannot get the scrap to pull the data. I am sure I formatted wrong

[jon00]

In the code you posted, what are you trying to scrape? Please highlight it in red.

[bmora]

Is there a way to create different devices if the pattern is the same? For example you have: <description> TEXT I WANT IN DEVICE </description>

Is there a way to pull out multiple instances using the same pattern? Maybe using a filter on the text I want for each device?

[jon00]

A pattern will automatically store all the data matches. You can then configure up to 100 devices per scrape with the data obtained.

Filtering would not be easy as you construct each device with the specific data from the scrape.

[ldragon99]

In this line:

<td> <a id="Link_Cancellations" class="Delays" href="Cancellations.aspx">Delays &amp; Cancellations</a></td>

I want: Delays &amp; Cancellations

This changes, I also beleive the id and class change

[jon00]

How about:

Pattern1=(?s)href="Cancellations.aspx">(.*?)</a></td>