Announcement

Collapse
No announcement yet.

Jon00 DataScraper/JSON Parser Script For Homeseer 3 and Homeseer 4

Collapse
This topic is closed.
X
This is a sticky topic.
X
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

    #61
    yes it gets populated perfectly


    i can you send the link in a pm if u want.
    Preferred -> Jon's Plugins, Pushover, Phlocation, Easy-trigger,
    Rfxcom, Blade Plugins, Pushbullet, homekit, Malosa Scripts




    HS3Pro 4.1.14.0 on windows 10 enterprise X64 on hp quadcore laptop 8 GB.

    Comment


      #62
      If it is populated perfectly, then it is just a matter of getting the pattern correctly defined for your extraction. Send me a copy of the text file and let me know what information you are trying to extract.
      Jon

      Comment


        #63
        i got it almost working , ive try it many times ,

        my scrape has text behind it .</td></tr><tr><td> how can i get rid of it ?

        i send u the text file


        Originally posted by jon00 View Post
        If it is populated perfectly, then it is just a matter of getting the pattern correctly defined for your extraction. Send me a copy of the text file and let me know what information you are trying to extract.
        Preferred -> Jon's Plugins, Pushover, Phlocation, Easy-trigger,
        Rfxcom, Blade Plugins, Pushbullet, homekit, Malosa Scripts




        HS3Pro 4.1.14.0 on windows 10 enterprise X64 on hp quadcore laptop 8 GB.

        Comment


          #64
          email sended.
          Preferred -> Jon's Plugins, Pushover, Phlocation, Easy-trigger,
          Rfxcom, Blade Plugins, Pushbullet, homekit, Malosa Scripts




          HS3Pro 4.1.14.0 on windows 10 enterprise X64 on hp quadcore laptop 8 GB.

          Comment


            #65
            this in my scrape : .</td></tr><tr><td>

            and showing in homeseer like : </td></tr><tr><td>

            But what encoding is it ? Im using utf-8

            i cant get rid of that code , it changes every time , maybe if i know wwhat encoding it is i can try that.
            Preferred -> Jon's Plugins, Pushover, Phlocation, Easy-trigger,
            Rfxcom, Blade Plugins, Pushbullet, homekit, Malosa Scripts




            HS3Pro 4.1.14.0 on windows 10 enterprise X64 on hp quadcore laptop 8 GB.

            Comment


              #66
              Can I scrape this site?

              I have been trying to figure out how to scrape this site but most of the elements I want to access - such as the instantaneous power - I can't actually find on the page. I think it is pulling images from another location or something? Any help would be great. The web page and html are below. Thanks.
              Attached Files

              Comment


                #67
                Looks like the page uses inline elements so cannot be scraped.
                Jon

                Comment


                  #68
                  Is there a way to capture it in image form?

                  Comment


                    #69
                    You can try my HTML to Image Utility. Make sure your copy of IE on your HS3 has Javascript enabled.
                    Jon

                    Comment


                      #70
                      Thanks for the advice. I tried the html to image - problem is that I have to login. I tried using the scraping script to force a login but this didn't seem to work. Is it a safe assumption that if I run the scraping script to login, and then the html to image script after it should still be logged in?

                      Comment


                        #71
                        No, that won't work as they use different methods to access the page.

                        The only other thing I have in the toolbox is this.
                        Jon

                        Comment


                          #72
                          I've got the sample deal working. Read all the documentation. I'm certainly going to stumble on the RegEx stuff, but your examples are excellent; I feel like I may actually be able to make sense of it all.

                          But the two cases I'm trying seem to have two different problems that I fear may not be possible.

                          1- Ubiquiti MFI Controller interface has all the info and it looks easy enough to grab... but I can't get any content from the https-only site which requires a password. How might I go about testing to make sure it's working with whatever method the script is trying to read?

                          2- Scrape of my Patiobar / pianobar / Pandora web interface (running on a Raspberry Pi) doesn't yield the content I'm looking for - artist, title, album, cover-art, etc. It doesn't seem to be in the grab2.txt file... just references to it as {{title}} for example. When I do "view source" in Chrome, it matches what's getting into the grab2.txt file. But the info /does/ show if I do "develop tools" (F12) in Chrome.

                          The "page source" content (and grab2.txt):
                          Code:
                          <!DOCTYPE html>
                          <html lang="en" ng-app="patiobarApp">
                          	<head>
                          		<meta charset="utf-8">
                          		<meta http-equiv="X-UA-Compatible" content="IE=edge">
                          		<meta name="viewport" content="width=device-width, user-scalable=no">
                          		<title>Patiobar</title>
                          		<link rel="stylesheet" href="/css/bootstrap.min.css">
                          		<link rel="stylesheet" href="/css/main.css">
                          	</head>
                          	<body>
                          		<nav class="navbar navbar-inverse" role="navigation" ng-controller="StationController">
                          			<div class="container-fluid">
                          				<div class="navbar-header">
                          					<button type="button" class="navbar-toggle" data-toggle="collapse" data-target="#stations">
                          						<span class="sr-only">Toggle Stations</span>
                          						<span class="icon-bar"></span>
                          						<span class="icon-bar"></span>
                          						<span class="icon-bar"></span>
                          					</button>
                          					<p class="navbar-brand">Patiobar</p>
                          				</div><!-- End navbar-header -->
                          	
                          				<div class="text-center collapse navbar-collapse" id="stations">
                          					<button type="button" ng-repeat="station in stations" class="btn btn-default navbar-btn btn-sm" ng-click="changeStation(station.id);">{{station.name}}</button>
                          				</div> 
                          			</div> <!-- End container-fluid -->
                          		</nav> 
                          	
                          		<div class="container" ng-controller="SongController">
                          			<div id="player" class="center-block">
                          				<div class="text-center">
                          					<img class="center-block img-rounded img-responsive" id="coverArt" ng-src="{{src}}" alt="{{alt}}">
                          	
                          					<div id="details">
                          						<p id="title" class="media-heading">{{title}}</p>
                          						<p id="albumartist">{{albumartist}}</p>
                          					</div>
                          	
                          	     		<div id="controls">
                          						<button type="button" id="love" class="pull-left btn btn-default" ng-click="sendCommand('+');"> <span class="glyphicon glyphicon-thumbs-up"></span> </button>
                          						<button id="pauseplay" type="button" class="btn btn-default" ng-click="sendCommand('p');"> <span class="glyphicon glyphicon-pause"></span> </button>
                          						<button type="button" class="btn btn-default" ng-click="sendCommand('n');"> <span class="glyphicon glyphicon-fast-forward"></span> </button> 
                          						<button type="button" class="btn btn-default" ng-click="sendCommand('(');"> <span class="glyphicon glyphicon-volume-down"></span> </button>
                          						<button type="button" class="btn btn-default" ng-click="sendCommand(')');"> <span class="glyphicon glyphicon-volume-up"></span> </button>
                          						<button type="button" class="pull-right btn btn-default" ng-click="sendCommand('-');"> <span class="glyphicon glyphicon-thumbs-down"></span> </button>
                          					</div>
                          				</div> <!-- End text-center -->
                          			</div> <!-- End Player -->
                          		</div> <!-- End container (SongController) -->
                          
                          
                          		<script src="/socket.io/socket.io.js"></script>
                          		<script src="/js/jquery-2.1.1.min.js"></script>
                          		<script src="https://ajax.googleapis.com/ajax/libs/angularjs/1.2.22/angular.min.js"></script>
                          		<script src="/js/bootstrap.min.js"></script>
                          		<script src="/js/main.js"></script>
                          	</body>
                          </html>
                          An example of what I see in the developer stuff that I want to scrape:
                          Code:
                          <img class="center-block img-rounded img-responsive" id="coverArt" ng-src="http://cont-sv5-1.pandora.com/images/public/rovi/albumart/1/2/9/6/074640926921_500W_500H.jpg" alt="Sounds Of Silence" src="http://cont-sv5-1.pandora.com/images/public/rovi/albumart/1/2/9/6/074640926921_500W_500H.jpg">

                          Comment


                            #73
                            1) If you can get a grabX.txt file with content, you know that your grab is successful. I assume you have inserted your user/ password in the config?

                            2) It looks like the web page is using Ajax and in-line elements which unfortunately cannot be scraped. Only items that can be seen via the browser's 'view source' can be scraped.
                            Jon

                            Comment


                              #74
                              Wow. Super fast response. Thank you!

                              Can't get that MFI one to get a "grab" file... and now that I look in the view-source, the content I want there is similarly not present. Bummer.

                              Any suggestions as to how I might go about getting this stuff otherwise? The browser obviously "sees" it to build the page... grrr.

                              Got a script that can grab content from the "rendered" version of the page?

                              Comment


                                #75
                                The browser is rendering the in-line elements which is why you see it. The script only has capability to download the HTML source.

                                I cannot think of another way to get direct access to the text. I do have utilities to grab/crop an image of a web page. This can be inserted in a virtual device or displayed in HSTouch. It may be of no use as you cannot act on it (as it is just an image).
                                Jon

                                Comment

                                Working...
                                X