Announcement

Collapse

Contacting HomeSeer This Week

HomeSeer is open and operational this week. All orders are being processed and shipped as usual. However, some staff are working from home. If you need to contact HomeSeer for support or customer service, please use our Email or Chat options. https://homeseer.com/contact-us/
See more
See less

Shifting to Nginx /1

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Shifting to Nginx /1

    Following a previous thread.
    While increasing my skills on HS3 script (I think about a year ago), I was struggling with memory leaks which overcame all my attempts to obtain a stable solid system let's say 2 weeks (my longest holliday) without being necessary to return to HS3 server management (I think its better now with last beta's). In my quest to memory leaks, garbage collector, the release of the objects,... I found that the problem was related to the behavior of the .NET environment in HS3.
    At this point came the idea of ​​keeping HS3 to its native work, event management, and other technological ZWave links, ... and transfer scripts to a more reliable engine. Since I had grown Javascript in my pages, I started working with nodejs. At that time, the HS3 guys released a gold coin aka JSON API. Bingo!
    The last piece of the puzzle was a front-end server to cope securely with the World Wild Web since as you know, all ips and ports are chased at random or by systematic hunting. This front-end server would of course serve HS3 and nodejs applications.
    In the following messages, I will describe the general scheme and the centerpiece of the configuration of nginx which is a file named "nginx.conf"

    Remember, I am not involved in computers, or not even working in the sector. All this is stuf without government guarantee, its all your responsibility to put your system down if you make some attempts, even after my example as tips. This is not advise this is experience, the one stuff that do not transfer between people. I am sure my system is not perfect, hides holes, thus enhanceme"nt and criticism are very welcome.

    And last I get no responsabilities for headache you catch reading my english.

    Stateof the art : today I am running fine with a front end NGINX server, serving HS3, JSON HS3 API, my own express server api. Basic auth is fully implemented, as for SSL i did succeed with a self signed certificate but not with an external certificate provider...

    My server is a W10 system on a cheap laptop. A few watts consumption, an included battery, access with microsoft distant connexion.
    Node js, Nginx at their last versions.
    Coming with nodejs and Nginx you get fancy addons like PM2 that I use for processes management, monitoring, relaunch of faulty process, easy logging of server errors...
    And tons of scripts to help, on the central bank of the system called NPM, and much more for people who get time. I lack!

    Let's go
    Jean
    Last edited by jmj09; January 12th, 2016, 03:01 PM.

  • #2
    Shifting to Nginx /2

    I shall try to explain there the general scheme
    Code:
    First came the Wild WAN
    ||||||||||||||||||||||||
    First funnel is the router which allows only ports 80 and 443
         |         |   only 2 ports remain open to my fixed ip
        80       443   ports are transfered to nginx server by port management in the router.
    Second funnel is in first block of code from nginx .conf which rewrites http to https
                    |
                  443 only remains.
    Then the query from internet is parsed to see if it is static stuff
    Then if it is HS3 page
    Then json api calls
    Then rss calls (my own api)
    
    Static files are served straight away 
    aiming at port 443 (ssl) with max cache
    
    Some variable files (eg. jpeg images from my ipcam)
    aiming at port 443 with no cache
    
    HS3 pages are served through a reverse proxy (internal to nginx) 
    aiming at HS3 server on port 81 (not accessible from outside)
    
    json calls are served straight
    aiming at port 81 (the one from HS3 server)
    
    rss calls are served through a reverse proxy 
    aiming at my own nodejs server on port 1000 with no cache
    
    No pasword for local port is on in HS3 setup.
    Please see config here after
    Last edited by jmj09; January 12th, 2016, 03:15 PM.

    Comment


    • #3
      Shifting to Nginx /3

      nginx.conf file
      This is the file where you handle the behaviour of your server.
      I shall add more comments if there is questions about pieces of code.


      Code:
      worker_processes  1;
      error_log  'C:\\Program Files (x86)\\nginx\\logs\\error.log';
      pid        'C:\\Program Files (x86)\\nginx\\logs\\nginx.pid';
      events {
          worker_connections  1024;
      }
      
      http {
      
        include       mime.types;
        default_type  application/octet-stream;
        sendfile      on;
        keepalive_timeout  65;
      
      	# enable gzip compression
          gzip  on;
          gzip_static on;
          gzip_comp_level 6;
          gzip_min_length 1400;
          gzip_types mime.types;
          gzip_buffers  4 32k;
          gzip_vary  on;
          gzip_http_version 1.1;
          gzip_disable "MSIE [1-6]\.(?!.*SV1)";
      	# end gzip configuration
        
        #rewrites http to https
        server {
          listen      80;
          server_name your.server.com;
          rewrite     ^   https://$server_name$request_uri? permanent;
        }
        
        #Main https server job default port 443
        server {  
          listen  443 ssl;
          server_name  your.server.com;
          ssl_certificate 'C:\\Program Files (x86)\\nginx\\ssl\\your.server.com.crt';
          ssl_certificate_key 'C:\\Program Files (x86)\\nginx\\ssl\\your.server.com.deprotected.key';
          root 'C:\Program Files (x86)\HomeSeer HS3\html';
          add_header X-Whom direct;
          auth_basic  "Acces reserve";
          auth_basic_user_file  'C:\\Program Files (x86)\\nginx\\conf\\.htpasswd';  
          satisfy any;
          allow 127.0.0.1;
          #allow 192.168.0.10;
          deny  all;
          
          #Errors -> Locations fancy own error page
          error_page 404 /error_404.html;
            
       
      	  #Prevents hidden files (beginning with a period) from being served
      	  location ~ \/\. { access_log off; log_not_found off; deny all; }
      
          #serve homeseer files the ones without dot in the path
          location ~* (?<![\.\/]...)$ {
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;      
            proxy_pass http://127.0.0.1:81 ;
            add_header X-Whom HS3;
            expires -1;
          }
      
          #serve static files 
          location ~* ^.+\.(?:css|cur|js|png|gif|htc|ico|txt|xml|otf|ttf|eot|woff|svg)$ {
            try_files $uri $uri/ /error_404.html;
            add_header X-Whom all;
            add_header Cache-Control public;
            expires max;
          }
      
          #serve jpeg, csv, html variables files including jpeg from my IPcam
          location ~* ^.+\.(?:jpe?g|csv|html)$ {
            try_files $uri $uri/ /error_404.html;
            add_header X-Whom vary;
            expires -1;
          }
      
          # serve HS3 json api via proxy
          location ~* \/(JSON|json) {
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_pass http://127.0.0.1:81 ;
            add_header X-Whom json;
            expires -1;
          }
      
          # serve rss via proxy, this is my own api (express node js) to deal with csv files
          location ~* \/rss {
          proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_pass http://127.0.0.1:1000 ;
            add_header X-Whom rss;
            expires -1;			
          }
      
        }
      
      }
      Last edited by jmj09; January 12th, 2016, 03:17 PM.

      Comment


      • #4
        I just implemented this myself - thanks, it works great!
        I also extended it to give med an encrypted connection for Geofence with the PHLocation-plugin.

        Comment


        • #5
          This works!

          But I have problems serving asp pages when the asp page is specified. I use Jon00s Quick page builder (with the ASP option as it uses much less RAM). If I set it up as as the loading page, then it is displayed without problems.

          If I specify the filename, it doesn't work.

          So this url:
          https://myserver/ <- works, and linking to a asp page
          https://myserver/buildpage.asp <- doesn't work!
          https://myserver/buildpage.aspx <- works!

          What am I not getting?
          HSPro 3.0.0.458, Z-NET with Z-wave plugin 3.0.1.190, RFXCOM + 2x RFXtrx433E, HSTouch, Squeezebox plugin, iTach IP/WF2IR & GC-100-6 with UltraGCIR, BLDenon, NetcamStudio, Jon00s Webpage builder, Harmony Hub plugin, SCSIP (with FreePBX), Arduino plugin, IFTTT, Pushalot plugin, Device History plugin.
          Running on Windows 10 (64) virtualized
          on ESXi (Fujitsu Primergy TX150 S8).
          WinSeer (for Win10) - TextSeer - FitbitSeer - HSPI_MoskusSample

          Are you Norwegian (or Scandinavian) and getting started with HomeSeer? Read the "HomeSeer School"!

          Comment


          • #6
            Since the begining of my post, things evolved...
            Let's encrypt came, my skill enhanced, though I still don't wanna cry...

            Thus I am now in my own domain name, with better nginx conf file I hope.

            Let,s encrypt certificates are a pain to manage (for me) on the HS w10 server, I did not succeed in automating the process.

            Regular expression are at least as efficient as sudoku to protect your brain from aging...

            I do not fully understand the deny/allow priorities...

            You can get ratings about ssl security here https://www.ssllabs.com/ssltest/analyze.html.
            With the following conf file I get a "A" which is not optimum, I think you can reach "A+" in this rating

            Comments Welcome as usual, specially security issues that could be detected in my conf file.

            Jean


            Code:
            worker_processes  1;
            
            error_log   'C:\\nginx\\logs\\error.log debug';
            pid         'C:\\nginx\\logs\\nginx.pid';
            
            events {
              worker_connections  1024;
            }
            
            http {
              #csp
                server_tokens off;
                add_header X-Frame-Options SAMEORIGIN;
                add_header X-Content-Type-Options nosniff;
                add_header X-XSS-Protection "1; mode=block";
                add_header Content-Security-Policy "default-src 'self'; script-src 'self'; img-src 'self'; style-src 'self' ; font-src 'self'; child-src 'none'; object-src 'none'";
                ssl_session_cache shared:SSL:50m;
                ssl_session_timeout 5m;
                # enables server-side protection from BEAST attacks
                ssl_prefer_server_ciphers on;
                ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
                # ciphers chosen for forward secrecy and compatibility
                ssl_ciphers "ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA:ECDHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES256-GCM-SHA384:AES128-GCM-SHA256:AES256-SHA256:AES128-SHA256:AES256-SHA:AES128-SHA:DES-CBC3-SHA:HIGH:!aNULL:!eNULL:!EXPORT:!DES:!MD5:!PSK:!RC4";
                # enable ocsp stapling (mechanism by which a site can convey certificate revocation information to visitors in a privacy-preserving, scalable manner)
                resolver 8.8.8.8;
                add_header Strict-Transport-Security "max-age=31536000; includeSubdomains;";
              #end CSP
            
              include       mime.types;
              default_type  application/octet-stream;
              sendfile      on;
              keepalive_timeout  65;
            
            	# enable gzip compression
                gzip  on;
                gzip_static on;
                gzip_comp_level 6;
                gzip_min_length 1;
                gzip_types "*";
                gzip_buffers  4 32k;
                gzip_vary  on;
                gzip_http_version 1.1;
                gzip_proxied any;
            	# end gzip configuration
            
              server {
                listen 80;
               #redirect all to https
                rewrite_log on;
                server_name myown.domain.com;
                return 301 https://myown.domain.com$request_uri;
              }
            
              #Main https server job default port 443
              server {
                listen  443 ssl;
                ssl on;
                rewrite_log on;
                server_name myown.domain.com;
                ssl_certificate 'C:\\pathToNginx\\ssl\\fullChain.pem';
                ssl_certificate_key 'C:\\pathToNginx\\ssl\\privateKey.pem';
                root 'C:\Program Files (x86)\HomeSeer HS3\html';
                add_header X-Whom direct;
                auth_basic  "closed website";
                auth_basic_user_file  'C:\\pathToNginx\\conf\\.htpasswd';  
                satisfy any;
                allow 127.0.0.1;
                allow 192.168.0.1;
                allow 192.168.0.4;
            #single and subnet deny ips selected from frequent raiders (out of logs)
                deny 198.0.236.0/24;
                deny 194.28.115.231;
                deny 185.159.36.0/24;
                deny 185.159.37.0/24;
                deny 176.157.128.233;
                deny 40.77.167.0/24;
                deny 89.163.242.226;
                deny 188.0.236.176;
                deny  all;
            
                location @404 {
                  return 302 '\\error_404.html';
                }
                error_page 404 = @404;
            
            	  #Prevents hidden files (beginning with a period) from being served
            	  location ~ \/\. { access_log off; log_not_found off; deny all; }
            
                #serve homeseer reqs
                location ~* \/JSON\? {
                  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; 
                  proxy_intercept_errors on;      
                  proxy_pass http://127.0.0.1:81 ;
                  add_header X-Whom JSON;
                  expires -1;
                }
                
                #serve homeseer files the ones without dot, = and ? in the path
                location ~* ^[^=?.]*$ {
                  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; 
                  proxy_intercept_errors on;      
                  proxy_pass http://127.0.0.1:81 ;
                  add_header X-Whom HS3;
                  expires -1;
                }
                
                #serve static files 
                location ~* ^.+\.(?:css|cur|js|png|gif|htc|ico|txt|xml|otf|ttf|eot|woff|svg)$ {
                  try_files $uri /error_404.html;
                  add_header X-Whom Static;
                  add_header Cache-Control public;
                  expires max;
                }
            
                #serve jpeg, csv variables files including jpeg from my IPcam
                location ~* ^.+\.(?:jpe?g|csv|html)$ {
                  try_files $uri /error_404.html;
                  add_header X-Whom vary;
                  expires -1;
                }
            
                # serve rss via proxy, this is my own api (express node js) to deal with own json files
                location ~ \/=(rss|ver|mem|cpu|osc|pas)$ {
                  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                  proxy_intercept_errors on;
                  proxy_pass http://127.0.0.1:1000 ;
                  add_header X-Whom api1000;
                  expires max;			
                }    
              }
            }

            Comment


            • #7
              Shifting to Nginx /1

              Very well written post. I locked down and centralized all the internal resources I wanted to make available on the pubic Internet using squid's reverse proxy and a let's encrypt ACME client. I opted to host my solution on my PFsense firewall using readily available packages that run in the PFSense system (squid reverse proxy and ACME).

              I never made the connection that you could run scripts this way. Would you confirm that you are running scripts within node.js outside of NGINX?


              Sent from my iPhone using Tapatalk
              Last edited by Kerat; May 23rd, 2017, 09:34 PM.

              Comment


              • #8
                Hi Kerat
                As soon as you can build a running server you can proxy it with nginx. Could be Python, Ruby on rails and nowadays of course node js.

                Here after, code for a node js server on port 1000, which answers for several metrics in parallel with HS3. My lib for file and error management is lacking, this is to give a glance of easy scripting

                Of course you can embed HS3 and nodejs through Json hs3 api calls.
                At this time all my csv files are built with nodejs (json calling HS3) and no longer HS3 .net, no more memory trouble and system is stable. Node js is easy to debug. I am using pm2 for app management.

                Jean


                Code:
                const five = require('take-five');
                const os = require('os');
                const exec1 = require('child_process').exec;
                const exec2 = require('child_process').exec;
                const util = require('util');
                const myF = require('./noisetfunc.js');
                const server = five();
                
                //tested in test.js
                // different metrics on host cpus see node docs
                server.get('/=osc', function(req, res) {
                  try {
                    const osCpus = 	os.cpus();
                    res.send(osCpus);
                  } catch(err) {
                    myF.processError('api1000-server.get-osc ', err, 55);
                    res.send([{"error":"api1000-server.get-osc"}]);
                  }
                });
                
                //tested in test.js
                server.get('/=ver', function(req, res) {
                  try {
                    const verRes = process.versions;
                    res.send(verRes);
                  } catch(err) {
                    myF.processError('api1000-server.get-ver ', err, 55);
                    res.send([{"error":"api1000-server.get-ver"}]);
                  }
                });
                
                //get atmospheric pressure from a fine yocto sensor
                server.get('/=pas', function(req, res) {
                  const options = {
                    timeout: 5000,
                    killSignal: 'SIGKILL'
                  };
                  try {
                    exec2('YPressure METEO get_currentValue', options, function (err, stdout) {
                      const line = stdout.toString();
                      let results = line.slice(-11,-3);
                      isNaN(results) && (results = 0);
                      res.send(results);
                    });
                  } catch(err) {
                    myF.processError('api1000-server.get-pas ', err, 45);
                    res.send(0);
                  }
                });
                
                //
                server.get('/=rss', function (req, res) {
                  const options = {
                    timeout: 5000,
                    killSignal: 'SIGKILL'
                  };
                  exec1('tasklist', options, function (err, stdout) {
                    const lines = stdout.toString().split('\n');
                    const results = [];
                    try {
                      lines.forEach(function (line) {
                        const proc = line.substring(0, 26),
                          pid = line.substring(26, 35),
                          util1 = line.substring(66, 69),
                          util2 = line.substring(70, 73);
                        results.push([proc, pid, util1 + util2]);
                      });
                      res.send(results);
                    } catch(err) {
                      myF.processError('api1000-server.get-rss ', err, 55);
                      res.send([{"error":"api1000-server.get-rss"}]);
                    }
                  });
                });
                
                //tested in test.js
                server.get('/=cpu', function(req, res) {
                  try {
                    const CPUout = process.cpuUsage();
                    const CPUUSER = + (parseFloat(CPUout.user) / 1000000).toFixed(2);
                    const CPUSYST = + (parseFloat(CPUout.system) / 1000000).toFixed(2);
                    const CPUFormat = {
                      user: CPUUSER ,
                      system: CPUSYST
                    };
                    res.send(CPUFormat);
                  } catch(err) {
                    myF.processError('api1000-server.get-cpu ', err, 55);
                    res.send([{"error":"api1000-server.get-cpu"}]);
                  }
                });
                
                //tested in test.js
                server.get('/=mem', function(req, res) {
                  try {
                    const memRes1 = util.inspect((process.memoryUsage().rss / 1048576));
                    const memRes2 = util.inspect((process.memoryUsage().heapTotal / 1048576));
                    const memRes3 = util.inspect((process.memoryUsage().heapUsed / 1048576));
                    const memRes = {};
                    memRes.rss = + parseFloat(memRes1).toFixed(2);
                    memRes.heapTotal = + parseFloat(memRes2).toFixed(2);
                    memRes.heapUsed = + parseFloat(memRes3).toFixed(2);
                    res.send(memRes);
                  } catch(err) {
                    myF.processError('api1000-server.get-mem ', err, 55);
                    res.send([{"error":"api1000-server.get-mem"}]);
                  }
                });
                
                server.on('uncaughtException', function(req, res, route, err) {
                  console.log('erreur api1000 ligne 90 : ' + err);
                  res.send(500, '0');
                });
                
                server.listen(1000);

                Comment


                • #9
                  Hi folks, I stumbled on this thread with a lot of interest. 2 questions for those more knowledgeable on Nginx config than I am:

                  1. Are you able to pass the original source IP to HS3 so that it knows whether to authenticate or not? All my efforts seem to pass the proxy address to HS3 so it thinks all request come from the local network and does not try to authenticate.

                  2. Could this be run on on a different server? I see the "root C:\..." setting, which obviously would need to be different if nginx was running elsewhere?
                  Author of Highpeak Plugins | SMS-Gateway Plugin | Blue Iris Plugin | Paradox (Beta) Plugin | Modbus Plugin | Yamaha Plugin

                  Comment


                  • #10
                    Shifting to Nginx /1

                    I know that in HAproxy for PFsense the options exist for IP pass through, but I still haven't worked it out yet either.

                    Edit: in my settings there is an option for "forward for" which appears to indicate that it adjusts the header to load the endpoint's IP address. This didn't do the trick for me. In the backend for my HS3 instance there is a setting for transparent clientIP that did the trick. Now when I login from work I am seeing my work's public IP address.

                    Sent from my iPhone using Tapatalk
                    Last edited by Kerat; June 12th, 2017, 05:48 PM.

                    Comment

                    Working...
                    X