Announcement

Collapse
No announcement yet.

Started getting lots of network problems and communication failures.

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

    #31
    Originally posted by rlrjupiter View Post
    Bob,


    Shutdown HS3, Installed new .exe, restarted HS3. No errors and no communications timeouts but also nothing is updating on HS3, yet I see the changes on the ISY via Mobilinc.


    <Subscriptions>
    <Sub isExpired="no" isPortal="no" sid="36" sock="29" isReusingSocket="yes" isConnecting="no"/>
    <Sub isExpired="yes" isPortal="no" sid="-1" sock="-1" isReusingSocket="no" isConnecting="no"/>
    <Sub isExpired="yes" isPortal="no" sid="-1" sock="-1" isReusingSocket="no" isConnecting="no"/>
    <Sub isExpired="yes" isPortal="no" sid="-1" sock="-1" isReusingSocket="no" isConnecting="no"/>
    <Sub isExpired="yes" isPortal="no" sid="-1" sock="-1" isReusingSocket="no" isConnecting="no"/>
    <Sub isExpired="yes" isPortal="no" sid="-1" sock="-1" isReusingSocket="no" isConnecting="no"/>
    <Sub isExpired="yes" isPortal="no" sid="-1" sock="-1" isReusingSocket="no" isConnecting="no"/>
    <Sub isExpired="yes" isPortal="no" sid="-1" sock="-1" isReusingSocket="no" isConnecting="no"/>
    <Sub isExpired="yes" isPortal="no" sid="-1" sock="-1" isReusingSocket="no" isConnecting="no"/>
    <Sub isExpired="yes" isPortal="no" sid="-1" sock="-1" isReusingSocket="no" isConnecting="no"/>
    </Subscriptions>
    Strange. The list above indicates that only one application has a subscription connection to the ISY. However your attached debug log shows the plug-in receiving updates from the ISY just fine. For example:

    parse_event : control = [DON] action = [1] node = [36 88 6A 1] eventinfo = []

    this is an update saying that Insteon device 36.88.6A sent an ON command.
    --
    Bob Paauwe
    ISYInsteon Plug-in
    http://www.bobsplace.com/ISYInsteon/

    Comment


      #32
      Originally posted by cerjzc View Post
      Still not once could I get a "good" connection where I could update something in HS and have it show the correct status, but it would turn the device on in the ISY it just wouldn't report back the status to homeseer so it would log the communication errors.

      I then tried changing the username/password on the ISY and then updated it in the plugin. As soon as that was done it started spamming these messages in the HS log:


      Jun-28 03:37:47 ISYInsteon Connect : Attempt to start already active WebSocket connection 0
      Yeah, restart was another part of the new code that I didn't implement yet. But it looks like it sort of recovered anyway. I think it probably started running a second thread so there's currently two threads running trying to process subscription messages. As long as only one of them is really working, it shouldn't cause any issues.

      But I was able to control the devices in Homeseer and it would update the status in the ISY and in HS. I also was no longer getting the communication or subscription errors in Homeseer.
      That kind of implies that either the username or password was wrong in the plug-in. I wouldn't have suspected that since it was sending commands and that shouldn't work if it can't authenticate. I also would expect an authorization error to show up in the logs.

      But this gives me something to experiment with. Maybe the plug-in isn't handing authorization failures correctly.
      --
      Bob Paauwe
      ISYInsteon Plug-in
      http://www.bobsplace.com/ISYInsteon/

      Comment


        #33
        Yes it is confusing. The username and password hadn't ever changed since I started using the ISY and that it would connect and work sometimes and not others and only started seeing this issue just over 2 weeks ago.

        As an update when I did get home today I found that while things were working they were extremely slow in homeseer for Insteon and z-wave. I was guessing it was the spamming of the messages from the ISY. So reluctantly I did go ahead and shut things down and put back the none debug version in the updater and restarted.

        Well so far to my surprise it did start up and was working when I last checked it. I do see now that I am getting reports from a single device that it can't communicate, but I'm guessing that is a real device issue I need to look into and other devices are currently working and updating.

        Jeff

        Quick update -- I did pull the air gap switch on the one device that was still logging the status update error and that appears to have fixed the problem with it. Since I'm heading to work before long I don't plan on cycling the plugin or HS unless required so we will have to wait and see how things start up after the next restart.
        Last edited by cerjzc; June 28, 2018, 04:20 PM. Reason: Update

        Comment


          #34
          I did experiment with using the wrong username and the plug-in failed to authenticate and wasn't able to query the device list or send commands. So whatever was happening, I don't think it was failing to authenticate.

          Interestingly though, I think I did find a bug in the ISY subscription process. Turns out my debug version was making the subscription connection before it had the username/password from the config file. Thus it was authenticating that connection with a blank username and blank password. The ISY responded correctly with a 401 (authentication failed) status, but the connection didn't close and the ISY starts sending status/updates. So that's a bit of a security issue if it will let clients subscribe even when authentication fails.

          I've resolved some of the restart issues and the authentication issue, but I'm not sure it's worth the effort to continue working on it. If the current release version continues to work for you, that pretty much rules out the plug-in's subscription code as the problem.
          --
          Bob Paauwe
          ISYInsteon Plug-in
          http://www.bobsplace.com/ISYInsteon/

          Comment


            #35
            Originally posted by rlrjupiter View Post
            Bob,

            Shutdown HS3, Installed new .exe, restarted HS3. No errors and no communications timeouts but also nothing is updating on HS3, yet I see the changes on the ISY via Mobilinc.
            As described in the previous post, the debug version does have a bug where it authenticates with a blank username and password. I did have one out of about 15 tries where the ISY actually closed the connection on me after the authentication failed. The other times, it stayed connected and and sent status updates. However, I was only running for a few seconds before I'd disable the plug-in and try something else.

            You might want to try re-entering your username/password in the plug-in and switch back to the latest version in the updater.

            If that doesn't work, I'll post a new debug version with authentication fixed.
            --
            Bob Paauwe
            ISYInsteon Plug-in
            http://www.bobsplace.com/ISYInsteon/

            Comment


              #36
              Originally posted by bpwwer View Post
              I did experiment with using the wrong username and the plug-in failed to authenticate and wasn't able to query the device list or send commands. So whatever was happening, I don't think it was failing to authenticate.

              Interestingly though, I think I did find a bug in the ISY subscription process. Turns out my debug version was making the subscription connection before it had the username/password from the config file. Thus it was authenticating that connection with a blank username and blank password. The ISY responded correctly with a 401 (authentication failed) status, but the connection didn't close and the ISY starts sending status/updates. So that's a bit of a security issue if it will let clients subscribe even when authentication fails.

              I've resolved some of the restart issues and the authentication issue, but I'm not sure it's worth the effort to continue working on it. If the current release version continues to work for you, that pretty much rules out the plug-in's subscription code as the problem.

              Bob I don't want to mess with it before I head to work so that it will for sure work through the night, but when I get home in the morning or if it acts up over night I will do some more testing and report back. I will try stopping and starting the plugin several times in the morning if it makes it over night and let you know.

              Thanks,
              Jeff

              Comment


                #37
                so confusing

                I've been very busy at work and haven't touched anything since installing your new EXE. No reboots of HS3 or ISY. When I left this morning it was not communicating, but come home today and its working, everything is updating in HS3 with no communication loss.

                What the heck? I'm going to let it run for few days, but then what?

                Roger

                Comment


                  #38
                  Originally posted by rlrjupiter View Post

                  What the heck? I'm going to let it run for few days, but then what?

                  Roger
                  I'm pretty sure the issue hasn't been with the plug-in (or at least not with the actual status/update code portion). Given that the release version is now working for Jeff.

                  There could be something going on with the authentication, maybe more specifically around the save/load of the username/password as there have been a couple of changes there more recently.

                  My suggestion would be to switch back to the release version when you have some time to mess with it. If you still see issues (I'd like to see the same debug log you collected above if this happens), try re-entering the username and password in the plug-in and then restart the plug-in.

                  If you're still having problems, switch back to the debug version. If that starts working again, I'll have to put some more time into finishing up the changes I started in the debug version.
                  --
                  Bob Paauwe
                  ISYInsteon Plug-in
                  http://www.bobsplace.com/ISYInsteon/

                  Comment


                    #39
                    Did not stay connected for long.

                    Well I spoke too soon as the connection lasted a few hours and now back to no connection.

                    Jul-01 3:31:13 PM ISYInsteon read : The subscription connection to ISY has failed.
                    Jul-01 3:21:13 PM ISYInsteon read : The subscription connection to ISY has failed.
                    Jul-01 3:06:57 PM ISYInsteon read : The subscription connection to ISY has failed.
                    Jul-01 2:49:29 PM ISYInsteon HTTP/1.1 401 OK Content-Length: 0 Connection: Keep-Alive WWW-Authenticate: Basic realm="/" Content-Type: text/html; charset=UTF-8 Cache-Control: max-age=3600, must-revalidate EXT: UCoS, UPnP/1.0, UDI/1.0 Last-Modified: Sun, 1 Jul 2018 14:44:20 GMT
                    Jul-01 2:49:11 PM ISYInsteon Waiting for connection status 4 or greater.
                    Jul-01 2:49:11 PM Z-Wave 1 Networks were restored from the data in the database... Building data connections.
                    Jul-01 2:49:10 PM Informational Ethernet connection established to 192.168.1.30:4998
                    Jul-01 2:49:09 PM Informational Ethernet connection established to 192.168.1.31:4998
                    Jul-01 2:49:09 PM Speech Listening for remote speaker connections on port 10401
                    Jul-01 2:46:15 PM ISYInsteon ReceiveLoop : The subscription connection to ISY has failed.
                    Jul-01 2:36:15 PM ISYInsteon ReceiveLoop : The subscription connection to ISY has failed.
                    Jul-01 2:26:15 PM ISYInsteon ReceiveLoop : The subscription connection to ISY has failed.
                    Jul-01 2:16:15 PM ISYInsteon ReceiveLoop : The subscription connection to ISY has failed.
                    Jul-01 2:06:15 PM ISYInsteon ReceiveLoop : The subscription connection to ISY has failed.


                    Just reinstalled latest plugin and will send you debug info. Rest Error in startup of lastest plugin install. Also reset username password with no effect.

                    Jul-01 2:57:20 PM ISYInsteon TriggerFromData : Leave
                    Jul-01 2:57:20 PM ISYInsteon TriggerFromData : Enter
                    Jul-01 2:57:20 PM ISYInsteon GetTriggers : Enter
                    Jul-01 2:57:20 PM ISYInsteon StartISYThread : Leave
                    Jul-01 2:57:20 PM ISYInsteon StartISY : Leave
                    Jul-01 2:57:20 PM ISYInsteon StartISY : ISY Initialize finish, status = 4
                    Jul-01 2:57:19 PM ISYInsteon RESTQueryAllVariables : Enter - Query for variables.
                    Jul-01 2:57:19 PM ISYInsteon RESTQueryNetworkResources: --> Data at the root level is invalid. Line 1, position 1.
                    Jul-01 2:57:19 PM ISYInsteon RESTQueryNetworkResources: REST network query failed. Corrupteded XML?
                    Jul-01 2:57:16 PM BLLAN Info Computer - Home Iphone 6 --- 1 PING failure
                    Jul-01 2:57:15 PM ISYInsteon GetModulesConfig : ISY module configuration Portal Integration - UDI ( 21075) is not currently supported.
                    Jul-01 2:57:15 PM ISYInsteon + 21075 Portal Integration - UDI
                    Jul-01 2:57:15 PM ISYInsteon + 21100 Z-Wave
                    Jul-01 2:57:15 PM ISYInsteon + 21040 Networking Module
                    Jul-01 2:57:15 PM ISYInsteon + 21020 Weather Information
                    Jul-01 2:57:15 PM ISYInsteon + 21010 OpenADR
                    Jul-01 2:57:15 PM ISYInsteon Modules installed on ISY:
                    Jul-01 2:56:59 PM ISYInsteon HSEvent : Leave
                    Jul-01 2:56:59 PM ISYInsteon HSEvent : Enter - Called for CONFIG_CHANGE
                    Jul-01 2:56:57 PM ISYInsteon ISY Firmware version: 4.6.2
                    Jul-01 2:56:57 PM Starting Plug-In Plugin ISYInsteon started successfully in 23 milliseconds
                    Jul-01 2:56:57 PM ISYInsteon InitIO : Leave
                    Jul-01 2:56:57 PM ISYInsteon InitIO : Enter - Called with COM1
                    Jul-01 2:56:57 PM Starting Plug-In Initializing plugin ISYInsteon ...
                    Jul-01 2:56:57 PM Info Plugin ISYInsteon has connected. IP:127.0.0.1:53800
                    Jul-01 2:56:57 PM ISYInsteon I have connected to HS
                    Jul-01 2:56:56 PM Plug-In Finished initializing plug-in ISYInsteon
                    Jul-01 2:56:56 PM Updater Re-initializing plugin after update.
                    Jul-01 2:56:54 PM Updater Install/Update of package ISYInsteon was successful.
                    Jul-01 2:56:54 PM Updater Copying file: C:\Program Files (x86)\HomeSeer HS3\HSPI_ISYInsteon.exe.config
                    Jul-01 2:56:54 PM Updater Copying file: C:\Program Files (x86)\HomeSeer HS3\HSPI_ISYInsteon.exe
                    Jul-01 2:56:54 PM Updater Installing package ISYInsteon
                    Jul-01 2:56:54 PM Updater Update HSPI_ISYInsteon.exe downloaded successfully
                    Jul-01 2:56:54 PM Updater Downloading update: ISYInsteon

                    Roger

                    Comment


                      #40
                      Just as an update. I never got a chance to try cycling HS as it was running well with the ISY so this morning while I had a few minutes and needed to reboot the homeseer machine I went ahead and did a reboot and HS restart. Well sure enough the issue came back with the new username and password. I tried cycling HS a couple more times just to see and each time it starts erroring pretty much right away with the ISY network communication failure messages and the ISY subscription: No data for 120 seconds.

                      I'm going to try changing the username/password again on both the ISY and HS plugin settings to see if that does anything before rebooting the ISY or anything else.

                      Jeff

                      Comment


                        #41
                        Well I changed the username/password on the ISY and the plugin, but still got errors. I tried cycling the plugin a few times, but no luck.

                        So next I had it reboot the ISY and then restart the plugin and still got errors right from the start.

                        So I went ahead and rebooted the homeseer machine completely and the ISY at the same time and on startup after that it started working and so far is stable. I don't want to stop it again while it is working, but will continue to monitor.

                        Jeff

                        Comment


                          #42
                          Jeff,

                          If re-booting is really what resolves it (even temporarily), the issue is far more likely to be hardware related. Unfortunately, the only way I know of to get a better idea will involve a lot of effort. Basically you need to try changing only one thing and do it enough times to be pretty sure that one thing isn't the issue. For example, restart the plug-in 20 time in a row, if it fails every time, move on to restarting HS and the plug-in 20 times, then rebooting the HS computer and restarting 20 times, then rebooting the ISY and so on. That's the approach I'd take.

                          You can also check some of the network state when it's not working to get a better idea of where the problem may be. Based on the plug-in's behavior, it looks like the network connection is dropping external to the HS computer which implies that the cause is really external to the HS computer. To test that I'd do the following:

                          1) With the plug-in *not* running, query the ISY's subscription list from http://isy.ip.address/rest/subscriptions and make note of any slots with isReusingSocket=yes.

                          2) Run the TCPView program (TCPView for Windows - Windows Sysinternals) to monitor the network connections.

                          3) Start the plug-in.

                          4) Query the ISY's subscription list again, a new entry should show up with isReusingSocket=yes, this is the subscription connection that the plug-in is using, again take note of this. In the TCPView program, look for a row that has the process name HSPI_ISYInsteon.exe, remote address of your ISY, and the state of "Established"

                          5) When the plug-in starts reporting timeouts, check both the ISY subscription list and the TCPView program. My guess is that the ISY will no longer show the subscription connection in it's list (I.E. it will look like the list from step 1) but TCPView will still show the connection as Established. If this is true, it means the computer with HomeSeer still thinks it has a network connection to the ISY but from the ISY's perspective, the connection has dropped.

                          If instead, you don't see a new subscription entry in step 4, that would imply that a connection is never really established. The plug-in should also be reporting an error indicating this.

                          I know you don't want to mess with it when it's working and that it will probably stop working at an inconvenient time since that's usually when things stop working for me, but hopefully, this provides some ideas to move forward when the issue occurs again.
                          --
                          Bob Paauwe
                          ISYInsteon Plug-in
                          http://www.bobsplace.com/ISYInsteon/

                          Comment


                            #43
                            I will keep an eye on it Bob. I won't have time tonight to mess with it and I go back to work tomorrow so I will have to wait until I'm off next time, but will keep what you said in mind. I have checked all the hardware and error logs I could and haven't found anything. I may swap in a different Intel nic card to see if that does anything or I have thought about isolating the ISY and Homeseer machine on their own network to see if that helps at all. I have already swapped the network cables, ports and switches they are connected to, but that didn't help last time. I also see no other issues on the ISY or the Homeseer machine besides this.

                            Thanks,
                            Jeff

                            Comment


                              #44
                              Well I was able to spend some time working on it tonight and think I have got it figured out. I did try swapping the NIC card and on the first boot up with the new nic it started working and worked for several hours until I was able to work on it when no one else was up.

                              I then started turning off all plugins which didn't seem to help and then any other software that was running on the system.

                              In doing that I came across a change in Malwarebytes that must have happened after a recent update. Although I have it installed I never have any of the active scanners running, but it appears an update some time back re-enabled or added a Web protect function. I first found with Malwarbytes turned off I could stop and start the plugin and it would work each time.

                              It was then I went through the settings and found that the "Web protection" was enabled. I didn't catch it at first because I found that it doesn't enable right away there appears to be a delay for it starting which probably explains that why on some reboots it would work and others it wouldn't.

                              But so far in testing with the Web Protection completely disabled again it has started and connected each time. With it running and the plugin starting I never get a connection to show up on the subscriptions page at all. While the ISY Admin console running on the same machine works fine the plugin can't connect. I didn't see any pop-ups in the program mentioning it was blocking a connection or showing I needed to approve it, but it must be doing something that doesn't allow that connection.

                              So thanks again Bob for all your help looking into this and I will continue to monitor it, but for me I think the issue is resolved.

                              Thanks,
                              Jeff

                              Comment


                                #45
                                I'm glad you found what appears to be the culprit.

                                I'll certainly keep this in mind if anyone else has this type of problem.
                                --
                                Bob Paauwe
                                ISYInsteon Plug-in
                                http://www.bobsplace.com/ISYInsteon/

                                Comment

                                Working...
                                X