Announcement

Collapse
No announcement yet.

New exception on Ambient weather polling interval

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

    New exception on Ambient weather polling interval

    I originally thought this was an issue with the OIP responding slowly, but it continued after a reboot of the device. Realized this error was occurring on the Ambient weather polling interval (every 3 mins). A restart seems to have fixed it, fyi in case it rings any bells.

    Also, not sure if its normal for you to always use a device enumerator so late in the cycle of the program (been running for some time when this started). I'd highly recommend keeping a local table of device mappings after the first enumeration if so. Enumeration is slow and fragile on systems with many devices (I have about 3000 and its more and more flakey I've noticed)

    Click image for larger version

Name:	Screen Shot 2019-01-03 at 9.41.45 AM.png
Views:	126
Size:	69.9 KB
ID:	1271960

    #2
    Still this after the plugin runs for some time, it appears after some time calling GetDeviceEnumerator fails. The call stack is different in this one. That call is VERY expensive, can you kindly look at caching the results?

    Click image for larger version

Name:	Screen Shot 2019-01-09 at 2.03.08 PM.png
Views:	95
Size:	144.8 KB
ID:	1274358

    Comment


      #3
      Originally posted by bsobel View Post
      Still this after the plugin runs for some time, it appears after some time calling GetDeviceEnumerator fails. The call stack is different in this one. That call is VERY expensive, can you kindly look at caching the results?

      Click image for larger version

Name:	Screen Shot 2019-01-09 at 2.03.08 PM.png
Views:	95
Size:	144.8 KB
ID:	1274358
      That is an interesting error. You're running Linux right?

      I can look into caching the device info however CPU cycles are more readily available than memory on some of the smaller systems.

      Comment


        #4
        Originally posted by Simplex Technology View Post

        That is an interesting error. You're running Linux right?

        I can look into caching the device info however CPU cycles are more readily available than memory on some of the smaller systems.
        Yes Linux. And I assume you are just looking up the deviceRef for certain devices, if so a Dictionary of your search key to the ints will be much much smaller. The GetDeviceEnumerator() call sends a very large serialization of ALL devices over the wire, its memory impact will dwarf anything you do to keep state locally for the devices your plugin owns.

        Comment


          #5
          Originally posted by bsobel View Post

          Yes Linux. And I assume you are just looking up the deviceRef for certain devices, if so a Dictionary of your search key to the ints will be much much smaller. The GetDeviceEnumerator() call sends a very large serialization of ALL devices over the wire, its memory impact will dwarf anything you do to keep state locally for the devices your plugin owns.
          Over what wire are you speaking?

          But yes I do agree the GetDeviceEnumerator() can be optimized.

          Comment


            #6
            Originally posted by Simplex Technology View Post

            Over what wire are you speaking?

            But yes I do agree the GetDeviceEnumerator() can be optimized.
            Wire in the TCP/IP sense but over local sockets in this case where the plugin is running on the HS box. But still a very large amount of data is wrapped up in the enumerator object, as the enumerator does not go back HS on every next, it already transferred the data in one shot. I haven't found a root cause, and I've talked to Rich about it, but I've seen random failures on that call since the .3xx series. My code actually retries it up to 10 times before giving up. Its pretty reliable locally, but for a remote plugin its (on my system with a few thousand devices) very very painful and fails a lot.

            Comment


              #7
              Originally posted by bsobel View Post

              Wire in the TCP/IP sense but over local sockets in this case where the plugin is running on the HS box. But still a very large amount of data is wrapped up in the enumerator object, as the enumerator does not go back HS on every next, it already transferred the data in one shot. I haven't found a root cause, and I've talked to Rich about it, but I've seen random failures on that call since the .3xx series. My code actually retries it up to 10 times before giving up. Its pretty reliable locally, but for a remote plugin its (on my system with a few thousand devices) very very painful and fails a lot.
              Ahh for a remote plugin I can see this. I admit I'm not thinking of these plugins being ran remotely. Something I should think more about.

              Perhaps habit but I do a device refresh before any call that will touch a device and stuff that into a variable to work against. On some of the plugins I do think I'm calling too often though and it could be optimized. I keep trying different tactics to optimize and then something changes and breaks and I have to go back and try something else or just do it 'ol fashion and make repeated calls.

              Comment


                #8
                Originally posted by Simplex Technology View Post

                Ahh for a remote plugin I can see this. I admit I'm not thinking of these plugins being ran remotely. Something I should think more about.

                Perhaps habit but I do a device refresh before any call that will touch a device and stuff that into a variable to work against. On some of the plugins I do think I'm calling too often though and it could be optimized. I keep trying different tactics to optimize and then something changes and breaks and I have to go back and try something else or just do it 'ol fashion and make repeated calls.
                What issues are you seeing, if you don't mind me asking, as I've not run into with my plugins. You should be able to cache the device until you get a device change callback for it (at a minimum). This probably explains the high cpu usage (this plugin is the highest cpu user on my system).

                Comment


                  #9
                  Originally posted by bsobel View Post

                  What issues are you seeing, if you don't mind me asking, as I've not run into with my plugins. You should be able to cache the device until you get a device change callback for it (at a minimum). This probably explains the high cpu usage (this plugin is the highest cpu user on my system).
                  I'm always open to learning new methods of working with the devices. There's still quite a bit of optimizing I need to do with this plugin as well.

                  Comment


                    #10
                    Originally posted by Simplex Technology View Post

                    I'm always open to learning new methods of working with the devices. There's still quite a bit of optimizing I need to do with this plugin as well.
                    Any progress here, I am still getting exceptions when your PI requests a device enumerator way too often (still a HS bug, but hoping if you cache the data it will help work around it)

                    Comment


                      #11
                      Originally posted by bsobel View Post

                      Any progress here, I am still getting exceptions when your PI requests a device enumerator way too often (still a HS bug, but hoping if you cache the data it will help work around it)
                      None at the moment. There's also some bugs in HS3 that I've been fighting with and having to rebuild/restore a large part of my environment. I've not been having a good new year.

                      Comment


                        #12
                        Originally posted by Simplex Technology View Post

                        None at the moment. There's also some bugs in HS3 that I've been fighting with and having to rebuild/restore a large part of my environment. I've not been having a good new year.
                        If I can help at all just lmk. Fought my way through a few of these with Hs, so know how you feel.

                        Comment


                          #13
                          Originally posted by bsobel View Post

                          If I can help at all just lmk. Fought my way through a few of these with Hs, so know how you feel.
                          Thanks for the offer. I'm trying to keep up with some work issues and rebuild. I'm hoping things at work also calm down too much from all directions all at once.... just one of those months....

                          Comment


                            #14
                            FYI since i am seeing the GetDeviceEnumerator() call constantly especially at startup, I've been talking to Rich to see if it could be something on the HS side. We just found one nasty one (DeleteEventByRef() didn't actually permanently delete the event, which explains a whole rash of other issues I was dealing with). Hoping we find something around the DeviceEnumerator too. Will LYK...

                            Comment


                              #15
                              Originally posted by bsobel View Post
                              FYI since i am seeing the GetDeviceEnumerator() call constantly especially at startup, I've been talking to Rich to see if it could be something on the HS side. We just found one nasty one (DeleteEventByRef() didn't actually permanently delete the event, which explains a whole rash of other issues I was dealing with). Hoping we find something around the DeviceEnumerator too. Will LYK...
                              Yeah keep me informed. I was actually doing a little checking earlier today and on the Windows box I was running on the CPU never went over ~9% during the PI refresh cycle against 2 different PWS's and OIP. I'm not doing a "normal" enumeration as I do the base enumeration and get ONLY my devices and then I use LINQ to query within that... I'm sure there's something that could be changed/optimized there always is.

                              I think I need to do a more thorough run with Linux though as I'm thinking this has a lot to do with Mono (again).

                              Comment

                              Working...
                              X