XAP HUB Issues

[jstaab]

As touched upon in another thread, I've been plagued with XAP issues and could never keep XAP running long enough to even get a feel for XAP. After a bit of research, here is a summary of my issues and what I've found.

First off, xFx-Express Hub reports the following:

09-Aug-13 23:15:40.2 ERROR Hub message loop socket exception (MessageSize): A me
ssage sent on a datagram socket was larger than the internal message buffer or s
ome other network limit, or the buffer used to receive a datagram into was small
er than the datagram itself
09-Aug-13 23:16:23.4 INFO Selected Ethernet network interface 'Local Area Conne
ction' AMD PCNET Family PCI Ethernet Adapter - Packet Scheduler Miniport

Now, when this happens, you can see by the time stamps, the hub is recycling and stops listening for about 40 seconds. Now, using Wireshark, I found that fragmented UDP packet are getting sent out by the mcsXAP plugin, which is what is causing the hub to drop and recycle, and is what is knocking an unrelated network device off my network.

Now, the interesting part, it appears that certain devices that are getting updated is what is triggering the bad UDP packets. I turned off one particular event that runs a script, problem solved. The script in question is non other then the Jon00PerMon.vben, Jon00's performance monitor. My conclusion is device strings that contain heavy HTML, maybe referencing images, are what's causing the UDP fragments. What I can tell you with certainty is as long as the script doesn't run, everything is fine, the second I manually run the script, I get the message from the hub, pasted above, thus the fragmented UDP packet.

Now, as near as I can tell, there is no way to stop device updates from getting sent out via XAP, as near as I can tell, every single device update gets transmitted, am I wrong here? If I could stop mcsXAP from sending the problematic devices, all would be fine.

Thoughts???

[Steve Q]

XAP HUB Issues

About 2 years ago, my xAP-Homeseer network was terribly unreliable. I could not keep it running for more than 24 hours. At that time I was running Homeseer on my primary computer that I used for general work. After spending lots of time trying to determine the cause, I came to the conclusion that anytime the CPU went to 100% for a few seconds, xAP would stop. Since I put Homeseer on a dedicated ( and faster) computer, I do not have problems.

Maybe your problem is similar. I know nothing about the internal workings of xAP, but I believe there are a lot of timing issues in the xAP protocol. When something impacts the timing of the message, things get out of sync and mcsxAP1wire.exe stops communicating.

Steve Q


Sent from my iPad using Tapatalk HD

[Michael McSharry]

The mcsxap setup allows used selection on a device by device basis which will result in xap mesages. Option exists to send hs log messages to so if logging device changes then thi will sesult in each log change. This can be disabled too.

The protocol has no time dependency. Different techniques hav been used over years with mcsxap and that could account for differences over the years

[jstaab]

I'm a little unclear, so are you saying there is a way to prevent particular device updates from being sent out? Out of the box, it seems like every single device change is being sent out, regardless of the device was accepted, this is on the HS2 machine.

In addition, this really doesn't remedy the fragmented UDP packets from the problematic devices. If I can select to NOT send out the device updates, great, it would at least be a band aide fix anyway.

The mcsxap setup allows used selection on a device by device basis which will result in xap mesages. Option exists to send hs log messages to so if logging device changes then thi will sesult in each log change. This can be disabled too.

The protocol has no time dependency. Different techniques hav been used over years with mcsxap and that could account for differences over the years

[billt]

I'm no expert and Xap is complicated, but I was under the impression that you had to select every item that you wanted to transmit xap messages.

The device has to have the (A)ccept box selected, and the Transmit BSC message box needs to be selected.

I am also running Jon00PerMon.vben and have looked at messages with xFx viewer, xFX express hub and the mcsXap viewer and am not seeing the errors that you are getting.

[CouchPotatoe]

Thanks for you research into this . xAP messages are limited to one packet of the underlying transport used which is typically ~1500 bytes (MTU size) on Ethernet. Larger fragmented messages should not be generated but I can see how they might if parameter values include a lot of html.

We do not have an automatic low level mechanism within the xAP protocol to handle fragmented packets so someone creating a device that sends copious amounts of data should utilise a schema that distributes the parameters over several adjacent messages. The xAP TV listings program had this issue for example with the long program descriptions and was modified accordingly. I suppose mcsxAP and other programs that expose devices via xAP could enforce this... I need to look at a couple I wrote too in this regard.

It's nice that the xFX hub identifies the issue but disappointing that it then stalls and restarts and so I will contact Edward and make him aware of this thread.

In the meantime as suggested, you should just be able to disable the problematic device in the HomeSeer xAP plugin. Ensure "Show HomeSeer Sourced BSC" is selected and then that 'A' accept is deselected for the problem device as well as perhaps all similar devices from that HS (Jon performance) plugin. These devices will be named mcs.xAP.something in the list. Hopefully you'll then be up and running with a stable xAP environment.

Steve.. re xAP and timing... because xAP is UDP broadcast and not point to point connected there should be no issues as devices come and go except of course that you lose the messages during the time they are absent. Lockups just shouldn't happen, that's one of the niceties of xAP.... although if a device or worse still a hub stalled and wasn't buffering then it would be missing xAP messages of course.

Kevin

[jstaab]

I appreciate your reply to this. Here is the problem though, NONE of the devices have been accepted, on the HS2 side, yet it appears all the devices are still being sent out. I've even tried to accept them, then un-accept them, yet they are still sent. It appears that any device that is updated in the background via HS events or plugins, are automatically sent, regardless of whether they were accepted or not. Now, what I can tell you, they DO NOT show up in HS3 until they indeed are accepted in HS2, but regardless, all devices spam the network. An example message, below.

xap-header
{
v=13
hop=1
uid=FF.000E:0000
class=Homeseer.Event
source=mcs.Xap.jesse
}
Event.String
{
Device=?6
Text=Online [443d,01h,35m]
}

Thanks for you research into this . xAP messages are limited to one packet of the underlying transport used which is typically ~1500 bytes (MTU size) on Ethernet. Larger fragmented messages should not be generated but I can see how they might if parameter values include a lot of html.

We do not have an automatic low level mechanism within the xAP protocol to handle fragmented packets so someone creating a device that sends copious amounts of data should utilise a schema that distributes the parameters over several adjacent messages. The xAP TV listings program had this issue for example with the long program descriptions and was modified accordingly. I suppose mcsxAP and other programs that expose devices via xAP could enforce this... I need to look at a couple I wrote too in this regard.

It's nice that the xFX hub identifies the issue but disappointing that it then stalls and restarts and so I will contact Edward and make him aware of this thread.

In the meantime as suggested, you should just be able to disable the problematic device in the HomeSeer xAP plugin. Ensure "Show HomeSeer Sourced BSC" is selected and then that 'A' accept is deselected for the problem device as well as perhaps all similar devices from that HS (Jon performance) plugin. These devices will be named mcs.xAP.something in the list. Hopefully you'll then be up and running with a stable xAP environment.

Steve.. re xAP and timing... because xAP is UDP broadcast and not point to point connected there should be no issues as devices come and go except of course that you lose the messages during the time they are absent. Lockups just shouldn't happen, that's one of the niceties of xAP.... although if a device or worse still a hub stalled and wasn't buffering then it would be missing xAP messages of course.

Kevin

[jstaab]

Have you accepted the devices for the performance monitor script? Also, I may have just discovered why all my devices are getting sent, I went in an unchecked every box in the Callback Event Notification, and everything indeed got quiet, now ONLY my accepted devices send out a message, what I'll have to do now, is accept a device that changes non stop via an event and see if it sends out messages on its own..

I'm no expert and Xap is complicated, but I was under the impression that you had to select every item that you wanted to transmit xap messages.

The device has to have the (A)ccept box selected, and the Transmit BSC message box needs to be selected.

I am also running Jon00PerMon.vben and have looked at messages with xFx viewer, xFX express hub and the mcsXap viewer and am not seeing the errors that you are getting.

[jstaab]

All, I believe I resolved the issue with all devices being sent, as mentioned in my prior post, I unchecked all the check boxes in that section, so good news there. However, as mentioned there is still the potential for the plugin to send out fragmented UDP packets with devices that contain large device strings. Anyway, I can now at least prevent the devices causing the problem from being sent out and continue running the scripts.

Now, back to the other thread, I can't get the device status to update in HS3, even though the last change column is getting updated....

[CouchPotatoe]

ahh , good find...... The 'A' checkbox likely then only applies to the xAPBSC schema , as does the 'Show HomeSeer Sourced BSC' as the name indicates. The copy of the message you posted is a different 'HomeSeer.Event' schema so I guess it's that one that's fragmenting and might be resolvable by turning off a callback event notification checkbox.

I'll defer to Michael on this...but in the meantime you should now be able to get transfer of some device status and control between HS3 and HS2 using the xAPBSC schema.

Just saw your post above this one...The device status will be updating, and so triggers will happen as you expect - however it appears the displayed status is not updating... which is a previously reported issue. Michael is awaiting HomeSeer to re-instate his licence expiry after the v15 / v16 change ,and until then he's a bit stuck I think on looking into this .... I saw the problems posted so I didn't update mine ...cmon HomeSeer ...

K

[Steve Q]

I'm trying to get a better understanding of the xAP interaction between 2 computers. I have mcsxaphub and xfx message viewer running on both computers and HS2 running on one computer. If I turn on an xAP device on the HomeSeer computer a message is sent to the network and I will see the message via xfx on both computers. No problem. Xfx has the ability to retransmit a message. When I resend the "on" message back to the Homeseer computer, I expected it to turn the device back on. But it does not! It appears on the Homeseer computer but it is not implemented. I have both send and receive xAP turned on in the Homeseer plugin. I have tried this with several devices. None of them work! Does anyone have any ideas why this is not working?

Steve Q

[CouchPotatoe]

You say that xFX Viewer running on the HS2 computer sees the message coming back (after being re-sent by XFX Viewer on the remote machine). Assuming you are running the latest version of xFX Viewer (?) then that proves that the hub is running on the HS2 machine and that it was started as the first xAP application. This is very important as otherwise xAP applications become 'deaf' although they can still send messages OK. This was my first thought as it is the most common issue when messages aren't being received ...but it seems you have that correct

Now on to the actual issue I think.. ... it's not just a case of re-sending the messages that HS transmits back to HS in order to control the device. I'm assuming you are referring to a xAP BSC message (there are other schema that HS can send) . Before I go into that let me just explain a feature in xFX Viewer that I think will assist you greatly.

If you click on the 'BSC' tab at the top of the xFX window you will be presented with a list of all the BSC endpoints on your network. To make things easier , if you select (click on) a name in the hierarchical left hand 'Sources' device tree then the BSC devices displayed will reduce to only those matching that selection. Now if you right click on the state column value for your device you will be presented with state => Off or state=> On choices and you can select one of these in order to control the device. Usefully this also works for controlling the level or text values too. This should now provide control for your HS2 device and you can examine the resulting xAPBSC.cmd message in xFX Viewer to see just what the message should look like.

BTW one nuance with HS2 is that sometimes the devices do change state but the displayed value in the HS web interface doesn't always reflect this change. Especially true when displaying levels. The correct HS triggers for device changes do fire though.

---------------------------------------------

Now the boring in depth which you can ignore if you're already sorted. There are 4 changes you need to make to the xAP message to change a status message into a command message

Now I'm assuming that the message coming out from the HS2 machine is of class=xAPBSC.event or class=xAPBSC.info which is providing information on the current status of the device in HS. This will not have a target= parameter in the header i.e. it is being sent to everyone/anyone who is interested....

However in order to control (command) a device you need instead to send back to it a messaqe of class=xAPBSC.cmd and it needs to be targeted at the specific HS device. So you need to add an appropriate target parameter in the header too. The target will be the original source address of the device.

Also the block name should be changed from output.state to output.state.1 and there must be an additional ID parameter in the block. The value of the ID parameter should exactly match the hex digits after the : in the UID parameter of the device you wish to control. However if you are being lazy you can instead just use ID=* although that is inefficient.... but that is what xFX Viewer does.

Here's an example of a message from HS and the corresponding message which if sent back to HS would control the device, switching it ON.

xap-header
{
v=13
hop=1
uid=FF.000E:5C03
class=xAPBSC.info
source=mcs.Xap.TRANQUILXP:Hallway.Plugwise.Hallway_Lamp
}
output.state
{
state=OFF
}

xap-header
{
v=13
Hop=1
UID=FFE2BC00
Class=xAPBSC.Cmd
Source=xFx.v4Viewer.Dell_8500
Target=mcs.Xap.TRANQUILXP:Hallway.Plugwise.Hallway_Lamp
}
output.state.1
{
ID=5C03
State=On
}

.. and just for completeness...

You should really change the source and UID of the message too as some devices filter out / drop messages from themselves.
I don't this matters with HomeSeer but it's good practice rather than spoofing another device. You'll notice that xFX Viewer uses it's own source address and UID in the example above.

Only output devices are controllable. These have xAP bodies named output.state. Inputs have bodies named input.state and are not controllable.

You can wildcard the target address, should you wish to control several 'matching' HS devices at once, and in such a case you would wildcard the ID as well i.e. use ID=*
For example if you had used the following it would turn everything in my hallway ON

Target=mcs.Xap.TRANQUILXP:Hallway.>

or all my ZigBee Plugwise devices on.

Target=mcs.Xap.TRANQUILXP:*.Plugwise.>

There is a full discussion of xAP wildcarding in the xAP specification document on the main xAP website, and also the xAP BSC schema.
http://www.xapautomation.org

[Michael McSharry]

A few concepts may help. xAP contains a header and body part. The header provides context information and the body provides the details. The xAP hub is a router so in the general case it does not care what is in the body and uses information in the header to route messages. In general it will distribute everything received on port 3639 to every other xAP application that is on the same computer. One of these is xFx Message Viewer. Another is Homeseer. Every xAP application will send only on port 3639 so the hub will receive all xAP traffic no matter if on the same computer or on another computer on the same network. This operation is per the xAP specification.

When a message gets to a xAP application then both the header and the body are used. xAP specification allows extensions so it is very powerful in this respect. It also defines certain schema (i.e. languages) that allow one xAP application to communicate with another with a degree of understanding and interoperability. One of the defined schema is Basic Status and Control (xAPBSC). There are defined rules that should be obeyed if both talker and listener actually communicate. Just as with natural languages not everybody that talks fully obeys the grammar of the language. This means that the understanding may be degraded in the communication. The "first language" used by mcsXap is xAPBSC. It uses other ones as secondary languages (schema) such as Weather.Report or Homeseer.Event. mcsXap can speak xAPBSC and Homeseer.Event. mcsXap can understand xAPBSC and Weather.Report. There are others, but these are illustrative.

Since xAPBSC is a defined standard schema for xAP there is a formal specification of how it is expected to be used. http://xapautomation.org/index.php?t...Control_Schema Homeseer.Event is one that I made up to characterize the event behaviors of Homeseer. Only other xAP applications that I happen to write will likely understand it.

In the header part of the xAP message the Class= line identifies what form of the language (schema) is being used. A declarative with xAPBSC.cmd where the talker commands the listen to do something. A question with xAPBSC.query where the talker asks for information from the listener. Notification with xAPBSC.event where the talker informs the listener that something changed. Notification is also possible with xAPBSC.info and in this case the same information is conveyed, but does not also indicate that it is a change.

If Homeseer wants to command a output to do something it will use xAPBSC.Cmd. If the value of a device changes in Homeseer then it provides notification using xAPBSC.Event. If you capture the xAPBSC.Event message that a device turned ON and then resend it via xAP Message viewer after changing ON to OFF you have only sent a notification (that is now a falsehood) that the Homeseer device is OFF. All the other xAP applications that are interested will now think the Homeseer device is OFF, but in reality it is still ON. Homeseer is not listening to itself so it will ignore the status provided by the xAP Message Viewer message.

If you did the same thing with the xAPBSC.Cmd message then you are sending a command to change the state. If you change the ON to OFF then you will be sending a request for the output device to be turned OFF. That output device is listening for commands so it will respond to whatever status is requested.

Per xAPBSC definition when a output device receives a command to turn ON, it will acknowledge with an xAPBSC.Event message to indicate that the state of the device has changed to ON. It is possible for you to Accept in mcsXap either the xAPBSC.Cmd or xAPBSC.Event from xAP Applications on the network. Since you will likely be interested in what the state of the device really is and not what the requested state is, then you will be Accepting the xAPBSC.Event messages. If the xAPBSC.Event message body is output.state then you will also be able to send xAPBSC.Cmd messages to command it to change state. If the start of the body was input.state then it is only a status device and xAPBSC.Cmd would not be appropriate and mcsXap does not provide controls to allow xAPBSC.Cmd messages to be sent to it.

[Michael McSharry]

My conclusion is device strings that contain heavy HTML, maybe referencing images, are what's causing the UDP fragments. What I can tell you with certainty is as long as the script doesn't run, everything is fine, the second I manually run the script, I get the message from the hub, pasted above, thus the fragmented UDP packet.

I believe the xAP specification allows 1024 byte messages. It does not explicitly allow HTML formatting but its common understanding is usually accepted.

No provisions exist in mcsXap to assure that messages are limited to 1024 bytes. This is something I should add to truncate very long text string to help avoid ancillary problems.

mcsXapHub does no enforcement beyond what it needs to understand the header for routing purposes. Other hubs may be better policemen and enforce the letter of the law such as not letting messages of greater than 1024 bytes through. The may fully parse both the header and body to assure they are in strict schema compliance.

The general philosophy I adopted was to be as tolerant as possible of xAP applications that did not have a good grasp of proper syntax.

[CouchPotatoe]

xAP message must be shorter than the MTU of the underlying transport which in the case of Ethernet is ~ 1500 bytes.

There is no lower level defined mechanism to support message spanning or sequencing within xAP although it could be handled somewhat inelegantly by creating a custom schema that appended parameter fragments . We did explore a seq= number in the xAP header but that is not formally specified. Message sequencing and spanning is more appropriate using TCP than broadcast UDP.

As Michael says it's really up to the originating application to avoid creating large messages but the HS plugin is just offering whatever values it is handed and in the case of html these can be long strings. If this then exceeds 1500 bytes then sending the information is not viable using xAP.. unless perhaps some reference to the string could be provided eg a URL for html.