Discussion Thread for Temperature Data Collection & Display

[Michael McSharry]

Jari & DavidL,
What interface are you using for the sensors? If it is a non-standard interface then the easiest method is to let this interface populate the virtual devices and use "custom" like the others using the TEMP05 plugin just like Bill suggested.

If it is a DS9097U then you can let the plugin collect and store the data for you. Unfortunately right now the TMEX driver for VB has a bug with A/D conversion drivers so the humidity and wind direction sensors cannot be processed with the DS9097U driver within the mcsTemperature plugin. I'm hoping to go to C for this part since the bug has been around for over 12 months and no fixes yet provided.

I provided a script interface to the database, but I would stay away from that right now since there is the potential that the schema may change a little if people want more capacity than is currently provided in the database table.

Version 3 is significantly improved over V2 and if this is something that is often used in your HA setup then the move is worthwhile. If you are happy with V2 then stay with it. The source is there so you are free to make any changes you wish and I'm happy to advise but I will not be supporting V2 per-see.

DavidL, I noticed that the TEMP05 plugin appears to be started on COM1, but you indicated that you have a DS9097U. This is not the problem you are having with mcsTemperature, but something that you look into.

The type of error messages you are getting occur very early in the processing. I will put an error trap at the plug'in entry point to give a little more info. The one setup item that will eliminate some potential problems is the Compatibility Mode checkbox. This should not be checked so only internal HTML formatting procedures will be utilized. If you have the ability to do a screen shot of your setup page then it can be emailed and I can look for anything I was not expecting. When these initial introduction problems get ironed out then messages like you are seeing should never appear.

A note for Bill and others about the non-Temperature devices. Only the temperature devices need to exist on the same house code. Bill is correct that the temperature devices start at unit code 1 and go forward up to unit 20.

[This message was edited by Michael McSharry on Sunday, 05 January 2003 at 04:50 PM.]

[Michael McSharry]

Ed,
I've not noticed any periodic dips such as you see with my setup. One thing to check is make certain the Temperature.asp selection aggregation selection is set to None. This is expecially true for such a short time duration of only 6 hours. This will eliminate the protential numerical rounding/truncation problems as the data samples are attempted to be grouped.

If you have a lot of data then you may want to aggregate to speed up processing otherwise None will provide the purest display on the line charts.

I have the Edit Sensor Map button on the setup page which shows all the temp sensors and how they are mapped to unique sensor id's. It would be very easy to add a bias column and add this bias, if non-blank, to any sensor reading. Let me know if you want it. When I brought up this page here it just had a matrix of labels. I do not know if that is because I have no data in the table or another bug. If/Whatever the problem is I'll get it fixed.

[bdickhaus]

Michael,

This is what I got when I tried the V2 database:

1/5/2003 5:05:36 PM~!~mcsTemperature Plugin Debug~!~Sample Timer=1, Comm Port=
1/5/2003 5:05:37 PM~!~mcsTemperature Plugin Debug~!~Log Temperature INSERT INTO Temperature(SampleDate,R1,R2,R3,R4,R21,R22,R23,R24,R26,R27,R 28) SELECT #1/5/2003 5:05:36 PM# AS Q0,"80.70" AS Q1,"74.17" AS Q2,"71" AS Q3,"72" AS Q4,"0" AS Q21,"0" AS Q22,"0" AS Q23,"0" AS Q24,"0" AS Q26,"71" AS Q27,"30.09" AS Q28
1/5/2003 5:05:37 PM~!~Temperature Plugin~!~ Log_Temperature: The field 'Temperature.R5' cannot contain a Null value because the Required property for this field is set to True. Enter a value in this field.

Also, I did know that the other house codes could be different, just gave an example with them the same, will think about that next time, as I did imply that everything had to be the same house code.

In reference to the temperature device codes, what about a way to set at least the start range of devices? Or maybe a list? My concern is that either the TEMP05 plugin, or other means of gathering data, will not necessarily start at device 1, or even be sequential, requiring that we copy data over into another house code (which I'm also willing to do if I have to).

Bill

[ekuykendall]

While I understand that you don't want this plugin to be soup-to-nuts, I believe a calibration feature would be useful. I suppose we are at that juncture where you are starting to implement hardware features that Ken has been addressing - all I can do is offer suggestions and see where this all leads. Now that I have your package working, I can always go back to the Temp05 plugin for collection. I leave this feature up to you mate! [img]/infopop/emoticons/icon_wink.gif[/img]

[Michael McSharry]

Version 3.1.0 Address PR16, 21, and 22.

This one is a little riskier because I made changes to the code area that addresses processing in the data stream between the TEMP05/ DS9097U and the virtual devices. I've got no way to test this while I'm away from home. Ed has the best chance of seeing if it is done correctly. He's also on the hook because he asked for the calibration feature which required the change. Note that before calibrations can be setup, the sensor IDs need to be recorded with the "Record IDs" button. The process is to select the "Record IDs", wait till the next TEMP05/DS9097 update cycle then go the "Edit Temperature Sensor Map". After a cycle the "Record IDs" status should revert back to normal and IDs will not be recorded until the button is pressed again.

Bill D, If you do not mind please try the old database with V3.1.0. In your case the R5 field has the "Required" propery set. I looked, but could not find out how to explicitly change it, but at my end it defaults to not required when new fields are appended. It turns out that existing field attributes cannot be changed programatically, but it is possible to append a new field in the database with the properties set as desired then copy the data between the original and new field. Hopefully the fields will default to not required for everyone when newly created as it did when I tested it here.

I currently have the abilty to totally remap sensor when using the internal data collection. It is the same page where I implemented the calibrartions. Simple remapping can also be done with the remapping string on the setup page. I do not see it as a big step to enable mapping for the custom data as well. I also started to implement a TEMP05 plugin autosetup to configure the devices to use the TEMP05 plugin device map, or to create a mcsTemperature device code with the same device offsets as used by the TEMP05 plugin.

I'm not aware if I have the ability to change the forum's privledges. I think the only additional privledge a Moderator has is the ability to edit all the messages that are in the forum. As far as I know only Rich and AJ have this ability. I did send Rich an Email on this, but I suspect he may be heading to the conference.

Tomorrow is a work day so hopefully the latest works out for everyone.

[This message was edited by Michael McSharry on Sunday, 05 January 2003 at 10:54 PM.]

[bdickhaus]

Michael,

Looks like Rich, or someone, enabled attachments.

3.1.0 is logging empty fields now, rather than 0's, for the temp devices I don't have, but everything is working fine. Tomorrow (and the rest of the week) are work days for me, so I'm shutting down for the night now. I will try to set aside some time later this week to try out the V2 database.

Bill

[jlikonen]

Mike & Bill D,

Thanks a lot for your help.

I am using nonstandard device (7 temp and 2 humidity sensors) controlled over serial line. I have used logtemperature script (for version 2), but I had to modify it manually in order to log data into the database and to strip out e.g. special characters. V2 has been working nicely for me but I can't change e.g. style settings (font colours) and therefore I would like to upgrade to version 3.

Mike, I didn't quite understand what you meant that I should use the interface to populate the virtual devices. Do you really mean that the interface should collect data from my temp measuring device? I think this won't work in my case because my device uses special commands for reading the values. I think I have to use my own script for collecting the temp and humidity values. I can log data into the database easily with the "old" logtemperature script.

Jari

[ekuykendall]

Michael,

"On the hook"? Yipes. OK - I loaded 3.1.0 before I left for work. I may try and check it via the web from there. Otherwise, I'll play with it tonight. Stay tuned...

- Ed

[Michael McSharry]

Jari,
As I understand it you have a little script or program that is running from a homeseer event or in its own process that is able to interact with your sensor interface(s). For this example I will assume you have a serial interface where you send a "Get Sensor 4" command and it will return the current reading for that sensor. Then the "Get Humidity 1" command, etc. When the value is returned from your interface you should store it in the device string of some virtual device. For example devices "Y4" and "Z2" for these two sensors. Do this same thing for all your sensors on this interface or set of interfaces that you have. With the "custom" interface to mcsTemperature you are responsible, by hook or crook, to put the current readings of all your sensors in a set of virtual devices. The mcsTemperature plugin will take it from there and do everyting else. The setup page of the plugin is where you associate virtual devices to the interfaced sensors.

For another example, assume you are using the weather station software provided with the VS3 weather station from Dallas/AAG(Mexico). It produces a log file that contains each update cycle's readings. You can configure the mcsTemperature plugin with the filename and the plugin will read the data from the file, populate the virtual devices, and then the database.

For a third example, assume you have a weather station that dumps data to a file and you have a custom serial interface such as yours that collects only temp sensor data and you have a DS9097U connected that collect a second humidity reading a few more temperature measurements. The mcsTemperature plugin can be configured to manage the DS9097U internally, read the data file, and accept the data that you put in virtual devices and organize this collection of stuff into the database.

In all these examples you tell the plugin the rate at which you want to transfer the data and it will make the tranfer from the virtual device, text file, or DS9097U interface at that rate.

It will also allow you to collect data yourself from a special interface using a script/exe, format the accumulated data per the same protocol used in V2 as a string of delimeted fields, and call the LogTemperature method within the plugin. The only difference between V2 and V3 operation is this last approach is that the call is hs.GetPlugin("mcsTemperature").LogTemperature rather than the LogTemperature script call (or hs.GetPlugin("mcsTemperature").LogData rather than the LogData script call). Note that this is from memory so the names and syntax may not be quite correct, but the concept applies.

In your case this LogTemperature interface will likely be the smallest change to migrate between V2 and V3, but I recommend that you use one of the other approaches because I suspect that the schema for the database may change either by design to support a future feature or dynamically as the plugin makes adjustments to accomdate the data that is presented to it. I envision that all you need to do in your script that calls LogTemperature (or your version of the LogTemperature script that is called out of a homeseer event) is put the temp and humidity data in a set of virtual devices rather than in a delimited string followed by a call to LogData. While I advised everyone else that they do not use the event/LogTemperature from V2 when running V3, you will need to have this configuration because it is the mechanism that handles you special interface, but you no longer call LogData.

I will be soon adding the option for the plugin to hold the virtual devices should you not want to use one of the 26 house code provided by homeseer. Also note that you can now add suffixes to any of you numbers (e.g. C/%) if you put them in virtual devices and the mcsTemperature plugin will strip these off before it stores the numeric part in the database. This allows the virtual device to show the units while the database is able to support numeric functions on the data.

If you would like more specific don't be shy about asking.

[ekuykendall]

Michael,

Ver. 3.1.0 seems to be working for me, once I discovered that the Temp05 stopped collecting yesterday after I upgraded it to Ver. 5.0 firmware (I forgot to INI it!). Both TempASP and ForecastASP are working, albeit with some quirks. Here are some observations to start with...

1. The plugin seems to be picking up 4 live 1820's and one HS, even though there are only 3 live 1820's and one HS, as verified in Hyperterm. I say live because the 4th temp is changing, not static. When I turn on Raw Data, the fourth unit shows up. I can't seem to correlate the ID's. I may have to unplug all sensors and try each one seperately so I can tell which one is which.

2. What is the Forecast setting on the TempASP page for? It doesn't work!

3. I discovered that although the ForcastASP was picking up the current weather from the right city, the forecast data was always static. I eventually traced it to your ACCID of WA55. Apparently, the field in the control GUI wasn't updating the script. Once I manually updated the script, everything snapped into place. BTW - my ACCID is USVA0645 - you're doc says it should be a "three or 4 character code".

4. Since the ForecastASP insists on displaying local data, even if you don't have any (is there a switch to turn this off?), I am considering getting a 1-Wire weather station. Can this data be collected through your plugin directly through the Temp05, vs. through a Dallas interface, Weather.exe and a CSV file?

5. Everytime I reset the Temp05 (via INI) and re-init your plugin, the virtual codes get remapped (I don't have remapping enabled). I would like the sensors to be assigned 1-up, vs. randomly in the sensor table. Is there a way to do that? I tried the "Use Serial Number for Device Index" switch, and it moved everything around with no apparent logic. Can you explain this feature?

6. My database is full of garbage data now from all the testing and moving about of virtual codes. Is there a way to clear it out? Is there a way to have it roll over after a certain amount of data? Eventually this file could get quite large (1 year @ 1 sample per sensor per 5 or 10 minutes times 20 sensors max plus forecast data).

7. Since I had to restart the collection, I don't have enough data to test the calibration feature. I looked at it in the GUI - it looks good. I'll test this tomorrow.

[ekuykendall]

Regarding observation #1 in the previous post, I discovered that Midon's Temp05 V5 now reads the temp sensor internal to the humidity sensor, thus adding an additional temp reading.

Regarding observation #5, I also discovered that Mitch has organized sensors on the Temp05 by type (humidity, baro, ...temps are last). Therefore, the MCS plugin is reassigning virtual codes to the end of the range (i.e. 4 sensors get mapped to R17-20, vs. R1-4). THis has my mapping all hosed up, and it forces you to have to set TempASP for all 20 sensors just so you can select the last 4 (in my case). I'm sure this can be solved with aliases, but it is late and I need to hit the sheets. More tomorrow.

[jlikonen]

Mike,

Thanks a lot for your help. I haven't done anything yet because I wanted to get some info at first how to migrate from V2 to V3 and didn't want to mess up the working V2 system (yet). I think it is now quite clear how to proceed.

Jari

[Michael McSharry]

Ed, you've been busy with all this info

I will first describe to you how it is suppose to work starting with the TEMP05 sensor mapping, then address the other good info and suggestion you are making.

5. When you do an INI command with V4 or V5 of TEMP05 it will go out and look for new devices on the bus and append those devices onto its list of things to query periodically. Once it is put in its internal table then its order does not normally change. If you have your virtual devices labeled based upon the order from the V4 firmware and then run the V5 firmware the order will likely change since V5 has to go out and look for itself and populate its internal list. When I was popping back and forth between V4 and V5 trying to debug configuration problems my list order was different, hence the collected data was not aligned in the virtual devices and database correctly.

While the TEMP05 sensor index changes, the sensor id's of devices remain constant. The "Use Serial Number for Device Index" switch will tell the plugin to use the sensor's serial number rather than the TEMP05 sensor index to identify the sensor. The setup page that allows you edit the sensor map now comes into play. It shows you the current mapping between the sensor id and the virtual device index. The virtual device index is the same as the database field index as well. The text boxes on that page can be changed to associate the sensor id with a specific virtual device unit code. When this is done, then if V4 or V5 or the DS9097 is used to collect the data it will always go into the correct virtual device. Until the mapping is edited, the relationship is based upon the sensor index from the TEMP05 at the last time the Record IDs button is pressed. In the case of the DS9097 the default is chronological order based upon when the sensor is detected. The only reason you would not want to use this mode of operation is if you are trying to wring every drop of performnace out of your computer. The sensor id lookup is one additional step, but at compiled code speeds it is trivial.

1. I think you found this one already

2. The Forecast timescale on the Temperature.asp page give you access to the forecast data collected in the database. The data in this table is normalized by date so that you are able to graphically compare the quality of the forecast. For example, on Jan 5 there are data points available for Jan 5'th max, the 1 day forecast on Jan 4th, the 2 day forecast on Jan 3, etc. You can now draw a graph that demonstrates the variance between the 5-day forecast and 1-day forecast. In a perfect world the lines would be the same, but in reality the variance is the quality of the forecast. It is likely that the selection does nothing for you because you do not have data collected to support it. Bill Walton was using this to demonstrate the subscript out-of-range problem so I believe it will work once the data is collected. The forecast data is updated by msnbc hourly so you should get some data to work with after you get some sleep.

3. You found a bug in the Forecast setup page. Same kind of bug that Bill found on the device parameters setup.

4. I can put in a switch so only the web data is shown. You can add a weather station and should be able to just attach it to the existing TEMP05 one-wire bus. I've had some problems with the physics of the connection that is likely related to reflections on the one-wire bus due to my specific setup and these problems were the motivation to provide other means of interfacing the data. Through one interface or another you will be able to connect a weather station and use the mcsTemperature plugin.

6. vrIllusions posted a database manager script in the library to weeds out some data based upon its aging. I was thinking about adding a similiar capability to the plugin. I implemented the plugin's database interface to only hold the connection to the database open when data is being written/read from it. This allows his script or an internal procedure to compact and selectively remove data in the background. If you do want this capability, then I suggest that the temperature database not be the same as the Ultralog database since Ultralog grabs a connection and does not release it until homeseer shutdown.

8. Mapping to devices 17..20 is not expected. The V5 prototype that Mitch provided would start numbering each sensor type from 01. Temp 01, Temp 02, Humidity 01, etc. Further info can be obtained by using the Show Raw Data button to confirm that Temp 01 rather than Temp 17 is being delivered by the TEMP05. A second remapping capbility exists within the plugin and that is the remap string where you can put pairs of number to remap the TEMP05 sensor index into a virtual device index. A string such as 1,17,2,18,3,19,4,20 would cause this behavior, but I suspect that this is not the case. I dont have hardware to work with while I'm traveling but if the symptions persist then maybe we can figure out what is happening by working together.

Bill pointed out that the mcsTemperature formum now supports attachments. It would be best if the followup to this thread can occur there and let this one die out.

[ekuykendall]

Folks,

Per Michael's request, please move all new mcsTemperature plugin discussions to the new Plugin forum located here... mscTemperature Plugin Discussion