Announcement

Collapse
No announcement yet.

Location Isolation with ESP32 Bluetooth Low Energy

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Michael McSharry
    replied
    I was not aware of the project, but did a look at the documentation. The passive approach is common between Find3 and what I have done. Both use RSSI as the primary measurement.

    There are two primary differences. The obvious is WiFi vs. Bluetooth and this really make little difference for this application.

    The second is that Find3 translates the set of RSSI into a specific X,Y (or room) as a fingerprint. There is a setup phase where you move the smartphone into each room that is a candidate for tracking and use the set of RSSI measurements from each RPi as the fingerprint for that phone in that room. In my implementation the RSSI is used to trilaterate an X,Y position using an algorithm similar to what is used to isolate earthquake epicenters from multiple seismic station readings. There is no learning phase with this approach.

    WiFi broadcast intervals are measured in minutes and are variable. Bluetooth scanning intervals are measured in seconds with the RPi being more capable of short intervals than the ESP32. If one's use case is to know when the tracked device is starting to move, is moving or has stopped moving then the faster interval is needed.

    I know in my household smartphones are usually stationary when in the house at some known location so when time to use it then will remember where it is. Others may carry there phone on-person all the time. Carrying beacons on a keychain or similar has the same issue. For the tracking to work the user behavior has to be that the device is always on-person.

    The Find3 document does acknowledge IOS MAC spoofing and my experience has been that Android does this as well to protect the user from passive scanning. IOS and Android do this with Bluetooth too. Beacons used with Bluetooth tracking by definition do not spoof their MAC address.

    Find3 looks to be a nice project, but with fundamentally different approaches to location identification. I do not see how what I have done could be integrated other than having two answers that would then be used by a higher level integration application. For me the BLE tracking effort was done as a learning vehicle for ESP32, Bluetooth, kalman filtering, and location isolation algorithms. It was an excellent learning experience. The need for on-person devices eliminates a practical application for me. For others this constraint will not be a problem.

    Leave a comment:


  • OliP
    replied
    Just found this project, looks really interesting.
    Earlier this year i started looking at Find3 - https://github.com/schollz/find3
    Wondered if you were aware of it, or if any part of it could help or assist this project.

    I've paused work on implementing Find3 for now as i have too much on my plate, but am still interested in this area.

    Leave a comment:


  • Michael McSharry
    replied
    I am familiar with ESP-NOW which is a capability developed Espressif. I had always considered it's benefit to be with battery operation associated with deep sleep so the ESP could be awaken and the WiFi on-time was significantly reduced largely due to reduction in the overhead to establish a connection. I had not considered the application where one server acted as a gateway in a star configuration. This gateway node would then still need to communicate with an application server such as HomeSeer using some other protocol (such as MQTT).

    What I cannot envision is how this becomes integrated into other HA components. All ESP processor products use a single WiFi radio and even the ESP32 shares the Bluetooth and WiFi antenna. Once data has been collected by the ESP-NOW server I do not see how it gets communicated to other parts of the HA environment. I suspect it may be possible to use something like RPi that has both ethernet and Wifi, but then the software would need to be developed to implement the Expressif protocol on the RPi hardware.

    There are other variants of communication protocols that have been developed to satisfy a specific niche. These typically exploit some aspect of a standard protocol to remove a capability and optimize another. When the standard is sacrificed then the breath of application is reduced. These solutions have their place, but they do not replace the predecessor protocol.

    In the general case of low power IOT devices one expects the devices to either lose connection because they are low-cost marginal hardware designs or they intentionally go offline to conserve battery. A system designed needs to consider the Quality Of Service that is being supplied. For example, consider an IOT device that turns on a light and it is important that the light does turn on. The application (e.g. HomeSeer) commands the light to the on state, but the IOT device it not connected at the time the command is sent. The MQTT protocol assures a QOS so that the command is held back until the IOT device has advertised itself as now being awake and connected.

    Leave a comment:


  • kiley.jerry@gmail.com
    replied
    I have your mqtt plugin but I am not sure mqtt is the most efficient protocol to use. Have you seen the ESP-Now protocol. It is made for stuff like this. 40ms from sleep to transmitting. Hardly any overhead so you could do wireless reveivers and have amazing battery life or much shorter intervals. Once a device is paired to the master it is a persistent connection. The master can be directly connected to the home seer servers and have 20 receivers If you need more receivers add another master. It has CRC and you could pull individual sensors on different frequencies. I love mqtt but this is what this protocol is designed for. 250 byte packet size. Basically a wireless can-bus system. I am not a programmer or I would try to do it myself but I have used ESP-NOW with amazing results. Hopefully this suggestion doesn't annoy you but you did say you were trying different things.

    Leave a comment:


  • Michael McSharry
    replied
    While I gave up on nodejs due to abandoned libraries, I did find libraries that can be used with C on RPi. I can do continuous polling of BLE beacons and report any change in RSSI. I am moving down the path of trying to implement the same business logic that I used in the ESP32 in the RPi. Short of this it should be pretty easy to just do reporting of RSSI/Distance from the RPi. I suspect the same will work with USB dongles as with the BLE built into the RPi.

    Leave a comment:


  • Michael McSharry
    replied
    Extending BLE location monitoring to RPi using nodejs seemed like a very good excuse to learn nodes just as it was to learn ESP32 development. Unfortunately it looks as if the nodejs Bluetooth libraries have been abandoned as many others seem to have the same install issue I have been having. I may try again in the future, but it is not looking very positive as the issue surfaced years ago and it seems to still exist. There was a lot of interest when Bluetooth first became available with RPi, but now the interest seems to have dwindled.

    Leave a comment:


  • ts1234
    replied
    Michael,

    Thanks for doing all you do. This is very over my head. Hopefully in a year or so I'll be a little more fluid. I do think long and short range BT have a lot of uses...but since home automation to me is your home responding to you automatically...stable tools for "where" you are in your house as well as arrival / departure sensing are really core/key to everything. I'm continuously amazed that major manufactures have not focused on this as about everything else could really be done with simple timers and hard wiring. That's a bit of an exaggeration. no doubt. However, some of the biggest complaints I've see posted center around things especially like the flakiness of zware arrive sensors. For example, I want my garage to open right when I drive up...not a 1/2 a block away...or sitting there waiting for it to open. I don't want it to open at 2 AM because my zwave arrival sensor disappears wjile I'm sleeping and then comes back. Clearly BT has many applications, but most (really all) reasonably priced approaches today are simply not stable and depend on cell phones and/or wireless TCP/IP and really work perfectly only about 30%-50% of the time. Cell service in my neighborhood sucks which make this even worse. Some of this might be OK for somethings, but for unlocking your house or any security related function it simply isn't.

    Tom

    Leave a comment:


  • Michael McSharry
    replied
    I found a very promising approach using node.js on YouTube. While I have never developed anything in node.js, it is used for zigbee2mqtt, node-red and many others. I followed the YouTube video, but ran into a library install issue that appears to be related to the version npm/node.js that is current. Node.js should provide an easy port to the API used for ESP32 so should be able to mix and match RPi/ESP32 with mcsMQTT. I will see if I find anything else following this thread tomorrow.

    Leave a comment:


  • ts1234
    replied
    That is pretty awesome!!! In my testing I couldn't connect either. I am guessing you'd see a more gradual RSSI value change with some of the long range ones. I'm way over my head here...just starting to load Visual Studio ...and already digging since some of the menus have changed in 2019 it looks like, but can't even find the Homeseer templates or even know if it's ok to run VS2019 or I really need to find the 2017 version. ...Going to have to start reading.

    Leave a comment:


  • Michael McSharry
    replied
    I have been poking around and see that RPi has bluetooth built-in, but the same should work with dongles as well. At https://www.raspberrypi.org/forums/v...ic.php?t=47466 is a good discussion of using linux script for bluetooth tracking. The core of it is the hcidump command. I have been playing with it and can get "hcidump lescan" to show the bluetooth devices. If I run "hcidump lescan --duplicates" then it produces a constant stream of output showing the nearby bluetooth devices. This is the continuous scan that you desire.

    I was not able to get the beacon to connect which I suspect is normal for a beacon, but to use the rssi hcidump command one needs to be connected. In this same thread I found "sudo btmon |grep RSSI &" to spawn off a process that dumps rssi info from bluetooth. I then setup a whitelist with one beacon and used hcidump with duplicates to repeat the scan. This resulted in the image below.

    Click image for larger version

Name:	Capture.PNG
Views:	80
Size:	53.5 KB
ID:	1324088

    I then moved the beacon 10 ft away and got RSSI around -90. Another 10 ft and the RSSI only marginally change to -92 or so.

    What this shows me is that one should be able to get BLE RSSI info for beacons relatively easily with the drivers built into the OS. More google seaching may even find more complete projects that may be able to be easily adaptable to work with HS.

    Leave a comment:


  • ts1234
    replied
    Again, thanks for EVERYTHING!

    I find it almost funny that one of my jobs was taking care of Sun Microsystems data network for 7 years and yet in that time I never learned UNIX (sigh). It seems far easier to go the RPi route as you state. I'll look over the SDK, but guessing it's not the way I'd go as one of my weakest points has always been ...read the darn manual.

    Again,

    Thanks for all the help!...especially on Labor Day...

    Tom

    Leave a comment:


  • Michael McSharry
    replied
    The ESP32 power is not the limiting factor, but I suspect its architecture is. It has a single 2.4 GHz antenna that is used for both Wifi and Bluetooth. Since Wifi is its method of interface it needs to keep this part relatively responsive. Something like a RPi with Ethernet for the interface and USB for the Bluetooth should have plenty of horsepower. I do not have experience working with hardware devices on Linux, but you may be able to do it at the Linux rather than source code level such as shown https://raspberry-projects.com/pi/pi...tooth-commands

    If I was to go the RPi route then I would not write the HS plugin, but use MQTT as the communication protocol and let mcsMQTT handle interface with Homeseer. This should also be doable at the Linux level by piping the BLE output to mosquitto_pub application. Of course a tighter and more flexible approach would be to develop source code that communicated directly with the devices. You could even follow the API that is documented in mcsMQTT.pdf so that the additional BLE features of mcsMQTT could be utilized. It just depends upon how fancy you want to get. It does seem, however, that just basic info is all you need. Since the RPi has the horsepower it seems most appropriate to maximize its share of the processing and only communicate event-level information to HS.

    If you do want to write a plugin, but the Homeseer SDK download provides a template. There are also users that have improved upon the HS template. I have no specific recommendations, but a board search should find some hits.

    Leave a comment:


  • ts1234
    replied
    Wow! Thanks for taking all that time! Great info!

    I'm guessing RFID probably is the answer, but I'm even less knowledgeable about that than BT. :-)

    So, I did a little testing of my own after you steered me in the right direction. I do wish I knew what you know as I fear this will end up taking a bit to learn about writing plugins. I have the outdoor version of the Feasycom 500 meter and a BP-109 which claims a 1000 meters I think. In my PC I have a TRENDnet Low Energy Micro Bluetooth 4.0 Class I USB 2.0 with Distance up to 100Meters/330 Feet. TBW-106UB. For this very simple test I loaded the Bluetooth LE Explorer from the MS APP store. I have it set for continuous enumeration and both of the Feasycom's set to around 150-100ms.

    I'm actually amazed at how many other BT devices are around including people who walk by on the sidewalk.

    I will say that I see RSSI value changes about every second via the app. Of course, this is a i7 processor not an ESP32. At least in initial testing I see fairly constant increases and decreases in RSSI as I move them around. I am trying to stay in the same line/plane as I move them away since angle does seem to have an impact. For my application this would be OK since the approach angle would be fairly constant.

    The variation in RSSI seems to keep a value around +/- 5 but does bounce around. I am guessing something like that the smoothing algorithm you mentioned in your doc could handle that easily.

    So, this brings up two questions...

    One, is there something out there that is basically a more powerful ESP32 board that I could use your current plugin somehow?

    Two, is there a primer somewhere I could start to learn how to write plugins myself?

    I used to program in C++ years ago before it became object oriented and been in networking and communications for 40 years, so I get abit about protocols. While I swore I'd stay away from tech.... I literally just retired and probably can find some cycles for things like this now. I've seen some code around for reading BT locally on a PC, but have a 1000 miles more to go to learn to take that (once/if I figured that out) and then write the Homeseer side of it.

    Leave a comment:


  • Michael McSharry
    replied
    Seems like what you are trying to do is better suited to Active RFID rather than Bluetooth. I recently did some development to replace the technology that was selling about 15 years ago that is described in Section 16.16.2 of http://mcsSprinklers.com/mcsMQTT.pdf. Active RFID is a RF signal, in this case 433 MHz, that contains a data pattern that is repeated every few seconds. As the transmitting device becomes within range the receiver will detect it and then when it goes out of range the lack of signal will be detected by the receiver.

    The pattern that I used consisted of 4 characters. The first three were constant and the last cycled through the alphabet. The first three are used to distinguish one transmitter from another. The last was used to evaluate if the receiver was missing transmissions. As transmissions were missed I knew the transmitter was at the end of detectable range. I little additional processing on the receiver could use the magnitude of the missing transmissions as a signal strength indication.

    I have CheaperRFID (old technology) units that had RSSI as well as those that did not. I though the RSSI would be able to improve the algorithm for presence detection, but found it not to be that useful. Nevertheless I got a receiver that was advertised as having RSSI output to evaluate this new CheapestRFID approach. When I received it, there was no pin that had any output other than the data signal.

    Today I did a little searching and found one from Sparkfun that actually had a datasheet with a analog output that is a signal strength indication. It may or may not work well, but I am curious as to what it will do. This one is 315 MHz rather than 433 MHz. This will give a little better range and be the same frequency that I have for CheaperRFID.

    ------------

    Thinking more about RSSI, then the same approach should be just as easily done with WiFi as Bluetooth as the RSSI is readily available. Both are at 2.4 GHz so should get the same range (which will be less than the 433 Mhz or 315 Mhz Active RFID). It would be less suited, however, for battery powered devices. This is something I can play with as well to satisfy my curiosity.

    ------------

    To answer your specific questions.
    1. I found that ping intervals in the 5 second range would choke the ESP32 program. I do not know exactly why but since it was not practical I did not look more deeply. The advertise request may not always be acknowledged by the tag or acknowledged with the same latency each time. When multiple tags are present then collisions become more of a concern as the request interval decreases. I believe that a single request is actually realized at a set of three or so RF transmissions spread over a random time interval (and I think channel in the Bluetooth 2.4GHz band). This Bluetooth design is explicitly to deal with RF signal collisions.

    2. I did not do explicit testing for RSSI/distance vs. motion, but generally my experience is that the RSSI does not change incrementally as distance changes. RF has many ways to get between point A and point B so while the beacon moves the actual signal strength may not change just because of distance. With trilateration from multiple ESP32 receivers there is a better chance that a motion change will be detected.

    Leave a comment:


  • ts1234
    replied
    Michael,

    I don't mean to be a bother, but before I jump off into this project I have a couple quick questions. I'm not that skilled with the protocol and devices yet so I'd like to be clear if what I want to work...will work.

    1) If both the "transmitter and receiver" plug into a power source (IE - Not battery dependent) How low can the ping interval be? I would like 5 seconds or less as I want what it controls to seem very responsive.

    2) Assuming that the Arduino board is kept in a fixed location and that a remote iBeacon always approaches from the same angle...can I depend on the RSSI value to be exactly/close to being the same value when a particular distance is reached between the ESP32 and the remote beacon (probably a FEASYCOM 500m Long Range Bluetooth Low Energy Waterproof iBeacon)?

    Thanks for your time!

    Tom

    Leave a comment:

Working...
X