Researchers at the University of Washington recently studied the potential medical uses of smart speakers, such as Amazon Echo and Google Nest. released a paper detailing their experiments with non-contact acoustic heart rate detection. Thanks to their sensitive microphones, which are typically used to help locate voice commands from the user, the team offers these affordable and increasingly popular smart home gadgets can lead a double life as unobtrusive life sign monitors. The document goes so far that even with a few people in the room, their technique can be used to monitor the heart and respiratory rate of a specific target individual.
These are some bold statements, but they are not unprecedented. Previous research conducted at UW in 2019 demonstrated how smart speaker technology can be used detection of cardiac arrest and monitor the baby’s breathing. This last article can be seen as the culmination of these earlier experiments: a single software that can not just monitor the vital signs of close patients, but actually find emergency medical care. The saving potential of such a program, especially for many young people and adults, would be incredible.
So when will you be able to install a heart monitor skill on the cheap Echo Dot you picked up on Prime Day? Well, as is often the case with this type of research, putting the technique into action in the real world is not as easy as in the laboratory. Although the concept is promising and more than worthy of further research, it may take some time before our low-intelligence speakers are able to Star Trek style detection of life signs.
Can you hear me now?
The researchers explained in the article that their work was based on earlier experiments that used powerful ultrasound transducers to detect small displacements in the patient’s chest, abdomen and neck. Relatively large movements are the result of normal breathing, while smaller sub-millimeter deviations correspond to the individual’s heart rate. While this study shows that it is possible to detect vital signs acoustically from a distance, the report says that the ultrasonic transmissions used are of a frequency and intensity that cannot be achieved by the consumer hardware of the smart speakers they examine.
The team’s sonar performance is designed around the inherent limitations of smart speaker hardware and generates 50 ms of frequency-modulated continuous waveform (FMCW) tweeting between 18 kHz and 22 kHz. This allows the software not only to calculate a very accurate distance to a stationary target, but to ignore an echo that returns outside a predetermined range.
Even in this case, the wavelength of the sound at these frequencies is much larger than some of the movements that the software is looking for. With a body deviation of only 0.3 mm to 0.8 mm, individual heart attacks are particularly difficult to detect. Several layers of filters are applied to try to improve the signal-to-noise ratio, and the adaptive beamforming allows the speakers to reset the target.
The document concludes that the pulse data obtained are not as accurate as the traditional electrocardiogram (ECG) readings used as a control, but are close enough to be a useful diagnostic tool for both healthy patients and patients suffering from various cardiac abnormalities. Especially when you take into account the advantages offered by the non-contact approach, especially for telemedicine applications.
Ideal conditions
Although the researchers were able to extract some impressive data with their test setup, it must be said that there are a few caveats that (at least for now) would make it difficult to put this technique into practice. On the one hand, the intelligent speaker should be aimed at the patient’s chest and placed at a distance of 40 cm to 60 cm (15 to 23 inches). Accuracy decreases with distance, and at least in this version of the software, any return over one meter is considered background noise and is completely ignored.
This is clearly at odds with the normal smart speaker placement. Even if the speaker was close enough to the patient, say at a table next to a bed or chair, it will usually be directed upwards to the ceiling. While testing showed that the software was able to function when the patient was left or right, the error rate increased significantly.
To be useful, the device will probably need to be placed in a custom stand to keep it in the ideal position while taking measurements. This is not a big burden and the costs would be largely insignificant, but this speaks to the relative flexibility of this approach.
Interestingly, the tests showed that the clothing worn by the patient had only a slight effect on the acoustic readings. While tight-fitting clothing gave the best results, even wide blouses and hospital gowns did not prevent the system from collecting useful data. However, thick fabrics or multiple layers of clothing weakened the signal, which could potentially be a problem depending on the climate.
Similarly, the paper found that the system could not reliably extract heart rhythm from extremely obese patients because excess tissue slowed down the underlying body movement. Accordingly, patients with a body mass index (BMI) of 35 or higher were excluded from the main test group. The software was evaluated separately with a group of five individuals who had a mean BMI of 38.6, of whom he was able to detect only one participant’s heart rate. The document notes that this is in line with the findings of previous experiments, and notes that obese patients are simply not good candidates for non-contact sensory technology at the moment.
Penetration into the garden with walls
The problems with the placement of speakers and the type of clothing that the patient wears are, by and large, quite minimal. Obviously some improvement is needed and obviously it won’t be as simple as just installing software on your Amazon Echo and seeing your baby’s heart rate on your phone without making any special considerations, but all the parts seem to be in place to someone can run with the idea. Unfortunately, despite such promising research, providing this capability to consumer intelligent speakers is simply not possible at the moment.
While the paper makes extensive references to these commercial devices and uses their hardware capabilities as a limiting factor in all their experiments, the researchers acknowledge that the software they develop cannot actually be used on any currently available product from Google or Amazon. The simple fact is that APIs available to developers do not provide low-level hardware access necessary for the application of their sonar technique. Providing third-party applications with direct control of the speaker and microphones would raise obvious security concerns, so this type of functionality will either need to be built into the system from the outset by the manufacturer, or potentially transferred to dedicated units. health care providers.
A long journey
Even if bureaucratic concerns prevent the team’s software from being sold on the smart speaker market, the research they conducted is still extremely valuable. The possibilities offered by an accessible, automated method for taking vital patient data without physical contact are unquestionably convincing, especially with the continuing threat of COVID-19. Technologies that can help monitor and diagnose potentially contagious patients without risking exposure to nurses and physicians who serve them are in high demand and will continue to be of great interest even when the current crisis is behind us. .
If then commercial smart speakers are not a viable platform for this software, then what? Well, on the one hand, you can just build your own hardware. As indicated at the end of the article, all experiments were performed with a UMA-8-S seven microphone arrays from miniDSP and a PUI Audio AS05308AS-R speaker connected to a desktop computer. The UMA-8-S is a kind of development kit and includes a built-in digital signal processor (DSP) with the ability to form a beam. Traded for both professional and DIY users, it looks a particularly interesting device in itself.
The team says Python source code for their project is also available on request, as long as you agree to a non-commercial license. Simply put, everything you need to recreate this project is easily accessible to anyone who wants it. Will it become a field of research for biohackers or if it is refined and marketed by a technology entrepreneur, it seems inevitable that this technology will surface in the coming years.