Staff at the West Point Army Cyber Institute’s Reality Lab explores every aspect of augmented reality technology, develops new tools, conducts research, and asks difficult questions.
The Laboratory of Realities is dedicated to research into what becomes known as augmented reality, or XR, a term that includes augmented, virtual, and mixed reality. Augmented reality technologies offer a wide range of military applications, including realistic training, available virtually anywhere, modeling and simulation to develop a weapon system, and real awareness of the battlefield situation.
But like most technologies, computer-generated realities can be a double-edged sword offering advantages while introducing vulnerabilities. The lab is designed to examine both sides of the XR domain.
Jan Kalberg is the man marked to ask tough questions in the lab. He is a social scientist with experience in cybersecurity and experience in red teams, which requires thinking as a potential enemy. “I think mainly about the surprise and what can go wrong. It may sound like a small thing, but when it happens, it’s very important, “he said.
Kalberg raises many concerns. The military can invest large sums of money in XR technology only to be thwarted by the commercially available $ 15 device. Also, combatants could be crippled by dependence on head-up displays (HUDs). “If the HUD disabled person depends on this equipment, can they switch and go old-fashioned? Do you fight a battle with regular iron targets, a regular battle without relying on this augmented reality support? “
Another concern is that sensors that transmit data to XR devices can be “tricked,” forcing commanders to make bad decisions or lead soldiers into traps. “There must be some sensors to power it. So if you submit it with the wrong information, things will be derailed. They can throw something, a $ 15 device, and you turn right into an ambush, ”Kalberg said. “As an adversary, I don’t need to enter your cloud or your networks. If I have a $ 15 device that can just distract you so that your sensor picks it up and misleads you, it’s a higher return on investment. “
Battery life for XR technology can also be a problem, especially in extreme conditions. “Battery life in the far north, like the Arctic, is 50 percent shorter. So if you fight in the Arctic or Korea at high altitude, will you run out of batteries? Will you be on a cell phone and ask, “Do you have a charger?”
The laboratory also draws on the experience of Major Victor Beitelman, USA, a signal officer who was deployed to special operations forces, worked in the defense industry and ran his own company.
His specialty is to look at XR technology with the gaze of a military fighter, assessing how useful the systems would be in combat scenarios. It also solves problems and equips the lab by integrating low-cost, easily accessible technologies such as the Raspberry Pi.
He once built “password-breaking equipment” for training and provided all the necessary documentation with instructions in the library of the US Army Special Operations Command for tactics, techniques and procedures. Major Beitelman also recently conducted training on the integration of various platforms, a starting point for conducting research at the Army Cyber Institute.
One of his current projects is to build a cyber and e-war package for the 82nd Airborne Division. The Cyber Electromagnetic Activity Skip Kit (CEMA) combines single-board computers and software-defined radios, readily available on the free market at a low cost and from multiple manufacturers. It provides a limited opportunity for 24 to 48 hours until more capable equipment arrives. “We attach antennas to it and this allows us to cover the spectrum and find out who is on the battlefield and get an idea of our own presence in the electromagnetic spectrum in this combat environment,” reports Major Beitelman.
The CEMA jump kit is not the only system being developed in the laboratory. Captain Victor Mitsuoka, USA, a researcher and logistician with experience in signaling, intelligence, coding and e-warfare, was one of the original developers of a radio frequency imaging project called RF Viz. It essentially detects radio frequency emissions and shows in the virtual reality system where these emissions occur.
The system can detect enemy or friendly radio broadcasts. If the commander sees that his tactical operations center is “shining like a Christmas tree,” for example, he will know how to take action. “This gives the commander a visualization that he must practice radio discipline. He has to tell the soldiers to stop using their mobile phones, because this whole radio show is being used for radio direction, and then the opposing artillery can put circles on that target, ”explains Captain Mitsuoka.
Captain Mitsuoka works on the nodes of the radio recognition sensor and the visualization for the simulation. He also built the network to connect the sensors together, the data collection system and data translators to take the data from the sensors and pump it to systems such as Microsoft HoloLens and Bohemia Interactive’s Visual Battle Simulation 3, which provides a complete virtual training environment for tactical training, experiments and mission rehearsal for land, sea and air.
“With your soldier avatar, you can see where the sensors are relative to you, and based on what they’re pointing to and the direction of that radio signal, you can also triangulate where the radio signal is coming from,” he said.
Arin Pike is researching whether it is better to integrate the RF Viz system into a heads-up display or on a tablet computer. Meanwhile, Michael Boyce continues to optimize the code, making significant improvements over the past two years.
Pike’s story is in cognitive science and computer engineering. Its goal is to make displays easy to use while identifying trade-offs between platforms. The tablets require soldiers to look down and use their hands, both of which can be harmful in battle. Meanwhile, the glasses reduce peripheral vision, which is also important on the battlefield. The latest versions of HoloLens have improved, so Microsoft is addressing the issue, Pike said.
“One of the projects we worked on was that the augmented reality tool hypothetically showed you the location of invisible threats, in this case sources of enemy radio signals,” Pike said, referring to RF Viz. “You can’t see those with the naked eye, but they can be detected by an antenna array, and that information can be superimposed on that location in your augmented reality system.”
Perhaps unsurprisingly, the cadets find the glasses cool, while the more mature soldiers prefer the tablets. “There is some knowledge of the use of equipment that you have used for several years such as tablets and mobile phones, and sometimes there is some skepticism or discomfort when using new technologies, so part of the research is to examine what the attitude of the subjects is. and how much training or what type of training may be needed to make subjects feel comfortable with new technologies, ”she explains.
Another challenge with current XR systems is that they work best when used in a familiar room. “There is a certain calibration that is expected, so augmented reality, as it is, can work best in an environment where you will always use it in the same room, and there are certain places for comparison … where the glasses can orient themselves, ”says Pike
Although it can be overcome by supplementing GPS information, this need for a familiar environment poses a challenge to using the XR on the battlefield, but Pike says the challenge is not “insurmountable.”
In addition to working on optimizing the RF Viz code, Boyce is working with the Army Simulation and Training Technology Center, Orlando, Florida, on the Cyber Battlefield or CyberBOSS operating simulation system. The system, built by Dignitas Technologies LLC, facilitates cyberspace effects and actions in training systems to provide in a multidomain operating environment.
Boyce is a human psychologist. His work at the Army Cyber Institute focuses on cyber training for brigade or senior commanders and commanders. It presents a perspective for information operations and explores technologies for the best display of readable data.
Currently, his focus is on visualizing large-scale training events. “One of the things we’re looking at is to take a lot of the existing simulation capabilities we have, like our augmented reality and virtual reality systems in Orlando, and make it scalable,” he suggests. It is one thing, he says, to talk about 10 soldiers, it is another to talk about 10,000 with radios, mobile phones and other equipment.
Boyce helped conduct a study using Battlespace’s visualization and interaction system. “We have combined multi-domain operations with the Air Force, so we have ground and air assets and we are trying to see how we can better understand the complex combat space through simulation. This is done through HoloLens, so we are looking at the effects of altitude and situational awareness on cadets’ ability to make tactical decisions, ”he said, adding that he is likely to report the results this summer.
Major Beitelman raises the possibility of future devices with 3D printers to repair or even create their own XR devices, such as virtual reality headphones. “If the knowledge base exists in the military to produce your own, the sky is really the limit. As long as you have the components and the soldering iron, you can assemble it, ”he says. “Using these things with cheap computers and systems … it’s entirely possible for units to be able to build them.”
The approach to building your device may be essential in the future, the main one points out. “Units that don’t innovate will have a hard time coping. Budgets are rising and budgets are falling, so if a unit knows how to produce things organically and can share, and it’s functional, I think that’s added value for taxpayers as well as for the country’s defense. “