In 2024, it seemed like you couldn’t go anywhere without hearing about the Apple Vision Pro or seeing content creators documenting their daily lives in this new mixed reality. I spoke with several clients about the benefits and potential applications on this new technology, but the U.S. Air Force was the first to dive in and want to test it out with their Air Education and Training Command (AETC). The core idea that the project was built on was “can the Air Force use remote sessions powered by the AVP to give their students in Texas access to the global network of skilled flight instructors?” It was ambitious and exciting to help them envision the future of flight training.

I built a scope that started with embedding our UX researcher with the Air Force instruction teams on base near Del Rio, Texas. Here he was able to observe in-person sessions and learn what tools are most important to the instructors and students and the overall script for these sessions. This was important in identifying what functions, information, and controls would need to be available within the Apple Vision Pro prototype.

Next we moved into design for the UI to give everything a sleek, but mission-focused style. We also proposed making use of the augmented reality space by having a 3D map of the virtual flight session to see the terrain, ground targets, and aircraft positioning. Our UI gauges and widgets would all be powered by API endpoints directly from the Air Force simulator so we had to be very aware of what data was available to us.

Finally, we engaged our engineers to build the AVP prototype. Initially we pulled in video streams of the simulator scene and gauges to test the flow of data between the simulator and our build. Once that was validated, our team began to work with the APIs to hook up our gauges and populate the 3D scene.

After testing and iterative updates, we delivered a fully operational prototype within the Apple Vision Pro for the U.S. Air Force. It is exciting to be on the forefront of integrating new technologies into building our armed forces for the next-generation.