A $4.7M defense spatial computing platform for the U.S. Army that provides field-level maintenance training through an AI-powered virtual environment. The OMNIVORE-IT system (Operator Maintainer Immersive Virtual Reality Environment for Intelligence Training) enables soldiers to train on emerging Intelligence Warfighting Function systems in an immersive, adaptable training ecosystem using spatial computing and augmented reality.
Contributions: Technical architecture, AI integration, C++ plugin development, cloud infrastructure design, team leadership, DoD stakeholder liaison
The OMNIVORE system needed to overcome critical challenges in military equipment training: limited access to physical hardware, deteriorating training equipment, and static training scenarios. Traditional training methods required soldiers to train on actual intelligence systems that were often in poor condition, limited quantity, or not customizable for diverse scenarios.
Our solution leveraged spatial computing and AI to create virtual representations of military equipment that could be manipulated, disassembled, and operated in a realistic virtual environment. This allowed for consistent, high-quality training experiences that could be rapidly deployed across multiple locations and easily updated as equipment evolved.
(onscreen content is a placeholder due to classification)
One of the most innovative aspects of the OMNIVORE platform was its integration of local language models for speech-to-speech interaction with training documentation. We built a custom RAG (Retrieval-Augmented Generation) workflow that allowed soldiers to query technical manuals and procedures using natural language while working hands-on with virtual equipment.
This AI system could process queries about maintenance procedures, troubleshooting steps, and equipment specifications, providing contextually relevant information without requiring soldiers to break training flow. The system worked entirely within a secure, whitelisted cloud environment, meeting the stringent security requirements of intelligence training while providing advanced AI capabilities.
Developing for military intelligence systems presented unique security challenges that required innovative solutions. I architected a custom C++ plugin that could parse and modify Universal Scene Description (USD) data at runtime within Unreal Engine—something that had never been accomplished before. This capability allowed for dynamic loading of equipment models and training scenarios while maintaining the integrity of classified information.
The system's distributed architecture supported multiple delivery methods: direct local rendering for on-premises training, CloudXR streaming for mixed reality devices, and pixel streaming for browser-based access from virtualized environments. This flexibility ensured that training could occur across various security domains and physical locations while maintaining a consistent experience.
OMNIVORE's highly customizable training interface was a major advancement over traditional military training systems. The interface allowed instructors to configure training scenarios based on specific equipment variants and operational conditions, adjust difficulty levels by introducing various equipment malfunctions or environmental challenges and create custom training sequences tailored to unit-specific requirements or rapidly update training materials as equipment specifications evolved.
This flexibility enabled the system to scale effectively across the Army, supporting hundreds of simultaneous users in secure distributed environments. Each training instance could be customized to the specific needs of different units while maintaining consistent learning objectives and assessment criteria.
The system is now being deployed to support training for both emerging and enduring Intelligence Warfighting Function systems across multiple domains, representing a significant advancement in military training technology.