NVIDIA Unveils Technology That Transforms Workers into 'Augmented Cyborgs': Here’s XR AI
Artificial intelligence is undergoing a significant transition: from conversational models and assistants integrated into digital workflows to systems capable of operating directly in the physical environment. In this context, NVIDIA has announced XR AI, a library for developers designed to build AI agents that can perceive, reason, and act in real-time through augmented reality (AR) and extended reality (XR) devices.
The platform revolves around four main functional areas. The first concerns the acquisition of signals from the real world coming from AR glasses and XR devices: video, audio, depth data, pose information, and various sensory readings are incorporated into the system to enable contextual understanding of the surrounding environment.
The second area concerns the connection of agents to specialized tools and services. In this area, XR AI integrates with NVIDIA Metropolis and the Video Search and Synthesis (VSS) module for visual analysis, as well as NVIDIA NeMo Retriever for retrieving information from enterprise systems through retrieval-augmented generation (RAG) techniques.
On the front of language and reasoning models, the platform supports a wide ecosystem that includes NVIDIA's Nemotron models, Cosmos Reason, and other compatible foundation models. Finally, regarding orchestration and runtime, NVIDIA NeMo Agent Toolkit manages the use of tools, reasoning flows, and coordination between multiple agents, while computational infrastructure is ensured by systems such as DGX Spark, DGX Station, and RTX PRO platforms, with support for cloud, datacenter, and edge deployment.
The declared goal is to reduce latency while maintaining a non-distracting user experience, two particularly critical requirements in high-concentration operational environments.
Several entities are already exploring or adopting the platform in concrete application contexts. Siemens is conducting research on the use of XR AI and DGX Spark in the manufacturing sector: a technician equipped with lightweight glasses can query an AI agent regarding an issue on a programmable logic controller (PLC) and receive real-time guidance, with contextual access to industrial systems, digital twins, and automation flows.
In the scientific field, Rana - a company part of the AutoBio group - has introduced its LabOS system on XR AI to bring spatial intelligence directly into research laboratories. LabOS provides real-time procedural guidance in hands-free mode for complex experimental workflows, initially focused on stem cell therapy and gene editing research, in collaboration with Stanford University's Cong Lab and Princeton University's Wang Lab.
The system is capable of identifying CRISPR samples and tools, guiding every stage of the experiment, and producing a structured and reproducible record of the entire process, even in scenarios of collaboration between human operators, robots, and AI systems. LabOS is compatible with smart glasses from Meta, Rokid, and VITURE; the latter has integrated XR AI into a wearable interface that allows workers to access the right context at the exact point where the task is taking place.
In the operating room, the Surreality Lab at the University of Pittsburgh Medical Center has presented an implementation of XR AI on DGX Station designed to support the surgical team with contextual assistance. A particularly relevant technical aspect of this implementation concerns the system's ability to determine which portions of the surgeon’s field of vision should not be occluded, in order to overlay useful information without interfering with the patient's view and the ongoing procedure.
In the field of automotive design, Innoactive has developed an experience based on DGX Spark that enables teams to preserve the context accumulated during immersive design review sessions, product showrooms, and interaction with digital twins, transforming isolated sessions into repeatable business processes.
Lastly, Atlantic Studios - a production studio with awards from the Academy Awards and Emmys - has integrated XR AI for an interactive immersive experience dedicated to the wreck of the Titanic, allowing users to navigate a three-dimensional scan of the site via voice commands and receive contextual information in real-time.