Background and Aim

Creating XR (Extended Reality) training for specific workplaces requires programming skills and expertise in creating virtual 3D environments. This makes it difficult for organisations to develop training for their own workplaces, equipment, and procedures.

IN+SIGHT aims to make this process easier. The project explores how people who know their workplace and its training needs (i.e. domain experts) can create immersive training themselves without programming or 3D development skills.

Procedure

IN+SIGHT is developing a process that allows people without programming or 3D development skills to create XR trainings for their workplace and training needs. As a first step, they record a real environment with a smartphone and explain important objects, processes or risks while doing so. These recordings are then transformed into digital twins: photorealistic 3D representations of rooms, machines, objects or outdoor areas, enriched with expert knowledge and learning content.

The planned workflow involves three main steps:

1. Capture: A domain expert records an environment with a smartphone and adds spoken explanations during the recording if necessary.
2. Author: The recording is transformed into a photorealistic 3D environment. Trainers review the captured content, suggest improvements, add documents and define learning activities.
3. Learn: Learners explore the environment on a smartphone, computer or immersive headset. They can access information, are presented with a guided learning path, complete tasks or answer questions.

The resulting virtual environments enable independent exploration, familiarisation with equipment, offer guided procedures and provide knowledge checks.

Methods and core technologies

IN+SIGHT combines 3D Gaussian Splatting (3DGS), artificial intelligence and web-based XR technologies in a workflow designed for users without professional 3D development skills.

A smartphone app guides experts through the recording process and helps them capture a real environment from all necessary viewpoints. The recorded visual data is then processed using 3DGS, including methods for cleaning and preparing the 3D representation.

An authoring tool allows trainers to correct transcripts, integrate information from existing documents, mark important locations and create quizzes or guided learning paths. Trainers remain in control of the content, while artificial intelligence and large language models (LLMs) assist with processing spoken explanations and documents.

Spatial AI helps the system understand where the learner is in the virtual environment. This means that the AI can take both the learner’s question and the area they are currently exploring into account when answering.

The tools are developed and tested with project partners from several professional fields. Their experts contribute real training scenarios and assess the tools through an iterative, user-centred development process.

Results and impact

IN+SIGHT provides a research prototype that is tested in real training settings. Evaluations reveal whether the tools are easy to use and how trainers and learners experience them in practice. They will also examine learning effectiveness and knowledge retention in comparison with established training methods. In addition, the project investigates how spatially linked explanations and documents can help organisations preserve and communicate expert knowledge.

Together, these activities help assess whether domain experts can create useful immersive training environments without support from specialist 3D developers.

Funding