Background

Creating a customised XR training application usually requires programming skills, 3D expertise and considerable resources. This makes the technology difficult to adopt for organisations that need training tailored to their own workplaces, equipment or procedures. IN+SIGHT addresses this barrier by researching how domain experts can create immersive learning environments without specialist knowledge of 3D development.

Project content and aims

Domain experts record a real environment with a smartphone and explain important objects, processes or risks while doing so. These recordings are transformed into digital twins: photorealistic 3D representations of rooms, machines, objects or outdoor areas, enriched with expert knowledge and learning content.

The planned workflow has three main steps:

1. Capture: A domain expert records an environment with a smartphone and can add spoken explanations during the recording.
2. Author: The recording is transformed into a photorealistic 3D environment. Trainers review the captured content, add documents and define learning activities.
3. Learn: Learners explore the environment on a smartphone, computer or immersive headset. They can access information, follow a guided learning path, complete tasks or answer questions.

The resulting environments support independent exploration, familiarisation with equipment, guided procedures and knowledge checks.

Methods and core technologies

IN+SIGHT combines 3D Gaussian Splatting (3DGS), artificial intelligence and web-based XR technologies in a workflow designed for users without professional 3D development skills.

A smartphone app guides experts through the recording process and helps them capture the required viewpoints. The visual data is then processed using 3DGS, including methods for cleaning and preparing the resulting 3D representation.

An authoring tool allows trainers to correct transcripts, integrate information from existing documents, mark important locations and create quizzes or guided learning paths. Trainers remain in control of the content, while artificial intelligence and large language models (LLMs) assist with processing spoken explanations and documents.

Spatial AI links information to specific locations within the environment. This allows the LLM-based conversation to consider both the learner’s question and the area they are currently exploring.

The tools are developed and tested with project partners from several professional fields. Their experts contribute real training scenarios and assess the tools through an iterative, user-centred development process.

Results and impact

IN+SIGHT will provide a research prototype and test it in real training settings. Evaluations will examine the usability of the tools and the experiences of trainers and learners. Selected studies will also compare learning effectiveness and knowledge retention with established training methods.

The results will show whether domain experts can create useful immersive training environments with less support from professional 3D developers. The project also investigates how spatially linked explanations and documents can help organisations preserve and communicate expert knowledge.

Funding