Product visualization in AR/VR applications requires a largely manual process of data preparation. Previous publications focus on error-free triangulation or transformation of product structure data and display attributes for AR/VR applications. This paper focuses on the preparation of the required geometry data. In this context, a significant reduction in effort can be achieved through automation. The steps of geometry preparation are identified and examined concerning their automation potential. In addition, possible couplings of sub-steps are discussed. Based on these explanations, a structure for the geometry preparation process is proposed. With this structured preparation process, it becomes possible to consider the available computing power of the target platform during the geometry preparation. The number of objects to be rendered, the tessellation quality, and the level of detail (LOD) can be controlled by the automated choice of transformation parameters. Through this approach, tedious preparation tasks and iterative performance optimization can be avoided in the future, which also simplifies the integration of AR/VR applications into product development and use. A software tool is presented in which partial steps of the automatic preparation are already implemented. After an analysis of the product structure of a CAD file, the transformation is executed for each component. Functions implemented so far allow, for example, the selection of assemblies and parts based on filter options, the transformation of geometries in batch mode, the removal of certain details, and the creation of UV maps. Flexibility, transformation quality, and timesavings are described and discussed.
This paper is based on the paper “Automated and adaptive geometry preparation for AR/VR-applications” (DETC2021-66731) presented at IDETC/CIE 2021 .
AR/VR applications can be used in a variety of ways in product development. The application areas range from VR design reviews (VRDR)  to inspections on real plants to training courses . Current AR/VR applications are predominantly based on game engines that visualize geometries based on polygon meshes, which are best suited for performant real-time rendering. The data transformation of the models from computer-aided design (CAD) systems into AR/VR applications is labor-intensive and requires expertise.
Mobile AR/VR applications in particular are subject to limitations in terms of the available computing power [4,5]. This results in limitations in the possible detailing of models. Since geometry preparation is mostly based on experience and is largely done manually, the transformation is often iterative and time-consuming. This circumstance complicates the integration of AR/VR applications into the processes of product development and use. To reduce these hurdles, automated geometry preparation should be strived for.
Furthermore, different use cases for AR/VR applications focus on different aspects of the product. Even within one use case like VRDRs the requirements toward the geometry preparation may differ based on the agenda of the VRDR. If, for example, only the latest changes to an assembly within the product are to be discussed, the remaining geometries can be simplified and reduced in quality but should be available very quickly. Therefore, automated geometry preparation needs to be adaptive and adjust depending on use case, target platform, and product characteristics.
VR design reviews in particular would benefit greatly from adaptive and automated geometry preparation, as they combine aspects that make manual geometry preparation especially disadvantageous. For example, AR/VR applications for VRDRs need to be suitable to products of any complexity while also enabling usability in early development phases, which are characterized by very frequent and sometimes drastic changes to the product. Accordingly, the time lag between changes to the product and the VRDR must be minimized to ensure actuality and relevance.
2 Related Research
Data preparation for AR/VR applications is dealt with in many ways in research. In engineering applications, not only the problem of transferring geometries arises, which is discussed manifold [6–8]. In addition, information such as product structure, kinematics, and calculation results must be integrated into AR/VR applications [9,10]. For a better integration into the product development process, the synchronization and coupling of AR/VR application, CAD system and PDM system is also crucial [6,11–13]. Research that focuses on geometry preparation [13–16] is sparse.
File formats for scene graphs provide an approach for transferring CAD data into AR/VR applications. However, important formats such as X3D , VRML , GLTF , JT , and STEP are not able to represent all relevant product information [9,10].
In Gebert et al. the concept of the Extended Scene Graph is introduced . This makes it possible to link geometry data, calculation results, configurations, etc. via the product structure and make them usable in AR/VR applications. In contrast to existing exchange formats, this generic approach is more flexible and can be adapted to the specific needs of engineering applications.
3D models that are to be visualized in game engines are usually modeled as polygon meshes. The parametrically defined models from CAD systems must first be transformed into this representation form. In this process, the analytically described geometry is approximated by tessellation methods [21–23]. The approximation quality can be controlled depending on the used algorithm. Common methods in CAD systems, for example, allow control via a maximum secant and angle deviation. The tessellated geometry must then be saved in a suitable file format.
Common game engines like Unity or Unreal can only read a few or none of the mentioned file formats for scene graphs directly  and also cannot import native CAD formats. However, suitable plugins are available. In the Unreal Engine, this is possible via the free plugin Datasmith . In Unity the PiXYZ plugin can be used .
These plugins do currently not support the loading of data at runtime, which is desirable for VRDRs and other applications where, for example, live synchronization of product data is required. The usage of natively supported file formats like OBJ, which can be loaded at runtime, can circumvent this problem.
CAD systems can also be used to prepare the geometry data by saving the analytical geometry as a polygon mesh. Mostly, the same parameters are used for all components of a storage operation. Without manual rework, this method results in unoptimized models in which up to 60% of the polygon count is irrelevant for later visualization . Thereby, performance on the end devices is reduced drastically.
CAD systems and similar tools with application programming interface (API) offer starting points for automation. Both PiXYZ  and CAD Processor  offer programming interfaces that can be used for batch data preparation. Regardless, the individual process steps run independently of each other here. Tessellation, for example, is carried out based on presets, which in turn have no direct reference to the actual performance of the target platform.
Approaches for a geometry preparation process that takes the performance differences of different target platforms into account can already be found in Graf et al. . Balzerkiewitz and Stechert and Salonen et al. also emphasize the relevance of optimized geometry preparation [30,31]. Furthermore, the work processes for movies and video games offer an orientation .
3 A Structure for the Geometry Preparation
The challenge of efficient geometry preparation is the reduction of manual work steps. In a best-case scenario, the process only needs one iteration and produces AR/VR data that meet an optimum between performance and visualization quality while also fulfilling the technical and content requirements.
Not all information included in the CAD data is relevant for our focus on the geometry preparation process. In this paper, CAD data are defined as the analytical geometry and the product structure. Other information such as calculation results, kinematics, etc. is not considered. The result of the geometry preparation is an AR/VR data set, which consists of the tessellated components, the product structure, and any metadata. A component is defined as a part or assembly and the corresponding geometry.
A generally accepted definition for geometry preparation does not exist. Therefore, a structure for the geometry preparation process for AR/VR applications is proposed, see Fig. 1. This structure aims to maximize the degree of automation of the process. The structured process is inspired by the processes suggested by Graf et al. and Salonen et al. [29,31]. Additional steps and functionalities of the geometry preparation were extracted from literature, e.g., [8,9,12,14,16,33] and the analysis of commercial software [25,26,28].
In the proposed structure, geometry preparation is performed in the three main steps of analysis, parameter selection, and transformation, which are explained below.
The CAD data represent at least one single part, but in general, a product consisting of assemblies, subassemblies, and parts in a multi-level hierarchical structure. The product structure is the central data structure for CAD modeling and also for visualization because it brings all product components into a context. It contains IDs, parent/child relations, and transformation matrices. During the analysis, structural data of the CAD file are collected. This includes, for example, the hierarchy of the product and corresponding metadata. Based on a rough examination of the components, duplicates and interior components can be identified. In addition, parameters such as volume, surface types, bounding box, etc. are collected, which are useful in parameter selection.
In addition, a segmentation is performed. The segmentation includes a more in-depth examination of the components, in which specific features are identified. This includes the search for features or surfaces that depict technical details, such as holes and fillets. The identification of shell geometries and possible substitutions of complex geometries by basic geometric bodies is another aspect. Another part of the segmentation is the search for features or surfaces whose geometry is suitable to be approximated with the help of textures.
3.2 Parameter Selection.
Depending on the use case and the target platform, it may be necessary or desirable not to prepare all components defined in the CAD data. An aspect of the parameter selection that influences all subsequent process steps of the geometry preparation is accordingly the component selection. The display quality and detail of the components are largely determined by the choice of feature selection and tessellation parameters. Feature selection includes the choice of geometry details (e.g., holes, fillets) to be removed.
There are interactions between the different parameters. For example, the performance profile of the target platform can influence the component selection, as well as the choice of tessellation parameters and/or feature selection. Changes to the component selection, for example, the exclusion of components in turn enable the increase of the display quality and detail of the components through adjustments to the feature selection and the tessellation parameters.
Based on the results of the analysis, the parameters for the transformation can be determined. Important influencing factors here are also the performance profile of the target platform and the requirements of the target application (see Additional Information on page 10).
Various tools are available during the transformation. These include simplification, tessellation, UV mapping, and texturing of the geometry data. Each tool can be understood as a module. If required, these modules can be implemented using various software solutions.
Simplification functions are often applied before the actual tessellation of the components. Simplification functions can be used to remove geometry details. The motivation for this can vary. A common reason is to optimize the performance in the AR/VR application. However, in certain use cases, it may also be necessary to remove geometry details to hide technical details. Common simplification functions include removing holes and interior geometry , removing fillets [34,35], and creating shell geometries .
The core of the geometry transformation is the tessellation of the parametrically described models. Unlike the other tools described, tessellation is not optional. During tessellation, the original analytically described geometry is approximated with a polygon mesh.
3.3.3 UV Mapping.
UV mapping involves the creation of an unwrapping for a polygon mesh (UV unwrapping). This unwrapping can then be stored in a UV map and is used to enable the visualization of textures on the polygon mesh. Without a suitable UV map, the texturing of complex geometries is not possible. The UV map accordingly serves as the basis for texturing.
Texturing includes functions to generate textures specifically for individual geometries. Textures can be used not only to represent the color and lighting of objects more realistically. With the help of, e.g., normal maps  and displacement maps, geometry details can be displayed that are no longer present in the tessellated components [38,39].
3.4 Additional Information.
In order to meet the demands of adaptive geometry preparation, additional information must be introduced into the process. This information includes the requirements of the target application and the already mentioned performance profile of the target platform. The requirements for the target application include information about which components are mandatory and whether the simplification of components is permissible. The performance profile describes the performance of the target platform's hardware and software. Suitable benchmarks must be used to determine the performance profile and platform-specific features (e.g., minimum required FPS) must be taken into account. The information described currently only flows into the preparation process through manual input, and usually not in the form of quantifiable data.
4 Achieving Adaptive and Automated Geometry Preparation
The main tools of the geometry preparation process described in Sec. 3 already exist. Simple improvements are possible through the automation of sub-steps of the geometry preparation.
A large potential for automation is apparent in the analysis. Both the structural information and the identified geometric properties from segmentation can be detected automatically. Applicable methods already exist for the recognition of geometric details/features such as holes and fillets [33,34].
The transformation can also be highly automated. Even running the tessellation in a batch process improves the geometry preparation process . Automated procedures for UV mapping are implemented in PiXYZ or Blender , for example, and can be integrated into the automated data preparation process . It should be noted, however, that automated UV unwrapping does not provide ideal results for all components. In case of high requirements on display quality and/or very complex components, manual editing may be necessary for optimal results.
However, the main challenge for adaptive and automated geometry preparation lies in the coupling of the different process steps under consideration of all relevant information. This coupling must be realized by evolved analysis functions and parameter selection. The parameter selection depends on various influencing factors.
If, for example, product data shall be prepared for a device with a typical polygon count limit of 100.000, compromises are necessary. By identifying product components that are not needed for the intended use case, the tessellation quality of the remaining components can be increased. Further analysis can detect features and components with a high impact on the polygon count. If these features and components are removed or simplified, the tessellation quality can be improved again.
Theoretically, the parameter selection process can be fully automated if all the required information is gathered during the analysis. The selection of required components can be performed based on the analysis results and the requirements of the target application using suitable filter functions. To make automation of the selection of tessellation parameters possible, suitable methods still have to be developed. These must be able to generate predictions for the resulting polygon count and approximation quality for specific tessellation algorithms (and parameter combinations). Ideally, these predictions can be generated for individual surfaces. In this way, it will also be possible to determine the influence of simplifications on the polygon count and include this information in the automated parameter selection.
The determination of the performance profiles for different target platforms can be largely automated with suitable benchmarks. The requirements for the target application, on the other hand, require manual input. Presets and suitable filters can accelerate this step. If the performance profiles and requirements for the target application are combined with the described potentials in parameter selection, automated and platform-specific data preparation for arbitrary products becomes possible.
The options and software offerings, discussed in Sec. 2, do not include all the required features and tools to implement the proposed geometry preparation process. For example, PiXYZ has limited simplification capabilities, while UV mapping is not available in CAD Processor and most CAD systems.
5 Tool for the Automated Geometry Preparation
The structured geometry preparation defined in Sec. 3 implies various requirements which a software implementation needs to meet:
– efficient and comprehensive data acquisition
– comprehensive range of functionality and tools (e.g., simplification, tessellation, UV mapping, texturing, filter functions aiding the component selection)
– high level of control over implemented functions and process steps through API to enable automation and coupling of process steps
– possibility to add additional algorithms (possibly altering source code) to implement needed functionality e.g., analysis and adaptive parameter selection
As already mentioned, existing software like PiXYZ and CAD Processor do not include all required tools and functions that are available for manual or semi-automated geometry preparation. Therefore, a solution containing several software products was necessary. Additionally, the research project requirements necessitated a reduction in license costs. Ultimately, a solution based on open-source software was favored.
To implement the concept described in Sec. 3, an application named GeoPrep was developed. It enables geometry preparation based on STEP and IGES files. GeoPrep is based on the open-source CAD system FreeCAD , missing functionality like analysis and simplification was added. GeoPrep is still under development, but offers many tools and includes the steps of geometry preparation for the realization of the concept. The GeoPrep core follows a modular design, that mirrors the structure of the proposed geometry preparation process, see Fig. 2. As a class library, the GeoPrep core can be integrated into other applications and process chains. GeoPrep is realized as a WPF application that implements the core.
The GeoPrep API exposes only the functions of the main modules. The main modules use the functionalities of the tool modules, which in turn use the APIs of the base software to realize their functionalities. The main modules analysis and transformation already work fully automated, while the parameter selection only offers limited automation.
In the analysis, GeoPrep realizes the structural analysis and aspects of segmentation. Figure 3 shows some results of the analysis. GeoPrep can detect duplicates, find interior components, and select components based on size ratios. These analysis results are subsequently usable during parameter selection.
During segmentation, the components are examined for possible simplifications. Usable simplification functions include the generation of shell geometries and the detection and removal of holes and fillets. The functions are implementations of topological detection methods based on the B-Rep model [33,34]. The simplification is executed by the “defeature” function of the OpenCascade kernel of FreeCAD. Figure 4 shows some results. The identified simplification options are not applied during segmentation but are added to the identified analysis data. The simplifications are executed after parameter selection in the transformation.
5.2 Parameter Selection.
After the analysis, GeoPrep allows the selection of the components that should be part of the transformation and/or are processed with simplification functions, see Fig. 5. To facilitate the selection, various filters are implemented.
The developed filters select components by
– preset (all components, only needed components without duplicates)
– assembly name
– hierarchy layer
– component size in terms of volume share of the total volume
– duplicate (e.g., all instances of a standard part)
– text search in the component name
– inside components
The default setting of GeoPrep selects only the minimum required components for the transformation. This means that duplicates are not transformed. The components are transformed only once and stored as a reference in all subsequent duplicates. However, it is also possible to select all parts for transformation if required. In addition, it is possible to completely exclude parts or assemblies from the data preparation. This means that they will not appear later either as geometry or in the resulting structural model. Small components and standard parts like screws might not be needed for the visualization and can be found with the hierarchy layer and component size filters. Components that are positioned inside the hull shape of another component and are therefore likely not visible from the outside can be found and excluded.
Before tessellation, the tessellation settings for the components can be selected. GeoPrep can save tessellated components as OBJ, STL, or VRML files. In addition, MTL files for the material description can be generated based on the color definitions.
Level of detail (LOD) models can be created for each converted component. LOD models are polygon meshes with different approximation levels that can be used to further improve the performance of the AR/VR application.
Automated UV unwrapping can be performed using an instance of the Blender software. The resulting UV maps are created by using the Smart-UV-Project function of Blender. Figure 6 shows an automatically generated UV map. The color of the polygons illustrates the distortion that occurs when a texture is projected onto the geometry. Here, black corresponds to no distortion and grey corresponds to a large distortion. The lack of grey in Fig. 6 confirms the assessment made in Sec. 3. For most geometries, automated UV mapping provides adequate results. In the AR/VR application, a texture can be used for visualization without any further steps.
The transformation and export of the components are done automatically, see Fig. 7. The structural data of the CAD file can be saved as xml file. To further speed up the transformation, GeoPrep can use multiple CPU threads. The generated data like tessellated components, LOD levels, and UV maps are saved separately and referenced in the xml.
Even with relatively small assemblies and the use of only one thread, a large time saving can be achieved. For example, the assembly analyzed in the previous figures includes 38 components. For the manual conversion of the components, an experienced user needs about 06:30 min. In contrast, the automated transformation requires only 3 s.
Figure 8 shows a comparison of the geometry preparation by GeoPrep with other software solutions. In all applications, the default tessellation settings were used and the use of any available tools for simplification or UV mapping was omitted. The PiXYZ Plugin and CAD Processor work faster than GeoPrep in this test. The main reason for this is the highly parallelized analysis and performance optimization in these commercial applications. However, both applications also bring disadvantages. For example, errors in the product hierarchy appear in the results (e.g., single faces that are listed as separate components in the hierarchy), which have to be corrected manually. These errors are a large drawback, as they can harm the information feedback and model consistency in product development. For example, automatic assignment of annotations that were created in the AR/VR application to the corresponding component in a PDM system might not be possible, because of the errors. Unreal Datasmith transfers the hierarchy without errors but takes more than twice as long as GeoPrep to do so. PiXYZ and GeoPrep are efficient in saving space on the hard drive. This is particularly important on mobile AR/VR devices, both to manage limited storage capacity and reduce loading times in AR/VR applications. While the subjective display quality is similar for all solutions, differences are apparent in the produced polygon count. The polygon count of the resulting AR/VR data of PiXYZ is less than half the size of the other applications results, this is partly due to the use of quads (that still have to be rendered as triangles by the GPU). However, GeoPrep produces comparable results to Unreal Datasmith and CAD Processor although no optimizations were made in this area, yet. Overall, the performance of the GeoPrep prototype is already comparable with commercial tools. The main advantage of GeoPrep is the wide range of functions and the accompanying flexibility in geometry preparation that was described previously.
6 Summary and Outlook
With the structuring of the geometry preparation process, an approach for the adaptive and automated generation of geometry data for AR/VR applications was presented. The current implementation status of the structured geometry preparation in GeoPrep bundles most of the functional scope of current software tools in one application.
To further improve the parameter selection, the next step is to develop a method that allows predictions for the resulting polygon count and tessellation quality. A promising approach is to identify geometry properties that can be used to describe the complexity of a geometry. A compilation of parameters suitable for describing the complexity of geometries can be found in Johnson et al. . Other approaches are offered by Sukumar et al. [43,44].
Further research is concerned with the development of benchmarks to quantify the performance of different target platforms. Renderers react differently to parameters such as polygon or object count. For this reason, it can be assumed that strategies for the development of suitable benchmarks for different hardware and software combinations are required.
Furthermore, texturing functions need to be automated. Potential is seen here above all in the implementation of procedures that enable the reorganization of suitable surfaces for substitution by textures. This would make it possible to preserve geometry details even on target platforms with low performance without negatively affecting performance.
Some of the results presented were developed as part of the ZIM cooperation projects ZF4123410SS9 and ZF4004704SS9, which were funded by the AiF.
Conflict of Interest
There are no conflicts of interest.
Data Availability Statement
The authors attest that all data for this study are included in the paper.