HM-SYNC: A Multimodal Dataset of Human Interactions With Advanced Manufacturing Machinery

Developing a deeper understanding of complex manufacturing systems, processes, and capabilities is a key to enhancing design choices and improving design efficiency. This realization is deeply rooted in practices such as design for manufacturing (DfM), whereby designers develop products to adhere to the realities of manufacturing and production capabilities. The efficacy of DfM relies in many cases on accurate and reliable analysis of a manufacturing process to inform design choices [1]. From this perspective, it is crucial to improve ways of understanding and modeling complex manufacturing systems for accurate prediction and integration with the design process. However, assessing manufacturing systems is becoming increasingly challenging as machines grow more interactive.

The manufacturing field faces significant changes due to influxes of advanced technologies such as industrial automation, robotics, and digital twins, as well as from pressures of increasingly competitive global markets [2,3]. Along with a technological shift in manufacturing systems, changes are also occurring in the role of human interactions within complex production systems [4,5]. Where humans once held primarily direct manufacturing roles through manual tool operation or hands-on product assembly, human interaction has expanded to include more indirect roles such as collaborating with robots or supervising automation systems [6]. These human–machine interactions are not yet sufficiently well characterized for integration into complex system models.

Humans play an integral role in many systems, and human performance is a determining factor in the outcomes of human–machine systems [7]. Human interaction and performance research are also critical in the areas of automation technology design for human use [8–10] and designing the layout of reconfigurable manufacturing systems [11,12]. Research in this area contributes to human-centered design and manufacturing productivity, especially in the ongoing industrial shift to Industry 4.0 and, eventually, Industry 5.0 [13]. Supporting this view, Gorecky et al. [6] and Lu et al. [3] demonstrated that understanding human–machine interactions can significantly enhance manufacturing systems by increasing flexibility, improving strategies, and facilitating expert knowledge transfer between workers and stakeholders. They also highlighted the need for more research on capturing diverse human interactions in manufacturing settings to develop a holistic approach for system performance improvement. Sadeghi et al. [14] emphasized the importance of considering human behaviors in design to achieve better integration of human interactions. Zhang et al. [15] underscored the need to capture human interactions in industrial assembly to reduce faults, minimize errors, and enhance predictability in manufacturing.

A fundamental step in addressing the challenge of observing and understanding human interactions in manufacturing environments is to establish reliable methods for sensing and interpreting workers’ actions. Human activity recognition (HAR) has emerged as a promising approach for reliably sensing and interpreting workers’ actions, particularly in industrial settings. A significant body of work has focused on HAR for human–robot collaboration (HRC), where the goal is to train robotic systems to recognize human actions and anticipate movements, facilitating more seamless collaboration. Notable datasets such as HRI30 [16] and InHARD [17] offer robust collections of data from users performing 30 actions and 13 actions, respectively, in HRC contexts. Similarly, Roitberg et al. [18] compiled a dataset encompassing 15 HRC-related human actions, including gestures for controlling collaborating robots.

While these datasets provide a strong foundation for industrial HAR, they primarily focus on HRC where data are collected in controlled laboratory settings, limiting their applicability to real-world industrial settings. Additionally, the data collection methods used, such as direct sensor application on humans or RGB video, raise privacy concerns. Another dataset, HA4M [19], compiled general industrial actions to support HAR. This multimodal dataset captured 12 manufacturing actions centered around the manual assembly of a small gear train system. Although more broadly applicable across various manufacturing contexts, these actions are limited to direct, hands-on tasks involving minor hand movements, excluding machinery interactions. Like the previously mentioned datasets, HA4M’s data were collected in a laboratory setting, making the transition to real-world applications challenging. The HUMM dataset [20] presents an interesting deviation from classical HAR, as it aims to understand human movement in manufacturing environments for trajectory prediction within HRC contexts. Although it focuses on human movement for robots transporting items, the insights from HUMM could be valuable for manufacturing layout design, potentially enhancing the movement efficiency of workers within manufacturing spaces.

In addition to capturing human interactions for the purpose of HAR, several studies have analyzed human–machine interaction with the goal of improving design. Notably, the dataset created by Carrera-Rivera et al. [21] observed human–machine interactions to support the development of adaptive, digital human–machine interfaces (HMIs). Similarly, Reguera-Bakhache et al. [22] presented data-driven methods for developing adaptive HMIs and collected a dataset of human–machine interactions through a computer interface. These studies involve extensive data collection from real machine use to support manufacturing design, specifically in the area of digital user HMIs. While many machines are used through digital interfaces, there is still a lack of data collected to understand how humans interact with more dynamic machinery that includes a broader range of human–machine interactions, such as hands-on tool use or assembly.

After reviewing this existing literature, summarized in Table 1, we identified significant gaps in the analysis and generation of datasets capturing human interactions within manufacturing environments, especially concerning manufacturing design. While some datasets are available for training algorithms to detect human industrial actions, they often lack observations of humans operating advanced manufacturing machinery. Most current datasets, as discussed earlier, capture either simple human gestures or general actions, like picking up and placing simple objects, which offer limited context for overall manufacturing processes. Moreover, datasets that do capture human–machine interactions mainly focus on digital HMIs and overlook the full range of interactions, from direct, hands-on operation to indirect digital interfaces. Additionally, all of the datasets in Table 1 were collected in highly controlled laboratory settings, reducing their relevance to real-world manufacturing scenarios. There is a clear need for datasets that capture human activity in manufacturing environments more representative of existing facilities, observing natural human use of machinery rather than predefined actions. Finally, there is a scarcity of datasets that capture human–machine interactions in a privacy-preserving and nonintrusive way, which is essential for protecting user privacy and ensuring authentic human behavior.

This article presents the collection and curation of a comprehensive dataset of human–machine interactions in an advanced manufacturing context. The goal is to support a data-driven understanding of how human interactions impact the manufacturing design process, and how this understanding can be used to improve design. This dataset, called Human-Machine interactionS in dYnamic advaNced manufaCturing settings (HM-SYNC), provides valuable insights to better inform product and manufacturing designers about the capabilities and uses of today’s manufacturing technologies and facilities. To achieve this effectively—and to align with the intelligent systems increasingly used in manufacturing analysis, such as machine learning—this work focuses on collecting a systematically derived dataset to support learning in these intelligent systems and enhance decision-making capabilities.

We address the aforementioned gaps derived from Table 1 by creating a dataset that captures human interactions with machinery in an active advanced manufacturing environment, utilizing noninvasive, privacy-preserving sensing technologies (i.e., depth sensors). Over a 6-month period, more than 1200 interactions were recorded between users and an advanced manufacturing machine—the wire-arc additive manufacturing (WAAM) machine—capturing a wide range of large-scale and small-scale activities, such as manually grinding down metal surfaces and adjusting machine calibration settings through a digital interface. A detailed analysis of machine selection and action diversity is presented in Secs. 2.1 and 2.5, respectively. HM-SYNC captures a broad range of human–machine interactions, focusing on understanding natural human use of machinery in a manufacturing setting. Additionally, it is one of the first datasets to achieve this in a privacy-preserving manner, with the goal of enhancing human-centered manufacturing design by analyzing human activity in manufacturing environments.

The remainder of the article is organized as follows. Section 2 details the data collection methodology, beginning with the justification for machine and sensor setup choices in Secs. 2.1–2.3. This is followed by an exploration of HM-SYNC’s format and organization in Secs. 2.4 and 2.5. The methodology concludes in Secs. 2.6 and 2.7 with a description of how the dataset adheres to proper data-sharing standards and ensures the quality for its intended applications. Section 3 discusses key potential applications of the dataset to support engineering design in manufacturing settings. Finally, Sec. 4 addresses HM-SYNC’s limitations and suggests future directions.

This section discusses and justifies the experimental setup and data design decisions. The primary considerations were (1) machine selection, (2) sensor selection, (3) sensor placement and hardware setup, and (4) data capture and formatting. We provide a detailed overview of the organization and curation of HM-SYNC, justify its content and size, and discuss its adherence to findable, accessible, interoperable, and reusable (FAIR) principles.

The primary criteria for selecting a suitable machine for this study were its capacity to support a diverse range of human interactions and its compatibility with sensor-based observations. The Lincoln Electric Sculptprint RND WAAM machine [23] was chosen as the focus of this work for the following two reasons. The WAAM machine is a large-format metal 3D printer housed in a 2.2 m × 4.1 m × 2.3 m (⁠$L×W×H$⁠) chamber. It features a robotic welder arm that deposits molten metal filament onto a specially configured build plate in a layered fashion. Images of the WAAM’s exterior and interior are shown in Figs. 1(a) and 1(b), respectively.

Interactions with the WAAM machine are highly varied, ranging from direct actions, such as manually adjusting the build plate for different projects or grinding the deposited surface with a hand grinder, to indirect actions, such as maneuvering the robotic welding arm with a mobile joystick controller or setting machine parameters on a control panel screen. A full listing and description of each interaction is presented in Table 2. This compilation of interactions is derived by referencing the technical Lincoln Electric Manual [23], observing interaction patterns from data and in-person shadowing, and consulting WAAM machine users about their routines and practices.

Interestingly—and underscoring the need for this work—many actions, especially the sequence of actions performed by WAAM machine users, were not formally outlined in the technical manuals provided by Lincoln Electric. Discussions with users revealed that they had developed their own techniques, including adding, reordering, or skipping certain actions, as a result of unique experimental cases with the WAAM machine and their discovery of more efficient workarounds. These realizations reinforce the idea that actual machine use differs from intended use from machine manufacturers. This also supports the view that more focus needs to be placed on observing humans interacting in real-world manufacturing processes such that designers upstream can be informed about accurate accounts of manufacturing performance, and machine producers can be informed about how their machines are being used downstream such that they can improve machine designs or keep technical documentation accurate and up to date.

Table 2 shows that the types of interactions effectively cover the various roles humans are assuming in manufacturing settings with the influx of sophisticated industrial technologies [4,5]. The WAAM machine demonstrates new techniques that are not covered in formal documentation, highlighting the need for updated observations and concurrent learning. With this in mind, compiling a dataset of human interactions with the WAAM machine will be relevant to human interactions with various machines in similar situations and across multiple manufacturing design contexts.

The Microsoft Azure Kinect [24] was selected to capture human behaviors due to its multimodal data collection capabilities—including a best-in-class 1 MP depth camera, $360$ deg microphone array, 12 MP RGB camera, and orientation sensor for building advanced computer vision and speech models—and comprehensive Software Development Kits (SDKs) that support skeleton-based HAR [24,25]. Key specifications for all integrated sensors are presented in Table 3. The multimodal nature of this sensor is essential for enabling the observation and labeling of the data before narrowing the modality choice for the dataset.

Another issue that was thoroughly considered was privacy and the value of information each of these sensors provides, as the more information a sensor offers, the less privacy it typically preserves [26]. On one end of this spectrum, RGB imagery includes the most rich and valuable information, enabling object recognition alongside human skeleton detection. However, it is privacy invasive, revealing detailed facial and characteristic features. On the other end of the spectrum, technologies like passive infrared sensors or radar are the most privacy preserving, but contain less rich data and require significantly more computation to interpret. Depth imagery strikes a balance between these two ends of the spectrum. The information provided by the data is sufficient for effective computational processing of human interaction, all while still maintaining user privacy. Hence, depth imagery was uploaded and used from the Azure Kinect sensors. This is due to the nature of depth imagery to support powerful computational techniques, such as skeleton-based HAR, while also supporting user privacy. Another advantage is that, since depth sensors use time of flight to measure the distance between objects and the device, they are not affected by visual artifacts or environmental interferences, such as shadows.

Since HM-SYNC consists of depth data, an appropriate processing technique for analysis is skeleton-based HAR. Skeleton-based HAR is an emerging HAR technique that utilizes depth imaging to extract a set of coordinates in 3D space—the second mode of data in the dataset—where each coordinate represents the location of a joint on the human body. This promising technique is privacy preserving and is being used extensively in many different HAR tasks (including industrial ones [18]) with high accuracy and computational efficiency [27–32].

In order to capture the full range of user activities and interactions with the WAAM machine, two Azure Kinect sensors were installed: one sensor capturing interactions with the machine’s exterior (referred to as the “outer sensor”) and one sensor capturing interactions within the machine’s interior (referred to as the “inner sensor”). Having both exterior and interior perspectives enables the system to capture a wide range of interactions that occur around and within the machine. Figures 2(a) and 2(c) show the placement of the two sensors, and Figs. 2(b) and 2(d) show each sensor’s field of view of the WAAM machine.

The placement of the sensors was determined through thorough consultation with skilled WAAM machine users. For the outer sensor, we focused on sides of the machine with attachments, such as gas tanks and the control panel, since activities around the machine typically occur near these attachments. The inner sensor covers most of the chamber, except for the bottom corner in Fig. 2(d), which is not included because there is a built-in box below the sensor (as shown in Fig. 1(b)) where no human–machine interactions can occur. The field of view was constrained to ensure that only relevant interactions and activities were observed from the two perspectives. Capturing both perspectives was critical for understanding both the “global” and “local” interactions, since HAR faces many challenges ranging from occlusion, to easily misinterpreted actions, to the uniqueness of each person’s movements, just to name a few. Capturing multiple views of a single interaction reinforces the ability to accurately interpret the activity through contextualization [33].

Both Azure Kinect devices were controlled by a local computer running a program that sampled a depth image from each sensor every second and performed basic body tracking using the Azure Kinect Body Tracking SDK. If the program detected humans in either camera’s field of view for 3 s, it triggered synchronized recording from both sensors at 15 frames per second (FPS). During recording, the program continuously monitored human bodies. If it failed to detect humans in both frames for three consecutive seconds, the sensors would stop recording. This ensured that recordings were made only when people were in frame and likely interacting with the machine or performing a relevant activity. All recordings were saved on a local hard drive and automatically uploaded daily to cloud storage. The automatic sensor triggering and data collection systems provided a minimally invasive process while allowing observations of interactions in a more natural environment. Capturing natural human interactions and behaviors in an environment is highly desirable but presents a significant challenge in HAR [34]. There is a trade-off between capturing human activity in real-life situations and the quality of automatic computational processing. Natural environments provide the most useful data as they directly represent the systems to be studied and modeled; however, there is very little standardization of actions and potential biases may remain unaddressed. In contrast, laboratory-based data is often more standardized and easier to interpret computationally; however, it is much more challenging to apply directly to real-world environments.

HM-SYNC was generated by collecting data over a 6-month period, which included times when the machine was idle (e.g., holidays, facility closures) as well as times when it was in use during regular operating hours. When the machine was active, the two depth sensors were automatically activated upon detecting users who were configuring and setting up builds. From the data collected when users were present, their interactions and activities were segmented and incorporated into the dataset. The time required to configure and setup the machine, especially the necessary interactions, was very short compared to the overall build time for a product.

Approximately 3.87 h of segmented interaction sequences were collected over the data collection window. At the depth sensor’s sampling rate of 15 FPS, this resulted in a total of 209,230 frames from each camera perspective, labeled to identify 1228 unique interactions between humans and the WAAM machine. The entire dataset was stored in a series of folders containing the captured depth frames and a metadata file in “.JSONL” format, which provided appropriate metadata for each image and frame labels. A sample frame of data is visualized in Fig. 3. Each frame contains the following information: a depth image from the optimal view perspective, a list of 32 skeleton joint coordinates in 3D space extracted from a detected human, a label indicating the action being performed, a label indicating the interaction location on the machine, a user ID assigned to each unique machine user, a label identifying which camera (outer or inner) captured the action, a label indicating which of the 1228 interactions the frame belongs to, and the timestamp.

Each Azure Kinect sensor stores captured data in the “.mkv” file format, consisting of multiple tracks of data: RGB, infrared, and depth images. Of these different tracks, only the depth images are used to extract 3D skeleton information, which this article uses to maintain user privacy. FFmpeg [35], an open-source, multi-media file stream processing platform, is used to extract depth images from each “.mkv” video clip. These images are converted from their original FourCC “b16g” codec to greyscale, 16-bit PNG format with a resolution of $320×288pixels$⁠.

The 3D joint data extracted from each depth image are stored as an array of 32 (⁠$X$⁠,$Y$⁠,$Z$⁠) coordinates. The labeling scheme for the extracted joints is shown in Fig. 4. Each skeleton consists of 32 joints with their respective coordinates, using the camera as the global origin. The joint coordinate values are measured in millimeters from this origin. The coordinates are intentionally left unedited so that users can normalize or recenter the origin according to their preference. Each label is stored as a simple text string (indicating the action type) or as an integer (indicating a location label, unique user identifier, view label, or action sequence number). The timestamp is stored as a text string formatted according to the ISO-8601 datetime standard.

After setting up the two sensors, creating the dataset required two main procedures, often performed in parallel. The first procedure involved reviewing the saved videos to identify and appropriately label high touch point sections of interest of the machine. During the labeling process, we combed through every video to correctly identify the class based on their characteristic features. The second procedure involved consulting with experts who used the WAAM machine regularly to validate the observations and deepen our understanding of the interaction patterns we observed. We refer to this method as “manufacturing ethnography,” which involved observing and consulting machine users in manufacturing settings to gain insights into behaviors and validate observations.

As summarized in Table 1, existing manufacturing datasets focus primarily on general human actions, small-scale human–machine interactions, and digital HMI. This work comprises the aforementioned activity types and also includes large-scale human–machine interactions. During the data curation process, 15 fundamental actions and 2 general human actions were identified to comprise the dataset (Table 2). The observed action groupings encompass a diverse range of direct and indirect interactions with the WAAM machine, which could serve as potential fundamental actions in other machine interactions. Actions related to human safety (e.g., donning and doffing personal protective equipment (PPE)) and general human actions (e.g., walking and observing) were included to promote well-being action recognition in manufacturing settings and to better contextualize human behaviors beyond machine interactions. The identified actions are listed in Table 2. A visual guide illustrating how these actions appear through the data collection methods is shown in Fig. 5.

Using similar manufacturing ethnography methods, eight points of interaction on the WAAM machine were identified as high contact areas, as shown in Fig. 6. Identifying these interaction points provides useful context for narrowing down possible interactions for action recognition tasks and helps in understanding how each action affects the machine system. Additionally, this information is valuable for analyzing interaction patterns, as users tend to interact with specific parts of the machine in certain sequences or contexts.

HM-SYNC primarily captured interactions from three main WAAM machine users, identified by user IDs 0, 1, and 2. User ID 3 is reserved for one-time use by other individuals. HM-SYNC focused exclusively on interactions between a single user and the WAAM machine. However, future work should explore better understanding interactions involving multiple users with machines and among themselves [36]. Labeling each interaction with the user performing it enables the identification of unique patterns, supports the prediction of operation sequences, and enhances machine security by detecting unauthorized users or anomalous activity.

Given the multiple interaction points around the WAAM machine (e.g., Fig. 6) and its large form factor, two depth sensors were positioned to capture activities occurring both outside and inside the machine. Section 2.3 presents details about the positioning of these two cameras. Capturing interactions from multiple angles can improve the robustness of human activity analysis. However, the machine’s structure can create occlusions from certain viewpoints. To address this, a view label is included to indicate which sensor perspective (i.e., outside or inside) provides the clearest and most direct line of sight for analyzing human actions. This label is 0 if the action is visible almost exclusively from the outer perspective, or 1 if it is visible almost exclusively from the inner perspective. While some actions were visible from both sensors, one angle consistently offered a more comprehensive view of each action.

In skeleton-based HAR, analyzing sequences of skeletons temporally is essential, as it provides crucial information and context for understanding the actions being performed [37]. Although HM-SYNC consists of individual frames, each frame includes an action number label indicating the action sequence it belongs to. The dataset comprises 1228 interactions, with each frame assigned an action number label from 0 to 1227. Frames sharing the same action number are part of the same complete action, allowing for analysis on a per-action basis by examining all frames with the same action number together. Additionally, some actions, such as “refit_buildplate,” can be long and repetitive. These actions can be further divided into more action sequences, increasing the sample size for that particular action. HM-SYNC also supports frame-by-frame analysis for continuous HAR or automatic segmentation, both of which are popular and emerging research areas in industrial settings where humans frequently switch between interactions [38].

As previously discussed, temporal context is crucial for interpreting human actions from skeleton sequences. Each data frame include an exact timestamp coded in the ISO 8601 datetime standard format. This allows for identifying patterns in the sequence of actions, providing context that reinforces action recognition and human–machine interaction patterns. The timestamps also help determine if interactions occurred on the same day, within the same experimentation session, or if they were part of a long sequence of experiments or a short check-up on the machine. This temporal information offers essential context for a more comprehensive understanding of human–machine interactions.

HM-SYNC comprises 1228 interactions with an advanced manufacturing machine, captured over 6 months, resulting in a total of 209,230 frames (approximately 3.87 h of information-dense footage). By capturing data over a 6-month period, the dataset captures a wide variety of high-level goals and objectives (performed by multiple users), each supported by distinct sets of interactions. Hence, HM-SYNC provides sufficient variety at different levels of abstraction: goals and objectives, interactions, and users. HM-SYNC is larger than similar datasets of human actions in industrial settings, such as those from Roitberg et al. [18] and Cicirelli et al. [19], and general daily actions, such as that from Chen et al. [39]. It captures human–machine interactions in a realistic manufacturing environment, which is critical since data collected in controlled, lab-based settings may not accurately represent real-world scenarios. Realistic data lead to more accurate real-world predictions and simulations, essential for engineering design. Additionally, HM-SYNC includes detailed labeling, providing abundant contextual information to learn patterns of action sequences and key points of interaction on the machine.

The goal of this dataset is to enhance the understanding of human–machine interaction patterns in advanced manufacturing settings, supporting the engineering design of manufacturing layouts and machinery interfaces, as well as expanding knowledge of manufacturing process capabilities for product design. HM-SYNC captures realistic use of advanced manufacturing machinery to uncover deeper insights in these areas. Consequently, the balance of actions within the dataset is not even, reflecting real manufacturing scenarios where some actions are performed more frequently than others. This imbalance provides valuable information about the manufacturing process with this machine and can inform designs that account for the varying levels of attention to different parts of the machine and manufacturing process.

Robust, data-driven research relies on high-quality and abundant data. It is essential to establish strong standards for accessing and sharing data, especially to support automated data utilization in our increasingly digitized scientific communities. To support these efforts, adherence to FAIR data practices is crucial [40]. HM-SYNC contains rich metadata, ensuring that it is accurately described and supporting indexing. Moreover, it is uploaded to an open and free platform for public access.³ Standard data formats are used to facilitate feature extraction and analysis while allowing for novel processing techniques. Additionally, the dataset meets typical community standards to ensure it can be used or combined in various settings and is provided under an MIT License.

By uploading HM-SYNC and providing detailed information on the data collection and curation, we aim to initiate a dialogue about—and directly support—its uses in data-driven design and engineering, as well as in enhancing the understanding of human–machine interactions for more human-centered manufacturing settings. To begin this dialogue, this section sketches four potential areas of application for this dataset.

HM-SYNC invites several meaningful questions regarding the nature of interaction in manufacturing environments: How does the nature of a point of interaction limit the forms of interaction that may occur? To what extent do individuals use parts of a machine differently? The nature of a point of interaction can significantly limit or enhance the forms of interaction that are possible. For example, a control panel with an intuitive layout may facilitate quicker and more efficient interactions, while a poorly designed interface may lead to frequent errors and inefficiencies. By using HM-SYNC to interrogate these aspects, researchers can gain valuable insights into the design of machine interfaces and manufacturing layouts. HM-SYNC allows researchers to study these interactions and understand how changes in the interface or machine parameters can modify the patterns of interaction, ultimately affecting the entire workflow. By collecting and analyzing patterns of use, this also supports the identification of opportunities for optimizing machine interfaces and manufacturing processes.

Additionally, HM-SYNC can also be used to support the development of tools for designing digital machine interfaces [41]. By capturing how people interact with machines in natural settings, one can decouple the interweaving of virtual versus physical interactions. This information can guide the design of digital interfaces, ensuring they are intuitive and aligned with the natural workflow of users. Understanding which types of actions and points of interaction are most common can help prioritize the design features that will have the greatest impact on efficiency and user satisfaction.

The use of advanced manufacturing machines to produce novel, innovative components is a constant challenge. One of the most significant challenges is establishing a standard or structure for machine operation in these new contexts. By analyzing the interactions of expert users with machines, like the WAAM machine, HM-SYNC can help identify patterns and develop new operating procedures tailored to specific contexts. This process involves observing and understanding how expert users adapt the machine’s use to fit new tasks or experimental setups, which can inform the creation of standardized operating procedures for these novel applications.

A critical aspect of this application is the ability to identify unique user interaction patterns. Each user may have a distinct style or method of interacting with the machine, and recognizing these patterns can serve multiple purposes. For instance, it can personalize the machine’s settings and parameters to match the preferences or style of the identified user, thereby optimizing the user experience and potentially increasing efficiency and safety. Moreover, HM-SYNC enables the identification of anomalous interactions, which are deviations from the established norms or patterns of machine use. Recognizing these anomalies is essential for several reasons. It can highlight potential misuse or errors, allowing for timely interventions to prevent accidents or equipment damage. It also provides insights into unusual but potentially innovative uses of the machine that could be standardized and incorporated into new operating procedures.

HRC is an emerging field in advanced manufacturing focused on enhancing the synergy between human workers and robotic systems [12]. HM-SYNC provides a rich source of data for developing and refining HRC systems. By analyzing human interactions with the WAAM machine, researchers can identify key behaviors and actions that inform the design of collaborative robots (i.e., cobots). Cobots can be programmed to recognize and anticipate human actions, facilitating seamless collaboration. For example, HM-SYNC can be used to train machine learning models to predict when a human worker needs assistance or when a robot should take over a task. This predictive capability enhances the efficiency and safety of HRC systems, ensuring that both humans and robots can work together effectively. By understanding the typical interaction patterns and contextual cues from human operators, cobots can be designed to respond proactively, thereby reducing downtime and improving the flow of tasks in a manufacturing environment.

HM-SYNC also supports the development of adaptive HRC interfaces that respond to human actions in real time. By capturing detailed interaction patterns, we can design interfaces that are intuitive and responsive, reducing the learning curve for human workers and improving overall productivity. This adaptive capability is crucial for creating flexible manufacturing systems where robots can adjust their behavior based on real-time human inputs. For instance, a cobot might learn to adjust its speed or force when handling delicate components based on the observed preferences and actions of human workers.

Ergonomics and human factors engineering are fields focused on designing systems that optimize human well-being and performance. Focusing on human-centered design (as opposed to machine- or product-centered design), HM-SYNC provides critical insights into the physical and cognitive interactions between humans and manufacturing machines, enabling the design of ergonomic workspaces and tools. By analyzing the dataset, researchers can identify common physical actions and postures that may lead to discomfort or injury. This information is essential for designing workstations and tools that minimize physical strain and enhance comfort [42] under varying environmental and operating conditions that influence behavior [43]. For instance, HM-SYNC can inform the placement of controls and displays to reduce awkward movements and improve accessibility.

Additionally, HM-SYNC supports the design of cognitive aids that enhance human performance. Understanding how users interact with complex machines allows for the development of user interfaces that reduce cognitive load and improve decision making. For example, visual and auditory feedback can be designed to guide users through tasks, by reducing errors and increasing efficiency. By analyzing interaction patterns, designers can create interfaces that are intuitive and easy to navigate, helping workers to complete their tasks more efficiently and with less mental effort.

A deep understanding of manufacturing processes is essential for advancing manufacturing-oriented design. As advanced manufacturing technologies have evolved, human interaction has emerged as a critical factor in both operation and design. Despite the increasing role of automation, humans remain central to manufacturing systems, often adapting processes and introducing efficiencies beyond what is prescribed. Quantifying and characterizing these human interactions is crucial for understanding true manufacturing capabilities, as well as for developing human-centered design approaches that optimize performance. Expanding this understanding to include human interactions within complex systems more broadly can yield valuable insights into productivity, adaptability, and system resilience, supporting the design and engineering of more efficient and responsive manufacturing environments.

This work, although specifically focused on advanced manufacturing environments, is crucial for understanding how human interactions can be monitored, analyzed, and utilized to improve the design of complex engineered systems. HM-SYNC is one of the first to extensively capture human interactions with an advanced manufacturing machine, supporting emerging, promising, and privacy-preserving skeleton-based HAR techniques.

The data collection techniques and dataset organization presented in this work offer a reproducible method for collecting, curating, and analyzing general manufacturing interactions, interactions with other specific machines, or human behaviors in other complex systems. This work aims to inspire and support design engineers and provide a framework to better understand how complex systems are utilized by humans, thereby improving efficiency and enabling accurate analysis of these human interactions within a design context.

HM-SYNC has certain limitations that suggest areas for further improvement. One consideration is the variability in manufacturing equipment design and usage. Since HM-SYNC focuses on a single machine—the WAAM machine—and a small group of people performing interactions, drawing broad conclusions about human interaction across different machines or settings may be challenging. Continued efforts will aim to capture more varied types of interactions, including a more diverse set of machine users, different types of machines, interactions involving multiple people, and a broader range of skill levels among users. Expanding the dataset in these ways would enhance the generalizability of any observations made and provide deeper insights into how particular actions may affect the machinery and overall process.

Additionally, the current dataset focuses solely on the human interaction input side of machine operation. Machines are complex systems with various loaded parameter inputs and environmental contexts, as well as a range of outputs such as error logs, real-time state data, and sensor readings. Future efforts should aim to gather a comprehensive range of data from the machine, beyond just human inputs. This expanded dataset would strongly support the analysis of how human actions affect machine health, production quality, and efficiency, through the mapping of input–output data. This would support goals such as enabling real-time performance corrections for machine users to prevent errors and inefficiencies during design.

Considering the potential applications and future work outlined, HM-SYNC is broadly envisioned as a tool for advancing the understanding of human–machine interaction patterns and for developing enhanced models and digital twins of advanced manufacturing systems. This would provide feedback for human operators to better understand how to use the machinery, avoid inefficiencies, and give machine and manufacturing designers insight into how humans are interacting within the systems of interest. Beyond these applications, we foresee HM-SYNC serving as a catalyst for pioneering advancements in human-centered design, fostering environments where technology and human ingenuity converge seamlessly. The insights gleaned from this research could spark innovations that redefine productivity and well-being in industrial settings, driving a new era of intelligent, responsive, and adaptive manufacturing systems. As we continue to explore and expand the boundaries of human–machine symbiosis, HM-SYNC will serve as a foundational tool in shaping the future of manufacturing, where efficiency, adaptability, and human centricity are paramount.

The authors would like to gratefully acknowledge the Carnegie Mellon University Manufacturing Futures Institute (MFI) staff and Brian Belowich for their support in configuring the testbed, as well as the students and researchers using the WAAM machine for allowing the use of their space and providing critical knowledge in our efforts to produce this dataset.

• Carnegie Mellon University Manufacturing Futures Institute.

There are no conflicts of interest.

All data and information supporting this article are freely available.⁴ .