Abstract

A deep understanding of manufacturing processes is essential for advancing manufacturing-oriented design and engineering complex systems. As advanced manufacturing technologies evolve, systems have grown more complex, and human interaction has become a vital component of both operation and design. This shift introduces new challenges as human roles within these systems extend beyond traditional boundaries and are not yet fully understood in current design processes. Characterizing human interactions within manufacturing systems is therefore critical to supporting further advancements. Additionally, human behavior plays a significant role in many engineered systems beyond manufacturing, underscoring the value of developing methodologies to better analyze human behavior and interactions within complex environments. These methodologies can broadly support and enhance diverse aspects of engineering design. This study presents HM-SYNC, which is a comprehensive dataset of human interactions with advanced manufacturing machinery, specifically a wire-arc additive manufacturing machine. Depth images and 3D skeleton joints are collected over 6 months using privacy-preserving pose tracking with depth cameras. HM-SYNC includes thousands of interactions across various contexts, goals, and users, providing valuable insights into patterns of human–machine interaction. By capturing a diverse range of interactions in natural settings, this dataset supports advancements in human-centered manufacturing design and facilitates the development of more effective manufacturing systems. This dataset can enhance models and digital twins of manufacturing systems, help operators optimize machinery use and efficiency, and guide designers in refining machine and system design, to name just a few applications.

1 Introduction

Developing a deeper understanding of complex manufacturing systems, processes, and capabilities is a key to enhancing design choices and improving design efficiency. This realization is deeply rooted in practices such as design for manufacturing (DfM), whereby designers develop products to adhere to the realities of manufacturing and production capabilities. The efficacy of DfM relies in many cases on accurate and reliable analysis of a manufacturing process to inform design choices [1]. From this perspective, it is crucial to improve ways of understanding and modeling complex manufacturing systems for accurate prediction and integration with the design process. However, assessing manufacturing systems is becoming increasingly challenging as machines grow more interactive.

The manufacturing field faces significant changes due to influxes of advanced technologies such as industrial automation, robotics, and digital twins, as well as from pressures of increasingly competitive global markets [2,3]. Along with a technological shift in manufacturing systems, changes are also occurring in the role of human interactions within complex production systems [4,5]. Where humans once held primarily direct manufacturing roles through manual tool operation or hands-on product assembly, human interaction has expanded to include more indirect roles such as collaborating with robots or supervising automation systems [6]. These human–machine interactions are not yet sufficiently well characterized for integration into complex system models.

Humans play an integral role in many systems, and human performance is a determining factor in the outcomes of human–machine systems [7]. Human interaction and performance research are also critical in the areas of automation technology design for human use [810] and designing the layout of reconfigurable manufacturing systems [11,12]. Research in this area contributes to human-centered design and manufacturing productivity, especially in the ongoing industrial shift to Industry 4.0 and, eventually, Industry 5.0 [13]. Supporting this view, Gorecky et al. [6] and Lu et al. [3] demonstrated that understanding human–machine interactions can significantly enhance manufacturing systems by increasing flexibility, improving strategies, and facilitating expert knowledge transfer between workers and stakeholders. They also highlighted the need for more research on capturing diverse human interactions in manufacturing settings to develop a holistic approach for system performance improvement. Sadeghi et al. [14] emphasized the importance of considering human behaviors in design to achieve better integration of human interactions. Zhang et al. [15] underscored the need to capture human interactions in industrial assembly to reduce faults, minimize errors, and enhance predictability in manufacturing.

A fundamental step in addressing the challenge of observing and understanding human interactions in manufacturing environments is to establish reliable methods for sensing and interpreting workers’ actions. Human activity recognition (HAR) has emerged as a promising approach for reliably sensing and interpreting workers’ actions, particularly in industrial settings. A significant body of work has focused on HAR for human–robot collaboration (HRC), where the goal is to train robotic systems to recognize human actions and anticipate movements, facilitating more seamless collaboration. Notable datasets such as HRI30 [16] and InHARD [17] offer robust collections of data from users performing 30 actions and 13 actions, respectively, in HRC contexts. Similarly, Roitberg et al. [18] compiled a dataset encompassing 15 HRC-related human actions, including gestures for controlling collaborating robots.

While these datasets provide a strong foundation for industrial HAR, they primarily focus on HRC where data are collected in controlled laboratory settings, limiting their applicability to real-world industrial settings. Additionally, the data collection methods used, such as direct sensor application on humans or RGB video, raise privacy concerns. Another dataset, HA4M [19], compiled general industrial actions to support HAR. This multimodal dataset captured 12 manufacturing actions centered around the manual assembly of a small gear train system. Although more broadly applicable across various manufacturing contexts, these actions are limited to direct, hands-on tasks involving minor hand movements, excluding machinery interactions. Like the previously mentioned datasets, HA4M’s data were collected in a laboratory setting, making the transition to real-world applications challenging. The HUMM dataset [20] presents an interesting deviation from classical HAR, as it aims to understand human movement in manufacturing environments for trajectory prediction within HRC contexts. Although it focuses on human movement for robots transporting items, the insights from HUMM could be valuable for manufacturing layout design, potentially enhancing the movement efficiency of workers within manufacturing spaces.

In addition to capturing human interactions for the purpose of HAR, several studies have analyzed human–machine interaction with the goal of improving design. Notably, the dataset created by Carrera-Rivera et al. [21] observed human–machine interactions to support the development of adaptive, digital human–machine interfaces (HMIs). Similarly, Reguera-Bakhache et al. [22] presented data-driven methods for developing adaptive HMIs and collected a dataset of human–machine interactions through a computer interface. These studies involve extensive data collection from real machine use to support manufacturing design, specifically in the area of digital user HMIs. While many machines are used through digital interfaces, there is still a lack of data collected to understand how humans interact with more dynamic machinery that includes a broader range of human–machine interactions, such as hands-on tool use or assembly.

After reviewing this existing literature, summarized in Table 1, we identified significant gaps in the analysis and generation of datasets capturing human interactions within manufacturing environments, especially concerning manufacturing design. While some datasets are available for training algorithms to detect human industrial actions, they often lack observations of humans operating advanced manufacturing machinery. Most current datasets, as discussed earlier, capture either simple human gestures or general actions, like picking up and placing simple objects, which offer limited context for overall manufacturing processes. Moreover, datasets that do capture human–machine interactions mainly focus on digital HMIs and overlook the full range of interactions, from direct, hands-on operation to indirect digital interfaces. Additionally, all of the datasets in Table 1 were collected in highly controlled laboratory settings, reducing their relevance to real-world manufacturing scenarios. There is a clear need for datasets that capture human activity in manufacturing environments more representative of existing facilities, observing natural human use of machinery rather than predefined actions. Finally, there is a scarcity of datasets that capture human–machine interactions in a privacy-preserving and nonintrusive way, which is essential for protecting user privacy and ensuring authentic human behavior.

Table 1

A comparison of existing state-of-the-art datasets focused on human activity in advanced manufacturing settings

DatasetNo. classesNo. videosControlledModalitiesActivity typeYear
Roitberg et al.[18]1524YesRGB+DGeneral2014
InHARD[17]134,800YesRGB+SAssembling2020
HRI30[16]302,940YesRGBGeneral2022
HA4M[19]124,124YesRGB + S + IR + P + RGB&D AlignedAssembling2022
HUMM[20]3551YesRGBTrajectory2023
Carrera-Rivera et al.[21]NA10,608YesNADigital HMI2023
DatasetNo. classesNo. videosControlledModalitiesActivity typeYear
Roitberg et al.[18]1524YesRGB+DGeneral2014
InHARD[17]134,800YesRGB+SAssembling2020
HRI30[16]302,940YesRGBGeneral2022
HA4M[19]124,124YesRGB + S + IR + P + RGB&D AlignedAssembling2022
HUMM[20]3551YesRGBTrajectory2023
Carrera-Rivera et al.[21]NA10,608YesNADigital HMI2023

Note: D, depth images; S, 3D skeleton joints; IR, infrared; P, point cloud; NA, not available.

This article presents the collection and curation of a comprehensive dataset of human–machine interactions in an advanced manufacturing context. The goal is to support a data-driven understanding of how human interactions impact the manufacturing design process, and how this understanding can be used to improve design. This dataset, called Human-Machine interactionS in dYnamic advaNced manufaCturing settings (HM-SYNC), provides valuable insights to better inform product and manufacturing designers about the capabilities and uses of today’s manufacturing technologies and facilities. To achieve this effectively—and to align with the intelligent systems increasingly used in manufacturing analysis, such as machine learning—this work focuses on collecting a systematically derived dataset to support learning in these intelligent systems and enhance decision-making capabilities.

We address the aforementioned gaps derived from Table 1 by creating a dataset that captures human interactions with machinery in an active advanced manufacturing environment, utilizing noninvasive, privacy-preserving sensing technologies (i.e., depth sensors). Over a 6-month period, more than 1200 interactions were recorded between users and an advanced manufacturing machine—the wire-arc additive manufacturing (WAAM) machine—capturing a wide range of large-scale and small-scale activities, such as manually grinding down metal surfaces and adjusting machine calibration settings through a digital interface. A detailed analysis of machine selection and action diversity is presented in Secs. 2.1 and 2.5, respectively. HM-SYNC captures a broad range of human–machine interactions, focusing on understanding natural human use of machinery in a manufacturing setting. Additionally, it is one of the first datasets to achieve this in a privacy-preserving manner, with the goal of enhancing human-centered manufacturing design by analyzing human activity in manufacturing environments.

The remainder of the article is organized as follows. Section 2 details the data collection methodology, beginning with the justification for machine and sensor setup choices in Secs. 2.12.3. This is followed by an exploration of HM-SYNC’s format and organization in Secs. 2.4 and 2.5. The methodology concludes in Secs. 2.6 and 2.7 with a description of how the dataset adheres to proper data-sharing standards and ensures the quality for its intended applications. Section 3 discusses key potential applications of the dataset to support engineering design in manufacturing settings. Finally, Sec. 4 addresses HM-SYNC’s limitations and suggests future directions.

2 Methodology

This section discusses and justifies the experimental setup and data design decisions. The primary considerations were (1) machine selection, (2) sensor selection, (3) sensor placement and hardware setup, and (4) data capture and formatting. We provide a detailed overview of the organization and curation of HM-SYNC, justify its content and size, and discuss its adherence to findable, accessible, interoperable, and reusable (FAIR) principles.

2.1 Machine Selection.

The primary criteria for selecting a suitable machine for this study were its capacity to support a diverse range of human interactions and its compatibility with sensor-based observations. The Lincoln Electric Sculptprint RND WAAM machine [23] was chosen as the focus of this work for the following two reasons. The WAAM machine is a large-format metal 3D printer housed in a 2.2 m × 4.1 m × 2.3 m (L×W×H) chamber. It features a robotic welder arm that deposits molten metal filament onto a specially configured build plate in a layered fashion. Images of the WAAM’s exterior and interior are shown in Figs. 1(a) and 1(b), respectively.

Fig. 1
The WAAM machine’s (a) exterior and (b) interior
Fig. 1
The WAAM machine’s (a) exterior and (b) interior
Close modal

Interactions with the WAAM machine are highly varied, ranging from direct actions, such as manually adjusting the build plate for different projects or grinding the deposited surface with a hand grinder, to indirect actions, such as maneuvering the robotic welding arm with a mobile joystick controller or setting machine parameters on a control panel screen. A full listing and description of each interaction is presented in Table 2. This compilation of interactions is derived by referencing the technical Lincoln Electric Manual [23], observing interaction patterns from data and in-person shadowing, and consulting WAAM machine users about their routines and practices.

Table 2

Fifteen foundational WAAM machine interactions and two general actions with accompanying description

Action labelDescription
using_control_panelInterfacing with machine start or stop controls and digital screen used for visualizing build files and configuring machine parameters
using_flexpendant_mountedFlexPendant being used in its control mode for loading build parameters and viewing machine output logs
using_flexpendant_mobileFlexPendant being used in its machine operation mode for moving the robotic arm with the attached joystick
inspecting_buildplatePerforming light build plate modifications and inspections before or after a build
preparing_buildplateClearing or moving the build plate to set up next build
refit_buildplateCompletely switching out the build plate configuration for a new project
grinding_buildplateGrinding down the new build plate to expose conductive metal and level surface
toggle_lightsTurning the internal WAAM light on or off
open_doorOpening the WAAM door
close_doorClosing the WAAM door
turning_gas_knobsTurning shielding gas on or off
adjusting_toolInstalling or modifying new or existing sensors on the robotic welder arm
wiringInstalling or adjusting wiring of tool sensors
donning_ppeUsers putting on personal protective equipment (PPE)
doffing_ppeUsers taking off PPE
observingaUsers looking around or watching WAAM activity
walkingaUsers walking around the WAAM
Action labelDescription
using_control_panelInterfacing with machine start or stop controls and digital screen used for visualizing build files and configuring machine parameters
using_flexpendant_mountedFlexPendant being used in its control mode for loading build parameters and viewing machine output logs
using_flexpendant_mobileFlexPendant being used in its machine operation mode for moving the robotic arm with the attached joystick
inspecting_buildplatePerforming light build plate modifications and inspections before or after a build
preparing_buildplateClearing or moving the build plate to set up next build
refit_buildplateCompletely switching out the build plate configuration for a new project
grinding_buildplateGrinding down the new build plate to expose conductive metal and level surface
toggle_lightsTurning the internal WAAM light on or off
open_doorOpening the WAAM door
close_doorClosing the WAAM door
turning_gas_knobsTurning shielding gas on or off
adjusting_toolInstalling or modifying new or existing sensors on the robotic welder arm
wiringInstalling or adjusting wiring of tool sensors
donning_ppeUsers putting on personal protective equipment (PPE)
doffing_ppeUsers taking off PPE
observingaUsers looking around or watching WAAM activity
walkingaUsers walking around the WAAM
a

These are two general actions.

Interestingly—and underscoring the need for this work—many actions, especially the sequence of actions performed by WAAM machine users, were not formally outlined in the technical manuals provided by Lincoln Electric. Discussions with users revealed that they had developed their own techniques, including adding, reordering, or skipping certain actions, as a result of unique experimental cases with the WAAM machine and their discovery of more efficient workarounds. These realizations reinforce the idea that actual machine use differs from intended use from machine manufacturers. This also supports the view that more focus needs to be placed on observing humans interacting in real-world manufacturing processes such that designers upstream can be informed about accurate accounts of manufacturing performance, and machine producers can be informed about how their machines are being used downstream such that they can improve machine designs or keep technical documentation accurate and up to date.

Table 2 shows that the types of interactions effectively cover the various roles humans are assuming in manufacturing settings with the influx of sophisticated industrial technologies [4,5]. The WAAM machine demonstrates new techniques that are not covered in formal documentation, highlighting the need for updated observations and concurrent learning. With this in mind, compiling a dataset of human interactions with the WAAM machine will be relevant to human interactions with various machines in similar situations and across multiple manufacturing design contexts.

2.2 Sensor Selection.

The Microsoft Azure Kinect [24] was selected to capture human behaviors due to its multimodal data collection capabilities—including a best-in-class 1 MP depth camera, 360 deg microphone array, 12 MP RGB camera, and orientation sensor for building advanced computer vision and speech models—and comprehensive Software Development Kits (SDKs) that support skeleton-based HAR [24,25]. Key specifications for all integrated sensors are presented in Table 3. The multimodal nature of this sensor is essential for enabling the observation and labeling of the data before narrowing the modality choice for the dataset.

Table 3

Sensors embedded in the Azure Kinect, including key attributes

Embedded sensorKey attribute(s)
RGB camera12 MP camera up to 4K quality at 30 FPS
Depth sensor1 MP, time of flight (ToF) depth sensing up to 30 FPS
Infrared diodesNear infrared (NIR) laser diodes enabling near or wide angle field of view
Inertial measurement unit (IMU)Accelerometer and gyroscope sampled at up to 1.6 kHz
Microphone arraySeven microphones supporting 360 deg audio
Embedded sensorKey attribute(s)
RGB camera12 MP camera up to 4K quality at 30 FPS
Depth sensor1 MP, time of flight (ToF) depth sensing up to 30 FPS
Infrared diodesNear infrared (NIR) laser diodes enabling near or wide angle field of view
Inertial measurement unit (IMU)Accelerometer and gyroscope sampled at up to 1.6 kHz
Microphone arraySeven microphones supporting 360 deg audio

Note: The depth sensor in bold is selected for this study.

Another issue that was thoroughly considered was privacy and the value of information each of these sensors provides, as the more information a sensor offers, the less privacy it typically preserves [26]. On one end of this spectrum, RGB imagery includes the most rich and valuable information, enabling object recognition alongside human skeleton detection. However, it is privacy invasive, revealing detailed facial and characteristic features. On the other end of the spectrum, technologies like passive infrared sensors or radar are the most privacy preserving, but contain less rich data and require significantly more computation to interpret. Depth imagery strikes a balance between these two ends of the spectrum. The information provided by the data is sufficient for effective computational processing of human interaction, all while still maintaining user privacy. Hence, depth imagery was uploaded and used from the Azure Kinect sensors. This is due to the nature of depth imagery to support powerful computational techniques, such as skeleton-based HAR, while also supporting user privacy. Another advantage is that, since depth sensors use time of flight to measure the distance between objects and the device, they are not affected by visual artifacts or environmental interferences, such as shadows.

Since HM-SYNC consists of depth data, an appropriate processing technique for analysis is skeleton-based HAR. Skeleton-based HAR is an emerging HAR technique that utilizes depth imaging to extract a set of coordinates in 3D space—the second mode of data in the dataset—where each coordinate represents the location of a joint on the human body. This promising technique is privacy preserving and is being used extensively in many different HAR tasks (including industrial ones [18]) with high accuracy and computational efficiency [2732].

2.3 Sensor Placement and Hardware Setup.

In order to capture the full range of user activities and interactions with the WAAM machine, two Azure Kinect sensors were installed: one sensor capturing interactions with the machine’s exterior (referred to as the “outer sensor”) and one sensor capturing interactions within the machine’s interior (referred to as the “inner sensor”). Having both exterior and interior perspectives enables the system to capture a wide range of interactions that occur around and within the machine. Figures 2(a) and 2(c) show the placement of the two sensors, and Figs. 2(b) and 2(d) show each sensor’s field of view of the WAAM machine.

Fig. 2
Locations of the (a) exterior and (c) interior Azure Kinect sensor placement, including sample images of the field of view for the (b) exterior and (d) interior cameras. The cameras were strategically positioned to ensure that no human or human–machine interactions occur within their blind spots, either because operator interactions do not take place there or because the physical layout cannot accommodate operators in those areas.
Fig. 2
Locations of the (a) exterior and (c) interior Azure Kinect sensor placement, including sample images of the field of view for the (b) exterior and (d) interior cameras. The cameras were strategically positioned to ensure that no human or human–machine interactions occur within their blind spots, either because operator interactions do not take place there or because the physical layout cannot accommodate operators in those areas.
Close modal

The placement of the sensors was determined through thorough consultation with skilled WAAM machine users. For the outer sensor, we focused on sides of the machine with attachments, such as gas tanks and the control panel, since activities around the machine typically occur near these attachments. The inner sensor covers most of the chamber, except for the bottom corner in Fig. 2(d), which is not included because there is a built-in box below the sensor (as shown in Fig. 1(b)) where no human–machine interactions can occur. The field of view was constrained to ensure that only relevant interactions and activities were observed from the two perspectives. Capturing both perspectives was critical for understanding both the “global” and “local” interactions, since HAR faces many challenges ranging from occlusion, to easily misinterpreted actions, to the uniqueness of each person’s movements, just to name a few. Capturing multiple views of a single interaction reinforces the ability to accurately interpret the activity through contextualization [33].

Both Azure Kinect devices were controlled by a local computer running a program that sampled a depth image from each sensor every second and performed basic body tracking using the Azure Kinect Body Tracking SDK. If the program detected humans in either camera’s field of view for 3 s, it triggered synchronized recording from both sensors at 15 frames per second (FPS). During recording, the program continuously monitored human bodies. If it failed to detect humans in both frames for three consecutive seconds, the sensors would stop recording. This ensured that recordings were made only when people were in frame and likely interacting with the machine or performing a relevant activity. All recordings were saved on a local hard drive and automatically uploaded daily to cloud storage. The automatic sensor triggering and data collection systems provided a minimally invasive process while allowing observations of interactions in a more natural environment. Capturing natural human interactions and behaviors in an environment is highly desirable but presents a significant challenge in HAR [34]. There is a trade-off between capturing human activity in real-life situations and the quality of automatic computational processing. Natural environments provide the most useful data as they directly represent the systems to be studied and modeled; however, there is very little standardization of actions and potential biases may remain unaddressed. In contrast, laboratory-based data is often more standardized and easier to interpret computationally; however, it is much more challenging to apply directly to real-world environments.

2.4 Data Capture and Formatting.

HM-SYNC was generated by collecting data over a 6-month period, which included times when the machine was idle (e.g., holidays, facility closures) as well as times when it was in use during regular operating hours. When the machine was active, the two depth sensors were automatically activated upon detecting users who were configuring and setting up builds. From the data collected when users were present, their interactions and activities were segmented and incorporated into the dataset. The time required to configure and setup the machine, especially the necessary interactions, was very short compared to the overall build time for a product.

Approximately 3.87 h of segmented interaction sequences were collected over the data collection window. At the depth sensor’s sampling rate of 15 FPS, this resulted in a total of 209,230 frames from each camera perspective, labeled to identify 1228 unique interactions between humans and the WAAM machine. The entire dataset was stored in a series of folders containing the captured depth frames and a metadata file in “.JSONL” format, which provided appropriate metadata for each image and frame labels. A sample frame of data is visualized in Fig. 3. Each frame contains the following information: a depth image from the optimal view perspective, a list of 32 skeleton joint coordinates in 3D space extracted from a detected human, a label indicating the action being performed, a label indicating the interaction location on the machine, a user ID assigned to each unique machine user, a label identifying which camera (outer or inner) captured the action, a label indicating which of the 1228 interactions the frame belongs to, and the timestamp.

Fig. 3
A representative data frame with all associated labels and metadata
Fig. 3
A representative data frame with all associated labels and metadata
Close modal

Each Azure Kinect sensor stores captured data in the “.mkv” file format, consisting of multiple tracks of data: RGB, infrared, and depth images. Of these different tracks, only the depth images are used to extract 3D skeleton information, which this article uses to maintain user privacy. FFmpeg [35], an open-source, multi-media file stream processing platform, is used to extract depth images from each “.mkv” video clip. These images are converted from their original FourCC “b16g” codec to greyscale, 16-bit PNG format with a resolution of 320×288pixels.

The 3D joint data extracted from each depth image are stored as an array of 32 (X,Y,Z) coordinates. The labeling scheme for the extracted joints is shown in Fig. 4. Each skeleton consists of 32 joints with their respective coordinates, using the camera as the global origin. The joint coordinate values are measured in millimeters from this origin. The coordinates are intentionally left unedited so that users can normalize or recenter the origin according to their preference. Each label is stored as a simple text string (indicating the action type) or as an integer (indicating a location label, unique user identifier, view label, or action sequence number). The timestamp is stored as a text string formatted according to the ISO-8601 datetime standard.

Fig. 4
Labeling scheme for the 32 joints extracted from the Azure Kinect Body Tracking SDK. Each joint is saved as a 3D coordinate relative to the sensor as the origin. The x-axis runs left and right in the plane of the sensor, the y-axis runs up and down in the plane of the sensor, and the z-axis runs longitudinally away and toward the sensor.
Fig. 4
Labeling scheme for the 32 joints extracted from the Azure Kinect Body Tracking SDK. Each joint is saved as a 3D coordinate relative to the sensor as the origin. The x-axis runs left and right in the plane of the sensor, the y-axis runs up and down in the plane of the sensor, and the z-axis runs longitudinally away and toward the sensor.
Close modal

2.5 Postprocessing, Data Organization, and Labeling.

After setting up the two sensors, creating the dataset required two main procedures, often performed in parallel. The first procedure involved reviewing the saved videos to identify and appropriately label high touch point sections of interest of the machine. During the labeling process, we combed through every video to correctly identify the class based on their characteristic features. The second procedure involved consulting with experts who used the WAAM machine regularly to validate the observations and deepen our understanding of the interaction patterns we observed. We refer to this method as “manufacturing ethnography,” which involved observing and consulting machine users in manufacturing settings to gain insights into behaviors and validate observations.

2.5.1 Action Labels.

As summarized in Table 1, existing manufacturing datasets focus primarily on general human actions, small-scale human–machine interactions, and digital HMI. This work comprises the aforementioned activity types and also includes large-scale human–machine interactions. During the data curation process, 15 fundamental actions and 2 general human actions were identified to comprise the dataset (Table 2). The observed action groupings encompass a diverse range of direct and indirect interactions with the WAAM machine, which could serve as potential fundamental actions in other machine interactions. Actions related to human safety (e.g., donning and doffing personal protective equipment (PPE)) and general human actions (e.g., walking and observing) were included to promote well-being action recognition in manufacturing settings and to better contextualize human behaviors beyond machine interactions. The identified actions are listed in Table 2. A visual guide illustrating how these actions appear through the data collection methods is shown in Fig. 5.

Fig. 5
Examples of depth images with overlaid skeletons. First row (left to right): using control panel, using mounted FlexPendant, using mobile FlexPendant, inspecting build plate, preparing build plate. Second row (left to right): refitting build plate, grinding build plate, toggling lights, opening door, closing door. Third row (left to right): turning gas knobs, adjusting welder tool, wiring, donning PPE, doffing PPE. Fourth row (left to right): observing, walking.
Fig. 5
Examples of depth images with overlaid skeletons. First row (left to right): using control panel, using mounted FlexPendant, using mobile FlexPendant, inspecting build plate, preparing build plate. Second row (left to right): refitting build plate, grinding build plate, toggling lights, opening door, closing door. Third row (left to right): turning gas knobs, adjusting welder tool, wiring, donning PPE, doffing PPE. Fourth row (left to right): observing, walking.
Close modal

2.5.2 Location Labels.

Using similar manufacturing ethnography methods, eight points of interaction on the WAAM machine were identified as high contact areas, as shown in Fig. 6. Identifying these interaction points provides useful context for narrowing down possible interactions for action recognition tasks and helps in understanding how each action affects the machine system. Additionally, this information is valuable for analyzing interaction patterns, as users tend to interact with specific parts of the machine in certain sequences or contexts.

Fig. 6
The labeled regions indicate the key interaction points on the WAAM machine
Fig. 6
The labeled regions indicate the key interaction points on the WAAM machine
Close modal

2.5.3 User Labels.

HM-SYNC primarily captured interactions from three main WAAM machine users, identified by user IDs 0, 1, and 2. User ID 3 is reserved for one-time use by other individuals. HM-SYNC focused exclusively on interactions between a single user and the WAAM machine. However, future work should explore better understanding interactions involving multiple users with machines and among themselves [36]. Labeling each interaction with the user performing it enables the identification of unique patterns, supports the prediction of operation sequences, and enhances machine security by detecting unauthorized users or anomalous activity.

2.5.4 View Labels.

Given the multiple interaction points around the WAAM machine (e.g., Fig. 6) and its large form factor, two depth sensors were positioned to capture activities occurring both outside and inside the machine. Section 2.3 presents details about the positioning of these two cameras. Capturing interactions from multiple angles can improve the robustness of human activity analysis. However, the machine’s structure can create occlusions from certain viewpoints. To address this, a view label is included to indicate which sensor perspective (i.e., outside or inside) provides the clearest and most direct line of sight for analyzing human actions. This label is 0 if the action is visible almost exclusively from the outer perspective, or 1 if it is visible almost exclusively from the inner perspective. While some actions were visible from both sensors, one angle consistently offered a more comprehensive view of each action.

2.5.5 Action Number.

In skeleton-based HAR, analyzing sequences of skeletons temporally is essential, as it provides crucial information and context for understanding the actions being performed [37]. Although HM-SYNC consists of individual frames, each frame includes an action number label indicating the action sequence it belongs to. The dataset comprises 1228 interactions, with each frame assigned an action number label from 0 to 1227. Frames sharing the same action number are part of the same complete action, allowing for analysis on a per-action basis by examining all frames with the same action number together. Additionally, some actions, such as “refit_buildplate,” can be long and repetitive. These actions can be further divided into more action sequences, increasing the sample size for that particular action. HM-SYNC also supports frame-by-frame analysis for continuous HAR or automatic segmentation, both of which are popular and emerging research areas in industrial settings where humans frequently switch between interactions [38].

2.5.6 Timestamp Labels.

As previously discussed, temporal context is crucial for interpreting human actions from skeleton sequences. Each data frame include an exact timestamp coded in the ISO 8601 datetime standard format. This allows for identifying patterns in the sequence of actions, providing context that reinforces action recognition and human–machine interaction patterns. The timestamps also help determine if interactions occurred on the same day, within the same experimentation session, or if they were part of a long sequence of experiments or a short check-up on the machine. This temporal information offers essential context for a more comprehensive understanding of human–machine interactions.

2.6 Dataset Size and Composition.

HM-SYNC comprises 1228 interactions with an advanced manufacturing machine, captured over 6 months, resulting in a total of 209,230 frames (approximately 3.87 h of information-dense footage). By capturing data over a 6-month period, the dataset captures a wide variety of high-level goals and objectives (performed by multiple users), each supported by distinct sets of interactions. Hence, HM-SYNC provides sufficient variety at different levels of abstraction: goals and objectives, interactions, and users. HM-SYNC is larger than similar datasets of human actions in industrial settings, such as those from Roitberg et al. [18] and Cicirelli et al. [19], and general daily actions, such as that from Chen et al. [39]. It captures human–machine interactions in a realistic manufacturing environment, which is critical since data collected in controlled, lab-based settings may not accurately represent real-world scenarios. Realistic data lead to more accurate real-world predictions and simulations, essential for engineering design. Additionally, HM-SYNC includes detailed labeling, providing abundant contextual information to learn patterns of action sequences and key points of interaction on the machine.

The goal of this dataset is to enhance the understanding of human–machine interaction patterns in advanced manufacturing settings, supporting the engineering design of manufacturing layouts and machinery interfaces, as well as expanding knowledge of manufacturing process capabilities for product design. HM-SYNC captures realistic use of advanced manufacturing machinery to uncover deeper insights in these areas. Consequently, the balance of actions within the dataset is not even, reflecting real manufacturing scenarios where some actions are performed more frequently than others. This imbalance provides valuable information about the manufacturing process with this machine and can inform designs that account for the varying levels of attention to different parts of the machine and manufacturing process.

2.7 FAIR Principles.

Robust, data-driven research relies on high-quality and abundant data. It is essential to establish strong standards for accessing and sharing data, especially to support automated data utilization in our increasingly digitized scientific communities. To support these efforts, adherence to FAIR data practices is crucial [40]. HM-SYNC contains rich metadata, ensuring that it is accurately described and supporting indexing. Moreover, it is uploaded to an open and free platform for public access.3 Standard data formats are used to facilitate feature extraction and analysis while allowing for novel processing techniques. Additionally, the dataset meets typical community standards to ensure it can be used or combined in various settings and is provided under an MIT License.

3 Dataset Applications

By uploading HM-SYNC and providing detailed information on the data collection and curation, we aim to initiate a dialogue about—and directly support—its uses in data-driven design and engineering, as well as in enhancing the understanding of human–machine interactions for more human-centered manufacturing settings. To begin this dialogue, this section sketches four potential areas of application for this dataset.

3.1 Machine and Manufacturing Design.

HM-SYNC invites several meaningful questions regarding the nature of interaction in manufacturing environments: How does the nature of a point of interaction limit the forms of interaction that may occur? To what extent do individuals use parts of a machine differently? The nature of a point of interaction can significantly limit or enhance the forms of interaction that are possible. For example, a control panel with an intuitive layout may facilitate quicker and more efficient interactions, while a poorly designed interface may lead to frequent errors and inefficiencies. By using HM-SYNC to interrogate these aspects, researchers can gain valuable insights into the design of machine interfaces and manufacturing layouts. HM-SYNC allows researchers to study these interactions and understand how changes in the interface or machine parameters can modify the patterns of interaction, ultimately affecting the entire workflow. By collecting and analyzing patterns of use, this also supports the identification of opportunities for optimizing machine interfaces and manufacturing processes.

Additionally, HM-SYNC can also be used to support the development of tools for designing digital machine interfaces [41]. By capturing how people interact with machines in natural settings, one can decouple the interweaving of virtual versus physical interactions. This information can guide the design of digital interfaces, ensuring they are intuitive and aligned with the natural workflow of users. Understanding which types of actions and points of interaction are most common can help prioritize the design features that will have the greatest impact on efficiency and user satisfaction.

3.2 Adaptive Operating Design Procedures.

The use of advanced manufacturing machines to produce novel, innovative components is a constant challenge. One of the most significant challenges is establishing a standard or structure for machine operation in these new contexts. By analyzing the interactions of expert users with machines, like the WAAM machine, HM-SYNC can help identify patterns and develop new operating procedures tailored to specific contexts. This process involves observing and understanding how expert users adapt the machine’s use to fit new tasks or experimental setups, which can inform the creation of standardized operating procedures for these novel applications.

A critical aspect of this application is the ability to identify unique user interaction patterns. Each user may have a distinct style or method of interacting with the machine, and recognizing these patterns can serve multiple purposes. For instance, it can personalize the machine’s settings and parameters to match the preferences or style of the identified user, thereby optimizing the user experience and potentially increasing efficiency and safety. Moreover, HM-SYNC enables the identification of anomalous interactions, which are deviations from the established norms or patterns of machine use. Recognizing these anomalies is essential for several reasons. It can highlight potential misuse or errors, allowing for timely interventions to prevent accidents or equipment damage. It also provides insights into unusual but potentially innovative uses of the machine that could be standardized and incorporated into new operating procedures.

3.3 Human–Robot Collaboration.

HRC is an emerging field in advanced manufacturing focused on enhancing the synergy between human workers and robotic systems [12]. HM-SYNC provides a rich source of data for developing and refining HRC systems. By analyzing human interactions with the WAAM machine, researchers can identify key behaviors and actions that inform the design of collaborative robots (i.e., cobots). Cobots can be programmed to recognize and anticipate human actions, facilitating seamless collaboration. For example, HM-SYNC can be used to train machine learning models to predict when a human worker needs assistance or when a robot should take over a task. This predictive capability enhances the efficiency and safety of HRC systems, ensuring that both humans and robots can work together effectively. By understanding the typical interaction patterns and contextual cues from human operators, cobots can be designed to respond proactively, thereby reducing downtime and improving the flow of tasks in a manufacturing environment.

HM-SYNC also supports the development of adaptive HRC interfaces that respond to human actions in real time. By capturing detailed interaction patterns, we can design interfaces that are intuitive and responsive, reducing the learning curve for human workers and improving overall productivity. This adaptive capability is crucial for creating flexible manufacturing systems where robots can adjust their behavior based on real-time human inputs. For instance, a cobot might learn to adjust its speed or force when handling delicate components based on the observed preferences and actions of human workers.

3.4 Ergonomics and Human Factors.

Ergonomics and human factors engineering are fields focused on designing systems that optimize human well-being and performance. Focusing on human-centered design (as opposed to machine- or product-centered design), HM-SYNC provides critical insights into the physical and cognitive interactions between humans and manufacturing machines, enabling the design of ergonomic workspaces and tools. By analyzing the dataset, researchers can identify common physical actions and postures that may lead to discomfort or injury. This information is essential for designing workstations and tools that minimize physical strain and enhance comfort [42] under varying environmental and operating conditions that influence behavior [43]. For instance, HM-SYNC can inform the placement of controls and displays to reduce awkward movements and improve accessibility.

Additionally, HM-SYNC supports the design of cognitive aids that enhance human performance. Understanding how users interact with complex machines allows for the development of user interfaces that reduce cognitive load and improve decision making. For example, visual and auditory feedback can be designed to guide users through tasks, by reducing errors and increasing efficiency. By analyzing interaction patterns, designers can create interfaces that are intuitive and easy to navigate, helping workers to complete their tasks more efficiently and with less mental effort.

4 Conclusion

A deep understanding of manufacturing processes is essential for advancing manufacturing-oriented design. As advanced manufacturing technologies have evolved, human interaction has emerged as a critical factor in both operation and design. Despite the increasing role of automation, humans remain central to manufacturing systems, often adapting processes and introducing efficiencies beyond what is prescribed. Quantifying and characterizing these human interactions is crucial for understanding true manufacturing capabilities, as well as for developing human-centered design approaches that optimize performance. Expanding this understanding to include human interactions within complex systems more broadly can yield valuable insights into productivity, adaptability, and system resilience, supporting the design and engineering of more efficient and responsive manufacturing environments.

This work, although specifically focused on advanced manufacturing environments, is crucial for understanding how human interactions can be monitored, analyzed, and utilized to improve the design of complex engineered systems. HM-SYNC is one of the first to extensively capture human interactions with an advanced manufacturing machine, supporting emerging, promising, and privacy-preserving skeleton-based HAR techniques.

The data collection techniques and dataset organization presented in this work offer a reproducible method for collecting, curating, and analyzing general manufacturing interactions, interactions with other specific machines, or human behaviors in other complex systems. This work aims to inspire and support design engineers and provide a framework to better understand how complex systems are utilized by humans, thereby improving efficiency and enabling accurate analysis of these human interactions within a design context.

HM-SYNC has certain limitations that suggest areas for further improvement. One consideration is the variability in manufacturing equipment design and usage. Since HM-SYNC focuses on a single machine—the WAAM machine—and a small group of people performing interactions, drawing broad conclusions about human interaction across different machines or settings may be challenging. Continued efforts will aim to capture more varied types of interactions, including a more diverse set of machine users, different types of machines, interactions involving multiple people, and a broader range of skill levels among users. Expanding the dataset in these ways would enhance the generalizability of any observations made and provide deeper insights into how particular actions may affect the machinery and overall process.

Additionally, the current dataset focuses solely on the human interaction input side of machine operation. Machines are complex systems with various loaded parameter inputs and environmental contexts, as well as a range of outputs such as error logs, real-time state data, and sensor readings. Future efforts should aim to gather a comprehensive range of data from the machine, beyond just human inputs. This expanded dataset would strongly support the analysis of how human actions affect machine health, production quality, and efficiency, through the mapping of input–output data. This would support goals such as enabling real-time performance corrections for machine users to prevent errors and inefficiencies during design.

Considering the potential applications and future work outlined, HM-SYNC is broadly envisioned as a tool for advancing the understanding of human–machine interaction patterns and for developing enhanced models and digital twins of advanced manufacturing systems. This would provide feedback for human operators to better understand how to use the machinery, avoid inefficiencies, and give machine and manufacturing designers insight into how humans are interacting within the systems of interest. Beyond these applications, we foresee HM-SYNC serving as a catalyst for pioneering advancements in human-centered design, fostering environments where technology and human ingenuity converge seamlessly. The insights gleaned from this research could spark innovations that redefine productivity and well-being in industrial settings, driving a new era of intelligent, responsive, and adaptive manufacturing systems. As we continue to explore and expand the boundaries of human–machine symbiosis, HM-SYNC will serve as a foundational tool in shaping the future of manufacturing, where efficiency, adaptability, and human centricity are paramount.

Acknowledgment

The authors would like to gratefully acknowledge the Carnegie Mellon University Manufacturing Futures Institute (MFI) staff and Brian Belowich for their support in configuring the testbed, as well as the students and researchers using the WAAM machine for allowing the use of their space and providing critical knowledge in our efforts to produce this dataset.

Funding Data

  • Carnegie Mellon University Manufacturing Futures Institute.

Conflict of Interest

There are no conflicts of interest.

Data Availability Statement

All data and information supporting this article are freely available.4 .

Footnotes

References

1.
Herrmann
,
J. W.
,
Cooper
,
J.
,
Gupta
,
S. K.
,
Hayes
,
C. C.
,
Ishii
,
K.
,
Kazmer
,
D.
,
Sandborn
,
P. A.
, and
Wood
,
W. H.
,
2004
, “
New Directions in Design for Manufacturing
,”
ASME 2004 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference
,
Salt Lake City, UT
,
Sept. 28–Oct. 2
, pp.
853
861
.
2.
ElMaraghy
,
W.
,
ElMaraghy
,
H.
,
Tomiyama
,
T.
, and
Monostori
,
L.
,
2012
, “
Complexity in Engineering Design and Manufacturing
,”
CIRP Ann.
,
61
(
2
), pp.
793
814
.
3.
Lu
,
Y.
,
Adrados
,
J. S.
,
Chand
,
S. S.
, and
Wang
,
L.
,
2021
, “
Humans Are Not Machines–Anthropocentric Human–Machine Symbiosis for Ultra-Flexible Smart Manufacturing
,”
Engineering
,
7
(
6
), pp.
734
737
.
4.
Romero
,
D.
,
Stahre
,
J.
,
Wuest
,
T.
,
Noran
,
O.
,
Bernus
,
P.
,
Fast-Berglund
,
Å.
, and
Gorecky
,
D.
,
2016
, “
Towards an Operator 4.0 Typology: A Human-Centric Perspective on the Fourth Industrial Revolution Technologies
,”
International Conference on Computers & Industrial Engineering (CIE46)
,
Tianjin, China
,
Oct. 29–31
.
5.
Hozdić
,
E.
, and
Makovec
,
I.
,
2023
, “
Evolution of the Human Role in Manufacturing Systems: On the Route From Digitalization and Cybernation to Cognitization
,”
Appl. Syst. Innov.
,
6
(
2
), p.
49
.
6.
Gorecky
,
D.
,
Schmitt
,
M.
,
Loskyll
,
M.
, and
Zühlke
,
D.
,
2014
, “
Human-Machine-Interaction in the Industry 4.0 Era
,”
2014 12th IEEE International Conference on Industrial Informatics (INDIN)
,
Porto Alegre, Brazil
,
July 27–30
, pp.
289
294
.
7.
Suh
,
N. P.
,
1998
, “
Axiomatic Design Theory for Systems
,”
Res. Eng. Des.
,
10
, pp.
189
209
.
8.
Pasandideh
,
S.
,
Pereira
,
P.
, and
Gomes
,
L.
,
2022
, “
Cyber-Physical-Social Systems: Taxonomy, Challenges, and Opportunities
,”
IEEE Access
,
10
, pp.
42404
42419
.
9.
Williams
,
G.
,
Meisel
,
N. A.
,
Simpson
,
T. W.
, and
McComb
,
C.
,
2019
, “
Design Repository Effectiveness for 3D Convolutional Neural Networks: Application to Additive Manufacturin.
,”
ASME J. Mech. Des.
,
141
(
11
), p.
111701
.
10.
Brennan
,
J. B.
,
Miney
,
W. B.
,
Simpson
,
T. W.
,
Jablokow
,
K. W.
, and
McComb
,
C.
,
2023
, “
Manufacturing Fixation in Design: Exploring the Effects of Manufacturing Fixation During Idea Generation
,”
ASME J. Mech. Des.
,
145
(
1
), p.
012005
.
11.
Maganha
,
I.
,
Silva
,
C.
, and
Ferreira
,
L. M. D. F.
,
2019
, “
The Layout Design in Reconfigurable Manufacturing Systems: A Literature Review
,”
Int. J. Adv. Manuf. Technol.
,
105
, pp.
683
700
.
12.
Agrawal
,
A.
,
Won
,
S. J.
,
Sharma
,
T.
,
Deshpande
,
M.
, and
McComb
,
C.
,
2021
, “
A Multi-Agent Reinforcement Learning Framework for Intelligent Manufacturing With Autonomous Mobile Robots
,”
Proc. Des. Soc.
,
1
, pp.
161
170
.
13.
Breque
,
M.
,
De Nul
,
L.
, and
Petridis
,
A.
,
2021
,
Industry 5.0: Towards a Sustainable, Human Centric and Resilient European Industry
,
Publications Office of the European Union
,
Luxembourg, Luxembourg
.
14.
Sadeghi
,
L.
,
Dantan
,
J.-Y.
,
Siadat
,
A.
, and
Marsot
,
J.
,
2016
, “
Design for Human Safety in Manufacturing Systems: Applications of Design Theories, Methodologies, Tools and Techniques
,”
J. Eng. Des.
,
27
(
12
), pp.
844
877
.
15.
Zhang
,
Z.
,
Peng
,
G.
,
Wang
,
W.
, and
Chen
,
Y.
,
2022
, “
Real-Time Human Fault Detection in Assembly Tasks, Based on Human Action Prediction Using a Spatio-Temporal Learning Model
,”
Sustainability
,
14
(
15
), p.
9027
.
16.
Iodice
,
F.
,
De Momi
,
E.
, and
Ajoudani
,
A.
,
2022
, “
HRI30: An Action Recognition Dataset for Industrial Human-Robot Interaction
,”
2022 26th International Conference on Pattern Recognition (ICPR)
,
Montreal, QC, Canada
,
Aug. 21–25
, pp.
4941
4947
.
17.
Dallel
,
M.
,
Havard
,
V.
,
Baudry
,
D.
, and
Savatier
,
X.
,
2020
, “
InHARD—Industrial Human Action Recognition Dataset in the Context of Industrial Collaborative Robotics
,” 2020 IEEE International Conference on Human-Machine Systems (ICHMS), Rome, Italy.
18.
Roitberg
,
A.
,
Perzylo
,
A.
,
Somani
,
N.
,
Giuliani
,
M.
,
Rickert
,
M.
, and
Knoll
,
A.
,
2014
, “
Human Activity Recognition in the Context of Industrial Human-Robot Interaction
,” Signal and Information Processing Association Annual Summit and Conference (APSIPA), 2014 Asia-Pacific, Monterey, CA.
19.
Cicirelli
,
G.
,
Marani
,
R.
,
Romeo
,
L.
,
Domínguez
,
M. G.
,
Heras
,
J.
,
Perri
,
A. G.
, and
D’Orazio
,
T.
,
2022
, “
The HA4M Dataset: Multi-Modal Monitoring of an Assembly Task for Human Action Recognition in Manufacturing
,”
Sci. Data
,
9
, p.
745
.
20.
Yao
,
J.
,
Li
,
K.
, and
Su
,
R.
,
2023
, “
HUMM: A New Dataset for Human Movement in Manufacturing
,”
2023 8th International Conference on Robotics and Automation Engineering (ICRAE)
,
Singapore
,
Nov. 17–19
, pp.
287
292
.
21.
Carrera-Rivera
,
A.
,
Reguera-Bakhache
,
D.
,
Larrinaga
,
F.
,
Lasa
,
G.
, and
Garitano
,
I.
,
2023
, “
Structured Dataset of Human-Machine Interactions Enabling Adaptive User Interfaces
,”
Sci. Data
,
10
(
1
), p.
831
.
22.
Reguera-Bakhache
,
D.
,
Garitano
,
I.
,
Uribeetxeberria
,
R.
,
Cernuda
,
C.
, and
Zurutuza
,
U.
,
2020
, “
Data-Driven Industrial Human-Machine Interface Temporal Adaptation for Process Optimization
,”
2020 25th IEEE International Conference on Emerging Technologies and Factory Automation (ETFA)
,
Vienna, Austria
,
Sept. 8–11
, pp.
518
525
.
23.
Lincoln Electric Holdings, Inc.
,
2021
, “
SCULPTPRINT RND - ADDSYS-1
,” https://engineering.cmu.edu/mfi/_files/documents/equipment/lincoln-electric-additive-solutions-sculptprint-rnd-description.pdf, Accessed March 3, 2024.
24.
Microsoft Corporation
,
2022
, “
Azure Kinect DK Hardware Specifications
,” https://learn.microsoft.com/en-us/azure/kinect-dk/hardware-specification, Accessed March 4, 2024.
25.
Microsoft Corporation
,
2024
, “
Azure Kinect Body Tracking SDK
,” https://microsoft.github.io/Azure-Kinect-Body-Tracking/release/1.1.x/index.html, Accessed March 4, 2024.
26.
Doctorarastoo
,
M.
,
Flanigan
,
K.
,
Bergés
,
M.
, and
Mccomb
,
C.
,
2023
, “
Exploring the Potentials and Challenges of Cyber-Physical-Social Infrastructure Systems for Achieving Human-Centered Objectives
,”
Proceedings of the 10th ACM International Conference on Systems for Energy-Efficient Buildings, Cities, and Transportation (Buildsys'23)
,
Istanbul, Turkiye
,
Nov. 15–16
, pp.
385
389
.
27.
Nikolov
,
P.
,
Boumbarov
,
O.
,
Manolova
,
A.
,
Tonchev
,
K.
, and
Poulkov
,
V.
,
2018
, “
Skeleton-Based Human Activity Recognition by Spatio-Temporal Representation and Convolutional Neural Networks With Application to Cyber Physical Systems With Human in the Loop
,” 2018 41st International Conference on Telecommunications and Signal Processing (TSP), Athens, Greece.
28.
Martins
,
J.
,
Flanigan
,
K.
, and
McComb
,
C.
,
2023
, “
Skeleton-Based Human Action Recognition in a Thermal Comfort Context
,”
Proceedings of the 10th ACM International Conference on Systems for Energy-Efficient Buildings, Cities, and Transportation (Buildsys'23)
,
Istanbul, Turkiye
,
Nov. 15–16
, pp.
377
384
.
29.
Lin
,
C.
,
Martins
,
J.
, and
Flanigan
,
K. A.
,
2024
, “
Read the Room: Inferring Social Context Through Dyadic Interaction Recognition in Cyber-Physical-Social Infrastructure Systems
,”
ASCE International Conference on Computing in Civil Engineering
,
Pittsburgh, PA
,
July 28–31
.
30.
Nguyen
,
H.-C.
,
Nguyen
,
T.-H.
,
Scherer
,
R.
, and
Le
,
V.-H.
,
2023
, “
Deep Learning for Human Activity Recognition on 3D Human Skeleton: Survey and Comparative Study
,”
Sensors
,
23
(
11
), p.
5121
.
31.
Lin
,
C.
,
Doctorarastoo
,
M.
, and
Flanigan
,
K.
,
2024
, “
Your Actions Talk: Automated Sociometric Analysis Using Kinesics in Human Activities
,”
Proceedings of the 11th ACM International Conference on Systems for Energy-Efficient Buildings, Cities, and Transportation (Buildsys'24)
,
Hangzhou, China
,
Nov. 7–8
, pp.
271
278
.
32.
Lin
,
C.
,
Flanigan
,
K.
, and
Munir
,
S.
,
2024
, “
Meaning Through Motion: DUET—A Multimodal Dataset for Kinesics Analysis in Dyadic Activities
,”
NeurIPS 2024 Workshop on Behavioral Machine Learning
,
Vancouver, BC, Canada
,
Dec. 14
, pp.
1
7
.
33.
Ji
,
Y.
,
Xu
,
F.
,
Yang
,
Y.
,
Shen
,
F.
,
Shen
,
H. T.
, and
Zheng
,
W.-S.
,
2018
, “
A Large-Scale RGB-D Database for Arbitrary-View Human Action Recognition
,”
Proceedings of the 26th ACM International Conference on Multimedia (MM'18)
,
Seoul, South Korea
,
Oct. 22–26
, pp.
1510
1518
.
34.
Vinciarelli
,
A.
,
Esposito
,
A.
,
André
,
E.
,
Bonin
,
F.
,
Chetouani
,
M.
,
Cohn
,
J. F.
, and
Cristani
,
M.
,
2015
, “
Open Challenges in Modelling, Analysis and Synthesis of Human Behaviour in Human–Human and Human–Machine Interactions
,”
Cognit. Comput.
,
7
(
4
), pp.
397
413
.
35.
Tomar
,
S.
,
2006
, “
Converting Video Formats with FFmpeg
,” https://www.linuxjournal.com/article/8517, Accessed March, 2 2024,.
36.
Doctorarastoo
,
M.
,
Lin
,
C.
,
Flanigan
,
K.
, and
Bergés
,
M.
,
2024
, “
A Bayesian Network Approach for Predicting Social Interactions in Shared Spatial Environments
,”
Proceedings of the 11th ACM International Conference on Systems for Energy-Efficient Buildings, Cities, and Transportation (Buildsys'24)
,
Hangzhou, China
,
Nov. 7–8
, pp.
279
286
.
37.
Sarker
,
S.
,
Rahman
,
S.
,
Hossain
,
T.
,
Faiza Ahmed
,
S.
,
Jamal
,
L.
, and
Ahad
,
M. A. R.
,
2021
, “Skeleton-Based Activity Recognition: Preprocessing and Approaches,”
Contactless Human Activity Analysis
,
M. A. R.
Ahad
,
U.
Mahbub
, and
T.
Rahman
, eds.,
Springer International Publishing
,
Gewerbestrasse, Switzerland
, pp.
43
81
.
38.
Gammulle
,
H.
,
Ahmedt-Aristizabal
,
D.
,
Denman
,
S.
,
Tychsen-Smith
,
L.
,
Petersson
,
L.
, and
Fookes
,
C.
,
2023
, “
Continuous Human Action Recognition for Human-machine Interaction: A Review
,”
ACM Comput. Surv.
,
55
(
13s
), pp.
1
38
.
39.
Chen
,
C.
,
Jafari
,
R.
, and
Kehtarnavaz
,
N.
,
2015
, “
UTD-MHAD: A Multimodal Dataset for Human Action Recognition Utilizing a Depth Camera and a Wearable Inertial Sensor
,”
2015 IEEE International Conference on Image Processing (ICIP)
,
Quebec City, QC, Canada
,
Sept. 27–30
, pp.
168
172
.
40.
Wilkinson
,
M. D.
,
Dumontier
,
M.
,
Aalbersberg
,
I. J.
,
Appleton
,
G.
,
Axton
,
M.
,
Baak
,
A.
, and
Blomberg
,
N.
,
2016
, “
The FAIR Guiding Principles for Scientific Data Management and Stewardship
,”
Sci. Data
,
3
, pp.
1
9
.
41.
Dhengre
,
S.
,
Mathur
,
J.
,
Oghazian
,
F.
,
Tan
,
X.
, and
McComb
,
C.
,
2020
, “
Towards Enhanced Creativity in Interface Design Through Automated Usability Evaluation
,”
The 11th International Conference on Computational Creativity (ICCC'20)
,
Virtual
,
Sept. 7–11
, pp.
366
369
.
42.
Doctorarastoo
,
M.
,
Flanigan
,
K.
,
Bergés
,
M.
, and
McComb
,
C.
,
2023
, “
Modeling Human Behavior in Cyber-Physical-Social Infrastructure Systems
,”
Proceedings of the 10th ACM International Conference on Systems for Energy-Efficient Buildings, Cities, and Transportation (Buildsys'23)
,
Istanbul, Turkiye
,
Nov. 15–16
, pp.
370
376
.
43.
Doctorarastoo
,
M.
,
Flanigan
,
K. A.
, and
Bergés
,
M.
,
2024
, “
Preference-Aware Human Spatial Behavior Modeling in Cyber-Physical-Human Systems
,”
IFAC-PapersOnLine
,
58
(
30
), pp.
115
120
.