Deep Learning-Driven Design of Robot Mechanisms

Neural networks (NNs) are well known as universal function approximators for a class of functions defined with the Euclidean space as their domain [1,2]. Classically, for approximating functions whose domain is not Euclidean [3], handcrafted feature vectors were used to map the data to a Euclidean space and then NNs could be used to learn or approximate the function. Consequently, the performance of NNs were limited by the human designers’ ability to capture or map the relevant features. The feature vectors are an Euclidean embedding of the data so that the relevant aspects of the data required for learning are captured. The modern revolution in deep learning (DL) came from the insight that the feature vectors should be learned from data as well, and different neural network architectures were designed for different data types that enabled this feature extraction or embedding of the data in a Euclidean space [4–7]. Examples include convolution neural networks (CNNs) [8] for image data, graph neural networks (GNNs) [9] for data obtained from systems most naturally modeled as a graph, and transformer networks [10] for sequential data, which have been used for learning large language models. Use of deep neural networks (DNNs) in engineering design is relatively new; a recent review on this topic is provided in Ref. [11] and in a special issue on Machine Learning in Engineering Design by the ASME Journal of Mechanical Design [12].

NNs have proven to be an interesting alternative in engineering design problems characterized by difficulty in formulating analytical representations, imprecise input, and large solution space. DNNs’ advantages over classical NNs have been that they allow one to automate the process of feature generation and provide a larger representation and mapping capacity. Let us consider an example of generating a four-bar mechanism for path synthesis problems. For designing a neural network to solve this problem, there are various ways of representing the input path that has been explored in the past, e.g., Fourier descriptors of the curve and wavelet descriptors of the curve. These are called different feature vectors for the path and even changing the number of Fourier descriptors or wavelet descriptors used changes the feature vectors. The feature vector choice is made by the human designer a priori and there is no clarity on which feature vector is preferable, if any. The choice of a feature vector is usually a trial-and-error process, and there is seldom any universally good choice (at least, such a choice has not been discovered yet). The use of DNNs allows the learning of the appropriate feature vector. The user can input the path as a list of points or an image and using a training dataset, the neural network will learn the features as well as the mapping from the features to the output space of mechanism parameterization. However, the cost to be paid for such flexibility is more training data and computational time. The paradigm of learning features instead of hand-coding them (which was also prevalent in computer vision and natural language processing (NLP) literature) has worked well across different application areas including computer vision and NLP. So the hope is, and preliminary results do support this, that the use of DNNs will make the learned models produce better quality mechanisms that follow the given input paths as closely as possible.

The design of robotic mechanisms from a given set of design specifications can also be formulated as a learning problem from the space of specifications to a parameterization of the space of mechanisms. Classically, design of mechanisms has been a two-step process of type and dimensional synthesis with scant attention paid to the type synthesis. As a result, the synthesis problems have been reduced to primarily computing dimensional parameters after selecting a type [13–15]. This approach has shown to be ineffective in creating practical mechanism design solutions from the given task [16]. In the context of machine learning (ML), the space of design specifications, which can be quite varied and application specific, is non-Euclidean in general. Examples of design specifications can be a desired planar curve (planar path synthesis problem), a subset of the group of rigid body configurations or SE(3)² (mechanism synthesis for a given workspace, which can be useful in exoskeleton robotic device synthesis), etc. [17]. Furthermore, there can be other geometric and kinematics constraints and objectives like the location of some of the pivots [18], moving envelope of mechanisms, relative link length ratios, defect-free generation [19], etc. The current DL literature does not deal with this kind of data that arise in mechanism design, and consequently there is no known way of neural network architecture design to embed the design specifications into a Euclidean space.

Within the mechanism design literature, CNNs have been used to embed design specifications given as planar curves into a Euclidean space, by converting the curve to a 2D image [20]. In the past, Fourier descriptors [21–26] and wavelets [27–31] have also been used to represent coupler curves. However, creating neural network architectures that can embed design specifications into a Euclidean space or learn features of the design task is a largely unexplored research challenge. More recently, it has been shown that geometric deep learning models could capture non-Euclidean domains more efficiently and effectively such as in graphs, trees, manifolds, and networks [32,33]. The joint representation of the design space along with the representation of known mechanisms that produces a design can lead to novel encoder–decoder architectures or deep generative networks that can facilitate novel mechanism synthesis [34]. This is analogous to the use of CNNs with variational autoencoder (VAE) [35,36] or generative adversarial network (GAN) [37] for DL-based image synthesis from known images of object categories. Moreover, it is well known that for simple one-degree-of-freedom linkage mechanisms, such as planar four-bar, Fourier descriptors of the path and the orientation are coupled, which can be exploited to synthesize mechanisms for motion generation problems [38,39]. An open question is whether such relationships can be captured by a NN by only providing a large amount of good synthetic simulation data? If so, this could lead to a breakthrough in developing artificial intelligence (AI) systems, which can predict critical kinematic properties of robot mechanisms. Enforcing a structure on the latent layers in VAEs and GANs could enable design synthesis under multiple constraints while helping such AI systems to be explainable and interpretable. In summary, there are great opportunities for researchers to (1) embed existing knowledge to create effective representations of mechanisms (type and dimension) and (2) use domain-specific algorithms in conjunction with ML models to process synthetically produced motion data from simulations or from physical motion capture. The outcomes could be component-based reusable unified NN models with sufficient granularity so that they can be applied in diverse synthesis applications and truly integrate machine learning with robotic mechanism design.

The rest of this paper discusses specific research questions and challenges, which could lead to new areas of investigation in deep learning of robot mechanisms. The foremost of these is the issue of representation of mechanisms and their properties. While the mechanisms themselves are defined by their structure, their properties are defined by the motion generated by a floating link of mechanisms. The space of design specifications primarily begins by defining desirable properties, but can also include structural parameters. We look at the representation aspects in Sec. 2. Next question is of the mapping from the design specifications to mechanisms. The goal is to generate a large number of conceptual design solutions, which satisfy design specifications while allowing users an intuitive way to explore the design space. The mapping should ideally generate both type and dimensions of single- and multi-degree-of-freedom mechanisms. This naturally leads to the questions of (1) representation of mechanism parameters, which encode both the type and dimensions, and (2) generation of a dataset for training DNNs with good quality of synthetic motion data for a variety of mechanisms. We discuss these issues in Sec. 3. Related to the question of mapping from design specifications to the mechanisms is the design of DNN architectures which would enable the mapping. While the recent research in DNNs has proposed a variety of generative frameworks, such as VAE, GAN, and their variants, it is unclear as to how they should be leveraged in mechanism synthesis! In Sec. 4, we discuss a few recent work using VAEs and possible directions for future efforts in designing DNNs for mechanism synthesis.

Linkage mechanisms are a collection of rigid bodies connected via joints; for example, the most ubiquitous of them is a planar four-bar mechanism consisting of four links connected by prismatic or revolute joints. Their topology is defined by the pattern of interconnection of their links and joints, while the length of the links and the location of joints define the dimensional parameters of the mechanism. Kinematic synthesis of such mechanisms is typically concerned with finding their topology and dimensions given design specifications, such as path of a point of interest on a floating link or the complete motion of the floating link as well as additional constraints. There have been several efforts in recent years to design NNs, which learn the mapping from the space of design specifications to the mechanism properties (type and dimensions). However, these efforts have largely been limited to a path or motion as the input and dimensional parameters as the output. Even then, several fundamental questions have remained unanswered, such as the choice of best representation for the path or the motion specifications for a given problem. Typically, one of these representations would serve as the input vectors for the DNNs. Fourier descriptors, wavelets, Cartesian coordinates, and curvature integral have remained the dominant choice for representing the path of a point; however, the choices made have been arbitrary without adequate justification. More recently, it was proposed that to leverage the learning capabilities of CNN, a path could be embedded in an image of fixed size, which would be independent of the algebraic complexity of the curve [20]. This area of research requires a careful investigation of different representations of the coupler curves and their comparative advantages and disadvantages. A key question related to their representation is if there could be a representation, which acts as a normalizer for the NN-based mechanism synthesis? In other words, could we use a unifying representation, which is invariant of these base representations and is insensitive to the generated mechanism solutions. This would be similar to the idea of a deep CNN extracting features from an input automatically.

While most work has focused on the path synthesis of planar mechanisms, their extension to the motion generation problem is even harder. That begets how the path and orientation together should be represented? Since they are unit incompatible, how to combine them in a single input vector for a NN is unclear! While it is well known that the path and orientation of the coupler of single-degree-of-freedom mechanisms cannot be decoupled, a straight-forward extension of the current approach would be to find Fourier or wavelet descriptors, Cartesian and curvature integral of path and orientation data independently and flatten the input vector to include both of them.

A third related question is how these representations should be extended for spherical and spatial mechanism motions, where the paths are in R³ and motions in SE(3). This debate is not limited just to the path and motion properties of mechanisms; it gets even more difficult when one needs to incorporate other geometric and kinematic constraints, such as location of the pivots, transmission angle, range of motion, or dimensional constraints. These design specifications form a non-Euclidean space for a NN and there is little literature on this.

It is heartening that alternative representations for the properties of a mechanism, such as path and motion of a mechanism, have been proposed. On the other hand, representation of the structure of mechanisms for NNs has been largely relegated to using an output vector of few joint positions. This may be fine for synthesizing dimensional parameters of a mechanism if the type of the mechanism chosen is accurate; however, it has been shown that a pre-selection of the type of mechanism often leads to no or poor solutions. Thus, there is a need to encode the type and dimensional information of a mechanism together for its representation in NNs. Kinematicians are familiar with the seminal work of Tsai [40], who modeled mechanisms as graphs and represented them using an adjacency- or incidence-matrix. While such representations may be useful for enumeration and evaluation of mechanisms, they form a non-Euclidean space of mechanism parameters, which makes it difficult to directly employ and interpret them in DNNs. What may be needed is a structured coupling of the type and dimensional parameters, which can be learnt by DNNs. We discuss such a possible representation in the next section in the context of motion synthesis using a DNN.

The ultimate goal is to learn a mapping from a diverse and often conflicting set of design specifications (including path or motion properties and additional constraints) to mechanism structure (type and dimensions). This mapping should yield a large set of conceptual designs satisfying specifications. Existing literature on using neural networks for mechanism synthesis is largely focused on the path generation problem, wherein a path is provided as an input to a NN using one of the aforementioned representations and the NN is trained to learn the mapping from the path information to the dimensional parameters of a mechanism. While these early efforts have produced desirable outcomes, the problem of synthesis is defined too narrowly and the results are far and few. Generally, these approaches yield one or a few mechanisms based on a specific task requirement, which is often a computational convenience rather than true representation of users’ requirements. For example, a given path or motion is converted into a precision point or pose problem with discrete data points, which can be embedded in a Euclidean space and a direct mapping is achieved using a NN. Moreover, typically the user will have to make an artificial choice about the type of mechanism and the goal will be limited to finding dimensional parameters. If the goal is to find a large set of feasible mechanism building blocks without assuming their types while accounting for amorphous, incomplete, and imprecise input, then the traditional or aforementioned NN-based synthesis approaches fall short.

It has been shown recently that the classical two-step paradigm of mechanism design, wherein the topology or type of a mechanism is first picked and then its dimensions are computed, is a flawed approach, which often yields no or highly suboptimal solution. Fruits of such endeavor will be sweet only if a true data-driven approach is incorporated in synthesis—one which includes simultaneous type and dimensional synthesis. However, the early work in this direction have focused on creating unified design equations for a class of mechanisms and then solving a low-degree polynomial to generate a limited number of solutions, which mostly produce defective mechanisms. There have also been several attempts to incorporate additional design specifications, such as location of pivots, but their extension to the more complex constraints or even higher-order mechanisms using a pure analytical approach has proven to be much more difficult.

Another set of key challenges that arise in forming the end-to-end system is the decision on architecture of the deep neural network, the design of the training algorithm, and obtaining the data to train the network for a particular task. The existing NN architectures for mechanism synthesis have been simple mapping networks from a path description to mechanism parameters. These have been limited to mostly planar four-bar mechanisms.

In the recent years, DL-based generative frameworks based on VAEs, GANs, and deep reinforcement learning have proven to be quite effective in generating a variety of synthetic data, such as handwritten digits, images, and videos. They have also been used in other fields notably autonomous driving, molecule design, anomaly detection, text classification, recommendation system, and many more. While autoencoders, a simpler variation of a VAE, can only compress the input data into a latent space, a VAE can also generate new data by sampling in the latent space. These generative models can be used for learning joint probability distribution of various mechanism parameters and their interdependence. With that, we can ask the following probabilistic inference questions: (1) what are the infeasible aspects of the input and how to modify them to make a more conducive input, (2) given a path or a motion task, how likely it is that a particular type of mechanism can perform the task, and (3) for a given task, what is the distribution of mechanism parameters with similar coupler motions (or, paths)?

A VAE-based approach could accept deliberately imprecise or inherently uncertain input from users and compute a distribution of conducive inputs and feed them to a synthesis solver to obtain a large variety of acceptable solution concepts. Having learnt the probability distribution of the coupler curves of mechanisms, the encoder part of the VAE predicts a 2D Gaussian distribution of latent features for each of the inputs. By drawing random samples in the vicinity of the computed latent code, the generator network of the VAE outputs the samples with closed paths that resemble the original input path. While a traditional VAE would simply generate outputs similar to the input, our synthesis goal requires that a VAE be augmented with another mapping to generate mechanism design parameters. Two potential approaches for this are to (1) map latent codes to mechanism parameters by a kNN search and a database query or (2) train another fully connected DNN to map from the latent code to mechanism parameters. While in the former approach, generated mechanisms could be tightly controlled, such as Grashof-only or of a certain complexity, in the latter approach, the capacity to represent a mechanism is completely built-in a NN, which could lead to more efficient and generalized query. Figure 1 shows an overview of this approach where a sequence of discrete points is the input to a trained VAE and the output are various mechanisms.

MotionGen Pro [17,42], a web-based app available,³ has implemented a VAE for path generation with a database and produces up to 30 four-bar and six-bar mechanisms for a user-sketched path. The training data were generated from a unified simulation algorithm presented in Ref. [42]. Figure 2 shows a few mechanism concepts produced by the MotionGen for a sit-to-stand motion path of hip joint.

While these VAE-based approaches have resulted in successfully generating a number of mechanism design solutions of specific type, they have still been restricted to approximating users’ task into a set of points embedded in a Euclidean space for the path generation problem. Several other questions as well remain to be answered: (1) how can this be scaled to other mechanisms without having to train separate NNs for each type of mechanisms, (2) how can motion generation problem be solved using such an approach, (3) how can additional constraints be handled, and (4) how can users visualize and filter out a large number of solutions? There are several variants of VAEs available, such as conditional-VAE (C-VAE), which enable conditioning input vectors on some random variables. A C-VAE could be used to condition the given task on, say, location of joints, which could enable users to directly synthesize mechanisms with specific properties.

The basic premise of the deep learning-driven robot mechanisms is that we want to create a probabilistic distribution of output motions of a variety of mechanisms including planar, spherical, and spatial ones. We are fortunate that we can algorithmically assemble different types of mechanisms and perform forward kinematics to get the output motions. So far, the NN literature has largely investigated a mapping between simple planar four-bar mechanisms and their paths. It is of course, but a prudent choice, to begin with something simple like planar four-bar mechanisms for which several simulation algorithms exist, which can create a large dataset of mechanisms. Their extension to the spherical and spatial multi-chained mechanisms, such as those found in exoskeleton devices for the different human applications, is hardly a straight-forward extension; see, for example, exoskeleton mechanisms discussed in Refs. [43–45]. This requires that new computationally efficient algorithms be developed, which can provide high-quality synthetic data associating mechanism designs with their output motions. Good quality data used in training a VAE or GAN or one of its variants will be amenable to exploration in latent space. Once we have learned this distribution, then the next question is of querying. While the user provides a raw input, a kNN or related search in the latent space of a generative network could find a plethora of design solutions, which can be further subjected to constraints on type, dimension, as well as range of motion. This is the basic idea, but a fundamental question is how to systematically explore the latent space, which requires that we have some understanding of the manifold structure of the latent space [46]. With that, it may be possible to impose the constraints ab initio in the design process, which has not been done before.

Apart from the ML approaches discussed so far in representation of the specification space, mapping from set of specifications to mechanisms, and DNN models, recently several other generative models have taken the world by storm. Diffusion models are a class of generative models and have primarily found applications in image synthesis [47,48]. The key idea is to iteratively add Gaussian noise to an input image to convert it into noise and also simultaneously learn the inverse conditional distribution function from the noise to an image iteratively. Within the image synthesis literature, diffusion models have been known to produce more photo-realistic images than GANs. Although the use of diffusion models has not been explored thus far in mechanism design, they can be potentially used in mechanism design problems where the use of GANs have been explored.

Transformer networks have been proposed originally for various learning tasks on sequential data, a hallmark of language processing applications [49,50]. However, more recently they have also found applications in image processing tasks and tasks involving multimodal input data (e.g., text and image). There are two key strengths of transformer networks. First, in principle, transformer networks allow one to encode global structure in the input, if present, which is usually referred to as long range dependencies or context in the NLP tasks. Second, they allow a common encoding (or embedding) of multimodal input data, which allows one to answer queries or give output for inputs in any modality. From the perspective of mechanism design, this raises interesting possibilities for embedding design constraints. For example, it may provide a way for stating qualitative constraints like “make the coupler shorter than the fixed link.” Furthermore, it can also provide ways to give constraints on locations of a fixed pivot by marking an area on an image. To develop such capabilities, the training dataset also has to be generated appropriately, based on the types of constraints and modes of specification that may be of interest to designers. The generation of such training sets itself provides an interesting opportunity and offers various possibilities of how designers can interact with a design software.

Physics-informed machine learning (PIML) approaches try to generate solutions for complex multi-scale systems by combining observational data with the mathematical models underpinning the physics of the system [51,52]. This field of research has gained momentum in many application areas including prediction of complex systems like atmospheric air-flow, design of functional materials, etc. PIML approaches can potentially lead to formulation and study of new problems in mechanism design that to the best of our knowledge are currently not well studied. For example, for given design tasks, there is very little in terms of principled approaches for synthesizing multi-link mechanisms with both kinematics and dynamics constraints as well as rigid and flexible components. In such scenarios, the physics could be modeled by (partial) differential equations governing the dynamics of motion along with algebraic constraints modeling the kinematic constraints. It is within the realm of possibility to formalize such problems as learning problems. Apart from architecture design and modeling challenges, a key challenge that arises here is in generating training data sets, which has to be curated carefully and possibly with a community-driven effort, with data sets coming from multiple researchers with their own application interest and expertise.

The formulation of the robotic mechanism design problem as a learning problem has immense potential of introducing deep learning techniques as a new tool to the design engineers toolbox. DNN-based approaches have the potential of solving problems in motion synthesis and path synthesis along with other scenario-specific constraints on the mechanism in a unified framework. However, there are key technical challenges that need to be overcome for DNN-based synthesis techniques to realize their potential. The key problems that we identified are that of representation of the design specifications and the mechanisms, development of the mapping from the design specification space to the mechanism parameterization that allows both type and dimensional synthesis, and design of the DNN architecture to learn the aforementioned mapping. We summarize key questions and approaches below:

1. The first area of investigation pertains to the representation of mechanisms, including their type, dimensions, and properties, as well as input specifications related to path, motion, pivots, and link length, among others. Specifically, there are several questions that need to be addressed:

   • How can one embed these properties and specifications in a Euclidean space, and can DNNs assist in the creation of invariant representations that do not require feature engineering?

   • What is the best way to develop a unified representation that combines inputs that are incompatible, such as path and orientation data?

   • How can these representations be extended from 2D to 3D spatial data for spherical and spatial mechanisms?

2. The second area of investigation focuses on NN-based mapping from the design space to mechanisms. This involves addressing the following questions:

   • How can a unified representation of the type and dimensional parameters of mechanisms be developed that can be learned by NNs? Can traditional representations, such as adjacency matrices, and newer techniques, such as GNNs, be useful?

   • What is the best way to leverage generative models to create a diverse range of mechanism design concepts?

   • How can conflicting design constraints be accommodated in synthesis and can multimodal models be useful?

3. The third area of investigation pertains to DNN architecture and training data. Here are some of the questions that need to be answered:

   • What is the best way to generate high-quality training data from simulations?

   • Which DNN architecture is most appropriate, and can end-to-end architectures be developed that meet design specification requirements?

   • Can a single DNN serve the diverse needs of generating mechanism design concepts?

   • How can the approach be scaled up to more complex mechanisms, such as spherical and spatial mechanisms?

This work was supported in part by NSF STTR phase 2 award #2126882 and NSF CMMI award #1853454. All opinions and conclusions presented in this paper are those of authors only and not of funding agencies.

The first author of this publication has research support from an NSF STTR phase 2 award (#2126882) as the Co-PI and also holds stock in Mechanismic Inc. The research findings included in this publication may not necessarily relate to the interests of Mechanismic Inc. The terms of this arrangement have been reviewed and approved by Stony Brook University in accordance with its policy on objectivity in research.

No data, models, or code were generated or used for this paper.