Reflect–Express–Transform: Investigating Speech-Based Iterative Digital Design for Young Designers

Our goal in this work is to explore verbal descriptions as a means to support iterative idea generation for young individuals. Iteration plays a central role in the early design process [1], where the goal is seldom to come to a conclusive solution to a problem. Instead, one is trying to refine their understanding of the problem through repetitive experiments and reflecting on the outcomes, thereby “moving through the state space of possible designs” [2]. As it is, providing meaningful computer-aided cognitive support for early design is a challenging task even for engineering students, given the divergent thinking involved in the quick exploration of a large quantity of ideas [3]. When considering younger students in the K-12 STEM (science-technology-engineering-mathematics) fields, the challenge increases even further. While these students may possess creative potential and design intuition, they do not necessarily have the formal understanding of the iterative nature of the design process. We specifically seek to investigate interactive methods for enabling design iteration for younger age groups.

Much of graphics and HCI literature on computer-support tools for design has, intentionally or unintentionally, adopted and embodied these principles of iteration and reflection. What is interesting is that the modality of “moving” (i.e. taking an action on the design) is almost always drawing with hands, as Schon imagines in his work [1]. There are numerous interactions, interfaces, and workflows that enable and/or study some form of design iteration through sketching [4–6], and even tangible and spatial interactions [7,8]. Verbalization, on the other hand, has been little explored as a way to directly enable design iterations. This form of interaction might afford more convenience to the younger audiences [9] in the design process, and as such, we primarily focus on verbalization as an input modality to enable 3D design iteration in this work.

While verbal communication in design has been quite extensively studied [10–17], it is surprising that there is little known regarding how to utilize speech as an input modality in creativity support systems [18]. Even the systems that explore speech as an input for design modification tasks [19–22], either explicitly or in combination with other modalities, have not studied the fundamental cognitive workflow that verbal interactions enable in association to the design process. Our work is motivated by the observation that in order to enable iterative design through verbal communication, there is a need for fresh perspectives on how to enable speech-based workflows as well as systematic study of what verbal input affords in an iterative design process. This work presented in this article is an extension of our recently published work [23].

Design iteration is an important part of the design process [24], where the designer goes through a series of steps and revisions to learn more information about their design problem and form a solution. The cognitive processes involved in design iteration are important to understand. Iterations can help in converting an ill-structured problem into a well-structured solution, which might not be obvious in the final design. Adams and Atman [25] studied this by first identifying transition behavior as the behavior displayed by the designer while making decisions between the design steps, which can be in the form of information processing activities and decision activities. Next, they conducted a study, where they used verbal protocol data to observe how freshman and senior students introduced new information and knowledge into their design process, and categorized the data using their transition behavior codes. By using this approach, they were able to qualitatively and quantitatively observe the cognitive processes that occurred during the design iterations. Faste [26] conducted a reflective study to understand the practice of intuition in design for an iterative aesthetic task. The users self-reflections regarding their iterations were documented over several weeks, where they observed four dimensions of intuition in design, namely efficiency, inspiration, curiosity, and insight. Enabling design iterations, therefore, is a key requirement to allow users to gain important information and knowledge about the problem and generate solutions.

As such, interfaces that provide computational support to the design process, need to, at the very minimum enable and support design iteration [27]. Sketch-based interfaces have been widely studied and used to support design processes [28]. Zhao et al. [5] developed a framework “skWiki,” which supported collaborative editing of sketches, texts, as well as photographs. Users had the ability to revisit not only their own but also their collaborators’ past iterations and integrate it into their own designs. Similarly, Piya et al. [6] developed a sketch-based collaborative 3D modeling software where design iterations could be easily shared, and re-used between users, while being stored and accessed in a hierarchical form. Vinayak et al. [8] developed “Mobisweep” to allow users to quickly generate iterations of 3D sweeps from sketches.

While sketching as an interaction has been explored extensively, the study and use of verbalization as a means to make direct design modifications have been limited. Language in design has primarily been used to extract information from textual databases [29,30] but very rarely used to support the design process. Prior works that make use of speech in design interfaces tend to focus either on multi-modal interfaces (e.g., gestures and speech), where the role of speech is diminished to simpler design tasks (such as instantiating primitive shapes) [22,31–34], or speech as an alternative mode of interaction to existing CAD software [19,21], where the software’s design process is not conducive to early stage design. Therefore, there has been a lack of a systematic study of how speech can be leveraged in computational support tools in a way that promotes the iterative nature of design.

The importance of verbalization during the formative years has been well studied and has shown to aid the process of learning and improving cognitive skills [9]. We learn through the process of verbalization even before having the ability to write or make sketches. In design, studies have shown the importance of verbal reflection in improving the design solution through the design iteration process [35,36]. The study by Bilda et al. [37] shows that the results that expert architects produced when verbalizing their ideas were similar to when they used sketching to communicate their ideas. Research in engineering design [38] has also shown evidence that while sketches are effective for communicating three-dimensional (3D) concepts, expressing the design intent to create those concepts is often more effectively initiated through verbalization, i.e., by using spoken language. The use of spoken language in relation to CAD functions (e.g., rotate, scale, move) have been studied by Khan et al. [39–41], where they conduct studies to understand how different groups of people use speech and gesture to perform specific CAD functions; however, this knowledge has not been operationalized into a computational tool. While prior works highlight the importance of verbalization in the design process, a deeper understanding of verbalization as an input modality to enable 3D design iteration is needed.

As a first step toward understanding whether verbalization for design transformation is suitable for the design process, our objective is to first study whether such an interaction enables design iteration. There are three technical/technological challenges that emerge when considering verbal input. First, language is a highly contextual modality and interpreting user’s intent even outside of design is quite challenging depending on the context. Second, verbal communication during the design process has been famously known to be ambiguous and messy [10,18,42]. Finally, how we communicate in design is still not well understood. Wiegers et al. [11] classified different verbal patterns for describing how people talk about shapes. However, the way in which designers talk about functions and their relationships with shapes is currently not available to the best of our knowledge. Therefore, within the broader vision of multi-modal (vision, speech, sketch, and gesture) conceptual design workflows, enabling a full-fledged natural language interaction is prohibitively challenging, if not entirely impossible. Toward this vision, we follow a constrained approach, where users are afforded semi-natural interaction as long as their design intents fall within a predefined vocabulary.

Inspired by Schon’s cyclical (“seeing-moving-seeing”) description of the design process [1], we formulate the design iteration process as a cyclic sequence of reflection, expression, and transformation (Fig. 1). Specifically, in our case, verbal input is central to all three activities. The user is reflecting on their past and future action by talking about the current design and expressing their intent verbally in terms of a specific design modification that should take place. We then envision an interactive workflow that interprets the verbal expression to effect a design modification on the fly. As a specific embodiment of our envisioned workflow, we implemented an example interactive system, ShapOrator, that takes users’ verbal inputs and transforms them into 3D shape modifications. Note that the purpose of our implementation is not to create a feature-rich system as such, but rather to enable the study of the reflection–expression–modification cycle. Starting from a seed design template, users can change shapes, sizes, and also the number of certain parts present in the template.

By using ShapOrator as an experimental workflow, we conducted a study wherein we invited middle- and high-school students with understanding of basic STEM concepts but no formal engineering or design course work. The students were given the task of designing spaceships under the “Star Wars” theme. The task was open ended, and the students were free to make any number of designs within a given time frame. In addition to the participants verbally expressing their design intents, we take inspiration from prior works [35,43] and use elements of participatory design to extract verbal reflections on their design transformations during the course of the study.

Our study showed that the participants were able to explore a wide variety of spaceship designs using ShapOrator. Furthermore, we explicitly highlighted users’ design iteration process through their reflective descriptions of their spaceship designs. We specifically showed the information and knowledge that users associated with their designs, in the form of functional and topical contexts, and how it motivated them to make further design iterations.

The main objectives for the design of our workflow are to study and enable the design iteration process constituting of reflection, intent expression, and design transformation. To study reflection, we incorporated aspects of participatory design to extract verbal reflection from users. We then developed an experimental interactive system, ShapOrator, to enable users’ to express their design intent and visualize the design transformations.

The key considerations for the design of the interactive system included: (a) the design context given to the users which, in our case, was meant to engage young individuals; (b) the process of converting verbally expressed intents into machine-understandable commands; and finally (c) converting the machine-understandable commands into 3D design transformations, providing users a visual representation of their expressed intents. We expand on each of these design considerations below.

Context plays a key role in the design process, as it motivates the users to make specific contextual design decisions. The users, in our case, were high-school students taking part in an engineering summer camp. To observe interesting interactions and an involved design process, our context needed to be engaging and easy to work with. Taking inspiration from prior works, spaceships were used as our design context, as they had similarly been used for interactive activities involving children [44,45]. We specifically used the “Star Wars” theme to provide further context to the users. Spaceships provided a good balance between the complexity and possible variety of designs that the users could make. To enable further modifications, the users could change specific components of the spaceship, namely, the fuselage (also referred to as the body), wings, engines, and missiles, thus allowing them to explore more unique designs.

Once the users had familiarized themselves with the context, their next step was to verbally express their design intents to the system. Our system took the users’ verbalized input and converted it into machine-understandable algebraic descriptions to make the desired design transformations. Therefore, understanding and semantically decomposing the design intent played an integral role in accurately representing the design transformations. Xue et al. [19] accomplished this by describing a verb-based CAD semantic search, where they searched for verb phrases (verbs and objects) and complements (parameters) in a command. Similarly, Khan and Tunçer [40] categorized words extracted from commands into object, dimensions, location, dimensional aspects, and modifiers. Taking inspiration from these prior works, our first step was to understand what object the users’ intents were referring to. In a multi-component design context like ours, we first searched the intent for specific components that the users wanted to change. The next step was to understand the type of transformation being requested for the identified components. Our system allowed three main types of transformations: cross-sectional shape changes, dimensional changes, and adding/removing components to the object. Once the type of transformation was identified, we checked whether the users had mentioned any specifications corresponding to the transformation. For instance, the users could mention the amount of change they wanted to make or add a number of components. This information was typically in a numerical form and usually appeared before or after the desired transformation. By using this knowledge, we decomposed the user’s design intents into meaningful hierarchical information that could conveniently be translated into machine-understandable commands.

To facilitate the design transformation process using our system, we took inspiration from several prior works in computer graphics that were based on component and assembly-based modeling [46,47]. These systems typically started with a template design, where parameters were used to define each component of the shapes present in the template, and a wide variety of designs could be created by simply changing these parameters. In our case, we closely followed the approach shown in Ref. [6] in terms of shape representation. Specifically, we considered the design of the spaceship to be a collection of swept volumes that were spatially configured to represent a spaceship itself. A fundamental reason for choosing swept volumes as our shape representations was that it gave an intuitive way in decomposing each individual surface direction of the shape into meaningful curve components, namely, the section, path, and guide curves (Fig. 2). While sweeps are popular in parametric modeling, they have a unique ability to allow intuitive design even for quick idea exploration, as has been shown by several works [8,48,49]. Therefore, we used speech as an input to make changes to parameters of swept volumes, allowing users to easily explore a variety of design transformations (Figs. 2 and 3).

Here, we reiterate that the design considerations for our workflow were influenced by our desire to study how verbalization could be used to enable design iterations for young designers. Our goal, therefore, was not to developing a feature-rich software.

Our user interface (UI) was developed using Unity scripted with the c# programming language. We used Azure’s Speech SDK and Custom Speech API for our speech-to-text recognition module. Intent classification for text input was done using Azure’s Language Understanding (LUIS) API. Our interface was deployed on an Asus ROG Zephyrus laptop with an AMD Ryzen 9 processor, 16GB DDR4 RAM, and NVIDIA GTX 3070 Laptop GPU. Built-in microphones were used to record voice.

Our shape modeling technique utilizes swept volumes to render the 3D parts of the spaceship. Sweeps enable us to easily explore a variety of different shapes by making changes to their sectional profiles and guide curves. By using this as the basis for our 3D modeling approach, we further discuss the shape manipulation functions present in ShapOrator.

We choose the superformula equation as it provides a convenient way of semantically mapping the shape descriptions to specific parameters corresponding to that shape. For instance, we can create a square by assigning the values of parameters m, n₁, n₂, and n₃ to 4, 1, 1, and 1, respectively. To create a four-pointed star, we simply reduce the value of n₁ to 0.5 while keeping the other parameters the same.

In addition to the superformula-based transformations, we also added the ability to make the cross-sectional shapes smoother, i.e., make the corners of shapes rounded. We use Laplacian smoothing using p_i = 0.5 (p_i−1 + p_i+1), where p_i is the smoothed vertex for neighboring vertices p_i−1 and p_i−1. The smoothing is performed on the vertices corresponding to the x-y (horizontal) plane where the cross-sectional shape lies. In order to obtain a visibly smoother shape, the smoothing function is iterated four times.

In addition to cross-sectional changes of shapes, we also allow users to change the dimensions of the parts in multiple different ways. The users can change the dimensions of the cross-sectional shapes and also the dimensions of the entire parts, which are defined by the guide curves of the sweeps (Fig. 2). The users can make the cross-sectional shapes wider, narrower, taller, and flatter. This is easily achieved by scaling the x and y coordinates as x = s × r(ϕ) cos(ϕ) and y = t × r(ϕ) sin(ϕ). Changing the values of s results in wider/narrower shapes, while changing the values of t results in taller/flatter shapes. The guide curves with the help of control points help us define the outline of the different 3D parts of the spaceship, i.e., the fuselage (body), wings, engines, and missiles. The control points were initially decided through a trial-and-error approach taking inspiration from the spaceships in the Star Wars movies, specifically the X-Wing design. The guide curves, therefore, enable a convenient way to change the dimensions of the 3D parts by simply changing the positions of the control points. For instance, if the user wished to make the parts radially bigger or smaller, the control points of the parts would be moved away or toward the central axis of the shape, respectively. Increasing or decreasing the length of the parts would subsequently result in the length between the control points to increase or decrease, respectively.

A major component of our modeling interface is the ability to understand and act upon the user’s verbally conveyed design intents. Speech recognition plays a significant role in this component. While most basic speech-to-text services are able to transcribe simple spoken sentences, they usually do not perform well for application specific commands that are not typically used in day-to-day life. For this reason, we used Microsoft Azure’s Cognitive Speech Services SDK as it allowed us to train custom speech models that catered specifically to our application. To train these models, we first collected and manually transcribed audio samples from eight individuals of varying backgrounds. Audio samples included sentences and commands that users’ would typically use while interacting with our application.

Our next step was to understand and act on the users’ design intents. For this, we needed to classify the users’ intents into different actions that our interface would perform. Recent advancements in natural language processing (NLP) [51] have made it easier to train models that parse and make sense of user intent from textual data. For our application, we used Microsoft Azure’s LUIS service, which enabled us to predict the overall meaning of the participants’ commands and gather relevant information from it. By using LUIS’ cloud service, we were able to train custom NLP models tailored to our use case. We were required to define Intents, which represent tasks or actions, and Entities, which extract specific information from the users utterances with the help of specific features (such as synonymous words). For our application, we defined four intents: changing cross-sectional shapes, changing dimensions, adding or removing parts, and undo. In order to improve the prediction of these intents, we provided the model with additional features to train on. The features included a list of different ways the parts could be addressed, a list of different shape descriptions, as well as a list of the different terms for adding or removing parts. The model was then provided with a wide variety of example utterances to train on in order to increase the accuracy. As the LUIS service is built on pretrained models, the quantity of examples mattered less compared to the variety. While our application did not support a completely natural interaction, natural sentences could be used as long as the necessary keywords were present.

Our interface comprises of two main modes: tutorial mode (no data are logged), and the study mode. Once a mode is selected, the users can see the object (spaceship) to modify in the center of the screen (Fig. 4(a)). The users can easily change the orientation of the object by rotating and zooming in/out with the help of the mouse. The parts constituting the object are labeled on the panel at the top of the screen allowing for easy reference. The bottom panel displays the transcribed texts and consists of multiple buttons to the left and right ends of the panel. To interact with the system through speech, users need to press and hold the space bar while speaking. The three buttons, starting from the left, are for resetting the shape to its default template, resetting the orientation to default, and saving the current object. Buttons on the right are to toggle on/off the lasers and flames. Having buttons and mouse-based interactions for nonessential functions allows the users to focus on the design task at hand.

Our user study was conducted during a university-hosted summer camp for middle- and high-school students. The summer camp aimed at providing exposure to these students to various hands-on engineering concepts through interactive activities. Our research group hosted the students for 2 days with the goal of introducing them to the different processes of engineering design. The theme for this session was Spaceships, as seen in science-fictional movies. With this in mind, the activities setup for the first day covered design ideation and prototyping, while the second day covered a 3D modeling tutorial to create spaceships using solidworks. We conducted our study with individual students during the second half of the second day.

The participants of our study were middle- and high-school students taking part in the summer camp. Ten students, in the age group of 13–17 years, voluntarily took part in the study (nine males, one female). They were studying in grades ranging from 8 to 11. The students did not have any formal coursework in engineering or design. However, seven out of the ten students had some basic experience in the past with 3D modeling tools such as maya, blender, solidworks, autodesk inventor, and fusion 360, which they had either used for school or personal projects.

Each user study lasted between 30 and 40 min. The participants were provided with a prestudy questionnaire to elicit their experiences with 3D modeling tools and their opinions on verbal communication during idea generation. Next, we gave them a detailed tour of our UI. The users were then asked to complete the following tasks:

1. Practice: Participants were given an elaborate tutorial of different speech-based interactions that they could use to explore and change the shapes of the spaceship. They were given 7–9 min to practice all the different available commands and to clarify any doubts they had regarding the interface. No data were logged during practice.

2. Study Task: The participants were given an open-ended task of exploring different spaceship designs by building upon the default template. They were asked to explore a variety of designs; however, no limit was set to allow a natural process of shape exploration. The participants were given the context of the Star Wars theme, but were not expected to create any specific designs. They were given 20 min for the task and were asked to save their designs whenever they felt that they had created a unique design of their liking.

3. Reflective Conversation: During the course of the study, participants were asked questions to make them reflect on their design choices and decisions [35] after every three to four major iterations. The questions mainly targeted the reasoning behind the design changes and their effect on the overall design of the spaceship. This helped us understand how participants added information and knowledge as they iterated through their designs.

4. Poststudy Questionnaire: Participants were asked to complete a poststudy questionnaire to evaluate the ease of use of our interface and its creativity support for the shape exploration tasks [52] and also included a study specific survey. They were also asked to provide general feedback regarding their experience.

We observed an overall positive user experience for the design task and the ShapOrator interface. The participants, in general, were able to explore a variety of spaceships by iterating through different designs. On an average, each participant made around 68 iterations to their spaceships (max: 111, min: 41). Any change that the participants made to the cross-sectional shapes, dimensions, adding/removing parts, or undoing their design change accounted for a single iteration. All participants preferred making the most iterations to the dimensions of the parts, followed by changes to sectional shapes. This may have been due to the extent of visual change corresponding to the functions. For instance, adding or removing wings made a significant visual change to the spaceship, while making the fuselage shorter by 20% did not have the same effect. With respect to the changes made to specific parts of the spaceship, the fuselage was iterated through the most and missiles the least by majority of the participants (six and seven participants, respectively). The reason for this could be the relative sizes of the parts and their relative importance in the spaceship.

Apart from the aforementioned quantitative results supporting the design iteration process, we wanted to develop a more fundamental understanding of the process from a qualitative point of view. To accomplish this, we analyzed the users’ reflective descriptions throughout their design process, which were verbally initiated by the study administrators’ questions. We found that the users associated relevant information and contextual knowledge to their spaceship designs that motivated them to make specific changes. We categorized the reflections into an hierarchical structure (Fig. 6) containing two broad categories: function-based contexts and form-based contexts. These categories are explained in detail in the following sections.

Function-based contexts referred to the descriptions where the participants focused on the functional aspects of the spaceship to describe and justify their designs. The functional contexts could be further divided into two broad categories: (a) capability and (b) usability-focused contexts. Under capability contexts, participants focused on the flight capabilities of their spaceships, while under usability contexts, their focus was on how their spaceship would be used (e.g., in battle). In the following sections, we highlight common functional contexts used by the participants in the form of their answers and interactions and also through their geometric decisions such as the common shapes and sizes associated with specific functions. We note that some contexts naturally fell under multiple categories; however, we included it under the category that most affected the design.

A few participants expressed that their design iterations were driven by the flight capabilities of their spaceship. More specifically, the participants reflected on the aerodynamics, control, and propulsion of their spaceship designs. We highlight some of the common contexts corresponding to the following specific functions:

(a) Aerodynamics

Aerodynamics was a common context that the participants focused on to create their spaceships. Three participants explicitly mentioned their desire to design more aerodynamic spaceships and attributed specific shapes and features of their designs to this function. When asked what design they were going for, P3 answered: “A sleeker more aerodynamic look.” Explaining further, they said: “The rounder wings and the triangular [fuselage] and smoother engines, just in my head that’s what contributes to aerodynamics. The sharper edges just don’t look as aerodynamic” (row 1, Fig. 7). Similarly when P9 was asked for the reasons for their circular shaped wings, they answered: “I guess the wind can pass through things that are circular more. I mean, yes, maybe angular would be better, but in some instances, it is better with a circular design.” For their triangular fuselage, they said: “Well, a couple of things, I guess the wind passes through it a lot and it passes through pretty well and it has the smallest surface area on the end of it so it’s the closest thing to a point, so I guess lesser the surface area, the more aerodynamic it is, the faster it can move.” The triangular fuselage was the most commonly used shape among the participants (Fig. 5). While some other participants did not mention the specific reason for the triangular shape, aerodynamics may have played a part in their decision.

(b) Control

We found that the context of control and maneuverability motivated some participants to make dimension and part-based changes. For instance, when P9 was asked for their reasoning for making a smaller spaceship, they said: “I guess it’s smaller and it’s able to move better..” On their second design, P9 said: “Now I’m going for a more lightweight [design] i guess. I made the wings a little bit thinner,” and for their third design, they added two more wings, and when asked for a reason, they said: “I was going a lot for looks I guess but I guess the bigger the body the more wings I think it requires you know. I could have made the wings bigger maybe but I went for that [more wings] instead and more wings equals more engines with it so that’s good” (row 3, Fig. 7). When P4 was asked to compare their spaceship designs, they said (about their final design): “I think this is probably the best ship that would actually work probably because everything’s more stable in this. The wings are well, there’s lots of wings, but they’re not like really thin or really large and they are balanced. Everything’s kind of balanced in this although the fuselage might be a little too big but all in all I really like it.”

(c) Propulsion

Propulsion was another functional context that some participants used as a motivation for their spaceship designs. For instance, P4’s reasoning for creating long engines was: “More fuel, no one will be able to be in it, but it’ll be fuel.” When P8 was asked why they added six extra wings to their spaceship, they said: “I like the shape. It looks like a star. There are more guns and engines so that helps.” The context of propulsion primarily led to the participants either changing the size of the engines or the number of engines. As the number of engines equaled the number of wings, some of the participants control-focused contexts also fell under the propulsion context.

The usability of the spaceship was a motivating factor for quite a few participants, where the common theme across participants’ reflections was the focus on the battle readiness of their spaceship as well as the human-centric approach of their design. We highlight some examples of these contexts.

(a) Battle

The context of battle was a common theme in few of the participants’ answers. The focus was divided between improving the defense of their spaceship and improving its attacking capability. However, there was no single approach taken by all the participants, and it led to a few unique design choices. When P10 was asked about the significance of the flat fuselage and large missiles in their spaceship, they said: “It’s hard to hit. It can just destroy everyone with the giant missiles” (Fig. 5). At a later iteration, P10 decided to make the wings much larger than the fuselage and said: “Essentially, the cockpit cannot be shot because it is behind the wings.” This was an interesting approach since they made a part much larger to act as a shield for another part. In a similar approach of increasing one dimension but decreasing another, P8 mentioned: “ … The long and thin body makes it harder to hit I guess.” We also had participants take metaphorical approaches to make design decisions (Fig. 6). One such instance was when P7 was asked if they were set on their cross-sectional shape preference, they answered: “I’m mostly trying to change the shape right now to where it’ll be overall smaller … I mostly want it smaller just because it’s harder to hit a small [thing]. I’m just sort of thinking about this metaphorically, it’s harder to hit a small fly than a big maybe six inch cockroach, so that’s why I’m going for more of the smaller wings, smaller body because it’s harder to hit” (row 2, Fig. 7). In total, four participants associated their designs with the context of battle or war.

(b) Human-Centered approach

Participants in general did not take the human-centered approach for designing their spaceships and instead gave more priority to the visuals and other functions. However, when they were asked questions about the human-centered approach, they had interesting insights on how that aspect could be incorporated in their designs or how they could circumvent it completely by defining a new function for their spaceships. For instance, when P5 was asked about the practicality of their design, they answered: “I think it would be best if it’s some kind of an unmanned thing with a lot of power, because you can’t really fit a person in there, probably lying down but it’s kind of hard to fit in that” (row 4, Fig. 7). Some participants, however, went with the dimension-based explanation for incorporating the human-centered approach. To carry more people, P1 said: “I would probably have it [fuselage] 30% larger.” P7 specifically designed a bigger fuselage to carry more people: “Yes, it is now carrying like a team of two and I’m not sure if you watched Star Wars but in the fifth movie they’re fighting on that ice planet and they have one person at the back for the gunner and one in front for the pilot and so that’s kind of where I’m going with it now and I could probably even extend it to where it’s a whole entire battalion even though it looks like an attack ship. I’m really just experimenting on how big it can get now.”

While functionality of the spaceship was an important factor in the participants’ design iterations, form-based contexts were more widely used to explore designs. We divide these contexts into (a) preferential and (b) experiential contexts. We give instances of participants using these to describe their design choices.

(a) Preferential

One of the most common motivators for participants was their personal preferences that were primarily driven by the visual aspects of the spaceship. When P1 was asked why they were making the engines small, they answered: “I just want the body to be big and the engines to look short and small. I don’t want them to stand out”(row 7, Fig. 7). P5 on the other hand made bigger engines and their reason for that was: “It’s mostly just visual it looks kind of cool.” P7, who had previously made functional and storyline-based design decisions, wanted to make their final design to be out of the ordinary: “I’m mostly just going for something really weird really out of the ordinary, because this is such a small ship and there’s so many features added to it. Because the missiles I added them to where they’d be a hundred percent bigger than what they originally were, I made the engines way bigger and I made the wings smaller for a ship that shouldn’t be that small and I made the body a lot shorter to where I’m not sure if we can fit one person into it. I’m really going for something unique and very weird”(Row 2, Fig. 5). The open-ended design task allowed users to be more visually creative, something that might have been less obvious in a specific task-driven study.

(b) Experiential

Apart from personal preferences, participants also took motivation from their experiences and the theme of the design problem (e.g., Star Wars and Guardians of the Galaxy). For instance, when P2 was asked why they preferred having a flat fuselage, they said: “I wanted to make one of those cool flat ships, like a pancake” (row 6, Fig. 7), where they related the flatness of the fuselage to the flatness of a pancake. Similarly, when P3 used the starry command on a cross-sectional shape and eventually changed it to a five-pointed star, we asked them if the term starry made them think of a five-pointed star, to which they responded: “Yes, when I think of the word star I think of a five-pointed star, because in general you see them on flags, you see them when you’re in school and you look at them in first grade when you’re looking at planets on the map and there’s planets and all the stars, and stars with astronauts, and they’re five-pointed stars. It’s just the general picture of stars in my head.” In regards to the theme of the design problem, P4’s explanation for making all the cross-sectional shapes pentagonal stars was quite interesting: “Yes, everything’s the star now. It’s called the ‘Star Wars’ for a reason” (row 5, Fig. 7). Star Wars was not the only fictional motivation for the participants; when P10 was asked what the reason for having long wings was, they said: “It looks like the Guardians of the Galaxy ship with the giant wings.” From these user reflections, we noted that the theme of the design task as well as their prior experiences played an influential role in the way users made design decisions.

One of our primary goals was to observe the evolution of design intent through iteration. In order to observe this visually, we created a plot highlighting the main context influencing each of the user’s design changes (Fig. 8). The first row for each user shows whether their iterations were motivated by function or form-based context (each column corresponds to a single iteration). The second row highlights the subcategories corresponding to function-based contexts (i.e., capability and usability), and the third row shows the specific function and form-based contexts, as seen in the leaf nodes of the hierarchical chart of contexts in Fig. 6. We classified each design iteration into a specific category based on their closest following reflection. For instance, if a user made a few design changes and reflected on those changes as a means to improve the battle readiness of their spaceship, then we classified those changes into function > usability > battle categories. Here, we note that while the users’ reflections might fall under multiple categories, we chose the one that influenced the users the most.

With the help of this visualization, we can observe some common trends across participants. For instance, quite a few participants started their design iterations through open-ended explorations (P1, P3, P4, P5, P6, P7’s second design), where they were not motivated by a specific purpose, but were trying to experiment with different shapes and sizes. For example, when we asked P1, at the beginning of the study, for their shape preference, they said: “I was testing it out. I don’t know what shape I want to go with.” Similarly, P5’s response to our very first question about their design was: “Right now I’m just mostly seeing what different things are there; I’m just exploring.” This insight of open-ended exploration can guide future speech-based design interfaces to focus on supporting more ambiguous and purpose-independent exploration in the initial stages of the design task. Another observation is the use of function-based contexts by all but two participants. The interesting thing to note here is that the majority of the function-based contexts are followed by preferential changes. This may suggest that once the participants were done making design changes to affect the functionality of their spaceship, they changed their focus to the visual aspects of their spaceship. We also note that out of the eight participants who made function-based changes, only three made consecutive changes influenced by different functions (P4, P7, and P9). For form-based changes, however, participants freely moved between preferential, experiential, and open-ended explorations. This may suggest that participants prefer focusing on individual functionalities of their design at a given time. Our workflow, therefore, revealed some interesting characteristics and insights into the iterative design process using a speech-based interface.

We received an overall positive feedback, where most participants responded warmly to the design activity and our modeling interface. Their responses to the creativity support index were also fairly positive (Fig. 9). All users agreed that the activity was engaging and made them feel creative, and their outcomes were also worth the effort. Two users found the exploration of different designs to be somewhat difficult. One of them found the exploration of shapes difficult as they used the plural form of shapes (e.g., squares instead of square) in their commands, and our interface was unable to act on those commands. The other user found that the interface was not able to recognize their speech very well. However, both the users found the experience fun and interesting and were able to create the shapes that they wanted. Users also commented on their experience using the system and completing the design tasks. P2 mentioned: “I felt the amount you could change with voice commands alone to be fascinating. I liked making the parts starry at first then turned away from that path.” P7 commented on the ease of using their speech to make changes and related it to their class activity: “I thought it was really cool, I haven’t done something like this before. I have done coding stuff in class and I have always wanted to use my mouth because its always more convenient. Because you have the thoughts in your mind and you can say it but can’t note it down sometimes.”

One of the limitations of our work was the disproportionate ratio of male to female participants taking part in the study. While we would have desired for an equal participant ratio, our participant pool was limited to the students who were attending the engineering summer camp and volunteered to be a part of the study. Unfortunately, the number of female students in this group was quite low, resulting in a lower number of female volunteers. Some other limitations of our work included our limited vocabulary for interacting with the system. This was constrained to maintain uniformity across participants. However, this could be solved in future works by using large language models [53] or databases [54] to find semantically similar words corresponding to the functions of the system and the parts of an object. We also showcased the speech-based design iterations on a single model of 3D spaceship as it allowed us to study the process while keeping the young designers engaged. In the future, we could look at datasets such as ShapeNet [55] to create a 3D model repository amenable to speech-based design modifications. Finally, we constrained the number of geometric operations that the users could perform, to prevent the students from having to remember a vast set of commands and functions. However, for more serious engineering applications, we could easily add more operations and features to our system.

Design transformation in our system is initiated through a “form-based” intent expression, i.e., we require the users to explicitly mention their desired form (shape) in their intent. The verbalization, therefore, is about the form and not the functionality of the object. However, as we noted in our results, the reasoning behind a lot of the shapes and forms that the users explored was tied to their functionality, which the users explained in the verbal reflection phase of their design iterations. While verbalization was about the form, it made the users think in terms of function. In interactions such as sketching, this aspect of functionality is internalized and is implicit. Verbalization, therefore, plays a critical role in expressing design intent. One way of expanding the users exploration options is by enabling verbalization of analogical reasoning and translating that to a form. We can take inspiration from design by analogy, a methodology that has been widely studied in design theory [56], where users generate designs based on analogies drawn from desired functions or structures and so on. Computational support of this can be very beneficial in helping users in generating new designs and concepts. From our observations through the verbal reflection process, we observed the participants derive analogies and develop analogical reasoning to explain and justify their design decisions. A wide range of opportunities can be made available by integrating analogical reasoning from fields such as bio-inspired design [57] into verbalization.

An important aspect of verbal inputs for design that was revealed in our study was that users invariably preferred exploring the design space by describing the function, behavior, and attributes of a design rather than directly specifying the form. This is quite natural as we typically think in terms of what we want from a design (reflection) and internally embody those preferences into the appropriate form. P3 referred to this in terms of “naturalness” of the speech interpretation, stating that: “I think there should be a sense of naturalness to it, but there should also be a sense of using percentages and using specific terms for the body parts, like using casual terms to address the [parts], so not like calling them natural things but just talking about them [parts], like make them bigger. It’s nice you incorporate bigger, smaller, narrower, terms you use in a regular conversation, to the thing [system], because that makes it significantly easier than saying ‘make it 20 percent larger in the x direction.’” Even though our system was based on semi-natural description of shapes, it offered us a mean to gain meaningful insight into how we may develop future verbal workflows that might be able to predict the form based on the description of function, behavior, and attributes.

Our main goal in this work was to operationalize and study verbalization as a medium to enable 3D design iterations, specifically for younger audiences. The motivation behind this work stemmed from the fact that externalization of verbal inputs to explore ideas, while important, has been little studied and understood. As such, we set out to accomplish two main tasks: first, develop a system that could enable simple geometric changes to 3D shapes using verbal inputs and second, to observe and form a fundamental understanding of how verbalization enabled young designers to make design iterations. As a result, we first developed ShapOrator, a workflow that allowed users to explore and transform 3D shapes by giving verbal inputs. Next, we conducted a user study with ten middle- and high-school students who were given an open-ended task of designing spaceships using our ShapOrator workflow. Through our user studies, we were able to show that verbalization, even when externalized through “forms,” allowed the participants to develop functional as well as form-based understanding of their designs. Furthermore, we were able to explicitly highlight the design iteration process that the users followed through their reflective descriptions of their spaceship designs. We specifically showed the information and knowledge that the users associated with their workflows and how it motivated them to make further changes to their designs. These observations make a strong case for utilizing verbalization in creativity support systems that typically use sketching, multitouch, and gestural inputs.

This work was partly supported by the Morris E. Foster Faculty Fellowship from the Texas A&M University. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the fellowship.

There are no conflicts of interest.

The datasets generated and supporting the findings of this article are obtainable from the corresponding author upon reasonable request.