Elicitron: A Large Language Model Agent-Based Simulation Framework for Design Requirements Elicitation

Requirement elicitation (RE) sits at the core of successful product design, yet it remains a complex and resource-intensive endeavor. Traditional RE methods, like interviews, focus groups, and prototyping, are invaluable but have inherent limitations. These methods are often time consuming, may not fully capture the diversity of user perspectives, and may miss underlying needs that are difficult for users to articulate [1,2]. The consequences of inadequate requirements elicitation can be significant, ranging from design misalignment to compromised product adoption.

Recent advancements in large language models (LLMs) present new possibilities for automating and augmenting requirement elicitation. LLMs, having learned the patterns and complexities of human language from vast textual corpora, seemingly possess a remarkable capacity for natural language understanding [3]. This potential can be leveraged to construct a simulated environment where LLM agents role-play a variety of potential users. These agents can simulate distinct viewpoints, engage in product experience scenarios, and participate in user interviews aimed at identifying user needs. By doing so, these virtual users could potentially uncover a wide spectrum of user needs, preferences, and pain points that might otherwise remain hidden or unexplored.

There are reasons to believe LLMs could offer unique advantages for identifying diverse and latent needs. The core capabilities of LLMs lie in their flexibility and ability to perform different tasks related to natural language. By incorporating contextual elements, LLMs can grasp nuances and make inferences through in-context learning [3], leading to more accurate and insightful analyses of user needs. Because LLMs have been trained on a vast amount of data, they likely have been exposed to a diverse set of user needs for a particular product. During training, LLMs may have also picked up behavioral patterns and subtle language cues that hint at underlying—sometimes even subconscious—needs experienced by users. Moreover, they may be able to perform analogical reasoning to relate experiences and needs identified across different products and uncover novel needs. However, a potential contradiction arises when considering the inherent nature of LLMs: They are trained to predict the most likely next word or sequence of words based on patterns in their training data [4]. On the surface, this focus on selecting the most likely outcome might seem at odds with the goal of uncovering diverse or latent needs. While LLMs are indeed trained to predict likely word sequences based on training data patterns [4], they generate responses by creatively combining and extrapolating from their training data, rather than simply retrieving and reproducing exact matches. This creative process, often labeled as “hallucination” when it produces factually incorrect information, is in fact the LLM’s core strength for generating novel and diverse outputs. The key lies in how we direct and constrain this generative capacity for a particular purpose. Through careful prompt engineering, contextualization, and the use of techniques like adjusting sampling parameters (e.g., temperature or top-P), we can guide the LLM to explore beyond obvious associations and produce outputs that reveal diverse and latent needs.

This research presents a new LLM-based framework, called Elicitron, for automating and augmenting the RE process. In Elicitron, LLM agents are constrained to produce structured outputs that are relevant to and useful for the requirement elicitation process. Elicitron also employs techniques to create diverse user-representing agents and simulate product experiences through the action, observation, challenge steps, inspired by chain-of-thought reasoning [5]. Finally, by creating agents with specific roles, Elicitron can discover interesting user needs that may otherwise be difficult to obtain with human interviews.

Elicitron’s ability to create diverse user agents can be leveraged to identify a diverse set of user needs. Because the process of creating and interviewing these agents is automated, Elicitron is highly scalable, unlike the traditional RE methods. We conducted an experiment to evaluate the capability of Elicitron to generate a diverse set of user needs and identified a context-aware generation method that maximizes need diversity.

In addition, we show how Elicitron can be applied to identify latent needs—unarticulated and unexpected user needs that strongly influence product desirability [6]—which are significantly difficult to obtain with the traditional RE methods. This can be achieved by either automatically or manually creating user agents with empathic lead user (ELU) roles [7]. We conducted a second experiment to demonstrate that Elicitron can generate a higher number of latent needs than human interviews. Finally, a third experiment was conducted to show that, given criteria and chain-of-thought reasoning, LLMs are capable of identifying and classifying latent needs from interview data.

LLMs are machine learning models that appear to exhibit the ability to understand, reason with, and generate natural language [3]. LLMs have been shown to engage in fluent conversations, translate languages, and write various types of creative content in different styles. The development of LLMs marks a significant step toward human-like agent capabilities within artificial intelligence. Beyond traditional applications, the use of LLMs is expanding into areas like software development, content creation, customer service, and mechanical design generation and understanding [8–13].

LLMs are trained using self-supervised learning. They are trained on vast amounts of text data to predict the next word or sequence of words in a sentence. This training process, along with architectural advancements like the transformer model [4], allows LLMs to grasp patterns, nuances, and context within human language, including learning implied meanings [3].

The knowledge acquired during training provides the foundation for LLMs to effectively engage in role-playing scenarios [14–16]. They can simulate various roles by using specific language patterns, word choices, and sentence structures learned from their training data. For example, if an LLM encounters a training dataset rich with technical communication, it can adapt its vocabulary and sentence structure to convincingly adopt the role of an engineer during role-playing sessions.

LLMs’ capability to role-play makes them useful tools in requirements elicitation. They can simulate a diverse set of users, including empathic lead users, providing valuable insights into their product experience and needs.

During the requirement elicitation phase of engineering design, empathy plays an important role in helping engineers better understand user needs [17,18] and consequently the design problem [19]. By interviewing, observing, and empathizing with users, designers can derive structured design requirements from unstructured feedback from product users, in a process called empathic design [20,21].

There are two types of design requirements or user needs: direct needs, which tend to be obvious to the customer and lead to incremental changes in a product, and latent needs, which may be nonobvious and difficult to uncover [22].

User interviews or observations may not reveal latent needs that consumers deem important in a final product [23]. Identifying latent needs early in the design process has been found to speed up the development process, and their discovery benefits the design engineer by providing insights into extreme use cases that might push the product to its limits [24–26].

Over the years, the design research community has experimented with various empathic design methods to improve latent need discovery. Hannukainen and Holtta-Otto used photo diary and contextual inquiry with disabled people to identify latent needs [20] for a mobile phone. Lin et al. elicited latent needs from ordinary users by simulating extraordinary situations, e.g. using a blindfold to simulate limited sight or oven mittens to simulate limited dexterity [7] during setting up a tent. Issa et al. prompted designers interpreting user interviews to “write a statement as someone with [specific experience]”, in an attempt to bias the designer to be more empathetic with a lead user [27].

More recently, LLMs have been evaluated on design requirement understanding and knowledge extraction tasks, and used for design requirement analysis to guide novice designers towards a conceptual design solution [12,28,29]. Zhu et al. theorized how artificial intelligence and LLMs might be leveraged to support data-driven user studies for empathic design, and later expanded their work to investigate the use of LLMs for inferring product users” goals and “fundamental psychological needs’” [30,31]. Outside of the product design domain, Barandoni et al. evaluated the performance of LLMs in extracting travel customer needs from travelers’ requests posted to a travel website [32].

While the design research community has identified empathy as an important component of design requirement elicitation [7,33–36], interviewing empathic lead users or observing them in user studies remains a time consuming and costly activity. The use of LLMs to simulate this process by creating and interviewing a diverse set of empathic lead user agents could address the gaps.

In the design methodology literature, design diversity is typically considered an extension of novelty from a set of designs. A novel design is often considered to be unique by the person who created it (psychological novelty) or more generally unique to the field (historical novelty) [37]. Evaluating the novelty of a design is a subjective task, and it is typical to leverage consensual assessment technique or similar methods that involve asking domain experts to rate designs on criteria such as novelty [38–41]. However, recent work in deep generative models, which leverages machine learning methods trained on large datasets of prior designs to create novel design solutions, has led to the development and adoption of computational novelty and diversity metrics. These include the convex hull volume and the mean distance to centroid metrics, which can be used to measure the average diversity of a whole set of designs [42–49]. In addition to those measures, we are also interested in measuring the diversity of possible clusters of design ideas, and not just the outliers; thus, the silhouette score typically used in cluster analysis [50] may also be applicable.

The convex hull volume is defined as the hypervolume of the smallest convex set that includes all of the samples. It has been used to measure the diversity in different disciplines, but is sensitive to outliers [42,43,51]. A larger convex hull volume indicates that, in the embedding space, the samples cover more space and are more diverse.

The mean distance to centroid of the embeddings is computed by averaging the distance of each sample to the centroid of the whole set [42,43,52,53]. A larger value indicates that the samples in the set are further from the centroid and thus are assumed to represent more diverse concepts. This metric is more suitable for more uniform distributions and is also sensitive to outliers.

In the context of clustering algorithms, the silhouette score is a metric used to calculate the performance of a clustering technique. Its value ranges from –1 to 1, where a high value indicates that the object is well matched to its own cluster and poorly matched to neighboring clusters. The silhouette score has been used to measure the diversity of recommendation systems, but has not been adopted in the field of design research [54,55].

Elicitron is designed to closely simulate real-world requirement elicitation processes. The architecture comprises four distinct components mirroring the phases involved in gathering requirements (Fig. 1). Each component is discussed in detail below. To maintain structural consistency and prevent workflow errors, we employ a Pydantic model to constrain LLM outputs, followed by a schema validation step.

A significant challenge in requirement elicitation (either with LLMs or traditional methods) is capturing a diverse range of user viewpoints. For this reason, our framework’s initial step generates a diverse set of agents to simulate users within the elicitation process. This mirrors real-world practices where a wide variety of users are deliberately selected for RE studies.

The LLM is instructed to generate three elements for each user agent:

• Name: A label representing the user agent.

• Description: A description of the user characteristics.

• Reasoning Chain: A rationale for creating this agent.

We have employed the following agent generation methods:

Here, the LLM receives $N$ independent prompts to generate $N$ user agents simultaneously. This method is advantageous for rapid creation of a large number of agents in parallel. However, due to the LLM’s lack of awareness of other agents being generated, diversity may be limited as the model could produce similar agents.

To mitigate this, we implement a filtering stage. We use a K-means clustering algorithm to group agents based on the similarity of their embeddings and then select only representative agents from each cluster to result in a diverse set of agents.

Given a set of generated agents $A=a1,a2,…,aN$⁠, where each agent $ai$ is represented by an embedding vector $vi$ in a high-dimensional space, the goal is to select a diverse subset of agents.

1. Assign Embeddings: First, assign an embedding vector $vi$ to each agent $ai$ description in the set $A$⁠. We used text-embedding-ada-002 by OpenAI for this purpose.

2. Perform K-Means Clustering: Apply a clustering algorithm, K-Means$(V,k)$⁠. $V$ is the matrix of all embedding vectors $vi$ and $k$ is a chosen number of clusters (⁠$k≤N$⁠). Given $k$⁠, the K-means algorithm assigns each agent to the cluster with the nearest mean embedding.

3. Select Diverse Agents: From each of the $k$ clusters, select one representative agent. The selected agents are deemed diverse, as they come from different clusters in the embedding space. The resulting set of agents is denoted by $A′$⁠.

This method may involve overgenerating agents, followed by filtering down to $N$ agents. While we used K-means for filtering, other clustering techniques are also applicable.

In this technique, the LLM receives a single prompt to generate $N$ agents. Here, the details of generated agents persist in the LLM’s context to promote greater diversity compared to parallel generation. A drawback is that it runs more slowly than parallel generation. Moreover, there is a practical limit on the number of agents generated determined by the LLM’s maximum token output length. At the time of this writing, most LLMs cap output at around 4096 tokens, which experimentally suggests a maximum of roughly 20 agents per call for each generation.

After generating a diverse agent pool, user agents are prompted to imagine their interaction with the potential product. This phase is essential for identifying specific usage scenarios that could lead to detailed and latent needs to be identified during the subsequent interview process.

1. Simulated Interaction: Agents receive an open-ended prompt to describe steps they would take to interact with the product. This might involve setup, specific feature usage, or troubleshooting. Agents are allowed to explore freely, simulating the varied ways real users would interact with the product.

2. Structured Response Generation: For each interaction step, agents provide responses organized into three elements:

   • Action: The description of the interaction step taken (e.g., setup, feature activation).

   • Observation: The agent’s reactions and perceptions of the step. This includes both favorable impressions and points of friction.

   • Challenge: Explicit articulation of obstacles or difficulties encountered. This is done to uncover pain points in the user experience.

These structured responses serve as contextual foundations for the subsequent interview phase, effectively contextualizing the agent’s experience. This approach mirrors the effectiveness of chain-of-thought prompting in generating more nuanced and in-depth responses.

An illustrative example of the product experience generation output can be found in Sec. 5.4. While it is not feasible to verify the factual accuracy of the output a priori, this methodology provides a structured framework that enhances the interpretability and reliability of the model’s responses. Empirical studies, such as the work by Holbrook [56], have demonstrated the efficacy of scenario-based methodologies in improving the quality of elicited information.

The agent interview step mirrors real-world user interviews. It prompts each agent to reflect back on their product experience and asks follow-up questions aimed at uncovering user needs and nuanced insights identified from their product experience. The process works as follows:

1. Question Pool Creation: A set of interview questions is prepared (human-developed or automatically generated by an LLM). These questions could be tailored to cover multiple aspects of the product, while also asking for innovative insights or improvement ideas.

2. Contextualized Questioning: Questions are asked to each agent, integrating their prior Q&A responses and simulated product experiences into the LLM’s context. This contextualization aims to alleviate the LLM’s tendency to provide generic responses and facilitates the answers to be based on the individual agent’s unique experience.

In this workflow stage, we leverage previously collected interview responses to isolate needs automatically. An LLM processes agent interviews, extracting expressed needs. It then provides step-by-step reasoning for each identified need, drawing from established latent need criteria and examples, provided by the human experts. Finally, the LLM compiles all findings into a detailed report, offering insights on both expressed and latent needs uncovered during the analysis.

After the responses are collected from all user agents, the designer can review them to identify user needs that could be utilized for the subsequent design process.

In this work, we conducted three experiments to examine the value of Elicitron, all using GPT-4-Turbo from OpenAI as the LLM [57].

To examine the value of Elicitron in terms of identifying diverse user needs, we evaluated the agent generation methods proposed in Sec. 3. We generated 20 user agents each using three conditions, such as serial, parallel, and parallel with the K-means filtering, and compared the diversity of generated agents and their responses using computational metrics.

As discussed in Sec. 2.3, there is no de facto method to computationally evaluate the diversity of interview participants and their corresponding responses. Thus, we leveraged three different methods used in the prior work that rely on embedding the generated responses in a latent space to measure the diversity of design solutions: the convex hull volume [42,51], the mean distance to centroid [42,53], and the silhouette score [50].

For all three metrics, we first generated embeddings for each of the role descriptions and responses to 12 interview questions using text-embedding-ada-002. Note the interview questions asked are presented in Section 5.1. This resulted in 13 sets of 20 embeddings (for the user role and for each of the 12 questions). After using each of our three methods to generate the data (“serial,” “parallel,” and “parallel with filtering”), we computed the convex hull volume and the mean distance to centroid from the individual sets of 20 embeddings and normalized the results from 0 to 1.

The silhouette score is defined as follows. If $a$ is the mean intracluster distance (the average distance between each point within a cluster) and $b$ is the mean nearest-cluster distance (the average distance from a point to the nearest cluster of which it is not a part), then the silhouette score $s$ for a single sample is given by the formula $s=b−amax(a,b)$⁠. The mean silhouette score for a set of samples is the mean of the individual silhouette scores for each sample. An appropriate number of $k$ clusters can then be selected by choosing the $k$ cluster with the highest mean silhouette score, indicating that each cluster is very compact and distinct from other clusters. We used this metric both to choose a number of clusters for K-means, as well as a standalone metric to quantify the diversity of the samples for each of our methods.

To further understand the differences along the diversity, we evaluated the content of the agent role descriptions and interview responses as follows:

1. Cluster the embeddings of all 60 user agents generated from the three conditions using K-means. $k$ is chosen based on the silhouette score to maximize distinct clusters.

2. Summarize the $k$ clusters of agents with the LLM with the following prompt, “Here are $k$ groups of users, give a theme for each group. Group 1: $⋯$” For the content of each group, the role descriptions of the agents in each cluster are used.

3. Examine the coverage of clusters for each condition using scatter plots after reducing the dimensions with $t$-distributed stochastic neighbor embedding (t-SNE) [58].

Table 1 shows that the convex hull volumes were higher for the serial method than the parallel and parallel with filter methods, indicating a significant increase in diversity. Also, on average, the filtering method improved the diversity of the parallel generation method.

Table 2 shows the mean distance to centroid values. The results show that the serial method led to more diversity than the parallel and parallel with filtering methods, but by a smaller margin than the convex hull metric. Again, the “filtering” method improved the diversity of the “parallel” generation by a small amount.

We expect that a high silhouette score across various $k$’s would indicate that the points are easier to cluster, or closer to each other, while a low silhouette score would indicate that the points are further apart, less clustered, and thus would represent a more diverse set. Figure 2 shows the computed silhouette scores for the three-generation methods. We can infer once again that the serial method produced the most diverse agents, compared to the parallel and parallel with filtering methods. There was no clear distinction between the “parallel” methods.

We qualitatively examined the categories of users created for each condition. We aggregated all 60 users’ embeddings created from the three-generation methods. We then chose $k=4$ based on the silhouette score, i.e., when $k=4,$ it gives the most distinct clusters. By using K-means with four clusters, we found the following groups of users, as shown in Fig. 3:

1. Service and Conservation: Roles including military, humanitarian work, and field research.

2. Outdoor Recreation and Camping: Roles with activities such as minimalist camping and stargazing.

3. Adventure and Exploration: Roles with activities such as hiking, backpacking, and mountain climbing.

4. Family Camping and Outdoor Activities: Roles emphasizing bonding through camping and nature activities.

It should be noted that neither the parallel nor the parallel with filtering method created any user roles belonging to the service and conservation groups.

From both computational and qualitative evaluation, the serial generation method led to the most diversity in the user roles and the responses to interview questions. The parallel generation method with K-means filtering helps improve the diversity than using parallel generation only, albeit not significantly. The serial generation method benefits from maintaining previous agent generations as additional context to the LLM, allowing the LLM to avoid generating repetitions.

In the second experiment, Elicitron was employed to automatically generate latent needs, which are difficult to identify using traditional interviews with human users.

To evaluate our method, the tent design example from Ref. [7] along with their reported results was used. In particular, we set our base condition using the ELU interview technique. This involves simulating extraordinary lead user conditions with regular users and interviewing them as the baseline condition. We then compare the number of latent needs identified with the ELU interview technique versus using three different conditions of our requirements elicitation method.

We set up three conditions to test the effectiveness of Elicitron. Each condition generated 20 user agents, the same number of people interviewed in Ref. [7].

• Condition 1: Automatic creation of user agents with the serial method

• Condition 2: Automatic creation of user agents with the serial method and addition of a steering prompt

• Condition 3: Manual creation of ELU agents

The serial method is used because it has been shown in Experiment 1 to create more diverse user agents. For condition 2, we provided the following additional prompt to encourage the creator agent to generate ELU agents. The text inside the double quotes is verbatim from Ref. [7].

For condition 3, we manually created ELU agents based on the deviations from the experiences of a typical customer in a typical application and usage environment listed in Ref. [7]. The list of all user agents generated for each condition is shown in Table 3.

For all three conditions, we prompted the agents to engage in simulated product experience scenarios. We then asked LLM agents the following interview questions in sequence. These are the same interview questions used in Ref. [7] with human subjects, but with some modification in wording to encourage the agents to provide specific needs and insights related to the question.

• Free style: “If you were to purchase an ideal tent, what main characteristics would you look for?”

• Categorical: “Focusing specifically on the [category], aspect of tent, can you tell me your needs and any innovative insights to address those needs?”

  Categories: size, shape, weight, material, safety, durability, aesthetics, ergonomics, cost, setup, transport.

The responses given by the user agents were analyzed to identify the number of latent needs suggested by each agent. Again, we followed the criteria used by Ref. [7] to label whether a particular phrase was a latent need:

Two raters performed the labeling task. Because determining a latent need is highly subjective and context dependent, the labeling was performed as follows. We began by randomly selecting 10% of the dataset for independent labeling by both raters. Following a calibration discussion to resolve discrepancies, we randomly selected another 20% of the data to calculate the interrater agreement score. Finally, the remaining 70% of the dataset was divided equally between the raters for independent labeling.

If both raters agreed on a particular phrase in the text as a latent need, it was counted as a true positive, tp. If the first rater identified it as a latent need but not the second rater, it was counted as a false positive, fp. If the second rater identified it as a latent need but not the first rater, it was counted as a false negative, fn. This choice of which rater’s classification to consider as a positive is arbitrary because only the sum of fp and fn contribute to the denominator term in (1). We found the F-score of 0.83 (⁠$tp=109$⁠, $fp=21$⁠, $fn=24$⁠), which indicated reliable agreement between the raters.

The results (Fig. 4) show that the number of latent needs identified was higher for all three Elicitron conditions compared to the baseline, demonstrating the potential of our LLM-based requirements elicitation framework in identifying latent needs. Because the prior work [7] reported the mean number of latent needs but not any measure of variance, we could not conduct any statistical test to show statistical significance.

One-way analysis of variance was conducted to examine the effect of the agent creation conditions on the number of latent needs identified, but no significant difference was found, $F(2,57)$⁠, $=2.3744p=0.1022$⁠. The fact that the difference in the average number of latent needs between conditions 1 or 2 (automatic creation) versus condition 3 (manual creation) was not statistically significant indicates that the automatic creation methods can be just as effective as the manual creation method that requires domain knowledge. In addition, the lack of significant difference between conditions 1 and 2 indicates that a steering prompt may not be necessary. In fact, many of the user agents created in conditions 1 and 2 are similar to the ELU agents manually created in condition 3, as shown in Table 3. Examples include those who are outside of the typical ages (“adventure-seeking teen,” “retired nature enthusiast”), those who are physically challenged (“person with physical disability”), and those facing extraordinary conditions (“expedition leader,” “high-altitude climber”), etc.

While the latent needs identified with condition 1 did not significantly vary against the two other conditions, an interesting observation can be made regarding the individual user agents. Table 3 shows that many of the agents created do not necessarily represent ELU agents. For example, while user agents such as “adventure racer,” “eco-tourism entrepreneur,” or “high school teacher” could be considered unique user types, they did not entail extraordinary usage conditions and therefore led to the identification of relatively few latent needs (6, 3, and 4, respectively).

Example outputs produced by LLM agents in the experiment are shown here. First, the product experience generated for a user agent titled “Elderly With arthritis”:

Here are example interview responses given by the same agent:

Q: If you were to purchase an ideal tent, what main characteristics would you look for?

Q: Focusing specifically on the setup aspect of tent, can you tell me your needs and any innovative insights to address those needs?

Finally, here are some examples of interesting latent needs identified from the experiment:

Analyzing interviews to detect latent needs is a challenging and time-consuming task that requires a deep understanding of the product and the customer’s requirements. It can consume valuable resources and may not always yield consistent results across different analysts.

In this experiment, we evaluate the performance of LLMs in automating the analysis of interview texts and detecting latent needs. To facilitate this, we create a dataset consisting of 20 latent needs and 20 nonlatent needs, based on the evaluations of human experts in Experiment 2. This dataset will be used to assess the LLM’s ability to accurately identify latent needs.

We conduct the evaluation using three different approaches:

1. Zero-Shot detection: The LLM is tasked with labeling latent needs without any additional context, simply by answering the question “Is this a latent need?” with a binary response (True or False). There is no extra information or criteria given. The LLM relies solely on its existing knowledge to answer “true” or “false.”

2. Detection With Latent Needs Criteria: In this approach, the LLM is provided with the criteria to evaluate latent needs. Given these criteria, the LLM is asked to answer the question “Is this a latent need?” with a binary response (True or False). The criteria are as follows (adopted from Ref. [7]), which are the same criteria used by the human evaluators in experiment 2:

   “Label the reported customer need as a latent need (latent = True) if either of the following conditions is met:

   • The need represents a significant change to the product design and does not fall into any of the following categories: size, shape, weight, material, safety, durability, aesthetics, ergonomics, cost, setup, or transport.

   • The need reflects an exceptionally innovative and clearly expressed insight regarding the product and/or how it is used.”

3. Detection with Latent Needs Criteria and Chain-of-Thought: In this final approach, the LLM is provided with the same criteria for latent needs as in the previous evaluation. However, the LLM is also instructed to use chain-of-thought analysis and think step-by-step to detect the latent needs. Given the criteria and the output of the chain-of-thought, the LLM answers the question “Is this a latent need?” with a binary response (True or False).

Figure 5 presents the confusion matrices for the three evaluations, while Table 4 shows the corresponding performance metrics. In the zero-shot detection scenario, the LLM achieves a precision of 0.7273, recall of 0.8000, and an F1-score of 0.7619. These metrics indicate that the LLM can identify latent needs with reasonable accuracy purely based on their internal representations, even without any additional context. However, there is room for improvement, as the model struggles to distinguish between latent and nonlatent needs in several cases.

When provided with the latent needs criteria, the LLM’s performance improves significantly. The precision reaches 10,000, indicating that all the needs identified as latent by the LLM are indeed latent needs. The recall increases to 0.8500, suggesting that the LLM can identify a higher proportion of the actual latent needs in the dataset.

The most impressive results are observed when the LLM is instructed to use chain-of-thought analysis in combination with the latent needs criteria. In this case, the precision, recall, and F1-score all reach 0.9500. This indicates that the LLM can accurately identify latent needs while minimizing both false positives and false negatives.

The observed improvements in the LLM’s performance across the three scenarios are logical and expected. Each successive approach provides the model with increasingly relevant context and guidance, enabling it to better address the task at hand.

First, by providing the latent needs criteria, the LLM is able to focus its attention on the specific aspects that determine whether a need is latent. This targeted context helps the model zero in on the most pertinent information for making its determination.

Building upon that, the chain-of-thought analysis takes things a step further by guiding the LLM through a structured reasoning process. By breaking down the analysis into a series of smaller, interconnected steps, the model is able to systematically consider each criterion and build a logical argument for its ultimate conclusion. This approach helps ensure the LLM’s output is well reasoned and grounded in the provided criteria.

The effectiveness of this approach is illustrated in Table 5, which presents two examples of the LLM’s chain-of-thought reasoning for a latent and a nonlatent need. In each case, the model’s step-by-step analysis clearly demonstrates how it arrived at its determination by carefully considering and applying each of the latent needs criteria in turn.

Beyond just improving performance, the use of chain-of-thought reasoning also enhances the interpretability of the model. Chain-of-thought opens a window into the LLM’s decision-making process and makes it possible to understand the apparent logic behind each of the model’s conclusions. This transparency is valuable in building trust and confidence in the model’s outputs, as it allows users to verify that the LLM’s decisions are based on sound logic and adhere to the specified criteria.

Cost is an important consideration in latent need identification studies, and the use of LLMs can be a cost-effective alternative. For example, GPT-4-Turbo, which was used in this study, costs 10 USD per 1 million input tokens and 30 USD per 1 million output tokens. This resulted in minimal costs for experiments 1 and 3, amounting to just a few cents, while experiment 2, which involved generating approximately 80,000 tokens, incurred a total cost of around 2.4 USD. These costs are significantly lower than those associated with traditional user studies and surveys, which often require substantial investments in participant recruitment, compensation, and data analysis. The cost-effectiveness of using LLMs for latent needs identification has important implications for the scalability and accessibility of this research, enabling researchers and organizations with limited budgets to conduct comprehensive studies and obtain valuable insights at a fraction of the cost of conventional methods.

The limitations of utilizing LLMs in requirements elicitation deserve detailed examination. One significant limitation is the inherent bias present in LLMs. These models are trained on vast datasets that are predominantly sourced from the Internet, which tends to reflect the biases and inequalities of the societies that produced the data. As a result, LLMs can inadvertently perpetuate stereotypes and may disproportionately represent certain cultural and socioeconomic perspectives while underrepresenting others. For instance, Western-centric viewpoints are often overrepresented in LLMs that are trained on mostly English text, leading to a skewed understanding of global user needs and perspectives.

Another limitation lies in the uneven representation of certain cultures and personalities. LLMs are more likely to generate content that mirrors the dominant socioeconomic narratives found in their training data. This means that the needs and preferences of less-documented communities may not be adequately captured or prioritized. For example, the design requirements of users from low-income regions may be overlooked, resulting in products that do not fully address the diverse needs of the global population.

Moreover, the ability of LLMs to identify latent needs is constrained by their training data. While LLMs can simulate a broad range of user interactions and generate diverse user personas, they are ultimately limited by the scope and diversity of the data they have been trained on. If the training data lack representation of certain user experiences or contexts, the LLMs’ output will similarly lack this diversity.

The process of creating user agents and simulating product experiences, as implemented in frameworks like Elicitron, can be automated and scaled. However, the quality and authenticity of these simulations depend heavily on the context provided and the diversity of the agents generated. Inadequate contextualization can lead to repetitive or generic outputs that fail to capture the varied needs of real-world users. Additionally, while techniques like chain-of-thought reasoning can enhance the interpretability and reasoning capabilities of LLMs, they do not fully mitigate the risk of missing out on unique or less obvious user needs that fall outside the patterns recognized by the model.

This article presents Elicitron, a framework that leverages LLMs to enhance requirements elicitation and uncover diverse user needs, including those that are latent. Our context-aware serial generation method proved most effective in creating diverse user agents. Elicitron successfully outperformed traditional empathic lead user interviews in generating latent needs. LLMs also demonstrate effectiveness in analyzing interviews and automatically classifying latent needs. Elicitron shows the potential of leveraging LLMs for requirements elicitation and user-centered design, providing an alternative and cost-effective method for designers.

While Elicitron shows promise, the quality of insights depends on the LLM’s capabilities, and prioritizing latent needs remains a designer’s task. Future work includes user studies to validate Elicitron’s ability to aid designers and exploring multiagent interactions to uncover broader unmet needs. Moreover, exploring the possibility of incorporating multimodal inputs and outputs in the process of eliciting requirements represents another prospective avenue for research.

There are no conflicts of interest.

The datasets generated and supporting the findings of this article are obtainable from the corresponding author upon reasonable request.