Electroencephalogram Experimentation to Understand Creativity of Mechanical Engineering Students

Defining creativity is challenging [1], and proposing a universal definition is nearly impossible [2]. One reason behind such difficulty is creativity is a multi-faceted and complex phenomenon [3–7]. Researchers have provided definitions of creativity from different perspectives, and in his booklet, Treffinger [8] collected 124 creativity-related definitions published between 1926 and 2011. Creativity definition varies based on discipline, for instance, creativity in music will have a different definition in science. Performance techniques, composition, novel use of rhythm, beat, pitch, improvisation, and expression may be some of the criteria of creativity in the music domain. Groundbreaking ideas, discoveries, and theories are accepted criteria for creativity in the scientific field [9]. New, novel, or original are broadly used across diverse definitions of creativity.

In the 21st century, “creativity” has become an essential skill, not only in the fields of arts and humanities but also in the science, technology, engineering, and economic contexts of human life [10]. In engineering, creativity has both novelty and appropriateness as key components. Studies have revealed through utilizing behavioral and neurological approaches that, after using creativity enhancing exercises and techniques, changes in behavioral outcomes and brain activity happen [11,12]. To study the impact of these exercises and techniques on creativity by using behavioral approaches is useful, however, it is not possible to direct way to investigate the neural mechanisms that underlie creativity from behavioral approaches. To study these underlying processes, neurological approach is a possible way. Researchers are able to use neurological approaches which allow them to obtain visible physical results. These results connect stimuli or prompt related to creativity to biological processes and structures. Also, whether or not methods claiming to improve creativity or aid in problem solving actually happening or not researchers are able to test through these approaches. Those methods which claim to aid in innovative design or problem solving could be hypercritically validated using neurological approaches that provide neurological and quantifiable measurements [13].

Several neuroscientific studies have been conducted to understand the underlying brain activations associated with creative idea generation [14]. Many aspects of cognitive activity are reflected by the changes in neural activity across different electroencephalogram (EEG) frequency bands [15–17]. EEG frequencies in the range of the alpha band (usually 8–13 Hz) have strong dominance during an individual’s creative ideation [18]. Alpha band power increases is tagged as synchronization of EEG response and alpha band power decreases is known as a desynchronization of the EEG response, which is generally associated with creative task. EEG desynchronization also indicates cortical activity or arousal. A cognitive task, during a reference interval, is usually assessed by alpha power in terms of changes in task-related power (TRP), event-related synchronization (ERS), or event-related desynchronization (ERD) [19].

Research has shown differences in alpha desynchronization for regions of the alpha band [19]. Klimesch [20] first provided evidence that general task (e.g., attention processes) is associated with lower alpha ERD. On the other hand, the upper alpha frequency band ERD is associated with intelligence-related demands, like creative thinking and divergent thinking [21–23]. Starting from the pre-stimulus reference up to task activation, if the alpha power activity increases, it is indicative of ERS, which signifies active information processing in neuronal networks [24]. Klimesch et al. [25] found that alpha power is a functional correlate of creative cognitive task demands.

Creativity is a multiple-stage (e.g., generative and exploratory) process [6,19,26,27] and alternate uses task (AUT) is a standard method utilized by researchers to assess divergent thinking. From EEG studies [28,29], AUT has been associated with TRP in the upper alpha band power. The process of idea generation to idea elaboration task-related performance has been correlated with distinct patterns of upper alpha power in frontal areas and decreases in centro-temporal areas [30]. Jauk et al. [31] had participants generate either common solutions (convergent thinking) or uncommon/unusual solutions (divergent thinking) to tasks, which included an AUT and a word association task. The authors reported that divergent thinking (uncommon responses) showed synchronization of alpha power on frontal cortical areas, whereas convergent thinking (common solutions) showed more desynchronization of alpha power. Wang et al. [32] confirmed the serial order effect in divergent thinking alongside the use of an AUT. The article reports, similar to previous findings, either early or late increases in upper alpha synchronization for participants with lower inhibition across epochs. Other studies [29,33] identified alpha power increases at the beginning of creative idea generation for the AUT but decreases during task involvement over time with a final re-increase at the end. This U-shaped alpha phenomenon is not uncommon [34] for creative individuals.

Machine learning (ML)/deep learning (DL) techniques have been applied in recent years to understand the nature of EEG signals of human emotions. In addition to ML approaches used for brain-computer interface (BCI) applications, motor cortical activity has also been classified and could control physical objects [35–38]. Recently, machine learning methods have been applied to creativity research for feature collection and selection from EEG band and frequency associated with creativity. Several research studies have assessed individuals’ creativity, but it is not obvious and certain that everyone’s level of creativity will be in the same manner [19,29,31,39]. It is difficult to measure the level of creativity of different groups of professionals. However, having certain conditions and assumptions for EEG features, which are subject-specific, makes ML approach possible to identify creativity.

Predictions and trained ML models may be pivotal players in identifying the creativity level of individuals. Among the various ML algorithms, support vector machine (SVM), k-nearest neighbors (kNN), artificial neural networks, and linear discriminant analysis are widely used [40]. Accuracy of the algorithms varies during training of the datasets. A recent AUT study [41] achieved 80.08% accuracy using SVM to classify 30 objects and items for two conditions (common or uncommon) to classify more creative from less creative persons.

To the best of the authors’ knowledge, the use of ML techniques to classify divergent thinking (e.g., creativity) for engineering students via a modified AUT is new.

Contributions from this paper are as follows:

• Experimental use of a modified AUT among engineering students, including 57 different objects with three use categories for each object (creative use, nonsense use, and common use), totaling 171 object/use pairs without any repetition (see  Appendix for a full list of object/use pairs). Additionally, there is an investigation of event-related potentials (ERPs) focusing on the N400 component.

• Use of ERP technique within the engineering disciple to understand creativity.

• Use of ERP techniques to understand neuro responses during cognitive activities.

• Enhanced understanding of how alpha band power varies during processing demands for creative tasks.

• Using statistical analyses to identify the significance of creative ideation based on hemispheres, temporal areas, and tasks.

• Successfully using kNN classifier to classify common, creative, and nonsense neuro responses for engineering students from multiple trials, and less possible temporal position data used to reduce the complexity.

The next section provides background on neuroimaging and combining neuroimaging with ML methods (Sec. 2), followed by the experimental setup and data acquisition procedure (Sec. 3). Section 4 discusses the EEG signal analysis, and the statistical analyses and results of this research are presented in Secs. 4 and 5. Section 6 presents machine learning techniques used to classify creativity. Limitations (Sec. 7) are followed by conclusions, summary, and future work.

Even though there are many different types of neuroimaging methods, functional magnetic resonance imaging (fMRI) and EEG are two techniques widely used for various reasons. When selecting these neuroimaging techniques, an important consideration is the trade-off between spatial and temporal resolutions. We discuss fMRI and EEG in this section.

fMRI detects changes in blood oxygenation and blood flow in the brain. Therefore, different active brain areas require more oxygen and more blood flow to sustain neural processes. Consequently, fMRI has high spatial resolution and indicates which brain areas are active. However, the drawback of this is that the temporal resolution (detailing exactly when activation happened) is poor because of delayed hemodynamic responses.

Functional near infrared spectroscopy (fNIRs) is similar to fMRI, which uses differences in optical absorption and to detect the changes of hemoglobin species inside the brain. The portability and potential for long-term monitoring capability are advantages of fNIRs. For applications where spatial and temporal resolutions are crucial, multichannel NIRs have already improved the spatial resolution of fNIRs for brain mapping. Limitations of fNIRs for clinical use include averaging and group analysis, along with the accuracy and precision [42]. One of the major disadvantages is oxygen absorption in blood takes time which makes fNIRs prohibitive to use in time-related neuro response experiments, such as ERPs. For fNIRs, it is critical to establish a stable contact between source/detector and skin, also, the color and layering of hair attenuate the light of NIR. fNIRs is time consuming to use with participants because of setup time needed [43].

EEG works by using electrodes placed on the scalp to detect changes in electrical activity as neurons release neurotransmitter [44]. EEG provides excellent temporal resolution in the millisecond range but lacks spatial resolution. Voltage fluctuations are minimal, and the signals are sent to an amplifier for analysis later. EEG signals are analyzed based on frequency, amplitude, and electrode position. In general, the groups of frequency bands include δ (0.1–4 Hz), θ (4–7 Hz), α (8–13 Hz), β (13–30 Hz), and γ (30–45+ Hz). However, these classifications can differ slightly based on the source and demographic parameters such as gender and age [45].

EEG is utilized to examine time-locked activity called ERPs, which are voltage fluctuations from responses to specific events or stimuli at a given time (down to the millisecond) ERPs [46]. ERPs are labeled according to positive or negative signal amplitude peaks or fluctuations, and according to the time when the peak occurs. The N400 and post-N400 components, which are the negative-peaking potential around 300–500 ms and 500–900 ms post-stimulus, respectively, have been tied to unusualness of stimuli and cognitive processes essential to creativity [47,48]. Previous research [48] indicates that the N400 is responsive to unusualness and novelty (two critical components of creativity [9]) stimuli presented during a modified AUT. Previous studies [49–51] have mainly focused on design, concept generation, and problem solving. Some new research focuses on studying divergent thinking, creativity, or novelty using ERP tasks. An approach using ERPs needs to be evaluated to better understand the timing of specific components related to unusualness, novelty, or creative stimuli for engineering. More research is needed to understand how these two components can assist in measuring conceptual expansion in engineers.

Research in neuroimaging techniques and machine learning or pattern classification has significantly increased since the early 2000s [52]. Majority of these researches utilizes fMRI alongside their machine learning method of choice to gather results. Many of the applications have been clinical up to this point, focusing on classifying patients with certain pathologies, like Alzheimer’s disease [53], or classification of patients with disorders, like schizophrenia, mood disorders, or autism [54].

While there is an extensive amount of research related to fMRI and machine learning, only a few studies utilize EEG with machine learning methods. Some examples include classification of emotional states [55], left- or right-hand movements [56], and depressed patients from non-depressed patients [57]. In recent years, machine learning techniques have been used in creativity research to classify between more and less creative individuals [58]. Our study uses machine learning techniques to classify the participants’ neuro responses (creative, common, nonsense) for the BCI applications.

This pilot study includes ten participants (n = 10). Nine participants were mechanical engineering students and one office administrator from mechanical engineering. Participants’ age is between 22 and 32 years (mean age = 26.6, SD = 3.13). Eight of the ten participants were male. Nine of the ten participants were right-handed. All participants self-reported normal vision or corrected to normal vision, currently not feeling any discomfort, and were not taking any medication.

A 24-channel EEG system from mBrainTrain was used to record data at 500 Hz with the corresponding SMARTING amplifier. M1/2 is the reference electrode of the EEG system. Appropriately sized caps were selected for participants and conductive gel was used to keep impedance low (5–10 kΩ). The placement of the 24 electrodes followed the 10/20 international placement system (see Fig. 1). Neurobs Presentation (Neurobehavioral Systems, Inc., Albany, CA) was used to synchronize the EEG acquisition with the stimulus presentation.

Participants were seated in a low-noise environment and fitted with the EEG cap. The experimental procedure was explained to the participants and what to expect, which buttons they would push, and any definitions they might need to complete the experiment. Participants then performed a practice experiment, followed by the actual experiment. The duration of the experiment including practice session was approximately 25 min.

This pilot study investigated conceptual expansion as a central component of creative thinking in engineering students. This study was based on work by Kröger et al. [47,48]. Using a modified AUT, Kröger et al. [48] related the N400 ERP component to varying levels of unusualness and/or novelty of stimuli. During the current study, participants were shown an object/function pairing as a stimulus, which differs from a traditional AUT, where participants are given an object and instructed to generate as many alternatives uses as possible for that object. The difference between the current study and Kröger et. al. [48] is that this study focuses on engineering participants to gain a better understanding of the neural responses of engineering students.

Each experimental stimulus started with a fixation cross (+) in the middle of the screen for 1000 ms. Participants then see a 500 ms blank screen, followed by the object/function presented for 2000 ms (object > function). Again, after a 500 ms blank screen, the participants are presented with the first question (“Unusual?”) for 1700 ms. The participants would answer “yes” or “no” to this question by using the left and right mouse buttons. The second question (“Appropriate?”) is presented after another 500 ms blank screen. Again, the participants answer “yes” or “no” using the mouse buttons. The trial ends with another 500 ms blank screen, and the cycle repeats for a new object/function pair. See Fig. 2 for a pictorial of the experimental design.

Each participant responded to 171 object/function pairs, 57 objects with three functions as stimuli (a creative function/use, common function/use, and nonsense function/use). Table 1 gives an example of what the participant might see. The participant ultimately decided function/use. To be categorized as common use, the participant would answer no–yes to the two questions, yes–yes to categorize it as a creative use, and yes–no for a nonsense use. There was no repetition on item-use pairs and pairs were presented randomly.

EEGlab, a matlab plugin, was used to pre-process the EEG data collected from the participants. A broad filter (finite impulse response (FIR) filter) from 0.5 Hz to 100 Hz was applied, followed by a notch filter from 58 Hz to 62 Hz to remove electrical noise. An independent component analysis, using FastICA algorithm, was performed to remove artifacts not related to brain data. The EEG data were prepared for ERP analysis by segmenting into 1200 ms blocks based on time-stamps indicating the start of each object/function stimulus pair. Each segment, or epoch, was baseline corrected with the 200 ms time window before the presentation of the object/function pair and the remaining 1000 ms segment was used for analysis.

A 30 Hz low-pass filter, with a slope of 24 dB/Oct, was applied to each segment and amplitudes exceeding approximately +/−100 μV were removed. Grand averaged ERPs from all participants were calculated for the 300–500 ms post-stimulus region. To be included in the grand average, participants needed to have selected a minimum of 15 object/function pairs for each of the three categories (common, creative, and nonsense).

From reference to activation, when there are negative values, it evinces decreases in power familiar with desynchronization. On the contrary, increased power with positive values is known as synchronization [24]. The electrodes were aggregated into the following temporal areas: anteriofrontal, frontal (F), centroparietal, parietotemporal (P/T), and occipital (O); odd numbers represent the left hemisphere and even numbers represent right hemisphere. The electrodes in the central positions (AFz, Fz, Cz, CPz, Pz, Poz) were not included for analysis because this study focused on potential hemispheric differences.

A 9 × 2 × 3 analysis of variance (ANOVA) with the within-subjects factors Area (nine electrode positions in each hemisphere), Hemisphere (left, right), and Tasks (creative, nonsense, and common) was statistically analyzed. Additionally, this study considers the continuous between subjects’ ideas in the ANOVA design to evaluate the task performance. According to Ref. [59], considering the violations of sphericity assumptions, this study employed the multivariate approach. A significance level of p < 0.05 (two-tailed) was applied for all statistical analyses. Statistical analysis was performed using by Minitab: Data Analysis, Statistical & Process Improvement Tools.

A two-factor repeated-measures ANOVA was used for analysis. The two factors for this study were: condition (common, creative, nonsense) and electrodes Cz, CPz, Pz, and POz. These four electrodes of interest were chosen based on p-values identified in previous studies [43,44], in addition to the known centro-pariatel distribution of the N400 effect [58] was monitored. Mauchly’s test of sphericity was first used to verify if the variances were equal. In this case, the sphericity assumption was not violated for the N400 time window for condition (Χ² (2) = 2.174; p = 0.337) but was violated for electrode (Χ² (5) = 14.820; p = 0.016). Therefore, degrees-of-freedom for the electrode factor were corrected using Greenhouse–Gesser estimates of sphericity (e = 0.354), and these corrected numbers are presented below.

The repeated measures ANOVA did not show significant main effects for the factor condition (F (1.320, 5.279) = 0.664; p > 0.05; η_p² = 0.142) or for the interaction of the factors condition*electrode (F (6, 24) = 0.992; p > 0.05; η_p² = 0.199). Main effects were significant for the factor electrode (F (1.063, 4.253) = 7.392; p = 0.049 < .05; η_p² = 0.649).

With more participants, it is probable that results similar to those presented here would be statistically significant. Even though not significant, results from the pilot study follow a similar pattern presented by a previous study [48]. The results indicate that stimuli classified as nonsensical or creative elicit larger N400 amplitudes (−1.107 mV and −0.755 mV, respectively) than common uses (0.0859 mV), as shown in Fig. 3. Figure 3 shows the mean amplitudes for all participants for each type of stimulus on electrodes Cz, CPz, Pz, and POz. Analysis of the N400 may be an approach to understanding the creativity of engineers. The grand average waveforms for all participants for each electrode of interest are presented in Fig. 4, with an outlined window of time where the N400 was examined. For more information on this study, see Ref. [13].

Task-related changes in EEG alpha power during the generation of creative/original, nonsense, and common uses in the modified AUT were calculated. Positive TRP indicates task-related alpha synchronization increases in alpha power relative to rest, negative values indicate desynchronization. Based on the originality of ideas, the individual participant’s neuro response for both hemispheres is shown in Fig. 5. Except for participant 1, all other participants showed strong increases in alpha power (relative to a pre-stimulus reference interval) over anteriofrontal sites. All the participants showed strong decreases over parietooccipital temporal area.

The 9 × 2 × 3 ANOVA revealed a significant main effect Area (F (8,486) = 18.22; p < 0.005; η_p² = 0.230), indicating decreased alpha power during creative ideation especially over (O1/2, P7/8,) area. The main effect Hemisphere (F (1,486) = 0.266; p = 0.287; η_p² = 0.00233) indicated stronger alpha power decreases over right (SE = 0.0294) than left hemispheric sites. The interaction Area × Hemisphere was not significant (F (8,486) = 0.69; p = 0.701; η_p² = 0.0112). No significant main effect Task (F (2,486) = 0.02; p = 0.984), creativity was associated with stronger alpha power decreases than common and nonsense task. The interaction Area × Task (F (7,486) = 0.02) revealed that strong power increases during three tasks at temporal sites (Tukey’s comparisons revealed significant task-related differences only at T7/8). No other effect involving the within-subjects factor task was significant. General model prediction on O1/2 temporal area, right hemisphere, and main effect creative task predicted.

Statistical analysis revealed a distinct pattern of alpha power—positive value of participants’ TRP shows ERS on the frontal and central cortical area, whereas the negative value of the TRP indicates ERD on the parietooccipital cortical area. The alpha power decrease indicates participants were task involved during the experiment, which is similar to results reported by Benedek et al. and Jauk et al. [19,31].

Time-frequency analysis of EEG is performed using wavelet transform because of the non-stationary nature of EEG. Transient features of the signal can be accurately detected in time domains by wavelet transform [60,61].

The wavelet analysis band the EEG signal for δ (0.1–4 Hz), θ (4–8 Hz), α (8–13 Hz), β (13–30 Hz), and γ (30–45+ Hz) bands. In this experiment, we analyze the α band (8–13 Hz) as slow temporal modulations of (<16 Hz), aiming the classification of AUT neuro responses primarily focused on creative response. We applied 00statistical analysis techniques to the wavelet coefficients, which are decomposed EEG signals, for feature extraction. The following statistical features were selected for α band (8–13 Hz) corresponding to each object category (creative, nonsense, common): (1) min value, (2) max value, (3) mean value, (4) standard deviation, (5) skewness, (6) kurtosis, and (7) variance.

Task-related alpha power showed significance area placed over parietooccipital (O1/2, P7/8,) area. Statistical results identified the temporal area electrode for classification. Figure 6 illustrates that creative task power has decreased more in O1/2 temporal position, which is the parietooccipital area of the brain. Furthermore, this study uses O1/2 temporal electrode position for kNN classifier to classify creative, nonsense, and common response from AUT tasks. For each participant, 57 trials for three object categories summing to 171 trials were included for classification. All participants’ data were included for the classification purpose from O1/2 area location.

Table 3 presents performance of the models in terms of accuracy, precision, recall, and F-1 score. The accuracy of kNN classifier was 99.92%. At the same time, this study also explored other classification models like SVM, Ensembles classifier, and Naïve Bayes classifier. However, these models performed worse in terms of accuracy, precision, recall, and F-1 score. kNN classifier was able to achieve 100% accuracy to classify creative neuro responses, 99.89% of nonsense neuro responses, and 99.89% of common neuro responses. Area under the curve (AUC) was 0.9995, which demonstrates that the classifier can probably distinguish the positive class values and negative class values. A recent study [41] presented that machine learning techniques (quadratic discriminant analysis and SVM) can classify more and less creative individuals and also classify more or less creative brain states from EEG responses. Studies [63–65] have reported results for EEG emotion classification from the aggregation of 12 or more pairs of electrodes to achieve high classification accuracy. This study achieved an accuracy of 99.92% from statistically significant temporal area position analysis, which is a faster classification than that of complex methods.

The aim of this study was to investigate the neurological responses on creativity task in the engineering domain. N400 component and alpha task-related power are analyzed to understand the cognitive process of creativity. The ERP results with highest negative values associated with nonsense functions and most positive values associated with common functions. These results support the main hypothesis of this study that novelty and appropriateness of a creative task will be exhibited in modulation of the N400 component, with highest negative values associated with unusual-inappropriate (nonsense) functions and most positive values associated with usual-appropriate (common) functions.

This is a pilot study to confirm modulation of the N400 with respect to novelty and appropriateness in engineering. The results show promise of this experimental design to further understand modulation of the N400 in an engineering creativity context. The findings allow for a more direct way to study the underlying cognitive process and help improve creativity training.

Considering the EEG results, task-related frontal alpha synchronization was observed which indicates high internal processing demands [19,66] during creative tasks. Low internal processing demands indicate strong desynchronization, especially in posterior brain regions (parietooccipital), which reflect stronger demands on the visual system during creative activity information processing [19,66]. Taken together, the results suggest that frontal brain regions may exert control by means of temporal synchrony of lower frequencies (especially α but also θ) with parietal brain regions. The findings of this study are in agreement with functional imaging and are also consistent with other studies [19,25,66,67].

Using the widely used Brodmann areas (BA), which relate cognitive functions to different scalp locations. Results from our study show EEG (de)activation in BA 09 and BA 10 areas in the right hemisphere. BA 09 and BA 10 areas are respectively associated with higher cognitive functions and decision-making [68]. Overall results from this study demonstrate that EEG is both a practical and relevant technique to studying neurophysiological activity of creativity in engineering. This study also showed that ML techniques can classify the neural responses (common, creative, nonsense—responses) from modified AUT experiments. This classification can be utilized to identify neuro responses associated with more or less creative brain states.

We present some limitations of our study to guide future works:

• Unbalanced male/female participants (80% male and 20% female) could be an issue in EEG response patterns for the classification of neural responses.

• This study did not consider categorical data such as age, gender, etc. Participants’ age group was between 22 and 32.

• This study did not include functional connectivity and did not explore the classification of more or less creative individuals.

• This study considered the preliminary screening of the participants, however, didn’t use psycho-behavioral questionnaires to profile the participants in terms of personality and attitude.

Although not statistically significant, our study found fundamental ERP effects for the N400 component of engineering students when modified AUT is used as stimuli. In other words, larger negative responses for nonsense uses, followed by creative uses, and then common uses with a positive response.

An ANOVA analysis of task-related power shows a significant main effect for parietooccipital temporal area and the main effect Hemisphere. The results indicated larger alpha power decreases over right than left-hemispheric sites. The distinct pattern of alpha power and positive values of TRP neuro responses show ERS on the frontal and central cortical areas. The negative value of participants’ TRP neuro response indicates ERD on the parietooccipital cortical area. Participants were actively involved during AUT tasks, as the alpha power decreased during the modified AUT experiment, as suggested by several previous studies.

Machine learning kNN model outperformed in terms of accuracy, precision, recall, and F1-score. kNN classifier successfully achieved high 99.92% overall accuracy to classify creative, nonsense, and common neuro responses of the participants using O1/O2 temporal area data. Other models like SVM, Ensembles classifier, and Naïve Bayes classifier did not perform well.

A future extension of this study will explore the classification between the more and less creative individuals. Also, future works will investigate the real-time ML based neurofeedback and the possibilities of DL techniques for human computer interaction.

This material is based upon work supported by the National Science Foundation under Grant Nos. 1561660 and 1726358. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

• National Science Foundation under (Grant Nos. 1561660 and 1726358; Funder ID: 10.13039/501100008982).

There are no conflicts of interest. All procedures performed for studies involving human participants were in accordance with the ethical standards stated in the 1964 Declaration of Helsinki and its later amendments or comparable ethical standards. Informed consent was obtained from all participants. Documentation provided upon request. Informed consent was obtained for all individuals. Documentation provided upon request. This article does not include any research in which animal participants were involved.

The datasets generated and supporting the findings of this article are obtainable from the corresponding author upon reasonable request.

The University of Oklahoma IRB# 11298 approved for this study.

Informed consent was obtained from all subjects involved in the study.

f_n =

false negative

f_p =

false positive

t_n =

true negative

t_p =

true positive

FPR =

false positive value

TPR =

true positive value

Item-use pairs were presented randomly to the participants. Number in the first column is solely for count. Use category (common, creative, or nonsense) was for data keeping purposes only. Category was ultimately decided by the participant.