Abstract
Undergraduate research continues to serve as an effective strategy for mitigating the effects of a leaky pipeline. Significant funding from institutions and government agencies has increased the number of students participating in undergraduate research. In this paper, we report on the six-year experience of a National Science Foundation funded Research Experiences for Undergraduates (REU) Site: Biomedical Engineering in Simulations, Imaging, and Modeling (BME-SIM). The operation and evaluation of the program are both described. We report on the results from 55 students over six summers from 2014 to 2019. Our program was successful in attracting a diverse group of participants including 46% under-represented minority students and 53% women. Based on evaluation results, students reported significant gains in technical skills, communication skills, and knowledge of graduate school. Our findings indicate baseline gender differences for several learning outcomes, where women and nonbinary students report lower levels of mastery. These gaps are closed by the end of the program except for confidence in skills, which is still significantly lower than those reported by male counterparts. The impact of the experience on ultimate career path is difficult to determine due to underlying biases and other motivating factors; however, 67.6% of graduates have entered graduate programs. Finally, we have provided lessons learned for those who are interested in building a summer research program. In conclusion, we have described the successful implementation of an REU site and the positive learning outcomes of the student participants.
Introduction
Undergraduate research experiences (URE) are one of several high impact educational practices that have been documented as successful and beneficial to a diverse range of college students [1]. Quality URE increase understanding of how to conduct research, confidence in skills, and awareness of graduate school [2]. In addition, such programs can clarify, refine, and reinforce STEM career path goals [3]; increasing the likelihood of pursuing a STEM graduate degree [4]. Research shows URE have the same effect or greater on under-represented minority (URM) students [3]. URE are also an excellent way to promote high-quality student-faculty interaction. Less than 20% of students have had research related interaction with faculty, with African American students having particularly low rates of participation [5]. URE that include professional development offer an opportunity to expand nontechnical or social skills—such as leadership and communication—often not a key focus in undergraduate curricula. Valuing the social dimensions of engineering identity may resonate more with underrepresented populations, thus promoting a more inclusive environment and increased sense of belonging [6].
Due to the positive and lasting benefits, universities have strategically invested in undergraduate research. This investment has been amplified by efforts from government entities. Since 1986, the Research Experiences for Undergraduates (REU) program funded by the National Science Foundation (NSF) has provided countless research experiences for students [7]. Currently, there are 151 engineering and over 700 total REU Site entries in the NSF directory [8]. This means approximately 7000 students participate in undergraduate research each year through the NSF program. In a review of the sites within the Engineering Directorate, only five were related to biomechanics, which involved rehabilitative engineering, robotics, injury mechanics, and mechanobiology. As each REU site is unique, evaluation of program effectiveness is limited to small cohorts.
The NSF-funded REU Site: Biomedical Engineering in Simulations, Imaging, and Modeling (BME-SIM) has been in continuous operation since 2014. Six cohorts with a total of 55 students have completed the program. This longevity provides a unique opportunity to evaluate the effectiveness of the program. The objective of this paper is to provide an example program framework, report outcomes for the past six years, and offer lessons learned.
Program Operation
Overview of Biomedical Engineering in Simulations, Imaging, and Modeling Program.
The REU Site in BME-SIM at East Carolina University was funded by NSF in 2014 (EEC-1359183), 2017 (EEC-1659796), and 2020 (EEC-1950507). The goal of the BME-SIM REU program is to broaden participation by providing authentic research experiences to students who are traditionally underrepresented in engineering. These research experiences along with mentoring and professional development have been shown to promote retention in STEM and matriculation into graduate programs.
The BME-SIM program is characterized by a unique home institution mentor model [9], methodological themed research experiences [10], creative and informative professional development activities, and conference attendance for all participants. The program provides research experiences for ten students (initially eight students) for 10 weeks each summer. At the end of the program, students should have a better understanding of how to conduct research, increased awareness of graduate school, clarification and reinforcement of STEM career path, and greater identification as engineer/scientist. Program operation can be divided into the following: recruitment, mentorship, professional development program, and research projects.
Recruitment and Selection.
To recruit students, the BME-SIM program has a two-pronged recruitment plan: a targeted regional component and a national campaign. To ensure a broad applicant pool, regional partners include the following Master's-level institutions: Fayetteville State University, North Carolina Central University, and Virginia State University, historically black institutions; and University of North Carolina—Pembroke, public institution serving a large Native American population. We have identified at least one faculty member at each partner institution who will help with recruiting students—advertising and facilitating site visits. To attract more community college students, we have recently partnered with the North Carolina Engineering Pathways program, a joint project between the North Carolina Community College System and the state engineering programs to enable students to begin engineering studies at community colleges and then transfer as seamlessly as possible to one of the UNC engineering programs.
While we are targeting four partner institutions (all of which have a large minority student population), most of our applications come from students at other institutions. Information about the program and the application is available on the program website and social media. Announcements are sent to numerous national listservs and social media platforms such as the Society of Women Engineers, Society of Black Engineers, Society of Hispanic Professional Engineers, American Indian Science and Engineering Society, Biomedical Engineering Society (BMES), Biomch-L, and NSF Louis Stokes Alliances for Minority Participation (LSAMP) programs.
In the last two years, we strategically expanded our recruitment campaign. Based on our successful partner institution visits, we started recruiting in-person at the BMES annual meeting. Our presence at BMES has increased awareness of the BME-SIM program. We also debuted annual video presentations which answered common questions and highlighted past participants. The videos were recorded using Mediasite (Mediasite, Madison, WI) in webinar format that included presentation slides and camera feed. The presentation covered how to apply, what to expect, and alumni accomplishments along with commentary from past participants. By filling the presentation with photos, we hoped to bring the program to life and help potential applicants visualize themselves as part of the program. To distribute the video, a link was included on the program flier, which is distributed broadly and also posted on social media. In the future, we will use a live webinar format to increase engagement while also recording for asynchronous viewing.
Because of the varied faculty expertise and multidisciplinary research, our goal is to bring in a diverse group of students across multiple identities who have a background in STEM and an interest in biomedical engineering. Specifically, we target under-represented minority, low-socioeconomic status, female, and veteran students, including students with disabilities, from institutions with limited research opportunities including community colleges.
Application materials including personal statement, recommendation letters, REU project rankings, and transcripts are then reviewed. The selection committee consists of the Principal Investigator (PI) and Co-PI. Occasionally faculty mentors are consulted on project-specific needs. Limiting the selection committee reduces the workload; however, the faculty mentors must trust in the leadership team to make good selections. This trust has been built over the years by listening to the faculty mentor needs and carefully matching students and mentors. We have found that high project interest corresponds to a successful research experience. Based on the above criteria, 30–40 applicants are selected for a phone interview. After the interviews, candidates who are likely to be significantly impacted by the experience and who demonstrated enthusiasm in the interview are matched to research projects based on their academic background, skills, and interests while ensuring a diverse participant group—race, ethnicity, gender, institution type, economic status, veteran status, and ability. We also acknowledge that students have multiple intersecting identities that make each candidate unique.
Mentorship and Relationship Development.
We have developed a mentorship model, previously discussed in the literature [9], that allows us to connect the students' home institution with the REU Site and faculty mentor. The mentoring structure is composed of a home institution mentor, REU faculty mentor, and peer mentors. The BME-SIM program aims to provide a community of practice (faculty, REU program directors, graduate assistant, and peers).
Home Institution Mentors.
To provide mentoring beyond the summer REU experience, all students are connected with a home institution mentor. The home institution mentors are faculty members in a related field that provide pre- and postprogram mentoring. The inclusion of home institution mentors in our program has several benefits: (1) provides a contact and mentor at the student's home institution; (2) extends the period of engagement for our participants; (3) allows students to hit the ground running at the beginning of the REU program—providing maximum benefit from their on-campus time; and (4) fosters collaboration between ECU and our partner institutions facilitating more research opportunities. For example, a home institution mentor worked with a student to learn matlab prior to the REU program, and another student joined the home institution mentor's research group after the REU program.
While we specifically target our recruiting effort at partner institutions, where we have already identified mentors, most participants come from our national campaign. For students not enrolled at a partner institution, home institution mentors are identified upon student acceptance. This was easily accomplished in our previous cycles. Upon initial contact, all 55 previous home institution mentors accepted our request. We recognize community colleges may not have research active faculty; therefore, we connect community college participants to a mentor at their intended four-year institution, in addition to a mentor at the community college.
The home institution mentor time commitment is limited to advertising the REU program on their campus (if a partner institution) and meeting with selected students once before and once after the program. The commitment is estimated to be a total of 2 h. It is our hope that by providing this point of contact, students will feel more comfortable asking for both academic and career advice and inquiring about research opportunities at their home institution.
Research Experiences for Undergraduates Faculty Mentors.
Over the past six years, a total of 19 faculty have served as mentors. This diverse group of faculty has spanned six departments (Engineering, Kinesiology, Health Education and Promotion, Physical Therapy, Communication Sciences and Disorders, and Physics) and four colleges providing a culture of multidisciplinary research.
Student learning gains are directly and indirectly related to their faculty mentor relationships [11]. To encourage positive student–faculty relationships, we require the following eight commitments from our REU faculty mentors: (1) provide a welcoming and inclusive environment; (2) convene weekly face-to-face meetings with students (our mentors average 3–4 meetings per week); (3) ensure proper training of lab techniques and software; (4) participate in both program classroom and social activities; (5) monitor and communicate progress toward project goals; (6) balance rigorous expectations with emotional support; (7) provide guidance and review of final poster and abstract; and (8) communicate with the student before and after the program. These commitments are based on the ten Salient Practices of Undergraduate Research Mentors [12]. Although graduate students are part of the mentoring structure, primary mentoring (both research and career) is accomplished by REU faculty mentors.
Upon program acceptance, students connect with their REU faculty mentor and home institution mentor to review pertinent literature and discuss the research process. In some cases, this interaction has been accomplished through web-based conference platforms such as Zoom. To facilitate relationship development, mentors are invited to several events—both formal and informal—throughout the summer experience. REU faculty mentors are encouraged to coach students beyond the program providing career advice and writing letters of recommendation.
Peer Mentoring.
The program fosters student–student relationships among the cohort as well as with graduate and undergraduate students in their research labs. The cohort will serve as a set of peers with similar interests and goals—a community of research practice [13]—that can provide support as students return to their home institutions. Although students will work in different labs across campus, the weekly professional development, social activities, and living communities support cohort camaraderie. All students are required to live on-campus during the summer program. However, internal students sometimes retain their off-campus housing in addition to the on-campus housing. When this happens, the student not residing on-campus demonstrates weaker connections to the other students. Students are housed in a suite-style dorm where other REU and summer research program students are housed. Fostering a strong sense of community facilitates long-term follow-up. Introduced in 2017, the BME-SIM graduate assistant acts as a cohort mentor and organizes informal activities.
Professional Development Program.
Our professional development series is designed to provide information on graduate school (application, funding, fellowships), career paths (program panel, graduate school student panel), and soft skills (communication, social media, networking). Based on feedback, special communication workshops have been developed and implemented in years 4–6. These workshops are offered by the University Writing Center and Speech Communication Center (SCC).
The structured components of the program include a bootcamp, weekly lunch seminars, communication development, and social activities. See Table 1 for an example schedule.
Example summer REU schedule
Week | Activities |
---|---|
1 | Research Training: 2-day Bootcamp |
2 | Writing workshop: Literature Reviews: Making Academic Synthesis Happen |
Lunch: How to Apply to Graduate School | |
3 | Communication workshop: Professional Communication |
Lunch: NSF Graduate Research Fellowship | |
Deliverable: Research Plan | |
4 | Writing Workshop: Personal Statements: Packing a Punch in 1–2 Pages |
Lunch: Graduate Program Panel | |
5 | Lunch: Graduate Student Panel |
Deliverable: Progress Monitoring | |
6 | Communication workshop: Elevator Speech |
Lunch: Professional Social Media Presence | |
7 | Lunch: How to Compose a Scientific Poster |
8 | Schedule individual practice with SCC |
Lunch: Speed-Networking Event | |
9 | Lunch: Student Oral Presentations |
10 | Lunch: Research Conferences—What to Expect |
Research training: REU Poster Session | |
Deliverable: BMES Abstract Submission and Updated Personal Statement | |
1–10 | Various social events (e.g., Kick-Off Cookout, Low Ropes Course, Ice Cream Social, Bowling, Game Night, Team Trivia, Women and Gender Minorities in STEM, and Final Celebration Lunch) |
Week | Activities |
---|---|
1 | Research Training: 2-day Bootcamp |
2 | Writing workshop: Literature Reviews: Making Academic Synthesis Happen |
Lunch: How to Apply to Graduate School | |
3 | Communication workshop: Professional Communication |
Lunch: NSF Graduate Research Fellowship | |
Deliverable: Research Plan | |
4 | Writing Workshop: Personal Statements: Packing a Punch in 1–2 Pages |
Lunch: Graduate Program Panel | |
5 | Lunch: Graduate Student Panel |
Deliverable: Progress Monitoring | |
6 | Communication workshop: Elevator Speech |
Lunch: Professional Social Media Presence | |
7 | Lunch: How to Compose a Scientific Poster |
8 | Schedule individual practice with SCC |
Lunch: Speed-Networking Event | |
9 | Lunch: Student Oral Presentations |
10 | Lunch: Research Conferences—What to Expect |
Research training: REU Poster Session | |
Deliverable: BMES Abstract Submission and Updated Personal Statement | |
1–10 | Various social events (e.g., Kick-Off Cookout, Low Ropes Course, Ice Cream Social, Bowling, Game Night, Team Trivia, Women and Gender Minorities in STEM, and Final Celebration Lunch) |
Preprogram Activities and Bootcamp.
Upon arrival on campus, students participate in a team building challenge course, which facilitates cohort identity. Students and REU faculty mentors are connected at an informal cookout at the PI's house. The program then kicks off with a 2-day bootcamp, which includes program expectations (lab etiquette, data storage rules, etc.), an overview of modeling in biomedical engineering, library and database resources, time management and goal setting, and responsible conduct of research. Additionally, the bootcamp includes our preprogram assessment as well as tours of campus and research labs. This bootcamp prepares students for their research experience by providing foundational skills and developing relationships within the cohort and with REU faculty mentors.
Weekly Lunch Seminars.
Weekly lunch seminars cover graduate school applications, fellowship and funding opportunities, graduate student life, and research conferences. To foster development of communication skills, lunch topics also include building a professional social media presence, networking with faculty from across campus, and oral presentations. Each week, students share something they learned, something that was challenging, and something fun they did. This allows us to celebrate accomplishments and support challenges. Often, the students report similar problems and thus do not feel alone in their challenges. The sum result is increased engagement and cohort identity.
Communication Development.
We have developed communication workshop focusing on oral and written communication. Our Speech and Communication Center offers workshops on professional communication and elevator pitches along with oral presentation preparation. Our writing center offers workshops on literature reviews, personal statements, and abstracts. These workshops complement the additional lunch activities described above. Students orally present their project in Week 9. The program concludes with a poster session, BMES abstract submission, and revised personal statement.
All students are funded to attend the BMES annual meeting, which provides an opportunity to share research with a larger audience and reconnect after the summer. At BMES, students can directly interact with representatives from ∼90 universities and engage in networking through BME-SIM program-coordinated events. Informal comments from participants describe their attendance at BMES as a powerful motivator for graduate education.
Social Activities.
A wide variety of social activities are planned to engage participants and REU faculty mentors: cook-out, ice cream social, bowling, and game night. Both formal and informal social activities were planned throughout the summer and coordinated with other programs on campus (see Table 1 weeks 1–10). Social activities are extremely important in bonding the students and building a sense of community.
Student Research Projects.
The core of the BME-SIM program is the individual research project. Instead of a single application or disease pathology, BME-SIM projects are bound by a methodological theme: simulations, imaging, and modeling. We have previously demonstrated that participants are able to see modeling as a common theme and are able to make connections across applications [10]. There are several central concepts to developing models, such as boundary or initial conditions, and validation. Developing an understanding of these unifying themes enables students to better comprehend and appreciate modeling in a broader context. Modeling and simulation techniques are used across biomedical engineering research and are employed across scales ranging from cellular to whole body. Biomedical imaging, a natural companion to modeling, is often used to inform models by providing a source for model parameters or a means for model validation.
Computational modeling offers several advantages over other approaches: (1) ability to generate and test new theories; (2) estimate difficult to measure variables; (3) incorporate subject-specific parameters, generally derived from noninvasive imaging techniques; and (4) inform experimental designs limiting the use of animals and humans. The power of modeling to span scales and blend different disciplines places this approach at the forefront for understanding mechanisms, system dynamics, and ultimately improving health. Government agencies have long recognized the importance of modeling with the formation of both the Interagency Modeling and Analysis Group (IMAG) in 2003—nine institutes of the NIH and three directorates of the NSF—and Multiscale Modeling (MSM) Consortium in 2006 [14]. To support growth in this field, students must be introduced to and trained in computational modeling: An area not often covered in undergraduate curricula.
Immediately following the bootcamp, students begin work in their research labs. REU faculty mentors carve out small components of active research projects to propose for REU projects. These projects are then vetted to ensure they align with the theme of the site and have a high probability of completion in 10 weeks. To increase student ownership, mentors must be flexible and adapt projects to students' skill set and interests.
During the program, REU faculty mentors oversee the students daily—guiding them through the research process. To provide an authentic experience, students are involved in all aspects of the research process including literature review, hypothesis development, experimental design, data analysis, and presentation of results. The experience should provide the student with an opportunity to make an original or intellectual contribution to the discipline. Students are expected to work approximately 40 h per week and participate fully in their faculty mentor's lab, working alongside the faculty mentor, graduate students, and other undergraduate students. Students participate in, and eventually present at lab meetings in their respective lab groups. To ensure students remain on track, there are several milestones: (a) in week 3, students complete a research plan; (b) in week 5, students and mentors discuss research progress, and both receive feedback; and (c) in week 9, students give oral presentations. At the end of the program, students present their research at a joint poster session with other summer programs. Students also submit a BMES abstract for the July undergraduate deadline.
Biomedical Engineering in Simulations, Imaging, and Modeling projects aim to improve modeling of physiological and pathophysiological conditions. The models included various organs or systems; from cells to blood vessels to muscles. The projects have resulted in 5 international, 74 national, 16 regional, and 10 local conference presentations. See Refs. [15–19], for example, abstracts. REU students have been authors or acknowledged on two manuscripts [20,21], with additional manuscripts in review (1) and in preparation (4). 96.4% of students presented their research at conferences after the program ended.
Program Assessment
Program assessment is conducted by an external evaluator and includes a pre/postsurvey and pre/postwriting sample assessment. Feedback on program components, mentoring, and environment is solicited.
This study was approved by the Institutional Review Board at ECU and informed consent was received by all participants. Student surveys were completed on the first and last day of the program. Students reported their progress toward mastery of the below knowledge and skills using a ten-point Likert scale. Students reported on their future career goals and past experiences. The presurvey asked about the mode and quality of interactions with their home institution mentor and REU faculty mentor before the program. Students also completed the Survey of Undergraduate Research Experiences [22]. The postsurvey instrument quantitatively and qualitatively assessed their experience in the program including the mode, frequency, and quality of interactions with their REU faculty mentor during the program. The Survey of Undergraduate Research Experiences reflection was completed along with the Mentorship Effectiveness Survey (years 5–6) [23]. The quality of the professional development and other program events were assessed. Baseline and postprogram measures were compared using a paired sample t-test while gender and race/ethnicity differences were compared using a two-sample unequal variance, two-tailed t-test.
To evaluate long-term program success, all participants are tracked through their undergraduate careers and postgraduation. Annually after the program, students complete follow-up surveys to track career trajectories and lasting program impact. For tracking to be successful, a strong sense of community must be established during the program. A BME-SIM LinkedIn group is used to facilitate long-term follow-up on graduate school, careers, and other achievements. Connecting REU cohorts across the years, the LinkedIn group also expands the students' professional network.
Results
Participant Demographics.
The primary goal of the REU Site is to provide research experiences to students who may not otherwise have the opportunity, especially URM students, women, and those at institutions with limited research. We have successfully met this goal as 46% of our students were from underrepresented minorities (27.3% were female URM), 53% of our students were female (1.8% were nonbinary), 1.8% were veterans, and 87% of our students were from R2 institutions or lower based on Carnegie Classification [24] (41.8% were from MS level or lower, one student was from a community college). Though data were not officially collected, our program has also served lesbian, gay, bisexual, transgender, or queer students and those with speaking disabilities. See Table 2 for full details.
Summary of participant demographics (n = 55)
Demographic category | Subcategory | Count (%) |
---|---|---|
Gender | Female | 29 (53%) |
Race/ethnicitya | African American | 17 (31%) |
American Indian | 2 (4%) | |
Hispanic | 10 (18%) | |
Total under-represented minorities | 25 (46%) | |
Asian | 7 (13%) | |
White/Caucasian | 31 (56%) | |
College level | Rising Junior | 25 (45%) |
Rising Senior | 27 (49%) | |
Institution type (Carnegie classification) | Associate's level | 1 (2%) |
Baccalaureate level | 6 (11%) | |
Master's level | 16 (29%) | |
Ph.D. level | 32 (58%) | |
R2 Institution or lower | 48 (87%) | |
Major | Engineering | 31 (56%) |
Other STEM | 19 (35%) | |
Other | 5 (9%) | |
Grade point average | 3.77 ± 0.18 |
Demographic category | Subcategory | Count (%) |
---|---|---|
Gender | Female | 29 (53%) |
Race/ethnicitya | African American | 17 (31%) |
American Indian | 2 (4%) | |
Hispanic | 10 (18%) | |
Total under-represented minorities | 25 (46%) | |
Asian | 7 (13%) | |
White/Caucasian | 31 (56%) | |
College level | Rising Junior | 25 (45%) |
Rising Senior | 27 (49%) | |
Institution type (Carnegie classification) | Associate's level | 1 (2%) |
Baccalaureate level | 6 (11%) | |
Master's level | 16 (29%) | |
Ph.D. level | 32 (58%) | |
R2 Institution or lower | 48 (87%) | |
Major | Engineering | 31 (56%) |
Other STEM | 19 (35%) | |
Other | 5 (9%) | |
Grade point average | 3.77 ± 0.18 |
Some students fall into more than one category.
Program Evaluation.
Program participation leads to increased knowledge of (1) the scientific and research process; (2) modeling within biomechanics; and (3) the processes necessary to enter the STEM fields. To these ends, data from the pre- and postsurveys clearly indicate that, on average, participants reported gains in all learning outcomes (Table 3). Across all learning outcomes, participants reported near mastery (10) upon completion of the program. Analysis showed no difference in pre-, post-, or gains for underrepresented minority students compared to majority students. LO6—understanding how to design and implement a research project—reached a near significant-level postprogram; although no difference was found at baseline. URM students reported higher mastery (8.29 ± 1.02) of LO6 compared to majority students (7.67 ± 1.30). Analysis by gender showed women and nonbinary students had significantly decreased pre- and significantly increased gains, compared to male counterparts, for LO3, LO4, LO5, LO8, and LO11. Therefore, by the end of the program, most of the gender gaps were closed. For women and nonbinary students, confidence in research skills (LO9) was significantly decreased both pre- and postprogram, despite lower male gains.
Summary of BME-SIM evaluation results with 1 indicating no progress/beginner competence and ten indicating mastery/completeness
All | Female/nonbinary | Male | ||||
---|---|---|---|---|---|---|
Question: Please indicate on a scale of 1 to 10 your response to the following questions | Baseline (n = 54) | Post (n = 54) | Baseline (n = 29) | Post (n = 30) | Baseline (n = 25) | Post (n = 24) |
Understanding the research process (LO3) | 4.83 ± 2.29 | 8.33 ± 1.09a | 4.21 ± 2.43 | 8.27 ± 1.09 | 5.56 ± 1.88b | 8.42 ± 1.08 |
My confidence in research skills (LO9) | 4.72 ± 2.29 | 7.98 ± 1.21a | 3.97 ± 1.92 | 7.63 ± 1.11 | 5.6 ± 2.37b | 8.42 ± 1.19b |
Understanding how scientists work on problems (LO4) | 5.11 ± 2.14 | 8.06 ± 1.16a | 4.55 ± 2.09 | 8.03 ± 0.95 | 5.76 ± 2.01b | 8.08 ± 1.38 |
Understanding how to design and implement a research project (LO6) | 4.41 ± 2.53 | 7.94 ± 1.22a | 3.93 ± 2.41 | 7.73 ± 1.24 | 4.96 ± 2.55 | 8.21 ± 1.15 |
The development of my identity as a scientist and/or engineer (LO8) | 4.85 ± 2.58 | 7.89 ± 1.44a | 4.07 ± 2.55 | 7.70 ± 1.19 | 5.76 ± 2.30b | 8.13 ± 1.67 |
Understanding the role of modeling in biomedical engineering (LO5) | 4.70 ± 2.44 | 8.09 ± 1.46a | 4.00 ± 2.41 | 7.87 ± 1.48 | 5.52 ± 2.21b | 8.38 ± 1.38 |
Understanding the importance of research experience for grad programs (LO7) | 6.48 ± 2.58 | 8.87 ± 1.04a | 6.45 ± 2.59 | 8.83 ± 0.93 | 6.52 ± 2.56 | 8.92 ± 1.15 |
Understanding the importance of grad education in STEM jobs (LO10) | 6.52 ± 2.62 | 8.39 ± 1.33a | 6.48 ± 2.84 | 8.37 ± 1.35 | 6.56 ± 2.35 | 8.42 ± 1.29 |
Understanding the various career opportunities in STEM (LO11) | 6.04 ± 2.43 | 8.17 ± 1.36a | 5.41 ± 2.46 | 7.93 ± 1.29 | 6.76 ± 2.20b | 8.46 ± 1.38 |
Development of meaningful/productive mentoring relationship (LO2) | 4.60 ± 2.48 | 7.89 ± 1.82a | 4.07 ± 2.36 | 7.67 ± 1.70 | 5.20 ± 2.47 | 8.17 ± 1.93 |
All | Female/nonbinary | Male | ||||
---|---|---|---|---|---|---|
Question: Please indicate on a scale of 1 to 10 your response to the following questions | Baseline (n = 54) | Post (n = 54) | Baseline (n = 29) | Post (n = 30) | Baseline (n = 25) | Post (n = 24) |
Understanding the research process (LO3) | 4.83 ± 2.29 | 8.33 ± 1.09a | 4.21 ± 2.43 | 8.27 ± 1.09 | 5.56 ± 1.88b | 8.42 ± 1.08 |
My confidence in research skills (LO9) | 4.72 ± 2.29 | 7.98 ± 1.21a | 3.97 ± 1.92 | 7.63 ± 1.11 | 5.6 ± 2.37b | 8.42 ± 1.19b |
Understanding how scientists work on problems (LO4) | 5.11 ± 2.14 | 8.06 ± 1.16a | 4.55 ± 2.09 | 8.03 ± 0.95 | 5.76 ± 2.01b | 8.08 ± 1.38 |
Understanding how to design and implement a research project (LO6) | 4.41 ± 2.53 | 7.94 ± 1.22a | 3.93 ± 2.41 | 7.73 ± 1.24 | 4.96 ± 2.55 | 8.21 ± 1.15 |
The development of my identity as a scientist and/or engineer (LO8) | 4.85 ± 2.58 | 7.89 ± 1.44a | 4.07 ± 2.55 | 7.70 ± 1.19 | 5.76 ± 2.30b | 8.13 ± 1.67 |
Understanding the role of modeling in biomedical engineering (LO5) | 4.70 ± 2.44 | 8.09 ± 1.46a | 4.00 ± 2.41 | 7.87 ± 1.48 | 5.52 ± 2.21b | 8.38 ± 1.38 |
Understanding the importance of research experience for grad programs (LO7) | 6.48 ± 2.58 | 8.87 ± 1.04a | 6.45 ± 2.59 | 8.83 ± 0.93 | 6.52 ± 2.56 | 8.92 ± 1.15 |
Understanding the importance of grad education in STEM jobs (LO10) | 6.52 ± 2.62 | 8.39 ± 1.33a | 6.48 ± 2.84 | 8.37 ± 1.35 | 6.56 ± 2.35 | 8.42 ± 1.29 |
Understanding the various career opportunities in STEM (LO11) | 6.04 ± 2.43 | 8.17 ± 1.36a | 5.41 ± 2.46 | 7.93 ± 1.29 | 6.76 ± 2.20b | 8.46 ± 1.38 |
Development of meaningful/productive mentoring relationship (LO2) | 4.60 ± 2.48 | 7.89 ± 1.82a | 4.07 ± 2.36 | 7.67 ± 1.70 | 5.20 ± 2.47 | 8.17 ± 1.93 |
Indicates a p-value < 0.05 when comparing all students baseline to postprogram.
Indicates a p-value < 0.05 when comparing genders at baseline and postprogram.
Evaluation results show our program to be largely successful; 29 of the last 30 participants rated the experience as extremely or moderately positive and 23 participants said the program far exceeds expectations. Based on participant feedback, our professional development has been continuously improved over the last six years, which has led us to a newly proposed leadership development program. Social activities have been highly rated and contribute to cohort development.
Student Outcomes.
Another important outcome is career path following graduation, and more specifically helping students make informed decisions about STEM graduate education. Data suggest the program structure was effective at achieving this goal as well. 67.6% of graduates have pursued graduate education. Of those attending graduate school, eight have pursued STEM Ph.D., 10 STEM MS degrees, and seven professional degrees. Students have attended top-ranked institutions such as NC State/UNC-Chapel Hill, Michigan, Duke, Mayo Clinic, and Penn State. Ten students have pursued careers in industry, one is in the AmeriCorps program, and one is unknown. The remaining 17 students are still enrolled in undergraduate STEM programs; one student tragically passed away prior to graduation. Students have remained in contact with the program. Within the last year, our follow-up response rate was 98.2% for all cohorts.
Discussion
As a high-impact practice, undergraduate research experiences have the potential to reinforce STEM career decision and to positively influence pursuit of advanced degrees. Many REU programs share common goals to provide an authentic research experience and encourage graduate education with an emphasis on recruiting underrepresented populations. However, each site has unique objectives and varying evaluation plans. Rorrer et al. point out the challenge of balancing the evaluation needs of individual sites for site context with the need for broader more generalizable outcomes [25]. However, even if generalizable outcomes are used, it is difficult to determine the program elements that contributed to those outcomes. Further research is needed to link program elements to common indicators. This will allow new programs to build their program based on successful practices.
Over the past six years, the learning outcomes have remained consistent and include student reported measures of the following: (LO2) mentoring relationship; (LO3) research process; (LO4) how scientists work; (LO5) modeling; (LO6) conduct research; (LO7) Importance of research for graduate school; (LO8) scientific identity; (LO9) confidence; (LO10) importance of graduate education in STEM careers; and (LO11) STEM career opportunities. Students reported significant gains for all learning outcomes. Additionally, our data show no difference in baseline or gains for under-represented minority students compared to majority students. These findings are consistent with other studies [3,26].
Differences in baseline and gains based on gender were found. The increased gains closed the gender gap for all learning outcomes except for LO9 (confidence). This occurs despite an increased gain in confidence (3.8±2.17 versus 2.48±3.37 for males). Harsh et al. also reported lower baseline measures of confidence in women; however, greater gains were reported by female students thus leveling the playing field [27]. Based on gender schemas and accumulation of advantages, males tend to accumulate advantages through implicit factors and women and nonbinary students accumulate disadvantages through subtle imbalances [28]. URE provide a way to counteract accumulated disadvantages for women and nonbinary students. However, it is important to note that efforts should also be made to address the systems and structures that perpetuate these gender schemas. Self-confidence has the greatest impact on the persistence of women [29], while interactions with faculty have a significant effect on self-confidence [30]. We plan to examine potential correlations between mentoring relationships and self-confidence, which may explain the reported gender differences. The fact that confidence remains significantly below male counterparts is troubling. Though, this self-reported confidence may not be grounded in actual ability. In order to address this, we have proposed changes to our professional development plan to shift focus to leadership capacities.
The importance of leadership development has been highlighted by prominent engineering education entities including ABET [31] and the National Academy of Engineering [32]. According to the Engineer of 2020, “engineers must understand the principles of leadership and be able to practice them in growing proportions as their careers advance [32].” The new program will include several workshops focused on implicit bias, fostering inclusive environments, and cultural champions. These activities will be required for all students so that we can engage in conversations across genders and other identities. This change was made based on direct feedback from participants when we held single gender discussions.
The new NSF program solicitation (19-582) [33] underscores the need for inclusive excellence. The BME-SIM program seeks to foster an environment where all students can be their authentic self and learning and living occurs in an atmosphere of mutual respect in pursuit of excellence. To cultivate an inclusive environment, the BME-SIM program will follow these eight practices adapted from Ref. [34]: (1) recognize participants as whole individuals; (2) outwardly recognize and value differences; (3) use gender neutral and inclusive language; (4) provide mechanisms to address confidential concerns such as housing or travel; (5) set expectations for professional behavior; (6) require all REU mentors and research lab members to complete diversity and harassment training; (7) reduce isolation through program social activities and networking opportunities; and (8) help participants find resources on our campus. Creating a welcoming environment and sense of belonging will facilitate long-term communication.
Another important outcome is enrollment in graduate programs. The specific impact of the URE on pursuit of graduate education, compared to other motivational factors, is difficult to measure. Programs typically use intention to pursue graduate school as a proxy as it is short term and intentions are predictive of actual behaviors [35]. However, if participants are followed long-term the impact of their experience may be able to be teased out. Harsh et al. Interviewed advanced members of the research community (those who had already completed advanced degrees) to identify the two most important reasons for attending graduate school. 42% of respondents reported that research experiences were a major factor, and this was more so true for women than men [27].
Of participants who have graduated from our program, 51.3% have attended STEM graduate programs (includes ten men, nine women, and five under-represented minority students). 21.6% attended professional programs such as physical therapy, physician's assistant, and pharmacy school (includes all women and four under-represented minority students). This means that 40% of male participants have gone on to STEM graduate programs, 56.6% of female participants have gone on to graduate school with 30% in STEM graduate programs and 26.7% in professional programs, and 32% of URM students have gone on to graduate school with 20% in STEM graduate programs and 16% in professional programs. Note one student has attended both a MS program and medical school. One of the challenges of determining the impact of URE on graduate enrollment is the presence of underlying bias as these participants sought out and were selected for URE. However, the identification of a control group is extremely difficult due to a variety of factors. Another limitation of determining the impact of URE is the typically small sample size. Programs like the NSF REU sites are typically funded for three years and include 30 participants. Sites like ours that have been in operation for longer periods of time provide larger samples sizes. However, because sites are continually trying to improve the program, the experience varies year by year. To overcome these limitations, NSF directorates like CISE have established common indicators of student outcomes [25]. These common indicators include the following construct variables: self-efficacy, intent to pursue graduate school, attitudes toward computing disciplines (discipline specific), academic help-seeking/coping skills, research skills, scientific leadership skills, scientific identity, and mentoring satisfactions. Adoption of common evaluation methods may allow for improved interpretations on the impact of the experience.
The current COVID-19 pandemic situation has impacted the BME-SIM program this year and potentially in the years to come. Based on national and institutional guidelines, we have decided not to hold the program in 2020. We imagine that many programs are operating similarly, which may lead to upward of 7000 undergraduates missing URE this summer. Due to the nature of our program's research, we considered the idea of a virtual program. Most of the planned research projects could have been transitioned to virtual projects. However, based on the award notice timing and the transition to on-line learning in the academic year, we felt that the quality of the program would suffer if we moved to virtual delivery. In particular, we were concerned about recruitment, cohort formation, student motivation, and faculty burn-out. We have also proposed significant changes to our professional development program and were hesitant to transition the delivery without attempting in-person first. In preparation for 2021, we will develop contingency plans for virtual or hybrid implementation. This will include using communication platforms such as Microsoft Teams. Because the majority of our projects are computational and could be completed remotely, our program could serve as a role model for virtual implementation.
Lessons Learned
Based on our six years of experience, we report on some of our lessons learned and current challenges.
Recruitment.
A concerted effort is required to recruit our target students (female, URM, low socioeconomic status, and veteran students along with students with disabilities and those from institutions with limited research opportunities). Program announcements are sent to national organizations targeting underrepresented populations such as the Society of Women Engineers, Society of Black Engineers, Society of Hispanic Professional Engineers, American Indian Science and Engineering Society and to discipline specific listservs such as Biomedical Engineering Society and Biomch-L, Based on our experiences this has not been a very successful strategy for recruitment. Only Biomch-L is frequently listed when asking students about how they learned of the program and students listing this generally are coming from research intensive backgrounds. To better reach our target audience, we will expand our social media presence to include additional platforms. Borrowing from social media marketing strategies, we will try to identify and collaborate with influencers—those who can reach our target audience and drive engagement to promote the program. These influencers would be those who are known in the biomedical engineering or academic diversity communities with active and far-reaching social media accounts.
We have found targeted recruiting through partner institutions, and programs that recruit or target similar students (NIH MARC, McNair Scholars, and NSF LSAMP) to be successful. In-person recruiting through site visits and conferences has also been extremely successful. Webinars provide the opportunity for viewer engagement. The platforms that we have tried did not result in a live webinar, but a recording that could be distributed. In the future, we will use other platforms to allow for the live experience. An up-to-date website and active social media are also ways to attract potential applicants.
Mentoring.
It is important to have clear expectations for the faculty mentors. Because our graduate program is relatively new, graduate students are part of the mentoring program, but primary mentoring (both research and career) is accomplished by our faculty mentors. Mentors are required to meet with students once per week. However, based on faculty mentor and participant surveys, we have found that our mentors voluntarily meet with students, on average, 3–4 times per week. Thus, our mentors exceed expectations. Most meetings may be short and impromptu, such as answering a question, while others may be several hours for data collection, debugging, or software training. We recognize that this is a substantial commitment, but quality mentoring relationships are a program strength. This high level of interaction is facilitated by most mentor offices being located near or in their research lab space. To complement our current mentoring plan, we are investigating cultural competence training programs to help create a more inclusive environment. It is important that all students feel like they belong.
Appropriately Scoped Projects.
It is important to provide students with appropriately scoped projects to be completed in 10 weeks. This has involved some trial and error and reflecting on what has worked in the past. Faculty mentors must also be willing to adapt projects to fit the skills and interests of their students. Students report a mixture of hands-on and computational activities is most engaging. If a student's project is solely computational, the hands-on component can be added by participating in another student's study or helping with other lab projects. Modeling can be accomplished with limited resources making it possible for students to continue to be involved in their projects after the summer is over. Most students have access to matlab at their home institution or remote connections are available to East Carolina University's network.
Administrative Burden.
Organizing program logistics (housing, dining, stipend, hiring, travel, programming) can be time-consuming, especially if there is not a central office or staff member to coordinate. Researchers interested in developing REU sites should determine what resources are available on their campus. While this is a summer program, work is required all year long with recruitment in late fall/winter, application review in early spring, preparation for the program in late spring, and conference travel in early fall. For the last three years, we have hired a graduate assistant to help with some of these logistics. The graduate assistants improved the experience for participants by providing an additional role model. Also, this was an excellent training opportunity for the graduate assistant as they gained experience in mentoring undergraduate students.
Follow-Up.
In order to evaluate the long-term success of the program, all participants must be tracked through their undergraduate careers and postgraduation. For tracking to be successful, a strong sense of community must be established during the program. The BME-SIM promotes early formation of this cohort identity by planning preprogram activities such as a low ropes team-building course, tours, and several social activities throughout the first week. Attending BMES together, following the program, reinforces the cohort identity and allows the students to connect one last time. As many students will graduate soon, alternate contact information should be collected including permanent address, nonschool e-mail, and phone numbers. In our last program cycle, we added a LinkedIn group for our program alumni and current students. Follow-up surveys are sent by e-mail, LinkedIn, then text, and finally a phone call. While our follow-up has been successful (98.2% overall response rate), we are working to include other social media to communicate with our alumni. We are currently working to develop our Twitter page to use to promote successes from the group. Additionally, based on feedback from students, we recommend sharing research updates and achievements from their ECU mentors' labs so that students can see how their work has contributed to the bigger picture.
Conclusions
In conclusion, the results of six years of the BME-SIM program demonstrate success in providing an authentic research experience during which students gain valuable skills. We have provided a program framework, which can serve as a model for new programs and have shared lessons learned from our experience. More research is needed on the differential outcomes based on gender and on common assessments so that the collective impact of the URE programs can be determined.
Acknowledgment
We would like to acknowledge our faculty mentors, home institution mentors, and participants. We would also like to acknowledge our external program evaluator, Nathaniel von der Embse, Ph.D.
Funding Data
National Science Foundation (NSF - 100000149) (Grant Nos. EEC-1359183, EEC-1659796, and EEC-195050; Funder ID: 10.13039/100000001) awarded to Stephanie George and Zachary Domire.