Designing evaluation for a high school research integrated learning outreach program

Abstract

BACKGROUND

Educational outreach programs that bridge university and high school contexts aim to increase students’ engagement and commitment to STEM career pathways (Tytler et al., 2017). Likewise, opportunities for students to engage in authentic scientific research (research integrated learning, RIL) have also been demonstrated to develop STEM skills and lead to increased STEM identity and aspiration (Beier et al., 2018; Stets et al., 2017). Outreach programs may also support access to STEM pathways for students from diverse backgrounds (Scull & Cuthill, 2010). However, these positive outcomes cannot be assumed from program design alone. Rigorous evaluation is required to ensure that goals are being reached and to support iterative improvements in program design (Australian Academy of Science, 2019).

AIMS

This study presents the design and preliminary results of an evaluation of a high school outreach program, SciX (this intervention is described in another ACSME 2023 abstract by Laura McKemmish). The evaluation approach focuses on the program’s impact on students’ perceived scientific skills, science identity, and commitment to pursue a career in STEM. It also explores differences in these effects between demographic groups (girls, students from rural locations, students from low SES schools). Other program goals including sustainability, scalability, and a positive impact on teachers and mentors are also incorporated. The goal of this research is to support iterative improvement of the intervention and identify transferrable principles to enhance the effectiveness and equity of similar student research programs.

DESIGN AND METHODS

An evaluation framework was designed around a program logic model that also incorporated elements of motivation theory and identity theory. Student surveys (University of New South Wales Human Research Ethics approved) were administered immediately before and immediately following the one-week RIL summer school in 2021-2023. A total of 238 students completed a survey following the intervention.

RESULTS AND CONCLUSIONS

The full evaluation framework will involve mixed methods studies that incorporate survey and interview data from students, teachers, and mentors, as well as key informant interviews and program administration data. Preliminary results will be presented from student surveys. These results show positive experiences (73% of respondents were extremely happy they attended) and significant increases in science identity, which were greater for girls. Survey results also reveal the key importance of the student-mentor relationship to the experience, which will be further explored in later studies. Initial results have informed program design and future iterations of the evaluation research.

REFERENCES

Australian Academy of Science. (2019). Women in STEM Decadal Plan. www.science.org.au/womeninSTEMplan

Beier, M. E., Kim, M. H., Saterbak, A., Leautaud, V., Bishnoi, S., & Gilberto, J. M. (2018). The effect of authentic project‐based learning on attitudes and career aspirations in STEM. Journal of Research in Science Teaching, 56(1), 3-23.

Scull, S., & Cuthill, M. (2010). Engaged outreach: using community engagement to facilitate access to higher education for people from low socio‐economic backgrounds. Higher Education Research & Development, 29(1), 59-74.

Stets, J. E., Brenner, P. S., Burke, P. J., & Serpe, R. T. (2017). The science identity and entering a science occupation. Social science research, 64, 1-14.

Tytler, R., Symington, D., & Cripps Clark, J. (2017). Community-School Collaborations in Science: Towards Improved Outcomes Through Better Understanding of Boundary Issues. International Journal of Science and Mathematics Education, 15(4), 643-661.