Measurement & evaluation for global health curriculum

wgha-500px

 

A statewide global health alliance that works to facilitate economic development collaborations and to engage, educate, and inspire the general public about global health supported the design and development of an interdisciplinary chemistry, algebra, and United States history curriculum organized around global health diseases. Twelve public high school teachers designed and implemented the curriculum in four high schools (two rural and two urban) across the state from 2009 to 2011.

GH-Careers-Poster1

Challenge:

Determine:

  • What did students learn from the curriculum?
  • How did the program influence high school students’ college and career readiness?

Improve:

  • Teachers’ ability to assess student learning

 

Solution:

Data2insight designed and implemented an appropriate evaluation designed to answer the key evaluation questions.

 

Results:

The written evaluation report was used to inform program funders and program leaders future grant proposals to continue the development of interdisciplinary education. Lessons learned were applied to program design and development. Program leaders also are in the process of publishing the evaluation findings, which showed that program participants had statistically significant learning gains compared to their peers in all three subjects.

The assessments designed by teachers were used both to evaluate what students learned from the curriculum as a whole and individual lessons. Furthermore, teacher-designed assessments were embedded in the final version of the curriculum now being used state-wide by Math Engineering Science Achievement (MESA) teachers.

 

Evaluation and learning assessment methods used:

Institutional Review Board exemption and approval

Western Institutional Review Board reviewed the evaluation methodologies for pretest-posttest and interim assessment secondary data analysis and approved an IRB review exemption request for that analysis. WIRB also reviewed and approved the process and impact evaluation study that included interviews with pilot school principals and counselors as well as a statewide global health alliance teacher focus group. For this study, consistent with the study protocol, all respondents voluntarily agreed to participate and their signed consent forms are on file. While results from interviews and focus groups are aggregated and study participants are not mentioned by name, confidentiality was not guaranteed due the small sample size (four principals, 10 counselors, and 10 teachers). Participants agreed to study results being published. Furthermore, summaries of the interview and focus group data have been shared with the participants for their review.

understanding following selected lessons in each of the three content areas administered during the academic year.

 

Interim assessment data collection

All statewide global health alliance teachers administered pen and paper interim assessments in their classrooms immediately following instruction of specified course lessons. One chemistry, math and history interim assessment was administered across all statewide global health alliance classrooms as a common interim assessment. These assessments were designed by teachers to measure student understanding of key learning objectives in the lessons being reviewed. Teachers scored these tests and included the scores as part of the student’s quiz grade.

Teachers mailed the completed assessments to a supervisory scientific agency where the tests were anonymized. A statewide global health alliance program staff member, masked to student identity, scored de-identified interim assessments using a scoring guide developed by statewide global health alliance teachers. The staff member then entered the test scores into a spreadsheet for analysis.

 

Field trip survey data collection & analysis

The statewide global health alliance staff administered the Site Exploration survey to students after the students completed of a half-day field trip including mini job shadows, conversations with the supervisory scientific agency scientists, tours of the agency’s research facilities and a hands-on lab investigation in the agency’s lab. This field trip took place at the supervisory agency’s lab in Seattle, WA. The pen and paper survey was designed to meet the following objectives:

a)    Assess student perception of the relationship between the statewide global health alliance course work and the Site Exploration activities

b)    Assess student understanding of the definition of global health

c)     Assess student interest in global health careers

Descriptive statistics were used to summarize student sample (N=242) demographics and answers to questions about course preparation and student perceptions about math, science, history and global health.

Student responses to the three open-ended survey questions were analyzed qualitatively. These questions asked students to define global health, share what is attractive to them about global health careers, and share their thoughts about Site Exploration activities. A coding scheme was developed based on the project objectives. All narrative responses were coded, using the constant comparative method, and emergent themes were identified.

 

Teacher, principal and counselor data collection

1)    Teacher focus group and principal interviews

The purpose of the focus group was to gain retrospective feedback from statewide global health alliance teachers about the statewide global health alliance program. Focus group discussion centered broadly on the following topics:

a)    Reflections about experiences during their 3 years of participation in the statewide global health alliance program.

b)    Which aspects of the statewide global health alliance curriculum were most effective.

c)     Which aspects of the statewide global health alliance program were least effective.

d)    Recommendations for future curriculum, instructional and professional development strategies and future teacher training.

 

2)    Counselor interviews

The purpose of the counselor interviews was to capture documented or perceived changes in statewide global health alliance student college readiness and/or global health awareness and interest. The counselor interviews were designed to provide the following data:

a)    Metrics of college readiness for Year 2 statewide global health alliance students included the following:

  • Colleges to which students were accepted
  • Whether or not a student completed a Free Application for Federal Student Aid (FAFSA)
  • Number and/or type of scholarships received by students

b)    Qualitative data about Year 2 and 3 statewide global health alliance students’ college readiness

c)     Qualitative data about Year 2 and 3 statewide global health alliance students’ interest in global health topics and/or careers

 

Pilot school principal, counselor interviews and teacher focus group data analysis

Principal, counselor, and teacher responses to interview and focus group questions were analyzed qualitatively. A coding scheme was developed based on the project objectives. All narrative responses were coded, using the constant comparative method, and emergent themes were identified.