How syllabi relate to outcomes in higher education: An evaluation of syllabi learner-centeredness and grade inequities in STEM
Data files
Mar 28, 2024 version files 16.26 KB
Abstract
Fostering equity in undergraduate science, technology, engineering, and mathematics (STEM) programs can be accomplished by incorporating learner-centered pedagogies, resulting in the closing of opportunity gaps (defined in this research as the difference in grades earned by minoritized and non-minoritized students). We assessed STEM courses that exhibit small and large opportunity gaps at a minority-serving, research-intensive university, and evaluated the degree to which their syllabi are learner-centered, according to a previously validated rubric. We specifically chose syllabi as they are often the first interaction a student has with a course and can serve to establish expectations for course policies and practices. We found that STEM courses with more learner-centered syllabi had smaller opportunity gaps. The syllabus rubric factor that most correlated with smaller opportunity gaps was Power and Control, which reflects the Student's Role, Outside Resources, and Syllabus Focus. This work highlights the importance of course syllabi as a tool for instructors to create more inclusive classroom environments, and the need for instructors to be supported by their institutions to create learner-centered courses.
README: How syllabi relate to outcomes in higher education: An evaluation of syllabi learner-centeredness and grade inequities in STEM
https://doi.org/10.7280/D1NH6N
DATA-SPECIFIC INFORMATION
Eslami_2022_How_Syllabi_Relate_to_Outcomes_in_Higher_Education.csv
Number of variables: 26
Number of cases/rows: 50
Variable names with descriptions and/or their values in parenthesis:
small_opportunity_gap; delta_GP (average grade point difference between minoritized and non-minoritized students in a STEM course); additional_item_Length_of_Syllabus (number of pages for each syllabus);
For the definition of rubric items, refer to the rubric designed by Cullen & Harris in 2009 published in Assessment and Evaluation in Higher Education journal; Rubric items are scored on a 5-point scale (0, 1, 2, 3, and 4): rubric_item_Accessibility_of_Teacher; rubric_item_Learning_Rationale; rubric_item_Collaboration; rubric_item_Teachers_Role; rubric_item_Students_Role; rubric_item_Outside_Resources; rubric_item_Syllabus_Tone; rubric_item_Syllabus_Focus; rubric_item_Grades; rubric_item_Feedback_Mechanisms; rubric_item_Evaluation; rubric_item_Learning_Outcomes; rubric_item_Revision
The 3 rubric factors are calculated as the average of the rubric items categorized under each factor in the rubric: rubric_factor_Community; rubric_factor_Power_and_Control; rubric_factor_Evaluation_and_Assessment (Community is the average of rubric items 1-3, Power and Control is the average of rubric items 4-8, and Evaluation/Assessment is the average of rubric items 9-13)
rubric_score_total; instructor_type (research faculty, teaching faculty, and lecturers); discipline (type of STEM discipline); average_PEER_representation (PEER= minoritized students); average_number_PEER; average_number_non_PEER (non-PEER= non-minoritized students); average_number_students
PEER = minoritized students including Black, Latinx, Pacific Islander, and American Indian students; non-PEER = non-minoritized students including White and Asian
Methods
This dataset is composed of rubric scores for 50 course syllabi of STEM classes in a research-intensive university with a large population of minoritized students as well as some institutional data (here defined as African-American, Latinx, Pacific Islander, and American Indian). We wanted to examine the relationship between racial grade gaps (here labeled as opportunity gaps) and the degree of learner-centeredness of the syllabi since course syllabi are good representations of classroom pedagogy according to the previous literature. The 50 syllabi were evaluated with a previously validated and peer-reviewed rubric designed by Cullen and Harris in 2009 and published in Assessment and Evaluation in Higher Education journal. The rubric measures the degree of learner-centeredness of syllabi. It has 13 items categorized under 3 factors plus the number of pages of the syllabi. We have modified the rubric to be on a 5-point scale (0-4). Zero represents the lowest degree of learner-centeredness and 4 shows the highest. The dataset includes the scores for the 13 rubric items, 3 factors, and the length of the syllabi (number of pages). Syllabus factors are calculated as the average of the rubric items that belong to a certain factor. The 50 syllabi were divided into 2 groups: Syllabi of courses with small opportunity grade gaps and those with large gaps (0 or 1 under the 1st column labeled small opportunity gaps). The gaps are calculated as the differences between course grades received by minoritized students vs non-minoritized ones on a 4.0 scale. All variable names in the dataset are clear and easy to understand. The dataset also includes institutional data such as information regarding instructor type, the number of minoritized students (PEERs), and non-minoritized students (non-PEERs) in a class.
Methods
Course syllabi were requested from the instructors by the study team. Instructors were contacted via email (as well as two follow-up emails for non-respondents). All instructors who responded were provided with the study information sheet, gave electronic consent for participation, and sent their course syllabi to a member of the research team. The syllabi were de-identified before analysis by eliminating any identifying names (of instructors, teaching assistants, and readers), emails, and course numbers to decrease bias during the coding process. We used a modified version of Cullen and Harris’s (2009) rubric to code the collected syllabi. There are 13 items included in the rubric: (1) Accessibility of Teacher, (2) Learning Rationale, (3) Collaboration, (4) Teacher’s Role, (5) Student’s Role, (6) Outside Resources, (7) Syllabus Tone, (8) Syllabus Focus, (9) Grades, (10) Feedback Mechanism, (11) Evaluation, (12) Learning Outcomes, and (13) Revision/Redoing. The 13 items are categorized under three factors: (1) Community, (2) Power and Control, and (3) Evaluation/Assessment.
Data Processing and Analysis
Syllabi were separated into cases and controls based on the size of the opportunity gap between minoritized and non-minoritized students for the respective course-instructor pairing. The difference in the grade points (on a 4.0 scale) received by minoritized and non-minoritized students is denoted as an opportunity gap. Syllabi in the large opportunity gap group (25th percentile) were designated as cases and syllabi in the small opportunity gap group (75th percentile) were identified as the controls. We used a logistic regression model using the stats and MASS package in R and performed the best subsets logistic regression using the bestglm package in R. We used a previously validated rubric to measure the degree of learner-centeredness of the syllabi. Two members of our group coded the syllabi after a 3-hour training session in which they reached 100% consensus.
The syllabi used for training were not used for study data. They scored the syllabi in sets of 10 and calculated the interrater reliability. There were calibration and consensus sessions after scoring a set of 10 syllabi. In only 3 cases where there was disagreement between the two main raters, a third rater also scored the syllabi. The Cohen's weighted kappa was calculated as a measure of the degree of interrater-reliability, provided in the preprint. The syllabi were deidentified before coding for fair scoring. The correlation matrix for the 13 syllabus items and also the 3 syllabus factors are provided in the preprint. Cronbach's alpha for the 13 rubric items and also for the 3 rubric factors was calculated and is provided in the preprint.