Our previous research [Kost et al., Phys. Rev. ST Phys. Educ. Res. 5, 010101 (2009)] examined gender differences in the first-semester, introductory physics class at the University of Colorado at Boulder. We found that: (1) there were gender differences in several aspects of the course, including conceptual survey performance, (2) these differences persisted despite the use of interactive engagement techniques, and (3) the post-test gender differences could largely be attributed to differences in males’ and females’ prior physics and math performance and their incoming attitudes and beliefs. In the current study, we continue to characterize gender differences in our physics courses by examining the second-semester, electricity and magnetism course. We analyze three factors: student retention from Physics 1 to Physics 2, student performance, and students’ attitudes and beliefs about physics, and find gender differences in all three of these areas. Specifically, females are less likely to stay in the physics major than males. Despite males and females performing about equally on the conceptual pretest, we find that females score about 6 percentage points lower than males on the conceptual post-test. In most semesters, females outperform males on homework and participation, and males outperform females on exams, resulting in course grades of males and females that are not significantly different. In terms of students’ attitudes and beliefs, we find that both males and females shift toward less expertlike beliefs over the course of Physics 2. Shifts are statistically equal for all categories except for the Personal Interest category, where females have more negative shifts than males. A large fraction of the conceptual post-test gender gap (up to 60%) can be accounted for by differences in males’ and females’ prior physics and math performance and their pre-Physics 2 attitudes and beliefs. Taken together, the results of this study suggest that it is an accumulation of small gender differences over time that may be responsible for the large differences that we observe in physics participation of males and females.