School reform efforts across the US have focused on creating systems in which all students are expected to achieve to high standards. To ensure that students reach those standards and to document what students know and can do, schools collect assessment information on students' academic achievement. More information is needed, however, to find out when such assessments are appropriate for English learners and can provide meaningful information about what such learners know and can do. We describe and discuss a study that addresses the question of when it is appropriate to administer content area tests in English to English learners. Drawing on the student database of San Francisco Unified School District, we examined the effect of language demands on the SAT/9 mathematics scores of Chinese-speaking and Spanish-speaking students. Our results showed that while the English language demands of the problem solving subscale affect all students, they have a larger effect on English learners' performance, thus rendering the tests inaccurate in measuring English learners' subject matter achievement. Our results also showed that this effect gradually decreases as students become more proficient in English, taking five to six years for students to reach parity with national norms. These results have important implications for the design of school accountability systems and policies with high-stakes consequences for English learners such as high-school graduation requirements based on standardized tests.