There are two recent "events" which lead to my writing this posting. The first was serving as a Reader (grader) for the AP exam for the course I teach. The second was the concurrent release of Jay Mathews' Challenge Index, which uses a ratio of number of advanced placement tests taken divided by number of graduating seniors to create a ranking. The combination of these two events has lead to my seriously thinking about the impact of Advanced Placement in our public schools, and thus to the writing of this post.
I invite you to read the results.
At the beginning of this month I spent a week in Florida serving as a Reader. I read several thousand answers to one of the Free Response Questions on this year's examination. Learning more precisely how the test is scored will be of value as I replan the course for next year. But that experience was eye-opening in many ways.
First, let me explain the structure of the examination for the AP course I teach. There are two sections. The first consists of 60 selected response (multiple choice) questions which, like the SAT, has a correction for guessing. Without totally explaining that correction, let it suffice to say that random guessing mathematically would lead on average to no raw points, but elimination of at least one answer per question puts the test taker ahead of the game. The students have I believe 45 minutes to answer the 60 questions, which represent half of the raw score.
The second section consists of 4 "free response questions" (FRQs) which are NOT essays - there is no credit for how well written, nor for topic sentences or conclusion. A rubric is established for each question, anchor papers (sample answers) are culled to guide the Readers who score them, and there are lists of what is and is not acceptable for each point of the rubric. While the questions may have varying number of points in the rubric, in theory from 4 to 10 (the highest this year was 6) the FRQs weight equally as 1/8 of the total value of the test. Students have 100 minutes to answer the 4 FRQs.
Our test had over 170,000 sets of papers. We had around 600 readers. We had in theory over 700,000 individual questions to answer, with each reader reading only one of the four questions. We are trained in multiple stages to ensure consistency, with multiple sets of eyes looking at each paper early on, then as we become more 'reliable' as scorers, a decreasing amount of downstream checking - by table leaders and/or question leaders. By then of the scoring period, because I read very quickly, I had read something over 2,000 sets of papers. Well, that is, I had processed over 2,000, since on more than a few students chose not to answer that question, or in some cases, any question, and/or offered nonsense information. Our question had a rubric of 6 points, and the average rubric score was around a 2.7.
I can imagine some of you asking why a student would sit through the 100 minutes of the test section for the FRQs and write nothing at all, or perhaps instead write deliberately totally off topic. Perhaps because the student was required to take the test by her school? Why? That leads to the challenge index.
Jay Mathews invented the Challenge Index, which rates high schools using a calculation of total number of Advanced Placement Examinations taken the previous year in each school divided by the number of graduating seniors. A ratio greater than 1.0 gets one on the list. Our high school has been on the list since Mathews first released it about a decade ago (and you can visit this page at the Washington Post for more information). The intent was to find those high schools which challenged their students. Mathews decided to use this measure as the result of an article by Cliff Adelman which argued that even sitting for an AP exam, whether or not one passes the exam, better prepares the student for college work. Here I note that the passing rate on the exam has no effect on the calculation of the index (and thus on the rating of the school). In the past Mathews has tweaked the calculation, in some cases uses International Baccalaureate (IB) exams as well as AP exams, and in some cases sitting for courses that are cross-registered with local institutions of higher education where a college level examination is required.
Mathews, for those who do not know his work, wrote a book on the achievements of Jaime Escalante at Garfield HS in Los Angeles, challenging his largely inner city Hispanic kids to take AP Calculus - you may well have seen the film that resulted, "Stand and Deliver," starring Edward James Olmos as Escalante. In introducing the index in 1998, Mathews wrote
Nearly every professional educator will tell you that ranking schools is counterproductive, unscientific, hurtful and wrong. But I am going to do it anyway, not because I believe my system is scientifically infallible, but because I think it provides insight into one of the most significant emerging issues in American education: whether our high schools are working hard enough to challenge and elevate students.From his experience as a reporter and as a parent, he began to question the restrictions some schools and systems used to limit enrollment in AP courses, which he viewed as more challenging than the course students would otherwise take. It is worthwhile to read the essay linked above, because it explains the rationales Mathews offers both for creating the index and for ranking the schools.
Here I should disclose several things. I got to know Jay Mathews about the time of the first index being published, although not for that reason. I had read a book he had written largely about a school attended by one of his children, and it was about my alma mater, which led to contact between us. Although I often find myself in disagreement with Jay, we developed a friendship and I have appeared in his column on several occasions. Thus I find myself conflicted because while I admire the intent behind the index, I find it destructive and misleading.
Let me first note the following - in the 1998 index the school at which I teach Eleanor Roosevelt, was ranked 16th in the DC metro area with an index of 1.173. That also, if memory serves, put us in the top 100 in the nation, excluding schools that got to totally select their student body (we admit about 1/3 of ours by exam to our science and tech program). In 2008, we have fallen to 82nd in the area, even though our index score has risen to 1.289. Having been at the school since 1998, I can assure you that our rigor has not decreased in any way, either on an absolute basis or in comparison to other schools in the region, even though many more now offer many more AP courses than before. It is the explosion of advanced placement that leads to the different results. And I believe that one can argue that explosions results directly from the Challenge Index. And I question whether it is the right course for our schools to follow.
Our school system, Prince George's County (MD), now requires all high schools to offer a base of 8 AP courses. The system pays for exams, which are required, because we give a weighted grade for an AP course - an A is worth 5 versus 4 for any other course, a B is a 4 versus a 3, and so on. At least now the teachers of the AP courses are required to receive appropriate AP training - that was not the case when we began to offer AP Gov in 10th grade. When we moved the course from 9th to 10th so that students would have had all of American History before taking government (previously they had the 2nd half in 11th grade, 2 years after taking government), I argued for allowing students to take AP in lieu of the regular course of Local, State and National Government. Our then superintendent decided all 21 high schools should offer at least one section of AP, but only 4 teachers,2 from our high school, were appropriately trained to do AP.
Many school systems began expanding their AP offerings after the Index first appeared. The College Board, which controls AP, began to question the validity of such courses, in part because of how many of the increasing number of exams scored so badly - here I will note that in 2007-08 academic year, in order to receive a passing score of 3 (out of a possible 5) on the exam, students needed less than 50% of the raw points. A few years back those wishing to use the AP designation on a course had to submit a syllabus to ascertain whether what was being taught was in fact at the college level. My syllabus was immediately accepted. Most of the AP government teachers in our system submitted a common syllabus prepared centrally, which was rejected as not meeting the requirements of the program.
Let me add one additional note: many colleges will limit the number of credits a student can receive via AP or IB, and will for AP courses only give credit for the maximum score of 5 on the zero to 5 scale. Now imagine you are a student who will graduate with ten or more AP courses, and you are required to sit for the AP exam in each. Why bother preparing for the exam, right? That is in part the explanation for some of the blank booklets or totally non-responsive answers we encountered.
What was far more troubling as a reader was seeing the number of answers where the student was attempting to answer the question but was not responsive to the prompt of the question, could not present a coherent series of statements, and/or lacked basic understanding of the material.
As noted, I read several thousand answers to one question. The rubric had six points. The number of papers I read which met all 6 points was in the single digits. And it is not that the rubric required us to be all that finicky in awarding points. Remember also that our average score was around a 2.7 out of those 6 points. And remember also that last year on our exam one needed less than 50% of the raw points to receive a "passing" scaled score of 3 (out of 5). If you find something wrong with this picture, so do I. There are far too many sitting for the AP examination who are not, as demonstrated by their FRQs, capable of working in this subject at a college level.
One can certainly argue for students being challenged. I do not believe an Advanced Placement course is the only way to accomplish that. In fact, I have found teaching AP to 10th graders has in some ways restricted my ability to challenge my students as much as I did when I taught the course to 9th graders, and did not have to worry about "coverage" - the amount of material my students had to learn to be prepared for what might appear on the AP exam. To give just one example, there is no defined universe of Supreme Court cases that could appear. I cover in some form or other almost 100. Yet this is not a course in legal history or in the Supreme Court.
Mathews does not currently include cross-registered courses in the calculation of his index. We have students who run out of the math, including Calculus B/C, sometimes by their sophomore year, often by the junior year. This is one case where we offer courses through cross registration with local universities. We have a man from Catholic U who teaches Calculus III and Differential equations. These course do not get a weighted GPA, nor do they count as part of the Challenge Index. Thus even though we are definitely challenging these students, the Index gives us no credit for doing so.
The students in our Science and Technology program can, if they are well prepared, do a Research Practicum in lieu of 3 of their courses as senior. They receive a grade, equal to 3 courses, but it is not weighted. They are working in laboratories with real-word scientists, often doing original work. They might be at the USDA lab in Beltsville MD, or working with someone in a university science department. They are doing work that is beyond the freshman level of college in its demands and challenges. Were they not involved in this program, most would be taking at least two and in some cases 3 more AP courses. The Index gives our school no credit for our challenging our students in this fashion.
I could go further. One of the most demanding courses in our school is Genetics, taught by a man with a Ph.D. It is the equivalent of a college level genetics course. The students do not receive a weighted grade. Nor is participation in the course included in the calculation of the index.
We have active parents at Eleanor Roosevelt. Some have argued with Mathews on both the calculation of the index and the concomitant ranking of schools by their scores. Mathews argues that any school with an index of >1.0 is already in the top 3% of schools in the nation, so that the rankings are not all that important. Then why publish rankings? I can remember when I was exploring teaching at a school in another system. The day I met with the Assistant Principal was the day the index was published, and the school in question was listed in the top 20 in the nation. As I was coming in to his office he was arranging to have the list reproduced and distributed.
We are obsessed with rankings in this nation. And that can lead to distortions. One of the more egregious examples of this has been in the rankings of Colleges and Universities, especially that done by US News and World Reports. As it happens, there is an op ed in today's Boston Globe entitled Our obsession with college rankings which came about in part because of how Clemson University successfully improved its ranking - they limited classes that would otherwise have 24-25 students to 19 in order to have a higher percentage of class with less than 20 students, which meant that the remaining classes often expanded to more than 50.
If you know how an index is calculated, you can do things to manipulate the score you receive. We have seen this outside of education, for example, in ratings of cities as places to live. All of these are yet again example's of Donald T. Campbell's 1976 postulate that
The more any quantitative social indicator is used for social decisionmaking, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.One statement from the Globe piece, while written with respect to universities, is I believe quite applicable to this discussion:
It is the public’s obsession with the numerical rank, and not the actions of US News or schools like Clemson, that misleads us into believing that some schools are innately better than others and pressures students into dreaming of schools that are not right for them.We have an obsession with reducing complex situations to single numbers so that we can do comparisons that provide much less meaningful information than the weight we give the results of those comparisons.
I am not opposed to Advanced Placement. I would not continue to teach my three sections, which next year may contain over 100 students, if I did not find value in the program. I know that some students sign up for the course for the weighted grade. Others do hope to receive the college credit. And a few sign up because they want me as a teacher, particular those who are younger siblings of students I have previously taught - although in a few cases that decision is made by the parent rather than the student. I always have some students who really are not prepared to do the work required. Still, I believe that I am able to stretch the vast majority of my students, and I always especially enjoy those who stretch me as a teacher - that helps keep me fresh. Insofar as I am challenging my students, Mathews is right about his emphasis on AP, although I believe I could challenge them even absent the AP designation.
And I do not disagree that one measure of a school is how much it is willing to challenge its students. I do not limit who can take my class. We do require a recommendation from the current social studies teacher, and check English grades and overall academic performance. Even in the few cases where a student is not recommended for AP, or recommended with reservations, the parent can override that recommendation and insist on the student taking the class. AP can provide a framework within which good teachers can find ways to challenge themselves and their students.
Still, I remember the large number of answers I read and rated during that week in Florida, and wonder if we may be overdoing it? I have had brilliant students, some of whom graduate with 14, 15 or more AP courses, with 5s on all their exams. So long as they have time to still be teenagers, to participate in activities like music, theater, sports, or simply hanging with friends, I don't object. I worry that we may push some into academic endeavors at the expense of developing the students in other ways.
If the reason is to improve how the school "ranks" on an artificial index, such a use of AP is truly a misuse, and may represent an abuse - not only of the academics, but of the students. If you think the word "abuse" too harsh, then perhaps substitute "academic malpractice."
I will continue to teach AP because it enables me to challenge some very bright students, and to experience how their minds and insights can challenge me, stretch me even further as teacher and as person. I will also continue to teach ordinary kids, those who might not be considered as academically able, because I know that they can surprise me if I can find the way of connecting them with the subject matter. Every year I get a couple of students who took regular government with me in a non-honors class as a sophomore who decide to come back and take AP as a senior or even as a junior.
I read over 2,000 answers. Less than a dozen got all 6 points on the rubric. Some of the papers I most enjoyed reading did not get all 6 points, but showed a depth of insight that blew me away. Reading so many papers could be mind-numbing, except so often I encountered students who were willing to stretch themselves. That is a proper purpose of Advanced Placement. AP is not the only way to accomplish that, but it can well serve that purpose, properly applied.
I do hope we do not continue down the path of our current obsession, believing that more AP is inevitably better. The quality of the instruction is not necessarily better nor more challenging merely because the course has an AP designation - after all, one can have the most wonderful syllabus and still not be able to communicate its contents to the students, nor to engage the students in the process of learning.
And I wish, almost certainly futilely, that we would stop distorting the meaning of AP by using it as a means of 'ranking' schools.
My school year is now officially over. This summer I will be working with students at the other end of academic performance, students needing extra help to meet state requirements in government to graduate from high school. I wanted to complete this reflection before I mentally transition next month.
All of my students deserve the best teaching I can give them. All students are capable of learning, of being stretched. I expect that this summer will stretch me as a teacher and person, albeit in a different fashion.
As a teacher, I find that I must take time to look back, to reflect, to learn. This essay is a part of that process. I hope it is of value to at least one person who encounters it.
Peace.
