Debate over test data signals serious approach

Published in The News Tribune, September 5, 2004

 The recent controversy over charter schools is something to welcome. While both sides of the dispute overstate what recent national test scores do and do not say about the effectiveness of charter schools, the debate over data indicates we may finally be getting serious about education policy.

The controversy started when the American Federation of Teachers (AFT) announced an apparent coverup by the U.S. Department of Education (DOE).  The AFT charged that the DOE had made data unavailable on the performance of charter schools showing that these schools were “underperforming”.

At issue was the (undisputed) fact that students in charter schools received lower scores on the National Assessment of Educational Progress (NAEP) test, than did those students in traditional public schools.  The NAEP is a well-respected national test given each year to a sample of 4th, 8th and 12th graders, and is used to gauge the academic abilities of our nation’s youth.

Generally, such pronouncements by the AFT are advanced and refuted off the front pages.  Amidst a very close Presidential election, however — with Bush an enthusiastic supporter of charter schools — the story wound up on the front pages of the New York Times.  This was immediately followed by an outcry from researchers and charter school proponents decrying the AFT’s misleading use of data, purportedly for its own political agenda.  They rightly charged that the AFT failed to examine factors other than the public school/charter school difference that might explain test score disparities.

Debate over what the NAEP scores reveal or fail to reveal, illustrates the challenge of education policy:  it is very hard to figure out for sure what policies are best for our children’s education.  But it’s not impossible!  The AFT made two crucial mistakes in its portrayal of what the data say – mistakes that unfortunately are made all too often in education.

Test Scores Say Little About School Performance   The data released by the AFT shows that 50 percent of 4th grade black students in charter schools, versus 46 percent in other public schools, have below basic-level math skills.  Among low income students, the gap is larger:  47 percent are below a basic level in charter schools, versus 38 percent in other public schools.

To the extent that an NAEP score offers an accurate portrayal of a student’s achievement, it represents the abilities and knowledge that a student has accumulated since birth.  If we want to account for the role of an individual school in this ability level, we would want to know about changes in ability during a student’s tenure at a school:  test scores reflect ability while changes in test score reflect learning.

Since education is about learning, assessing the effectiveness of our educational system should involve assessing its role in changes in ability.  To compare charter schools with non-charter schools, we would thus want to compare how much students learned in each environment, not how much they know.  While an NAEP score tells us the latter, it tells us very little about the former and thus, is not very useful in answering the question:  how do charter schools compare with non-charter schools?  Yet this is precisely what the AFT did.

While learning shouldn’t be confused with ability, we unfortunately confuse the two all the time.  The federal No Child Left Behind legislation assesses schools – and labels many as failing — based on students’ test scores (their ability).  But an individual school plays a small role in this overall score.  Examining changes in test scores is a more accurate way to assess learning and the effectiveness of different schools or policies.

Peer Effects Matter  Critics of the AFT also argue that apples should be compared with apples.  A black or poor child in a charter school should not be compared with a black or poor child in other public schools without controlling for peer effects.  That is, an important factor determining how much an individual child learns is who her peers are. Children learn more if they are with higher achieving, more motivated peers, factors that (unfortunately) are related to the socioeconomic composition of schools.

Adjusting the AFT data for the socioeconomic composition of charter schools versus other public schools leaves little difference in test scores.  The implication is that the lower test scores found in charter schools can be attributed to the greater challenges they face from the populations enrolled in them, and not to any intrinsic weakness or failure in the charter schools.

Neglecting the importance of peer effects is a common error we make in assessing our schools.  Almost everyone would agree, for instance, that private schools offer a better educational opportunity than do the public schools.  A common refrain is “why can’t the public schools be as good as the private schools?” Yet once you take into account the “peer effects” of the public schools, one finds that there isn’t much difference between public and private schools.  That is, children do learn more in private schools, but this is because of the characteristics of children (and their parents) who attend these schools, not to any particular advantage that private schools per se bring to education.

In the end, what the public wants is an excellent school system, however that might be achieved.  But in truth, there is no consensus as to what would lead to such a system.  The only way to find out is to carefully examine data comparing one policy – such as charter schools – with another.   The great misfortune of education policy in the United States is that to date, policy has not paid much attention to data and research, at least not in a way that is very instructive.

And so, believe it or not, a very public debate over the intricacies of data analysis is something to cheer!  In the end, improvements in our public school system will depend on the public’s understanding of how test scores can be used – and misused – and on less rhetoric and more hard evidence from our education leaders.