Reading the opinions and rants of various math curricula, I see how hard it can be to evaluate the strengths and weaknesses of a given math textbook. Is Saxon better than Everyday Math? What’s the best intervention for pre-K? When someone tells me that my school’s math book is “bad,” should I believe them?

A place to look for real data is here: the US Dept of Ed’s What Works Clearinghouse, which evaluates studies, compiles data, and compares the results. While it’s far from complete–many curricula have never been tested at the scale that the Clearinghouse requires–it gets us out of the muddy territory of personal anecdote and into data driven decision making. To answer our questions:

**Is Saxon better than Everyday Math?**

No. It looks like the opposite is true. In fact, Everyday Math is one of the few math curricula that seem to have a positive effect on children’s understanding of math. Saxon is mixed, and in the upper grades, seems to be downright negative.

**What is the best intervention for Pre-K?**

According to the Clearinghouse, the SRA Real Math Building Blocks PreK is the clear winner. If you do a little research into what it’s about, you’ll have ideas for what types of activities work to get 4 year olds on the right track in math. Here’s a quote from some press on the authors:

Douglas H. Clements and Julie Sarama have spent the past decade developing a curriculum that seeks to cultivate young studentsâ€™ math skills through the types of games, artwork, songs, and puzzles that those children enjoy, as well as through computer software.

**When someone tells me that my school’s math book is “bad,” should I believe them?**

Nope. For some reason, math education is one of those areas where even the brightest, normally scientific-minded people get swayed by personal anecdotes and cherry-pick data to suit their own personal needs (like wheresthemath.com promoting a 2009 study that found Saxon and Math Expressions to be more effective than Investigations and Addison-Wesley, then ignoring the updated, much more comprehensive study that found the four curricula not statistically different for first graders (though Saxon does beat out Addison-Wesley in 2nd grade. My cursory reading of the study leads me to be pretty skeptical of Addison-Wesley as a text).

Be skeptical. Doubt the peddlers of magic bullets. Look at the data. These are hard questions, and the decisions ahead about math education are too important to be made from a position of fear and vaguely informed opinions.

“Math Expressions,” which has recently been adopted by the Northshore School District (where my Pre-K and K children attend) is not included in the database.

So additional questions parents might ask are:

What does it mean if my child’s math curriculum was not included?

What does it mean to include interventions with “small evidence?”

So far I like what I’ve seen of the new math program, but also consider our family more math-saavy than typical. What works for some families may not work for others.

Unfortunately, a lot of the current curriculum hasn’t been studied at the scope necessary to get it on the Clearinghouse. And, as you point out, this is a statistical measure anyway. If you have a student you’re trying to pick a curriculum for, their individual response to a particular curriculum should matter more. If you’re making policy decisions (what book should my district be using?), then I think this data is more useful.

It can also help provide support for different pedagogical perspectives.

It does seem that for a certain group of teachers and students, one particular curriculum may be more effective, but unfortunately the reporting of such studies often causes parents to unduly worry about the curriculum their child uses and the effect it has on their academic performance. In reality, there are so many factors at play not least, teacher ability/enthusiasm, students’ learning style/ability/enthusiasm and parental support. I agree that reports like this, although useful, should be used and reported on cautiously.

Really good point, Caroline. In fact, my real motive for writing this blog was to respond to the hysteria around the sense that some curriculum is “bad” and needs to be replaced by the “good” one. One frustration is that the people who are doing that often are in favor of the worse curriculum. But the deeper problem is the point you bring up: learning is multifaceted, and too much focus on any one part leads to the magical thinking that a single change will solve every educational problem perfectly.

You left out a key piece of information: Of the 72 studies which were evaluated by WWC for Everyday Math, only 1 met their evidence standards and that 1 was with reservations. I would not venture to form an opinion about any text based on 1 study, even if it was found to have positive effects.

Deanna: thanks for that tidbit. You’re absolutely right that you shouldn’t form an opinion based on one study. However, given the high evidence standard that leads them to disqualify the other 71 studies, presumably the study that qualified has some statistical relevance. So it should, possibly, inform your opinion, if not form it.

The larger point is that it’s easy to have your mind made up before you look at the data, and pick and choose as it suits you. Data should make you less sure of your preconceived ideas, and open your mind to the possibility that there’s more to the story than you realized.

The one study you quote for EM says it has “potentially positive effects,” not that it offers a complete “positive.” Secondly, that particular study involved students who were white, upper-middle class, and had been using reform math materials prior to the study. You need to find an equal study of EM across all sub-populations of math learners.

Thus, using ONE very narrow study to say any curriculum is better than another is silly. In addition, education research in general has taken some major hits as being unscientific and weak in its calculus for 50 years, which included the timeframe in which the EM and Saxon studies were conducted. Only in recent years has WWC admitted the fact that their research methodologies were producing unverifiable data.

The one way for parents to determine if their child is in a “bad” math curriculum is to ask the child in the 5th grade if he/she can recall multiplication facts through the 12′s with no hesitation (automaticity), if they can add and subtract without counting on their fingers (also with automaticity), and if they can use standard algorithms (which will be needed in high school and college). Can they pass an on-level entrance math test without a calculator with at least a grade of 80?

My students on an Indian reservation and then in an all-white Seattle, WA, elementary school could do these tasks because of Saxon Math. For the first time, some of my Indian kids actually enrolled in college. My high school students in Seattle , however, who came from a reform math background were unable to take advance math and science courses, which greatly reduced their career opportunities.

Saxon research isn’t anecdotal. It’s been irrefutable and verifiable since it first hit the market in 1981. You might try reading the biography, John Saxon’s Story, a genius of common sense in math education.

Niki: It would be silly to judge one curriculum as superior to another based only on this kind of statistical evidence. On the other hand, it seems equally silly to toss out this evidence because it doesn’t support your views coming in.

I’m inclined to agree with KimS (below) on the greater importance of teachers over curriculum. I’ve kept myself away from the “math wars,” in large part because I’m not particularly convinced that a win from either side would have much of a positive effect on the ground. I’m not saying that EM is better than Saxon. My point is that it seems far from clear that Saxon is better than EM.

You’re clearly a big fan of Saxon (given that you wrote a book about him and all), and just for the record, I’ve got no axe to grind with the man or the curriculum. In fact, if you have research or data to steer me toward (anything available online?), I’d love to see it. I have no preconceived notions about which curriculum is better than which. (Though I should say that my high school used Saxon when I was a student, and it struck me at the time as an incredibly boring, unmotivated curriculum. I imagine the elementary level stuff is better.) The fact that you’ve used Saxon to such a good end is great.

Even though we’ve got a math war on our hands, I’m going to continue to try to be a moderate, and take the best all sides have to offer. Watching what should have been a conversation degenerate into a “math war” has been dispiriting, to say the least.

I can’t agree with your method of assessing curriculum via your individual child though. Any curriculum, any method you put in place–especially at a systemic level, is going to work for some kids and not for others. Everyone will always be able to show the kid their preferred curriculum works for, or the kid that their opponent’s fails. I also don’t think your points of assessment are complete. As a mathematician and teacher, I’m a huge fan of kids knowing their multiplication tables, knowing how to add and subtract, and having command of algorithms. When I’m trying to assess where they stand in math, though, I’m interested in bigger questions: can they generalize, simplify, question, recognize patterns? Do they know when to multiply, divide, add and subtract, and not just how?

In lieu of the fact that so few textbooks have been “assessed” by the WWC because the studies on their effectiveness don’t meet WWC standards or they’re simply too new to have completed studies, it’s difficult to place any degree of reliance on the data. If you’re trying to choose between Investigations, Math in Focus, and enVisions you’re out of luck as none of the programs has anything in the WWC.

One factor I consider important in considering any textbook series or instructional program for use in a public school setting is teacher professional development. Are the PD courses for the instructional program focused on the instructional philosophy or the mathematical content?

So many PD programs these days are strong on philosophy and weak on content. With so many elementary teachers openly admitting that they are “math phobic” and were poor math students, content poor PD courses put their students at a disadvantage.

A teacher who never got math and still doesn’t get it or like it much, will follow the instructional philosophy without really understanding why she is teaching what she is teaching. She’ll move her students along to the next unit as per the program dictates because she doesn’t have sufficient knowledge of the subject to determine whether her students are ready to move on or how to adapt instruction outside the proscribed program. Teachers with strong content knowledge can determine the objective of each lesson, assess whether her students really understand the content well enough to proceed to the next level, and adapt instruction as needed to meet her students’ needs.

KimS: I absolutely agree. I taught a few Math for Elementary Teachers courses as a grad student, and for many I think it was the first time they got to actually DO math (at least in a context that didn’t entirely alienate them). I’d guess that having elementary teachers who know math and are comfortable doing it is primary factor in kids learning math.

Nicki Hayes has some good points, though. The state of education research is deeply murky–there are very few really solid studies… Possibly none. There are a lot of poorly done studies. Most of the studies find that the [new curriculum] is better than what it’s being compared to by a small but statistically significant margin on one criteria, and is not statistically worse on the other criteria. Interesting, but not convincing. Something I’ve noticed is that almost all of these studies are done to look at the effects of changing from [old curriculum] to [new curriculum], and there is very little that lasts beyond the first 2 years of implementation. I have spent some time at the WWC, and I’m not convinced that there are any answers to be found there.

I almost have an opinion in the math wars, and I’m getting there through reading curricula. It’s my job (I teach math for elementary ed relatively regularly) to know what’s being done by whom at what grade level. It’s an interesting perspective, looking for: how does each series teach multiplication, and a what grade level? Fractions are even more telling (they are more often done poorly). If you want to form your own opinion, I recommend that you pick a math content topic that you care about (multiplication, fractions, whatever), and go to a university library and sit down with several different textbook series, and see what they do with it. It helps even more if you work with children of about the right age to be using those textbooks (helps you imagine better how the lessons would play out). That’s my strategy, and until the state of education research improves, it’s the best I’ve found for comparing apples to apples among text series.

LSquared: I think your strategy of informing yourself about the specifics of each curriculum is the great way to go, especially when it comes figuring out how and what you want to teach.

Still, for those making policy decisions, I think you have to look at the data, even if you need to take into account weaknesses in how it may have been gathered. I’d like for those determining policy to be informed by studies (though critical of them), and not just going from the gut.