Follow-up for those who care: The limits of baseball statistics

Some follow-up thoughts from the previous post on the limits of baseball statistics, as emailed back-and-forth between trusty argument partner cannatar and myself (before this came out, incidentally). I think it follows logically and enhances the discussion, probably clarifies some things that were not clear in my post. This also has a nice cast of characters including Steve the old-school baseball guy, Mike the mathematician and Andrea the cook (she's a great cook). Also, there's a next-level article at Edge magazine here that discusses the general limits of statistics and has some overlap, but you're going to need a fairly advanced degree to read it.

From: cannatar
To: Bryan

"I'm also mystified at the premise that the 'better' team doesn't always win the championship, when in fact the point of the championship is to determine the best team."

When did you turn into Steve? [ed's note: very anti-sabermetrically-inclined fellow, tends to turn quite disagreeable to the point of yelling at the mention of VORP. Sometimes threatens physical violence. Am not kidding.]

Look, it depends on what you want the word "better" to mean. Does it just mean who did a better job during the 7 games?

Or does "better" mean something more - that one group of men is inherently more skilled at the game of baseball than the other? We use the word "better" all the time in life. You're better than me at stickball, Andrea is a better cook than me, Mike is better at math than both of us.

When I say you're better at tennis than me, I mean that in my opinion (based on years of observation), on any given day, you have a better than 50% chance of beating me in a game of tennis.

If I beat you 10 times in a row, I'd probably change my opinion. If I beat you in one game of tennis, I won't. Even if we call this game a "playoff game" and both agree that this match is extra-important and attempt to play with more focus and determination than we would on another day, the outcome isn't going to change my opinion that if we played again the following week in such a match, you'd be more likely to win. And that's sort of how I choose to think about what baseball team is better. I think the Rays are a better team than the White Sox (particularly since the White Sox don't have Carlos Quentin). But, I realize that baseball is a somewhat random game and that the inferior White Sox had about a 1 in 3 chance of winning the 5-game series. If they had won the series, that would've meant to me that they played better for 5 games, but it wouldn't have changed my opinion about who would win if they played another 5 game series. In other words, I would still think the Rays are "better". I assume Steve would think that if the White Sox won the series, they would win if they played it again the following week. And again, and again. In other words, I think Steve believes that the series reveals the inherent truth about which team is better.

Basically, I think the whole argument is circular. There are 30 baseball teams. The commissioner decides to come up with a construct to determine which team is the best at playing baseball in a given year. Then, whichever team wins under that construct is in fact the best. That requires us to make two assumptions: (1) it is possible to come up with a way to determine which team is better in the absolute sense (Sheehan and I don't think it is) and (2) MLB has actually come up with the best method for doing so.

Even if I presuppose that #1 is true, all I'm doing is accepting that if the White Sox beat the Rays, that means they're better at this point in time in a series made up of 5 games with two days off in between. These distincions are important - the Dodgers probably were actually a better team than the Cubs in October even though they were worse during the 162 game regular season. They were different teams at different points in time - Manny Ramirez and Rafael Furcal weren't around in June, but were in October. Carlos Zambrano and Rich Harden were healthy for most of the regular season, but not in October. The Red Sox and Angels both used a 5-man starting rotation during the regular season. The playoff structure allowed them to use 3-man rotations in the first round of the playoffs. "5-man rotation Red Sox" is essentially a different team than "3-man rotation Red Sox."

But, even if I accept that it's possible for MLB to come up with a system to determine which team is better at a given point in time, I think we'd both agree that MLB has not in fact come up with the best method to determine such a thing. They're running a business - their primary goal is to come up with a season and playoff schedule that makes the most money for MLB. I think they're concerned with fairness in determining a champion, but that's clearly secondary. Even assuming that the 162, 5, 7, 7 format is the best, it's clearly not the case that the system is designed so that the best 8 teams advance to the playoffs. If the goal was to determine who's the best, all 30 teams would play a balanced schedule and the 8 teams with the best records would advance to the playoffs. Instead, some teams play much harder schedules than others, and teams with inferior records make it to the playoffs because they happen to be grouped with other crappy teams. The 2006 Cardinals had the 13th best record in baseball. In addition, their 83-78 record was inflated by the fact that they played a disproportionate number of games against teams in their own division. Not only did their division contain only one other team with a winning record, it also contained the two teams with the worst records in the National League. They also played the vast majority of their games against other National League teams, which as a group are inferior to American League teams. I think there's a compelling argument that the Rangers (80-82), Indians and Mariners (both 78-84) played better than the Cardinals in 2006, they just had much harder schedules. So, there's at least a reasonable argument that the Cardinals were the 16th best team during the regular season, and there's not much of an argument that they were better than 13th.

From: Bryan
To: cannatar

I'm not Steve. Whatever I am, I'm not Steve. That is to say, I understand (and respect) the angle from which you approach this argument, which I did NOT understand, in all its facets, three years ago. The discussion I've started now proceeds from our previous, similar discussion. I understand the statistical undercurrent of your objection and begrudge your opinion in no way, shape or form (To that end, this is the first time I've written something stats-y without the sort of implicit knowledge that I was writing it to you; I was just writing about things that were bothering me. I appreciate your taking the time to read and respond).

That said, let's take the Andrea-as-a-cook example. Andrea is a better cook than me. What if Andrea was on Top Chef? The only goal of Top Chef is to win Top Chef, not to win Top Chef if it were played again, and again, and again. Now I agree that being the better chef coming into Top Chef will give you a better chance to win, the same way I think a team full of Albert Pujii would be better than a team full of Justin Mornii. But Top Chef exists so someone can win it, the same way Major League Baseball exists so that someone can win the championship. The same way there are hundreds of thousands of independent chefs around the country who are NOT on the show, if Major League Baseball did NOT exist, there would be a surfeit of independent leagues and teams and no real place for them to come together. MLB brings them together to determine who is best. That is the entire idea.

And so, I am taking the not-all-that-revolutionary step of saying that the team that wins the championship is the best team, given that is the sole goal of the entire season. You say that you and Sheehan do not believe that there is a way to determine who the best team is, yet in this very email you say that you think the Rays are better than the White Sox, and Sheehan argues quite forcefully that the Cardinals were not the best team in 2006. But hold on, if you do not believe there is a way to determine who the best team is, how can you determine who it's not? It seems, actually, that you have a very good idea of who you believe the best team is and are frustrated with/dismissive and ambivalent of results that do not conform thusly. Sheehan has no problem saying the Red Sox were the best team last year. So I think there's no question that what you've written is not exactly true.

Whether or not MLB has come up with the best method is up for debate, but what's not up for debate is that it is the method. Teams plan ahead at the beginning of the season because they know what the playoffs will look like. When things change during the year, like when the NBA moved to a 7-game first round series in the middle of the season, or, you know, our tennis matches, that's what I find fundmentally unfair. As you know. That ain't what we're discussing. It's just that winning short series is known to be part of winning the championship. Do I hate the 5-game series? You bet. But it's like a Supreme Court Decision — it's the law of the land until it's changed. The law of the land is that if you want to be named champion, you win a five game series. Better have some good starters.

And I don't buy the MLB-as-a-business aspect of your argument. Yes, I understand that MLB is a business and these are business decisions. But trying to separate any of this from the games seems like a fruitless and ultimately foolish pursuit.

In short, I feel like your arguments are mostly political — taking on the established law of the land, so to speak — and I actually agree with some of them. What I'm saying to Joe (and, I guess, you) is that it's fine to take on the orthodoxy but be careful about tearing to shreds people who are merely accepting the orthodoxy, mostly in light of the fact that the numbers do not describe anything in "the fog" that Bill James warned about that lies between axiom and theory. There is a point, it seems, when the numbers guys will throw up their hands and say, "Well, what happens, happens, we can't predict it all." I agree. I don't think numbers or words or observations alone or even paired can describe baseball; that's what makes it great. It takes ALL of them, and more than that. I don't want to disqualify any terms outright. And when it comes down to determining the "best" team, either we can — by the numbers OR the championship — or we can't. If we can't, I don't see the point in playing the games. Ask Herm Edwards why. If we can, I'll take what actually happens. It may not be perfect, but it DID happen, each of us only get so many of these championships in my lifetime. The champions deserve their due. The history of anything isn't perfect; we know only what we really need to know. We know the champions of baseball because it is, even in shorthand, the easy way to tell who the best team was in a given year. It's fighting for that right that makes the playoffs so dynamic, and what makes them the playoffs even if the numbers think it's just another game.

To me, at least.

From: cannatar
To: Bryan

I think most of our argument (and part of your issue with Sheehan) boils down to a definition of the word "best." I am using the word to mean something inherent about the person/team that can really only be "known" by God, but can be determined to some level of certainty by looking at data. You're using the word to mean who did the best at winning the World Series. My take on it is that we already have a word for that: "World Series Champion." Now, if you want to also call the Champion, "the best," you can, but I think you need to acknowledge that you're using the word "best" in a particular way and that other people are entitled to use it in a different way. Stephanie is the "Top Chef Season 4" champion, but I don't think we have any basis to declare her the "best" chef of the group.

"when it comes down to determining the "best" team, either we can — by the numbers OR the championship — or we can't. If we can't, I don't see the point in playing the games."

I guess part of the difference between us is that if the Mets won the World Series, I would still be just as happy even if I didn't think they were the best. It does seem that if I think it's all pretty random, I shouldn't care who wins, but I still care for some reason. I'm more puzzled by the fact that I care about the success of a random group of 25 guys who are employed by Fred Wilpon.

minor point:

"You say that you and Sheehan do not believe that there is a way to determine who the best team is, yet in this very email you say that you think the Rays are better than the White Sox,"

I don't think I contradicted myself. I THINK the Rays are better based on all the evidence, but I don't have a method to determine it with 100% certainty, and I don't believe such a method exists. They're close enough that I probably can't say it with even a very high percentage of certainty (in contrast, I think there's sufficient evidence to say that Albert Pujols is a better baseball player than Andy Phillips with at least 99% certainty, but there's a minute possibility that Pujols's career has just been random chance).

From: Bryan
To: cannatar

My main point is that it's more about the quality of ideas on any side — the numbers, observations, or words — that it is about the ideas themselves. I don't think this is a radical notion, and I don't think Joe Sheehan or anyone at BP would think so either. So if idiots use the terms "veteran leadership" wrongly, they're still idiots. If they use the term correctly, good. Same thing with determining what team is the "best." I think there's a compelling argument — obviously — that mysticism w/r/t unplayed, nonexistent sets of baseball games does not automatically outweigh the results of the actual World Series, no matter how good the numbers are w/r/t these magical unplayed series. I don't think it's a silly thing to argue about, though, and I think that many, many people do. I think this is the main problem. Of course, if you really think that the "best" team can only be known by God, maybe that supports my theory — I'm just going with what us humans have come up with. God's almost certainly smarter than me. I'm just doing the best I can here.

The limits of baseball statistics

I love baseball, I love baseball writing and I love baseball statistics, so I’ve been mulling a subject that has bothered me for a long time: the assault on some baseball terminology as propagated by Joe Sheehan at Baseball Prospectus.

Baseball Prospectus has revolutionized baseball’s record-keeping over the last few years — or at least popularized a long-simmering underground revolution. Put simply, baseball’s most cherished statistics — batting average, runs batted in, pitcher wins and losses — tell us a lot more about what happened in a game than they do make an accurate measure of a player’s contribution. It sounds tricky, but the best way to look at it is that the statistics that Baseball Prospectus compiles from games have a much better track record of predicting what will happen in the future (hence, “Prospectus”) for any given player than the “traditional” stats, which are measures of things that are often beyond a player’s control. That is, the total number of runs batted in a player will accrue during the season relies heavily on the quality of a player’s teammates, and a pitcher may win a game in which he gave up 10 runs and lose one in which he gave up 1. It doesn’t even take a baseball fan to divine the dubious value of such a statistic.

As the BP crew has grown in stature and number over the past few years — primarily since the publication of the 2002 book Moneyball, which highlighted the “newfangled” methods — it has been under nearly constant attack from baseball lifers and “purists,” who argue, basically, that the number-crunchers are a bunch of dweebs who long to make passionate love to their computers. It’s was unfair even before the BP numbers turned out, on a macro level, to help teams to such a degree that it’s pretty much accepted that they were right; the criticism from the old guard now is fairly passive/aggressive and limited to veteran announcers and writers who make claims that the “stats don’t always tell the story” and that there are intangibles involved with winning baseball games. The numbers guys don’t deal with intangibles. If it can’t be measured, it is not important.

This dichotomy rears its ugly head, as it were, every October during the baseball playoffs. Inevitably an announcer will make a comment that a team’s “veteran leadership” will prove decisive, or that their “heart” will lead them to victory. Just as inevitably, Sheehan will write a column excoriating the mouthpiece. Here is an excerpt from this year’s column:

Post-season baseball is just baseball with more media credentials and fewer games between flights. Pressure? There may be more, but is it any more than that faced when you're trying to get drafted? Make a team? Win a playoff spot? Does this week really feel more pressure-packed for the Brewers or White Sox than last week, every game a must-win game, did?

The stock storylines don't add anything to our enjoyment of the game. Whether it's "post-season experience" or "veteran leadership" or "pitching and defense" or "small ball," all these attempts to fit the postseason into boxes limit our knowledge rather than expand it. If we're going to break down these games, and figure out why players do well and poorly, why teams win and lose, let's wipe the slate clean and focus on what's happening on the field.

After years of struggling to figure out exactly what my problem is with this line of reasoning, I think I’ve finally found it: it renders words meaningless. Sheehan’s problem is not that these terms are misapplied but that they are applied at all. This gets to the heart of what Baseball Prospectus is all about: predicting the next set of numbers. There is one set of numbers, a game happens, and then there is a new set of numbers, both for the game itself and one that incorporates all previous games. The numbers do a fairly good job of predicting what the results will be on a macro level, but as Sheehan notes above in reference to the playoffs, post-season baseball is fundamentally no different than regular-season baseball; that is to say, and I’m sort of quoting from memory from hundreds of other articles that he’s written, there is nothing about post-season baseball that makes the numbers any less capricious than they are in May. That is to say, October baseball is subject to the same forces as any other game, with respect to creating new sets of data. You can predict what might happen, and be correct a good percentage of the time, but the game itself — the number of outs, the rules, etc. — is no different in the playoffs than it is in September, in that it’s damn near impossible to predict anything with certainty. The otherwise dogshit Cardinals won the World Series two years ago due to a strong October run. Here’s what Joe wrote to crown them:

Fans, and the less-critical corners of the media, are welcome to embrace the Cardinals and create storylines about raising their level of play and coming up big when it counted and grit and guts and what have you. It might ring more true if it wasn’t the standard storyline for every single team that wins a championship: they’re better people than the guys who lost.

I think he means “better baseball players,” but that’s not my point. Look at sentence fragment that talks about how writers will “create storylines about raising their level of play and coming up big when it counted and grit and guts and what have you.”

Do you notice anything wrong?

If you do, awesome.

If you don’t, let’s start from the beginning. Of baseball. Baseball is a human construct. Or at least we assume it is (ha!), not knowing its precise origins. (The Cooperstown moment is a myth, but one that will do. Like a lot of history.) But let’s just be clear: there’s nothing inherently special about baseball any more than there is anything inherently special about anything: any meaning it has is what we give it. The pre-season, 162-game schedule and post-season are completely arbitrary, save for the meaning we give it. The “championship of baseball” is a construct that, like the sport itself, has no inherent meaning whatsoever. I suspect Joe would agree with me on this, and why the numbers don’t play any different in the post-season than they do the regular season. The numbers don’t know it’s the playoffs.

But the numbers don’t play the game.

This is an incredibly important distinction that has been made many times, by many people, the only difference between them and myself being that they are usually trying to discredit BP’s stat-heavy mission. I am doing no such thing. I love the numbers. I play in a fantasy baseball league that is entirely situation-neutral numbers heavy — that is, the numbers which are BP’s bread-and-butter — and wouldn’t trade the numbers for anything. But there’s a reason that the numbers can only predict what will happen in a given game, series, or season, I dunno, 60 percent of the time (to randomly choose a fairly generous number) — the game is played by people. Or, as Billy Beane, master of the numbers, said in Moneyball, “My shit doesn’t work in the playoffs.” People play the game, and sometimes the favorites win, and sometimes they don’t, like the Cardinals in 2006. And the people, unlike the numbers, know it’s the championship. While Joe is perfectly fine with making his own, completely subjective value judgments on how much “pressure” playing the playoffs actually brings (Despite his distaste of subjectivity, remember: “There may be more, but is it any more than that faced when you're trying to get drafted? Make a team? Win a playoff spot?”), he a) intentionally overlooks the fact that the World Series is, by acclimation if not definition, the most important baseball played each year and thus likely subject to the most pressure; and b) follows it up with, “If we're going to break down these games, and figure out why players do well and poorly, why teams win and lose, let's wipe the slate clean and focus on what's happening on the field,” which has nothing to do with his anti-“veteran leadership” et al. screed. When people are talking about “postseason experience” and “veteran leadership,” this is exactly what they are trying to do.

The numbers, with their gap in accuracy between predicted results and actual results, don’t do the trick. Observation closes the gap. In the Cardinals/Tigers series, Sheehan talks about how the Cardinals got lucky that the Tigers made so many errors, and that the Tigers lost the title more than the Tigers won it. This is likely because the Tigers made such a shockingly high number of errors (seven, I believe) in a short series, and errors are largely unpredictable, so the random sequence of events — the errors — tilted the series toward the Cardinals. All this talk of randomness and capriciousness, which creeps up every year, viz:

I keep coming back to the central theme of any baseball postseason. The champion isn’t necessarily the best team, but it is almost always the team that plays the best in the short series of October. The Rays aren’t getting "lucky" in any sense other than they’re playing well when playing well has some excellent rewards. The Red Sox aren’t getting "unlucky," other than that they’re playing poorly at the same time. The Rays are playing better baseball, and thanks to that, they’re one win away from something that would have seemed preposterous to all but one man and his trusty CPU seven months ago.

Just for a quick side-trip, let’s look up “champion” in Webster’s. It’ll be important:

2. One who by defeating all rivals, has obtained an acknowledged supremacy in any branch of athletics or game of skill, and is ready to contend with any rival; as, the champion of England.

Getting back to the Tigers/Cardinals, the question Joe posed is not how the Tigers lost but why they lost, for the how is obvious — it was the errors. Neither Joe’s statistics or observations begin to provide the why he seeks. Isn’t that something? In fact, the only two sources of why are the injury report and the work of writers, who try to use the tools at their disposal (words) to describe why what happens, you know, happens. Words like “heart” and “passion” are perfectly applicable in baseball because if they are not they would cease to exist. They would be meaningless. As a quick exercise, think about your day right now for one second. How many things are going through your mind? Now imagine a sport that takes, at the least, 18 people to complete one game. How many processes, spoken or unspoken, would contribute to the outcome? It would have to be infinite, right?

I think it’s time that Joe and the other hardcore number-crunchers realize that we have created baseball, but the numbers merely describe the numbers, and nothing else. The words we use have meaning, so when we call a team a “champion,” they are the best team because we say they are. Everyone knew what they were playing for when the season began, and only one team achieved it. It happens because of great players, veteran leadership, tactical decisions, experience, and features from across the spectrum of what it means to be human, some of which are quantifiable, some of which are not. Sometimes the words are wrong (Your best bet would be to ask for examples of veteran leadership). But sometimes the numbers are wrong, too. We’re trying to describe what makes our champions our champions. If the “champion” is merely a construct and doesn’t mean anything, you shouldn’t care. If it does mean something, then you’ve admitted defeat. The words don’t predict or describe as well as precisely as the numbers, but that doesn’t mean they’re less important. Baseball is one of the most dynamic games ever created, but it's not one-one hundredth as dynamic as the human brain and human emotion. Champions are champions for a reason; we made up the word the same way we made up the game. Let us tell the stories of why the champions became who they are. We'll use the numbers and use words. It's an imperfect exercise. But we're trying.

UPDATE: Wow. This now exists. It's all there.

Also: Follow-up emails for those that are interested.