Boyd's World-> They're Looking in the Wrong Place

About the author, Boyd Nation

They're Looking in the Wrong Place

This wasn't really the way I planned it, but it might just work better. Boyd's World tries to look big, but in the end it's just me working with what I've got, and what I've got at the moment is a home broadband connection that's really dodgy, left me off the air completely for a few days last weekend and could go out at any time again. I'll try to keep the ratings updated as I can, since I know some of you rely on them, but in the mean time that leaves with the question of what to do with some columns that I had planned.

In trying to think about what was actually needed, I realized that you don't particularly need me to put together a list of what the field would look like if seeded by the ISR's; you can do that yourself pretty easily if you want to, and the results are predictable, with more West Coast teams in and seeded higher but not many extra major conference teams this year. You certainly don't need to know who I think is actually going to be in the field; that's a whole cottage industry by now, and everyone's sure about the same 60 teams and thoroughly guessing on the rest.

What you do need to know, though, is why the current system doesn't work, so let's talk about that for a while. First of all, there's the question of the parts that don't involve the selection criteria at all. Anyone who's ever spent much time working on committees will recognize the output of the NCAA baseball committee if they look at it for a while, because it's got all the hallmarks of any other dysfunctional committee -- everyone comes in with their own agenda, and they bargain pieces of those agendas until everyone's slightly unhappy and then come up with rationales for public consumption for the results. No one's evil here, especially since the system is designed for committee members to serve as regional advocates currently, but in the absence of clearcut rules, these are the results you get.

That said, those rationales do have to make some sense, so there are some fences put around their behavior. These are set up in the committee manual when it talks about the factors that the committee is to consider. They theoretically have some leeway on this, and they use that for considerations like discounting the RPI or trying to grow the game, but there's no evidence that they look at any statistical evidence other than what's on that list, so let's look at the factors that are included with a series of studies.

First of all, there's the RPI. It's not the source of all evil, but it really doesn't work accurately for baseball. Just as a quick proof that there are better systems out there, here's an example of one of the two types of studies that I'm going to present; in this case, it's identifying games between teams where the methods disagree on placing them. From 1999 to 2007, there were 160 games in the postseason where one team had a higher ISR and the other had a higher RPI:

Better ISR, Worse RPI: 88-72, .550

I'm just going to do the ISR for simplicity (and I'm not going to look at deeper issues, like the fact that the gap widens if you just look at cases where the two systems disagree by at least 10 spots), but I'm sure you'd get similar results with any of a half-dozen or so Internet-available systems, notably Ken Massey's, since the NCAA already has experience with him through the BCS. The RPI just doesn't work well.

But you know what works even less well? Everything else they're using. We'll start with that starter of bar fights everywhere, conference standing (or conference winning percentage, since that accomplishes the same thing and is easier to compute). For this type of study, since there aren't enough pairs for a head-to-head comparison, we'll compare the full postseason records of the two teams from each set of pairs with opposing results:

Better ISR, Worse CWP: 350-228, .606
Better CWP, Worse ISR: 235-213, .525
Better RPI, Worse CWP: 403-275, .594
Better CWP, Worse RPI: 283-251, .530

Conference winning percentage is a poorer predictor of postseason success than the RPI and is much worse than the ISR, beginning a trend that we'll continue here.

Next up is the darling of those who believe in the hot hand, the reason that this week gets overemphasized so much, record in the last 15 games. This time we've got enough matchups to do the head-to-head comparison:

Better ISR, Worse L15: 328-111, .747
Better RPI, Worse L15: 334-133, .715

OK, how about non-conference RPI?

Better ISR, Worse NCRPI: 127-102, .555
Better RPI, Worse NCRPI: 98-89, .524

Road record?

Better ISR, Worse RWP: 322-116, .735
Better RPI, Worse RWP: 312-127, .711

Now, we get to the stratification table, with records against the RPI top 25, 26-50, and so on:

Better ISR, Worse vs. 1-25: 199-98, .670
Better RPI, Worse vs. 1-25: 192-103, .651
Better ISR, Worse vs. 26-50: 197-84, .701
Better RPI, Worse vs. 26-50: 169-75, .693

I could go through the rest of the list, but it doesn't get any different.

When comparing two teams directly, they can also look at head-to-head or common opponents, but those are harder to test (although they're unlikely to be better predictors). There's also an ill-defined "record against teams under consideration", which is probably something like the RPI top 75 and autobid winners, but who knows?

I'm well aware of the value of multivariate analysis, having done a good bit of it myself, but that's not what the committee does -- they're just taking the potentially useful pieces and then using the ones that they need to justify their decisions, which is backwards from the way it should work. Part of the reason for that is that they don't have the analysis that they need going in; they have a very hard job to do this weekend, and they're not going to be given the tools to do it with.

I'll see you Tuesday, hopefully, to talk about the results.

Boyd's World-> They're Looking in the Wrong Place

About the author, Boyd Nation