Silver on Model Overfitting

« President Obama Again Rules Out Ground Forces – Should We Believe It?

The CEA and the Long-term Fiscal Crisis »

Silver on Model Overfitting

March 24, 2011 by Jason Sorens

Over at 538, Nate Silver has an excellent discussion of the perils of “overfitting” statistical forecasting models. It’s good enough that I could see assigning it to my students in methods courses. Incidentally, I would argue that the opposite peril (“underfitting” if you will) is more common in standard, hypothesis-testing political science research. Because the goal is to establish a particular hypothesis, authors face a temptation to exclude relevant control variables and maximize degrees of freedom.

Posted in methodology | Tagged regression analysis, statistical methodology | 5 Comments

5 Responses

on March 24, 2011 at 10:16 am | Reply Phil

That was a good post.

I have to respectfully disagree about the question of whether the modal political scientist is using too few control variables. I do think we ought to worry more than we do about omitted variable bias, but the standard practice of including everything that other authors have found to also correlate with your DV doesn’t guard against that, and it’s not very well appreciated that adding control variables not only reduces degrees of freedom, but can also increase bias on your estimate of the coefficient for the variables your most interested in.

If you’re interested, I’ve blogged about this here:

http://fparena.blogspot.com/2010/11/what-your-control-variables-dont-do-for.html

and here:

http://fparena.blogspot.com/2011/03/what-your-control-variables-dont-do-for.html
- on March 24, 2011 at 10:40 am | Reply Jason Sorens
  
  Interesting & important stuff. I’ll have to go through the Clarke piece. Certainly I’m as guilty as anyone for not thinking very much about how measurement error in control variables can introduce new bias. And the “collider variable” problem looks really disturbing! Economists are slightly ahead of political scientists in treating omitted variable bias with instrumental variables, but IV has its own pitfalls.
  - on March 24, 2011 at 10:54 am Phil
    
    Yeah, it’s unfortunate the Clarke piece hasn’t gotten more attention.
    
    I agree, economists tend to handle the issue better, but are a bit blind to the shortcomings of IVs. A lot of them tend to think instruments are magic.
    
    Of course, it’s easy for me to criticize standard practice given that I hardly ever touch real data anymore…
on March 24, 2011 at 10:34 am | Reply Phil

This one wasn’t about control variables per se, but if you found the first two links of any interest, you might also want to check this one out:

http://fparena.blogspot.com/2010/11/were-all-modelers.html
on March 24, 2011 at 10:28 pm | Reply Sven Wilson

The central problem here was Arthur Golderger referred to as “micronumerosity,” which was his mocking term for just not having enough data. The best name for a model with 16 observations is “creative description.”

Goldberger was mocking those who think that multicollinearity is a “problem.” As I like to say, multicollinearity is a problem only in the sense that it makes researchers very upset. It is also a problem because it makes them do very naughty things like dropping relevant variables, thus turning a non-problem (collinearity) into a serious problem (omitted variables).

A fairly good-sized chunk of political scientists would do better research if they just dropped the idea of confirming theories through statistical analysis and concentrated on telling better stories by combining quantitative stories with qualitative ones, since they just don’t have enough data.

Comments RSS

Pileus

A Classical Liberal Blog on Political Science, Economics, Philosophy, Law, and More