Unknown unknowns

Larry Summers talked to our class today. His talk is usually wonderful. Though sometimes I disagree with what he said strongly, I love his talk nevertheless. Just as he said, a valuable paper/talk makes points that some people would disagree with rather than obvious things. I guess the public does not feel this way, or perhaps with the distortion inherent to the media, this advice might not be golden as far as public relation is concerned.

He mentioned unknown unknowns. Basically he is trying to illustrate that there are many factors (unknowns) that we are not even aware of when we make a judgment/estimate. Consequently, our estimates are overly confident. The obvious implication is that, when we make estimates, we should be cognizant of the fact, and in addition to allow for uncertainties around the factors we consider, but also allow for further uncertainties left out in our model (I usually call it "model risk").

I wish to make two points.

Three years ago, when I was writing a paper for a tutorial with Jerry Caprio, I thought about this and actually sent out a survey to investigate this further. I was shocked to find that, even when I remind people of the model risks, people choose to ignore it by and large. Most people felt pretty good about the decision, and when I did follow up, they usually tell me 1) yeah, I know, but I don't know how much difference the model risk gonna matter so I just ignored that 2) oh, I though model risk does not make much difference--models might be wrong, but it is not that wrong, right? so I ignore that. Well, as I noted in my paper that surveys are surveys, you cannot read much from this, because things might be different when people have real stakes involved.

Another thing is personal. I was quite hesitant to come to harvard for econ before August (I do not wish to discuss what changed my mind). I wrote down on a scale of 1-100 how happy I expect myself to be in different places and programs. Well, I probably should not disclose the results as one number that I did get to observe is grossly wrong. One important thing is the weather, I feel as outdoorsy as I was, I would be happier with the gorgeous weather in stanford. What is more, I love rural areas more. As it turned out, I am much happier than I could ever imagine after coming here. In the end, factors I never considered mattered dramatically. The people in my office are awesome, and I got to make friends with some people outside the program and outside my year. There are some other factors that I never thought would matter. but they mattered, to put it mildly. When I tried to make the decision, I probably realized that people around me matters, but not as important as it turned out to be. Part of it might be because I do not know anything about the people about me, so I choose to ignore it, or more precisely I choose to focus on what I can see concretely. I slightly considered the possibility that I might not have a clear picture with all factors considered, but in retrospect, I allowed too little room of error. In the end, it is a combination of some factors and my own realization that my forecast has always been wrong in the past that pushed me to discard my own forecasts and made the choice I did. What I learnt? Actually, not much or not that useful. at least it is not very reassuring to realize that we often optimize for nothing. but I probably will not through away an option that seems crappy at first sight. Yes, we never know.