The Gold Standard for What?

If you know anything about causal inference, you probably know that randomized trail is the gold standard. Period?

That is very incomplete. If you want to answer the question how effective penicillin can fight infection, doing a randomized trial is the gold standard for answering that particular question. Period.

However, things get tricky when we have a different question. How effective is a Fosamax in promoting bone health? For beginners, it seem a no-brainer. Do a randomized trial, and  measure the bone density at the end of the trial. Oops, you are changing the question. We were talking about bone health, not bone density. Bone density is a surrogate for bone health. In fact, research discovered that Fosamax might make bones more fragile while at the same time increasing bone density. This phenomenon is called surrogate paradox. Excluding surrogate paradox is not easy, see the paper (full disclusure: my girlfriend is a co-author). What about measuring bone health directly? It is hard. You cannot take someone's bone and apply pressure to it and see what is the maximum pressure under which it won't break. Measuring bone density is the best you can do for randomized trials. Running a randomized trial is the gold standard for answering the question about bone density, it might not be the gold standard for answering the question about bone health.

Randomized trail can only answer a certain set of questions. One should be able to manipulate the treatment in a controlled setting.

The fetish with randomized trials is dangerous. It makes research question subservient to methodology. We move towards research questions that is admissible to randomized trials. We could be missing important questions.

Development is an interesting topic. When I think of it, I always think of industrial policies that make the economy really take off. These are of first order importance. However, lately, research in this area becomes poverty-relief program evaluations. Does giving an egg daily to the poor help them earn more money? People should care about such questions, but ultimately, people should ask is there something we can do so that an African country, could like China really experience industrialization, and lift its people out of poverty en masse and permanently?

Another area is education. "Active learning" is the fad. People like seminars not lectures. Research show them to be more effective. Whatever. When it comes to education effectiveness, the format of class is not independent of many other factors, like the mentality and incentive of the student and the quality of the teacher. They are hard to measure and manipulate in a lab setting. So we take them out. Maybe, with a good teacher and motivated students, a lecture is more effective than a seminar format, while with crappy teacher and unmotivated student, seminar is better? Manipulating teaching format is easy, a low-hanging fruit. Picking low-hanging fruit does not help us find the root cause of our problems (bad teacher and wrong incentive).

Once you choose a question, if randomized trial is an option, do it. That is the gold standard. But if you change the question so that you can do a randomized experiment and pretend to have answered the previous question, I am not sure it is the gold standard anymore.

I wrote about gold standard. But the problem of subordinating research question to methodology is more general. For methodological superiority, we answer question B and pretend we answer A. Gold standard, silver standard, whatever.....

Let me end with a story: I have a servant. I ask him to do things for me. I asked him to do the laundry. He said yes and did it. I asked him to mop the floor, and he told me "actually, I am not good at mopping the floor, let me do something else." ...now, every morning I get up, I call my servant, and ask him politely, "sir, what can you do for me?"