Pretense of Selectivity Under Censored Ability

Thinking about real word is fun. Thinking about probability is more fun. Applying a probabilistic thinking to  real word phenomenon is most fun. I have written in the past about Admission bias, applying a probability model to admission process. Today I hope to look at a quite related topic from a very different perspective--signalling, in this case and as in real world, noisy and censored signalling.

Motivation

This is probably the most abstruse title I have ever written.  What I mean is simple. When a college decides to admit students, or when an employer decides to recruit people, they will have to rely on some signal. For example, in college admission, the SAT, in business school admission, the GMAT. Through these scores, they hope to have a better understanding of the underlying Ability of the applicant. However, in many cases, the tests are not challenging enough. For example, many of us probably won't find the SAT math section tough, we can easily answer all the questions right, if we do not make careless mistakes. Mistakes we make, if any, are due to carelessness or exhaustion, not our inability to answer them. This happens when our ability is beyond what the test is aimed to test. Formally, if we can measure ability and denote it as a random variable T, then the test score could be modeled as
\[
Y_i=min(T, \text{cutoff})+\epsilon_i
\]
when our ability increase beyond the cutoff, it no longer helps us perform well on the tests.  We generally think there is diminishing marginal returns on ability in terms of score on a test. This cutoff assumption is just an extreme version of diminishing marginal returns. The $\epsilon$ correspond to the factors unrelated to ability, like how one feels on the test day, carefulness, or random luck.

Now many "elite universities" want to be selective and only admit the best and brightest. Logically, they choose students in the highest percentile in terms of test score. What is interesting is among the students they select, how many of them are truly that good.

As a person who cannot tolerate non-quantifiable claims, I decide to do a numerical simulation.

Model Details

Suppose the elite university is interested in selecting the top 1% of the population. Suppose further that the university just admit the students with score in the top 1%.

Now we need to specify the data generating process.Suppose both ability and error term comes from a normal distribution
\[
T_i \sim N(0,\sigma^2)\;\; \epsilon_i \sim N(0,1)
\]
The $\sigma$ will determine the relative weight of ability and error in determining the test score. Let us say $\sigma^2=10^2$.

Let us say once your ability is 1 standard deviation above average, the test does not reflect further increases in ability.

Results

 So the elite university will admit a bunch of kids, all of which has score in the top 1%. But among them, how many of them has ability that is in the top 1%? about 5.7% from a numerical simulation with $10^6$ draws.  Not so impressive. In fact, conditional on being admitted, we can plot the percentile of the students' ability within the general population.

Alternatively, consider a school that only admits students with score falling between 90% to 91%. Among those students 5.2% of them rank higher than 1% in terms of ability. In fact, if we plot the histogram of the percentiles of these students ability, it is very similar. So an Ivy league dude should have no right to feel any pride before those who go to State Universities.

To investigate further, let us consider other cutoffs. In Figure 2, the solid black line plots the percentage of students among admitted whose ability are in the top 1% of the population (refereed to as "the percentage" afterwards). As we can see, as cutoff increases, the percentage increases. As a robustness check, I have varied the weight of the ability in the score, namely $\sigma$. The pattern we see is robust to this change. Further, we see that when cutoff is low, increasing the weight of ability is not useful. But when the cutoff is higher, increasing the weight increases the percentage even further. When we make a test easy, the less informative the test becomes, the less efficient the matching algorithm.

This paints a more optimistic picture then it really is. It seems easy simply to increase the cutoff by making the test harder. However, if we instead plot the percentage against the percentile of the cutoff (for example, if your ability is above 95% of that of the population, any increase in ability no longer helps the test score), we see a different picture.

Suppose the test is so challenging that unless your ability is higher than 95% of the population, increases in ability still helps. Even in that situation, "the percentage" is still low.

Conclusion and Discussion

What we can conclude is that when ability is censored in the test, or more generally, when there is diminishing returns to ability in terms of test score, elite colleges, medical schools or elite employers are not as  selective as they claim to be. The population admitted to elite institutions and "good institutions" differ more in their luck than abilities.

Is this an simplification of matching process. For sure yes. There are usually multiple signals to be sent---if you are truly the top 0.01%, you can win some competition. Some might argue that the college admission in elite schools are "holistic". I do not entertain such Bullshit. This so-called holistic evaluation has racist origin, and is practised today with dishonesty and arrogance. Who are the admission officers to make judgement about an applicant's insight, or character, not to mention whether that student is a good match for the school, when at elite universities, many admission officers would not have been admitted themselves. No matter what, in each group of students who are comparable on other dimensions (race, legacy status, and perceived "character"), they will need to be sorted on academics terms and admitted. They undergo the same process I modelled. The inefficiency now shows up in each such group/stratum. The holitic admission additionally introduce more inefficiency.

Appendix: R code used

 

#set up
n <-10^6; #number of simulations
Traw <- rnorm(n,0,1) #simulate raw talent, will vary scaling
error <- rnorm(n,0,1) #simulate error term, variance normalized to 1

testcutoff <- 1
Btestcutoff <-Traw>=testcutoff #binary whether talent was cutoff for test
WTraw <-10 #weight of talent
score <- WTraw*(Traw*(1-Btestcutoff)+Btestcutoff*testcutoff)+error
admitcutoff <-quantile(score,0.99)
Badmitted <- score >=admitcutoff
TofAdmitted <- Traw[Badmitted]
percentileadmit <-pnorm(TofAdmitted)
png("Histogramofpercentile.png")
hist(percentileadmit,probability=TRUE,breaks=seq(from=0.7,to=1,by=0.02),main="Histogram of Ability Percentile Among Admitted",xlab="Ability Percentile")
dev.off()
mean(percentileadmit>0.99)


stateschool <- (score <=quantile(score,0.91))  &(score> quantile(score,0.9))
stateschool <-Traw[stateschool]
statepercentileadmit <-pnorm(stateschool)
hist(statepercentileadmit,probability=TRUE,breaks=seq(from=0.7,to=1,by=0.02),main="Histogram of Ability Percentile Among Admitted",xlab="Ability Percentile")
mean(statepercentileadmit>=0.99)
#func
tion
trueselective <- function(testcutoff, WTraw, selectivelevel){
Btestcutoff <-Traw>=testcutoff #binary whether talent was cutoff for test
score <- WTraw*(Traw*(1-Btestcutoff)+Btestcutoff*testcutoff)+error
admitcutoff <-quantile(score,0.99)
Badmitted <- score >=admitcutoff
TofAdmitted <- Traw[Badmitted]
percentileadmit <-pnorm(TofAdmitted)
return(mean(percentileadmit>selectivelevel))
}

# Plot 2---selective level at different cutoff
cutofflist <- seq(1,3,0.1)
selectivelist <- lapply(cutofflist,trueselective,WTraw=10,selectivelevel=0.99)
selectivelist2 <- lapply(cutofflist,trueselective,WTraw=5,selectivelevel=0.99)
selectivelist3 <-lapply(cutofflist,trueselective,WTraw=15,selectivelevel=0.99)
png("selectivecutoff.png")
plot(cutofflist,selectivelist,lwd=2,lty="solid",type="l",main="Percentage of Qualified Students Among Admitted",xlab="Ability Cutoff Level For Test")
lines(cutofflist,selectivelist2,lty="dashed",col="red",lwd=2)
lines(cutofflist,selectivelist3,lty="dotted",col="blue",lwd=2)
legend(1,0.8,c("Weight = 10","Weight = 5","Weight=15"),lty=c("solid","dashed","dotted"),col=c("black","red","blue"),lwd=2)
dev.off()

# Plot 3---selective level at different cutoff percentile
cutoffpercentilelist <- seq(0.81,0.99,0.02)
implicitcutofflist <- qnorm(cutoffpercentilelist)
selectivelist <- lapply(implicitcutofflist,trueselective,WTraw=10,selectivelevel=0.99)
selectivelist2 <- lapply(implicitcutofflist,trueselective,WTraw=5,selectivelevel=0.99)
selectivelist3 <- lapply(implicitcutofflist,trueselective,WTraw=15,selectivelevel=0.99)
png("selectiveatcutoffpercentile.png")
plot(cutoffpercentilelist,selectivelist,lwd=2,lty="solid",type="l",main="Percentage of Qualified Students Among Admitted",xlab="Ability Cutoff Percentile For Test")
lines(cutoffpercentilelist,selectivelist2,lty="dashed",col="red",lwd=2)
lines(cutoffpercentilelist,selectivelist3,lty="dotted",col="blue",lwd=2)
legend(0.85,0.7,c("Weight = 10","Weight = 5","Weight=15"),lty=c("solid","dashed","dotted"),col=c("black","red","blue"),lwd=2)
dev.off()

The case against google

Tim Wu of Columbia has recently published a study with Micheal Luca of HBS on how Google is violating antitrust rules. (See new coverage here and here). While one can debate about the shortcomings of the study, some critics of the study obviously failed to understand both the law and the study. After all, on internet times, not everyone would leisurely read through such a "long" study. I hope to summarize the findings here.

What is the central claim?

The claim is Google has downgraded its search result in local searches to consumers' harm in its attempt to extend its monopoly power from general search to more specialized search.

Backgrounds

Just as Google's general search displaced directory services (as provided by portals like Yahoo), it began to face challenges from specialized search, that is search engines that aims to provide information on a specific category of information. For example, Yelp specialized information on restaurants and Kayak on flights. To deal with such challenges, after some failures to clone such specialized searches, Google leveraged its dominant position in general search to exclude its competitors from its search results. The tool for this tactic is called "universal search": once Google detects that a natural search returns a specialized competitor like Yelp as a leading result, it automatically turns on OneBox (see Figure 1), which exclusively shows results from its own specialized search services.

Figure 1: Google search results displays sponsored ads as the first result, followed by its OneBox, which is powered by both Google+ and Zagat (acquired by Google)

The authors showed that by extending OneBox's source to include its competitors like Yelp and ZocDoc, and ranking the results using Google's own algorithm, consumers experienced an improved product (as measured by click-through rate). The design shows that Google has the capacity to improve its product (its own algorithm works) and could easily improve its product by abandoning exclusivity. Yet, it insists on exclusivity at the cost to consumers.

 Why should we be concerned?

It is Google's page, so it should be able to do whatever it wants right? Usually the answer is yes. But "no" in this case. The rational is the following: Google has achieved a monopoly in general search. Fine, this is because it has delivered an excellent product and this monopoly is likely to perpetuate because this industry is a natural monopoly. However, it is unlawful for Google to extend its market power from general search to specialized search. An analogy here. Microsoft has achieved its monopoly power in operating system because of its innovation. However, it violated antitrust laws when it tried to use its market power to get rid of Netscape, an internet browser. Yes, it is on Microsoft's operating system, but it cannot just do what it wants. Note that, many times in technology sector, innovation takes place by creating new markets or changing the paradigm--Google created the market for general search, so did specialized search engines. If we allow such thins to happen, then a company that happened to achieve monopoly in one market can essentially block innovation in emerging adjacent market.

What are the actual laws?

In cases arising under Section 2 of Sherman Antitrust Act, the so called "Rule of Reason" shall be applied: The law acknowledges that sometimes for efficiency reasons, exclusion might benefit consumers.  It is necessary to examine intent and motives and assess its overall impact on the market.  Courts have have isolated several cases where exclusion violates the law.

Naked Exclusion

 "Exclusion of competitors is not justified by any real efficiencies or benefits for consumers". The authors claim that in local search the specific implementation of OneBox  (exclusively drawing from Google's specialized searches not its competitors) is naked exclusion.

Neglect of less restrictive alternatives

 "A dominant firm forgoes  an obvious, less restrictive alternative course of conduct that would be equally, or more effective in serving the pro-competitive goals articulated". That is even if the exclusion could be justified on efficiency grounds, there is an obvious way to achieve the same efficiency, but being less exclusionary. In here, Google might justify the need of OneBox on the grounds that univeral search that returns information directly rather than returning links to the information is a useful innovation. Nevertheless, this could be done in a less exclusive way.

sacrifice of product quality

"The sacrifice of profit or product quality so as to damage competitors".  The authors demonstrated that consumers preferred a universal search that is inclusive. 

Final thoughts

Google obviously can argue that this study has flaws---the experiment participants might not match the actual users of Google. Click-survey might not tell the whole story. Nevertheless, I deem this study as  a proof of concept---that Google's claim that its exclusionary behaviour is efficient, can be scrutinized. Such A/B testing could be carried out on Google, and used as evidence in antitrust trials. Even short of that, this study, or more accurately the plug-in tool the study used--Focus on the User Local, demonstrated that there is less restrictive way to implement OneBox.

Am I fully convinced by the study? I am convinced of what it claims, but beyond that, I could still believe that there is some efficiency reasons for exclusion. One might wonder, is it really efficient to have that many specialized search engines owned by different companies? Maybe as the internet evolves, one company owning both general search and specialized search could make bigger innovations possible. After all, there is multiple equilibria problem in markets with network effects, and for review website that relies on user-generated content, network effect is huge. Such an exclusion by Google simply moves the equilibria--in the interim, there might be some efficiency loss, as users readjust, but once they readjust, they might reach an equally efficient equilibrium. With all search engines owned by one firm, there might potentially be synergies.

Admittedly, I am too much an idealist and central planner in writing the previous paragraph. Without punishing such exclusion, innovation will be stymied. Without innovation putting on a competitive pressure, incumbents will have no incentive to innovate. Despite all the great potentials for synergy, innovation will not happen. Alternatively, even if we believe in the very best of incumbents that wish to innovate, it simply might not be innovate quickly enough. The literature of crowd-sourcing demonstrated the power of the crowd vs. the established elites. With more brains trying to innovate, it is more likely that there will be some entrepreneurs that come up with a better plan. Without laws to protect them from being ostracized by exclusionary behaviours of the incumbent, they will not innovate, leaving us with a slow-moving incumbent behemoth.