I attended a talk by Xiao-Li Meng yesterday. The talk was organized by the librarians here at WashU as part of a week-long celebration titled “Love Data Week”. Xiao-Li is a well known statistician(and now the founding editor-in-chief of the Harvard Data Science Review), I was really excited to attend his talk. The talk was not technical in an attempt to be accessible to a larger fraction of people. Here are my notes from the talk.

Xiao-Li started by asking the audience to ponder the definitions of artificial intelligence and data-science. The term “artificial intelligence” was first used in a Dartmough meeting in 1956. There’s a commemorative plaque (at Dartmouth?) that honors this origin. The definition of AI that stuck with me after the talk is some a machine that can mimic human intelligence. A more recent perspective seems to want AI to go beyond that, not just mimic human intelligence but augment and go beyond human intelligence. I agree these are all pretty vague ideas, I’d argue this is because we still don’t have a grasp on how to model human intelligence. The leaders in AI, according to Xiao-Li, agree that we are no way close to achieving AI at the moment, though the media would want you to believe otherwise. I recalled that this also fits with Judea Pearl’s observation from The Book of Why that current machine learning algorithms can learn patterns but can’t really go beyond that, they cannot reason for example.

Xiao-Li defined data science in terms of what it’s not. He called it the mathematician’s approach of defining something by defining it’s complement. If I remember correctly, Xiao-Li says data science is not just machine learning or statistics, data science is not just prediction, data science is not just analysis and data science is not a STEM field. He mentioned that currently several universities are racing to setup data science schools, Berkeley for example seems to have got it close to right. Xiao-Li believes data science should not be a department in itself but rather a school, just like engineering, science or humanities. At this point I’m not sure I understand what data-science is to be honest, is it some form of lightweight statistics?

Xiao-Li did a great job of talking through how we started using small samples to estimate population parameters(statistics) - starting with Bernoulli and the law of large numbers in 1713, De Moivre with the Central Limit Theorem in 1733, Anders Krier from Norway who was one of the first to propose surveys, and Morris Hansen who used sampling in the US census for the first time in the 1940s. After the US government started using surveys apparently everyone followed. Surveys were initially referred to as a “Intellectually violent revolution”.

The talk got a little technical after this(which was great, Xiao-Li breaks things down to be intuitive). He delved into why most surveys got the predictions wrong for the 2016 elections. It’s now known that there was a correlation (negative) between people who responded to surveys and people who voted for Trump. This sort of correlation is really bad for surveys, this correlation reduces the effective sample size a lot. Xiao-Li’s calculations show that even if 2.3M people might have been sampled by the surveys, this correlation reduces the effective sample size to 404 (reminds me of a 404 error). In other words the survey data is pretty close to useless. The kicker is that this correlation can only be detected in hindsight, so this could happen again. This leads to Xiao-Li’s, now popular, point - big data alone is not enough, the data needs to be representative. The size of the data can ensure a small variance but might still have bias, this can throw the survey estimate way off. I think in many ways the 2016 elections was very educational for many of us, especially the statisticians. Exit polls in many countries are still learning to optimize the methodology, I remember atleast one national election in India where the exit polls got it horribly wrong. I can predict with certainty that sites like 538 are only going to become common in every democracy.

Some of the lessons from the talk were - data quality matters nad not just quantity, don’t ignore seemingly tiny probabilistic datasets while combining datsets, look at the relative size and not just absolute size of the data. Xiao-Li also spoke about the limits of cross-validation without an independent test set. This is a fascinating topic that probably needs a lot of theory to understand properly. The gist of his argument was that cross-validation can be cherry-picked. You can do the cross-validation many times, get different performances and only report the best performance. One way to get around this is to use an independent test set that’s used just once. There’s a paper by D. Draper on the 3 CV approach that might be interesting to read. Once you have the model from cross-validation, test it on the independent data-set and see how well it does. Put your keyboards down after that, the model cannot be trained on the independent test set, it loses its value if the model is estimated using the independent test set. This is apparently what the physicists do, if the independent test set shows poor performance then the model is probably not a good one.

I had a question for Xiao-Li about the limits of prediction. In hindsight it wasn’t a well thought out question, but basically the answer depends on whether we are in the noise regime or estimating some underlying true value. If a majority of the voters continously go back and forth on their choice of candidate their final choice would be hard to predict. It was a fantastic choice of speaker by the organizers, well done!