Looking critically at rankings

“Mediocrity cannot be what we aspire to!” Anja Karliczek, Germany’s Education Minister at the time, was alarmed at her country’s performance in the 2018 Pisa Study. In the international reading rankings, Germany occupied 15th place. Estonia came top. However, Professor Daniel Wilhelm has his doubts about how conclusive this ranking table really is. The statistician and economist concerns himself with econometrics, a field of research at the point where empirical economics, statistics and the social sciences intersect.

“Estonia led the rankings, but only based on the data that was available for the Pisa Study,” he notes. And Wilhelm is convinced: “The random samples used for the different countries do not allow rankings to be assigned with absolute certainty.” Following this logic, the results were based on estimates.

Wilhelm and his team developed a statistical method that they applied to the score achieved in the reading study. What are known as confidence intervals allow the statistical uncertainty in the rankings to be quantified. These intervals have an upper and a lower limit between which the countries’ true rankings lie with a probability of 95 percent. This computation would put Estonia somewhere between ranks 1 and 5, while the confidence interval for Germany admits a ranking between 7th and 20th place. “As a rule, this kind of uncertainty in studies is not addressed in political debates,” Wilhelm says.

That said, the paper published by his working group has had repercussions: “We are talking to the OECD about implementing our method in the next Pisa Study,” says the clearly happy professor.

Before moving to Munich, he was most recently Professor of Economics at University College London. Wilhelm appreciates the good research environment at LMU, especially the opportunity to better integrate empirical research and statistical methodology. "There are very good researchers in Munich working on interesting topics. Networking with them and researchers at the Ifo Institute or the Technical University give me the feeling that we can build an excellent group here."

Distorted data as the basis for decisions

Under the aegis of his ERC starting grant project MEimpact, one of the things Wilhelm is developing is a set of methods to quantify the possible influence of measuring errors – i.e. data that do not measure what one ideally wanted to measure – on empirical findings and the decisions that are based on them. Why? Because these errors can distort empirical findings and ultimately lead to the wrong political decisions.

For example, it is difficult to measure people’s abilities, motivation or intelligence. Data culled from survey or test results can only model these qualities in part. Yet assessments of how successful a given educational policy measure has been depend on such data. And it is possible that evaluations of a measure could lead to a very positive result based on the data observed, even though the true effect was less positive or even negative. The newly developed statistical methods make it possible to find out whether measuring errors may have distorted the empirical findings in this kind of way.

“The special thing about what we do is that we work with societal data – data from people – where statistical analysis is fundamentally different to what we find in many areas of science, for example. In the realms of science, you might be able to take repeat measurements under changed circumstances. But you can’t have a child grow up again in a different environment to see what would have happened differently in how their life developed,” says Wilhelm, who pursued undergraduate studies at the University of Ulm and Yale University before earning a doctorate at the University of Chicago. Before his move to Munich, his most recent position was that of Professor of Economics at University College London.

Daniel Wilhelm appreciates the positive research environment at LMU – especially the chance to dovetail empirical research more closely with statistical methodologies. “Munich has very good researchers working on interesting topics. Links to them and to researchers at the ifo Institute and the TUM give me the sense that we can set up an excellent group here."