Part II — Every attempt to refill it has failed, in seven ways
Chapter 4
Aggregate confusion
Suppose you wanted to know whether a company is good for the world. Not profitable. Good. Does it treat people decently, does it look after the places it operates, is it honestly run.
You wouldn't be the first to ask. There's a whole profession that answers the question for a living. They're called ESG raters, for environmental, social and governance, and they score companies on exactly this.
Here's the trouble. Ask six of them about the same company and you get six different answers.
Six raters
In 2022 three researchers, Berg, Kölbel and Rigobon, published a study in the Review of Finance with a title that tells you how it came out: "Aggregate Confusion." They took the ratings of six firms, KLD, Sustainalytics, Moody's ESG, S&P Global, Refinitiv and MSCI, and compared them company by company.
If two raters agreed perfectly, the match between their scores would be 1. If their scores had nothing to do with each other, it would be 0. Here is what the study found: "Correlations at the ESG level are on average 0.54 and range from 0.38 to 0.71."
To give a sense of how low that is, the authors ran a comparison. Credit ratings, the grades that tell lenders how likely a company is to pay its debts, come from competing agencies too. The authors took the credit ratings from a 1998 study, turned the letter grades into numbers, and worked out the same kind of match. It came to 99 percent. They are careful to say they did this "For the sake of illustration," because credit grades aren't really numbers and researchers don't usually treat them that way. Fair enough. As an illustration it does its job. Professional raters looking at the same companies can agree almost perfectly. On whether a company is good, six of them averaged 0.54.
So it isn't that rating companies is impossible. Something about this question comes apart in their hands.
Where it comes apart
The study didn't stop at showing the disagreement. It went looking for where the disagreement comes from, and it found three sources.
The first is what's on the list. The paper calls it scope: "ratings are based on different sets of attributes." One rater includes something another leaves off.
The second is how you count a thing that's on both lists. The paper calls it measurement: "rating agencies measure the same attribute using different indicators." Two raters care about the same quality in the same company, look at different evidence for it, and come back with different scores.
The third is how much each thing matters. The paper calls it weight: "rating agencies take different views on the relative importance of attributes."
Then they worked out how much each one contributes. "Measurement contributes 56% of the divergence, scope 38%, and weight 6%."
Look at where the small number is. You might have guessed the fight was about priorities, that one rater cares more about the environment and another cares more about workers. That turns out to be the smallest part of it.
More than half of the disagreement is the second kind. The raters look at the same quality in the same company and count it differently. Most of the rest is the first kind. They aren't working from the same list.
The authors put the split in one sentence: "Scope and weight reflect what an ESG rating intends to measure, whereas measurement reflects how it is measured."
So the raters disagree mostly on how to count, and also on what to count. That's the average across the six. For one of them, MSCI, the bigger share came from what was on the list.
I'm going to call this failure Same Word, Different Things. Six firms use the same three letters. They publish scores that look comparable. And underneath the word, they are not counting one thing.
Six people watching a door
Let me make that concrete with something that never happened. I'm imagining this, so treat it as a picture and not as evidence.
Put six raters in a lobby and ask them to score door-holding.
The first counts doors held. The second counts seconds saved. The third counts how many strangers were involved, on the theory that holding a door for a friend is just manners. One act, three ways to count it.
Then there's the list. Does the door you didn't want, the one from Chapter 2 that made you jog, count as a door held? One rater says of course, a door is a door. Another says no. A third never thought about it.
Then the weights. Is the trust worth more than the seconds? How much more?
By the end of the day they would hand you six scores for the same lobby, and the scores wouldn't match. Not because any of them was sloppy. "Door-holding" sounds like one thing until you try to count it. Then it's three questions: how to count it, what counts, and how much each part weighs.
Keep that lobby in mind. It's the picture this book comes back to more than any other.
The same thing, wider
ESG is the cleanest case because somebody measured the disagreement. It isn't the only case.
The best-known attempt to rank countries by something other than income is the Human Development Index, first published in 1990. It combines income with health and schooling. Behind it is serious thinking by Amartya Sen about what people are actually able to do and be, and it is a working alternative to judging a country by its income alone. I don't want to wave it away.
But an early critique of that first index reported that the index was correlated with income per person at .87. That's one critic, in 1991, on the first edition. I mention it only for the pattern. When you build a new measure and it moves almost in step with the old one, it's fair to ask how much new you are seeing.
Counting only some people
There's a second failure in this chapter, and it's quieter.
Bhutan is known for measuring Gross National Happiness. It's a real survey, carefully done. The 2022 round interviewed 11,052 people aged fifteen and over, in randomly sampled households, and found 48.1 percent of them happy by its own standard.
A household survey reaches the people living in the country. That isn't a flaw in the survey. It's what a survey is.
But in the years before, a large number of people had left Bhutan. When resettlement began in 2007 and 2008, the United Nations refugee agency counted about 107,000 Bhutanese refugees in seven camps in Nepal. Most are Lhotshampa, the Nepali-speaking people of Bhutan's south. Whatever their happiness was, it isn't in the number.
I want to be even-handed about this. I'm not saying the survey was built to leave them out, and I can't tell you what the number would have been with them in it. Nobody can. What I can tell you is the plain fact of it: the people who left are outside the population that got measured.
I'll call this one Counting Only Some People. Every measure of how people are doing starts with a decision about which people. Usually nobody announces that decision. It's just where the survey went.
What this chapter leaves you with
Two failures, then. In the first, honest professionals use one word for different things, and you can't tell from the scores that they're doing it. In the second, the count is careful, and the question of who gets counted was settled before anyone started.
Now suppose you fixed both. Suppose everyone agreed on the list, the method, the weights and the people. Suppose the number was good.
What happens to a good number once people can win or lose by it?