“Aggregate Confusion: The Divergence of ESG Ratings”

“Aggregate Confusion: The Divergence of ESG Ratings,” written by Florian Berg, Julian F. Kölbel, and Robert Rigobon, was published in Review of Finance in 2022. With a rapid increase in sustainable investing and investors’ reliance on ESG ratings to obtain a third-party assessment of a firm’s ESG performance, ESG rating providers have become influential institutions with potentially far-reaching effects on asset prices and corporate policies.

The divergence in ESG ratings created by various existing ESG rating institutions brings several impacts. First, there is difficulty to evaluate the ESG performance of firms, funds, and portfolios - which is the primary purpose of ESG ratings. Second, ESG rating divergence decreases firms’ incentives to improve their ESG performance, and markets are less likely to price firms’ ESG performance based on actual results. Third, the divergence presents a challenge for empirical research, as using one rating system versus another may alter a study’s results and conclusions.

This paper investigates what drives the divergence of sustainability ratings. Previous studies provide two major reasons for such divergence: what ESG raters choose to measure and the consistency of measurement. Yet, it remains unclear whether a better articulation of what is measured in ESG ratings could resolve the divergence or whether the measurement itself is the central dilemma for confusion.

The paper decomposes the divergence into the elements of scope, measurement, and weights.

1. Scope divergence: ratings are based on different sets of attributes. Each rating institution chooses to break down the concept of ESG performance into various indicators and organizes them into different hierarchies.

2. Measurement divergence: measuring the same attribute using different indicators. Each rating agency chooses how to assess ESG performance using other methods and indicators.

3. Weight divergence: aggregating the same indicators using different weights. There are substantial differences in the weights for different rating institutions, as the top three most significant rating categories are different among agencies.

The scope and weight reflect what an ESG rating report intends to measure, whereas measurement reflects how it is measured. Using data from six notable ESG rating agencies: KLD, Sustainalytics, Moody’s ESG, S&P Global, Refinitiv, and MSCI, the paper finds that the divergence exists and is substantial. Such divergence is not merely a matter of varying definitions but a fundamental disagreement about the underlying data, producing not merely different but opposite conclusions.

The paper also introduces the rater effect, describing a bias in which performance in one category influences perceived performance in other categories. Simply put, when rating institutions provide a positive rating for a firm’s particular category, the institution tends to also provide a positive rating in other categories. This paper also directs future studies to investigate additional reasons why ESG ratings might deviate systematically in their assessment.

The paper also provides several suggestions for researchers, investors, firms, agencies, and regulators.

For researchers: Researchers should carefully select the data that underlie future ESG studies. They can deal with the ESG rating divergence in the following ways.

1. Include several ESG ratings in the analysis.

2. Carefully explain why the chosen rating methodology/agency is the most appropriate for the specific study, especially when measuring a specific firm characteristic.

3. Construct hypotheses around more specific sub-categories on ESG performance.

For investors: Investors can reduce the discrepancy between ratings by obtaining indicator-level data from several rating agencies and then imposing their own scope and weight.

For firms: Firms should ensure that the metrics used for their own purpose support their underlying goals, and reaching those goals is also recognized by the rating institutions - the values must align.

For agencies: Agencies should clearly deliver and communicate their definition of ESG performance in terms of scope of attributes and aggregation rules and be significantly more transparent concerning their measurement practices and methodologies.

For regulators: Regulators can make ESG rating divergence more intelligible and foster competition on measurement quality by requiring agencies to map their data to a common taxonomy, which enhances the comparison of ESG performance.