Measuring Mechanistic Interpretability at Scale Without Humans

The goal of understanding the inner workings of a neural network is inherently human-centric: Irrespective of what tools have been used, in the end, humans should have a better comprehension of the network. Removing the need for human labor by automating the interpretability evaluation can open up multiple high-impact research directions: One benefit is that it enables the creation of more interpretable networks by explicitly optimizing for interpretability — after all, what we can measure at scale, we can optimize. Moreover, it allows more efficient research on explanation methods and might lead to an increased overall understanding of neural networks.

You can find more detailed and interactive versions of some of the paper's visualizations here.

We extend work by Borowski et al. 2021 which introduces a setup that allows quantifying how well humans can infer the sensitivity of a unit in a vision model, e.g., a (spatially-averaged) channel in a CNN or neuron in an MLP, from explanations: They leverage a 2-AFC task design in a psychophysics experiment (see left side of the above figure) to measure how well humans understand a unit by probing how well they can predict which of two extremely activating (query) images yields a higher activation, after seeing visual explanations. Specifically, two sets of explanations are displayed: highly and weakly activating images, called positive and negative explanations, respectively.

We automate this evaluation by passing the explanations and query images through a feature encoder to compute pair-wise image similarities (DreamSim) before using a simple (hard-coded) binary classifier to solve the underlying task. The Machine Interpretability Score (MIS) is the average of the predicted probability of the correct choice over multiple tasks for the same unit. The MIS proves to be highly correlated with human interpretability ratings and allows fast evaluations of new hypotheses.

Our proposed Machine Interpretability Score (MIS) explains existing interpretability annotations (Human Interpretability Score, HIS) from IMI well: It reproduces the ranking of models presented in IMI while being fully automated and not requiring any human labor, as evident by the strong correlation between MIS and HIS.

As the previous results were of a descriptive (correlation) nature we now perform a causal (interventional) experiment. We use MIS to perform a causal intervention and determine the least (hardest) and most (easiest) interpretable units in a GoogLeNet and ResNet-50. We then use the psychophysics setup of Zimmermann et al. 2023 to measure their interpretability and compare them to randomly sampled units. Strikingly, the psychophysics results match the predicted properties: Units with the lowest MIS have significantly lower interpretability than random units, which have significantly lower interpretability than those with the highest MIS. More details on the experiments and the result cab be found in the full paper.

Equipped with a fast and cheap measure of human-perceived interpretability we can now perform experiments considered infeasible before. We begin be substantially extending the analysis of Zimmermann et al. 2023 from a noisy average over a few units for a few models to all units of 835 models. Note that performing this analysis with human evaluations would have amounted in costs of around one billion USD. See the full paper for more details on the experiment and the results.

We also show how our proposed measure allows even more fine-grained evaluations by tracking the interpretability of a model during training. Among other things, we find that the interpretability of a model decreases during training. For more insights, see the full paper.