Outline

I - What Does “Feature Importance” Really Mean?

II - Measuring Importance Through Variance

III - Sobol Decomposition: Theory and Interpretation [Sob. 01]

IV - Local Feature Importance via Perturbations [Fel. 21]

V - Sobol Indices Estimation

VI - What Sobol Explains (and What It Does Not)

VII - Take-Home Messages

Not really mathematic…

• Consider a model $f:\mathbb{R}^d \to \mathbb{R}$ (e.g. a classification score).
• Intuitive definition:

• Gradient-based importance (infinitesimal changes)
• Occlusion / masking (finite removal)
• SHAP (coalitions / marginal contributions)
• Sobol (variance decomposition + interactions)

Example 1: Linear model

• Consider $f(x_1,x_2)=3x_1+0.1x_2$.
• Changing $x_1$ affects the output much more than changing $x_2$.
• In this special case: importance $\approx$ coefficient magnitude.
• This is a global notion (same for all inputs).

Example 2: Gradient (local)

• Consider $f(x_1,x_2)=x_1^2+x_2$.
• $\nabla f(x) = (2x_1,\,1)$.
• At $x=(0,1)$: the sensitivity to $x_1$ is $0$, to $x_2$ is $1$.
• We have a local notion of importance.
• It depends on the local metric / scaling.

Example 3: Occlusion (finite change)

Occlusion = remove some input information and compare the prediction before/after.

• Image: hide a patch
• Audio: remove a frequency band / TF region
• Text: delete a word

Let $f(x_1,x_2)=x_1x_2$ and input $(x_1,x_2)=(1,1)$. Then $f(1,1)=1$. If we remove $x_1$ (set $x_1=0$), then $f(0,1)=0$. If we remove $x_2$, $f(1,0)=0$.

Hidden interaction (Sobol motivation)

In $f(x_1,x_2)=x_1x_2$, neither variable has a strong individual effect everywhere: the effect comes from their interaction.

Feature importance (in this course)

Variance-based feature importance

• To define "importance", we must define input variability.

Setup: make the output random

• Deterministic model: $f:\mathbb{R}^d \to \mathbb{R}$
• Introduce random input: $X=(X_1,\dots,X_d)$
• Output becomes a random variable:

Notation

• For a given index $i$, we denote:

Variance as global sensitivity

• Mean and variance:

Law of total variance

• For any $Y$ and any variable $X_i$:

Main effect vs total effect

• Main-effect variance contribution:

• Total-effect variance contribution (includes interactions):

Normalized indices

Toy example A: additive model

• $X_1,X_2 \sim \mathcal{U}[0,1]$ independent, $Y=X_1+X_2$

Toy example B: pure interaction

• $X_1,X_2 \sim \mathcal{U}[0,1]$ independent, $Y=(X_1-\frac{1}{2})(X_2 - \frac{1}{2})$

Functional ANOVA / Hoeffding decomposition

• Assume $X_1,\dots,X_d$ independent and $f\in L^2$.
• Assume that $\int f_u(X_u)d\mathbb{P}_{X_i} = 0, \forall i \in u, \forall u \subset \{1, \dots, d\}$
• Then $f$ admits a unique decomposition:

Centering constraint for variance additivity (1/2)

• Without constraints, the decomposition is not unique (constants can be moved between terms).
• The two previous assumptions (aka. zero-mean constraints) make terms identifiable

Centering constraint for variance additivity (2/2)

• Under independence + centering constraints, ANOVA components are orthogonal:

• Therefore, variance decomposes exactly:

Sobol indices for subsets

• For any subset $u\subseteq\{1,\dots,d\}$:

• Properties:

Alternative expressions (useful)

• Main-effect function (intuition):

• Main-effect variance contribution (from Part II):

Total-order index and interactions

• Total-order index aggregates all terms involving feature $i$:

• Quick diagnostic:

Local explanation (1/2)

• We want to explain a single input $x$ (image / audio / text).
• Introduce a random mask $M=(M_1,\dots,M_d)$ controlling visibility of regions.
• Define a perturbation operator $\Phi(x,M)$.

Local explanation (2/2)

• We want to explain a single input $x$ (image / audio / text).
• Introduce a random mask $M=(M_1,\dots,M_d)$ controlling visibility of regions.
• Define a perturbation operator $\Phi(x,M)$.

Perturbation operator: baseline inpainting

• Typical continuous perturbation used in the paper:

• $\mu$ is a baseline / reference input (black image / blur / mean / inpainted content).

From perturbations to a saliency map

• We sample many masks $M^{(1)},\dots,M^{(N)}$.
• We build perturbed inputs $\tilde x^{(n)}=\Phi(x,M^{(n)})$.
• We evaluate the model outputs $Y^{(n)}=f(\tilde x^{(n)})$.
• We estimate Sobol indices for each mask component $M_i$.

Local Sobol importance (Total-order)

• For a given region/feature $i$ (mask component $M_i$), we use the total-order Sobol index.

Why total-order is the default in this paper

• First-order $S_i$ measures the isolated effect of region $i$.
• Total-order $ST_i$ measures everything that involves region $i$.
• If $ST_i \gg S_i$, region $i$ mostly acts through interactions.

Signed importance (optional)

• Sobol indices are non-negative: they measure magnitude (how much it matters), not direction.
• A simple signed variant attaches a sign using an occlusion test:

Why we need estimators

• Definitions involve conditional expectations / high-dimensional integrals.
• We want a black-box estimator using only evaluations of $f(\Phi(x,M))$.
• Key trick: build pairs of perturbations that differ in one component.

Saltelli sampling: A, B, and mixed C

• Sample two matrices of masks: $A,B\in[0,1]^{N\times d}$.
• For each feature $i$, build $C^{(i)}$ by replacing column $i$ of $A$ with column $i$ of $B$.

Jansen estimator: total-order

• Estimate the output variance (using $A$):

• Total-order estimator:

Jansen estimator: first-order

• First-order estimator:

Computational cost

• Evaluations required: $N(d+2)$ forward passes.
• Typical setting in the paper: mask grid $11\times 11$ $\Rightarrow d=121$ and $N=32$:

Quasi-Monte Carlo (Sobol sequences)

• Instead of i.i.d. Monte Carlo masks, use a low-discrepancy sequence.
• Better coverage of $[0,1]^d$ $\Rightarrow$ lower estimator variance in practice.

White-box vs Black-box explanations

• White-box methods use gradients / internals:
• Examples: Saliency maps (∇), Integrated Gradients, Grad-CAM, LRP
• Require access to model structure + backpropagation


• Black-box methods use only model outputs:
• Examples: Occlusion, RISE, SHAP, Sobol perturbation-based saliency
• Only require forward passes: $f(x)$ queries

Evaluation #1: Pointing Game

• Goal: check whether the saliency map points to the object of interest.
• Requires: ground-truth object location (bounding box or segmentation mask).
• Protocol:
• Compute a saliency map $S(x)\in\mathbb{R}^{H\times W}$
• Find the most salient pixel / region:
$$p^\star = \arg\max_{p} S_p(x)$$
• Hit if $p^\star$ lies inside the ground-truth object mask / box

Evaluation #1: Pointing Game

Evaluation #2: Deletion score

• Goal: measure how “faithful” the saliency map is to the model decision.
• Idea: remove the most important pixels first and track the output drop.
• Protocol:
• Rank pixels/regions by importance (descending): $p_1,p_2,\dots$
• Create a sequence of perturbed inputs $x^{(k)}$ by deleting top-$k$ pixels
• Track the score (e.g., class logit/probability) along the deletion path:
$$s_k = f_c\!\left(x^{(k)}\right)$$

• Typical metric: Area Under the Curve (AUC) of $s_k$ vs. deleted fraction
• Lower AUC = more faithful (faster score collapse)

Evaluation #2: Deletion score

Fel et al. (2021): what to remember

• Local Sobol saliency maps are obtained by applying Sobol indices to a perturbation model.
• Total-order maps naturally capture interactions between regions.
• QMC sampling improves stability / convergence for a fixed evaluation budget.
• Provides a principled “variance budget” view of explanations.

Interpretation: sensitivity, not causality

• Depends on $p(M)$ (mask distribution) and $\Phi(x,M)$ (perturbation operator).
• Depends on baseline $\mu$.

Limitations / pitfalls

• Off-manifold: perturbations may create unrealistic inputs.
• Feature dependence: classical Sobol assumes independent variables (true for masks; not always for raw features).
• Granularity: pixel vs superpixel vs patch changes the explanation.
• Faithfulness metrics: deletion/insertion depend on the perturbation protocol.

Time-frequency features

• Mixture: $x(t)=s(t)+n(t)$
• STFT: $X(\omega,\tau)$
• Mask TF regions:

Why interactions matter in audio

• Speech/music structure:
• harmonics (correlated frequencies)
• formants (bands)
• time coherence (onsets/transients)
• Many features are informative only jointly.

Spectrogram in audio

Take-Home Messages

1. Feature importance must be defined: it depends on “how you vary the input”.
2. Variance is a principled sensitivity measure (scalar, stable, decomposable).
3. Sobol = functional ANOVA: exact decomposition into main effects + interactions (under assumptions).
4. Total-order indices are key in interaction-heavy models.
5. Local XAI: apply Sobol to $g(M)=f(\Phi(x,M))$ using perturbation masks.

References

Outline

I — Why counterfactuals?

II — Formal definition

III — Single counterfactual = optimization view

IV — DiCE: multiple diverse counterfactuals

V — Distances, constraints, mixed data

VI — Evaluation + limitations

VII — App: Counterfactuals for medical image

VIII — App: Counterfactuals in text-audio

Explanation as “minimal change”

• We have an instance $x$ and a model prediction $y=f(x)$.
• We want a new point $c$ such that the decision becomes $y^\star$.

Why one counterfactual is not enough

• Many different changes can flip the decision.
• We want multiple options to provide flexibility.

Model + target outcome

• Let $f:\mathbb{R}^D \to \mathcal{Y}$ be a black-box predictor.
• Given an input $x\in\mathbb{R}^D$, we want $c\in\mathbb{R}^D$ such that:

Core desiderata

• Validity: $c$ achieves the target outcome.
• Proximity: $c$ is close to $x$ (small change).
• Feasibility / actionability: respect constraints (immutable features).
• Interpretability: sparse changes preferred (few features).

Constrained formulation

• Minimal change subject to achieving the target:

Relaxed (penalized) formulation

• Convert constraint to a loss term:

Discrete / categorical constraints

• Some features are categorical ({CDI, CDD, FreeLancer}) (one-hot) or ordinal {Bac < Bachelor < Master < PhD} or continuous {salary etc.}.
• Practical solutions:
• continuous relaxation (categories becomes real) + projection (on category space)
• heuristic search (beam search, simulated annealing etc.)
• optimize in a continuous latent space of a generator

K counterfactuals

• Find $\mathcal{C}=\{c_1,\dots,c_K\}$ such that each $c_k$ is valid and close.
• Additionally: counterfactuals should be diverse.

DiCE objective in a nutshell

• Optimize all CFs jointly:

DPP-style diversity (determinant)

• Define a similarity kernel between counterfactuals, e.g. RBF:

• Diversity proxy:

Practical note: gradients

• If $f$ is differentiable (NN), optimize by gradient descent in input space.
• If $f$ is non-differentiable, DiCE can use model-agnostic search / sampling.

Continuous features: robust scaling (MAD)

• For continuous features, normalize by median absolute deviation (MAD):

Categorical features: mismatch penalty

• For one-hot encoded categorical features:

Immutable / actionable constraints

• Partition features into:
• immutable: cannot change (e.g., age, past defaults)
• actionable: can change (e.g., savings, debt)
• conditionally actionable: changes allowed only in one direction

Sparsity

• Interpretability often requires changing few features.
• Proxy: encourage sparse deltas $\Delta = c-x$ via $\ell_1$ penalty:

Standard evaluation metrics

• Validity: fraction of CFs achieving the target.
• Proximity: average distance $d(c_k,x)$.
• Diversity: determinant-based or pairwise distance statistics.
• Sparsity: average number of changed features.

Local decision boundary approximation (intuition)

• Counterfactuals approximate the local boundary around $x$.
• One can fit a simple surrogate near $x$ (e.g., 1-NN on generated samples).

Main limitations

• Off-manifold: CFs can be unrealistic (not resembling real data).
• Feature dependence: changing one feature may require changing others.
• Causality: actionable recourse should respect causal relations.
• Multiple solutions: must balance proximity vs diversity vs sparsity.

Take-Home Messages

1. Counterfactuals explain by giving changes that flip the outcome.
2. Mathematically: solve a constrained optimization (validity + proximity).
3. DiCE: generate K CFs and add a diversity term (determinant / DPP-style).
4. Mixed data + constraints are central (actionability, immutability, categorical).
5. CFs are only meaningful under an explicit feasibility model.

References

• DiCE: Mothilal, Sharma, Tan. “Explaining ML Classifiers through Diverse Counterfactual Explanations.” FAT* 2020.
• (Optional background) Wachter et al. “Counterfactual explanations without opening the black box.” 2017.

Problem setting (medical imaging)

• We have an image classifier / regressor: what minimal change flips the decision?
• We want counterfactual image $x_{cf}$ such that:

Diffusion Autoencoder (DAE) as unsupervised feature extractor

• Train a DAE on unlabeled images → semantic latent space z_sem.
• DAE learns a compressed representation where interpolations are meaningful.

Linear decision boundary in latent space

• Encode labeled samples: $w = z_{\mathrm{sem}}(x)$
• Fit a linear classifier (SVM / linear layer): hyperplane

• $n:$ semantic direction corresponsing to the pathology existence
• $b:$ bias term

Binary counterfactual = reflect across hyperplane

• Given latent code w, the paper proposes a closed-form CE in latent space.

• Then decode: $$x_{cf} = \mathrm{decode}(w_{cf})$$

Ordinal counterfactuals(severity progression)

• Severity measure: severity is defined as the signed distance to the hyperplane $$ \text{dist}(w,P) = \frac{n^\top w + b}{\|n\|} $$
• Calibration to ordinal grades: $$ \text{grade}(w) \approx \alpha\,\text{dist}(w,P) + \beta $$
• Controlled severity traversal: we explicitly parametrize a latent trajectory along the normal direction $$ w(t) = w_0 + t\,\frac{n}{\|n\|} $$
• Key property: along this path, $$ \text{dist}(w(t),P) = \text{dist}(w_0,P) + t $$ $\Rightarrow$ increasing $t$ increases severity linearly.
• Counterfactual generation: decoded images form a smooth severity progression $$ x(t) = \text{decode}(w(t)) $$

Results of the generator

Results of the generator

Results of accuracy, grade etc.

Limitations

• Linearity assumption: a single hyperplane direction may not match complex pathologies.
• Confounders in latent space: “move to healthy” may introduce spurious anatomical changes (bias).
• Calibration: mapping distance→grade can be imperfect (especially under label scarcity / imbalance).

Main idea

Can be a chosen paper for the evaluation !

Why prototypes?

• Standard deep nets rely on abstract latent features.
• Prototype nets explain by analogy: “this looks like that”.
• Interpretability is intrinsic (part of the model), not post-hoc.

Outline

I — Motivations

II — Prototypical networks: $\texttt{[Sne. 17]}$

III — ProtoPNet: patch prototypes $\texttt{[Che. 19]}$

IV - Prototypical and Counterfactual

V — Prototypes in Sound Classification $\texttt{[Zin. 21]}$

VI — take-home

II – Prototypical Networks

Few-shot classification setting

• We consider an episodic learning setup.
• Each episode mimics a few-shot task.

Learning an embedding

• Inputs are mapped to a representation space:

• $f_\phi$ is a neural network (CNN, ResNet, …)
• All reasoning happens in the embedding space

Class prototypes

• Each class is represented by a prototype.

Classification rule

• Given a query point $x$:

• $d(\cdot,\cdot)$ is a distance in embedding space

Episodic training objective

• Minimize negative log-likelihood on queries:

• Episodes are sampled during training
• SGD on $\phi$

Why is the prototype the mean?

• Assume the distance is a Bregman divergence ($\phi$ strictly convex on a convex domain).

Connection to mixture models

• Exponential family density:

• One component per class
• Equal mixture weights

Squared Euclidean distance

• Corresponds to spherical Gaussian clusters
• Prototype = empirical mean

Equivalence to a linear classifier

Few-shot terminology and datasets

• N-way: number of classes in a classification episode
• K-shot: number of labeled examples per class

• Omniglot: handwritten characters, many classes, low intra-class variability
• miniImageNet: natural images, fewer classes, high variability

Experimental validation (ProtoNet)

• Benchmarks: Omniglot and miniImageNet
• Few-shot classification tasks:
• $N$-way, $K$-shot (e.g. 5-way 1-shot, 5-way 5-shot)
• Episodic training and testing

Results

Key takeaways

• Classification via distances to prototypes
• Prototype = mean (theoretically optimal)
• Distance choice encodes distributional assumptions
• Model is simple, stable, and interpretable

Why ProtoNets matter for interpretability

• Each decision is explained by distances to class prototypes
• Prototypes live in the same space as data embeddings
• No opaque classifier head

Patch embedding map

• Backbone (ResNet, VGG ...) outputs feature map: $F_\theta(x)\in\mathbb{R}^{H\times W\times d}$
• d: channel number
• Patch vectors: $z_{ij}(x)\in\mathbb{R}^d$

Matching: max over patches

Softmax (log-sum-exp) smoothing

Objective: accuracy + interpretability

• $\mathcal{L}_{cls}$: cross-entropy on logits
• $\mathcal{L}_{p2d}$: each prototype close to at least one real patch (same class)
• $\mathcal{L}_{d2p}$: each sample covered by at least one prototype (same class)

Prototype-to-data term (patch version)

Data-to-prototype term

Prototype projection (key XAI step)

Prototype-driven counterfactual (idea)

• Target class $y'$ has prototypes $\{p_k:c(k)=y'\}$.
• Seek $c$ close to $x$ but near a target prototype:

What APNet tries to fix (audio XAI)

• Deep audio classifiers are accurate but opaque.
• Post-hoc saliency (gradients / LRP / etc.) can be unstable or hard to interpret in TF.
• APNet: explanation is case-based: prediction is based on similarity to a small set of learned prototypes.

Architecture at a glance

• Input: log-mel spectrogram $X_i \in \mathbb{R}^{T\times F}$
• Encoder: $Z_i = f(X_i)$ with $Z_i \in \mathbb{R}^{T'\times F'\times C}$
• Prototype layer: store $M$ prototypes $P_j \in \mathbb{R}^{T'\times F'\times C}$
• Similarity $\rightarrow$ weighted sum over frequency $\rightarrow$ linear classifier

Autoencoder objective (makes prototypes audible)

• Decoder reconstructs $\hat X_i = g(Z_i)$
• Reconstruction loss:

Prototype loss = “prototypes are real” + “data are covered”

• Distance from each sample to each prototype (latent space):

• Prototype objective:

Audio-specific twist: frequency-dependent similarity

• They compute similarity per frequency bin $f$ (latent TF structure):

• Then aggregate frequency with a learnable kernel $H_j[f]$:

Classifier + training objective

• Linear layer on similarities (no bias for interpretability):

• Cross-entropy:

• Final loss:

Datasets (3 tasks)

• UrbanSound8K: 10 urban classes (≤4s clips).
• Medley-solos-DB: 9 instruments, 3s clips .
• Google Speech Commands V2: 35 keywords, 1s clips.

Results

• They compare to SB-CNN, Att-CRNN, and OpenL3 features + MLP.
• APNet is competitive on all tasks.

Inspection: what makes it interpretable? (1/2)

• Decode prototypes $\to$ mel-spectrograms (and audio) to inspect “what a class looks like”.

Inspection: what makes it interpretable? (2/2)

• Inspect $W$ (prototype-to-class connections): prototypes mostly connect to their own class.

Refinement (because you can debug it)

• Prototype redundancy pruning using prototype-prototype distances.
• Channel redundancy pruning by analyzing prototype distances per channel.
• After pruning + short retrain: fewer params and sometimes better accuracy.

Interested by it ? Then select his extension for the project :-)

Take-home (APNet)

1. Prototype networks give intrinsic explanations: “this sounds like that”.
2. Audio needs domain-aware similarity (frequency matters).
3. Decoding prototypes to audio makes explanations tangible (listen + view).
4. Interpretable structure enables pruning + debugging beyond accuracy.