What is SHAP and How Does It Work?
SHAP, or SHapley Additive exPlanations, is a game theoretical approach that aims to explain the output of any machine learning model. By linking optimal credit allocation with local explanations, SHAP uses classic Shapley values from game theory and their related extensions to provide explanations for machine learning models.
The basic idea behind SHAP is that when a machine learning model gives a prediction, it has assigned some amount of "credit" to each feature or input that was used to make that prediction. SHAP breaks down the prediction into the individual features and assigns credit to each feature in proportion to its influence on the prediction.
To approximate these Shapley values, SHAP uses two main algorithms: Kernel SHAP and DeepSHAP. Kernel SHAP uses a weighting kernel to approximate the Shapley values, while DeepSHAP uses DeepLift to approximate them.
Why is SHAP Important?
SHAP is important because it brings transparency and interpretability to machine learning models. In the past, black box models were commonly used in machine learning because they had high accuracy, but it was not always clear how they came to their decisions.
With SHAP, it becomes possible to understand why the model made the decision it did. This can be especially important in situations where the model is making a prediction that has significant consequences, such as in medical diagnoses or financial decisions.
SHAP also helps with debugging and improving machine learning models. Because it provides a clear breakdown of how the model is making decisions, it can help identify errors or biases in the model, and suggest ways to improve it.
Kernel SHAP
Kernel SHAP is one of the main algorithms used by SHAP to approximate Shapley values. The basic idea behind Kernel SHAP is to create a weighted average of possible credit assignments.
For each possible subset of features that could have been used in the prediction, Kernel SHAP calculates the contribution of that subset to the overall prediction. Then, it weights each contribution by the probability that that subset was actually used in the model.
The weighting kernel used by Kernel SHAP determines how to assign weights to each possible subset. There are different kernels that can be used, such as the Gaussian kernel, that have different properties and trade-offs.
DeepSHAP
DeepSHAP is another algorithm used by SHAP to approximate Shapley values. It uses DeepLift, a layer-wise decomposition method, to estimate the Shapley values for each neuron in a neural network.
The basic idea behind DeepSHAP is to create a set of reference activations for each feature, and then compare the actual activations to the references to determine the contribution of each feature to the overall prediction.
At each layer of the neural network, DeepSHAP makes a local linear approximation of the function using the reference activations to estimate the contributions of each neuron. It then combines these estimates recursively to get the overall contribution for each neuron.
SHAP is a powerful tool for understanding and improving machine learning models. By providing explanations for each feature's contribution to a prediction, SHAP brings transparency and interpretability to models that were previously considered black boxes.
The Kernel SHAP and DeepSHAP algorithms used by SHAP provide a set of tools for approximating Shapley values, and can be used to debug and improve machine learning models. With SHAP, it becomes possible to make accurate predictions while also understanding why those predictions were made.
