What is the goal of this paper?
This paper presents a case study on how different explainability methods are being used by stakeholders in real world use-cases . The authors interview multiple folks who either want to or currently use explainability techniques in their work. It highlights the issues people face and presents some great insights on how explainability methods aren't really able to fulfil their original goal of explaining model's decisions to laymen. Taking all this into account, they lay out a framework of goals that explainability methods should try to achieve.
Use Cases
The paper highlights the following set of general uses for explainability in Machine Learning/Deep Learning-
Model Debugging Use the explanations given by the model to figure out why the model is performing poorly in certain cases. Once you know why the prediction was made, you can go back and forth between the model and features to improve.
Model monitoring Model drift can happen when the model you have deployed is now predicting on test data that has different distribution than the original data which it was trained on. Using explainability on the deployed model can let you know why predictions are going awry when they do.
Model transparency A data scientist building models needs to communicate the results of the model and justify the outcomes given by the model to business units. Using explainability, the data scientist can figure out the important features for the prediction and suggest those intuitive features as justification to higher-ups and decision makers.
Model audit In areas like finance, models are audited by regulatory agencies for compliance. Explainability can be used to test the soundness of the model. That is, if the explanation suggests that a prediction is given for some xyz reason, does that xyz reason being satisfied in some other data point result in same prediction.
Stakeholders
The authors put forward an exhaustive set of stakeholders that will be affected by these explainability methods. These are-
Executives To keep the organisation's AI arm accountable, executives find it important to employ and advertise the use of explainable AI in their organisations.
ML Engineers These people use explainability for the purpose of model debugging as detailed in the previous section's first use-case.
End Users These are the most obvious stakeholders in the explainability process. A person wanting to know why his loan was declined or why he was diagnosed with a disease. Explanations of model decisions help these end users understand and change the things they are in control of to get a better outcome next time.
Others(Regulators & Domain Experts) The use case of model auditing highlighted in the previous section is employed by folks like regulators. It also helps domain experts make peace with the fact that the model is being affected by the features it should be affected by in the particular domain.
Deployment
Depending on the specific combination of use-cases and stakeholders that might be suitable for the organisation, one or more explainability techniques can be deployed. These techniques usually aim to achieve one of the following end goals-
Feature Importance Get a value corresponding to each feature of the data point indicating how it influenced the model prediction. This is the most common end goal. This can be used for model debugging, reporting to business units and even to provide explanations to end users. However, issues like that of spurious correlations and non-trivial nature of aggregation of individual feature importances make it challenging to use this to provide explanations to end-users.
Counterfactual Explanations If the test point yields a prediction of class X through the model, then report another generated data point that will yield a prediction of class Not X through the model. This helps users identify what changes they need to make to reverse the decision. This is used when the stakeholders are end users since these are intuitive to understand and actionable as well. They suffer from a drawback wherein it is unclear what the optimization function should try to maximize or minimize while generating counterfactuals. This restricts the solutions to only approximate methods.
Adversarial Training Make explanations robust to little changes. Two data points with very little difference in feature values shouldn't have drastically different explanations. If they do, the explanations and model are fragile and unreliable. This overlaps with model robustness and is therefore used by ML Engineers when they desire more robust models.
Influential Samples Find a set of data points that make the most impact on the model's predictions. In other words, if these data points were to be removed from the dataset while training, the model would perform very differently than it does when they are present. This technique was deployed by an insurance provider in hopes to provide the end users with the information of how the premiums for similar drivers are priced. It did not work out well due to computation load involved which is one of its major drawbacks especially when the goal is to give real time explanations to end-users.
Recommendations
Based on their conversations with multiple organizations, the authors come up with a set of recommendations pertaining to explainability techniques.These are-
Establish Clear Desiderata
In simpler terms, know what you want from the explanations and know who you want to provide those explanations to. These are not easy questions to answer. Therefore, as a guidance they lay out the following points- a. Identify stakeholders based on who will consume the model output. b. Talk to each stakeholder to understand what THEY want. c. Figure out if the explanations will be consumed statically or dynamically. Static explanations are ones that you would provide once to the end user and be done with it while dynamic explanations will be used in situations like model monitoring wherein you need a continuous stream of explanations.
2. Address Concerns
Explainability as an area in itself poses some concerns and these need to be properly addressed. These are factors like explanations not being causal in nature, affecting privacy and also being used to promote undesirable behaviour just to increase model performance.
Conclusion
In conclusion, the authors suggest that even though explainability first came up as a means to provide the layman end user with the reasoning for a model's decision, it has not yet lived up to it. Major uses of explainability are still restricted to model debugging and there is a lot of gap between what organisation's set out to do and what they end up doing with this tools. Keeping the end goals in mind and being thoughtful about the use-case and stakeholders will help mitigate this.
Resources
Last updated