Demystifying Explainable AI (XAI) for Better Decision-making
Explainable AI (XAI) bridges the gap between complex AI models and human understanding by providing insights into how and why decisions are made. It matters because XAI can prevent errors, biases, and adversarial attacks in AI systems. Complexity in AI models affects accuracy and explainability, with trade-offs to consider. Understanding XAI is crucial for improving system performance and avoiding potential legal and reputational risks.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
A solution to the blackbox challenge 5.2 EXPLAINABLE AI (XAI)
What is XAI? AI held to superhuman standards? A sense of irony How easy is it to explain how and why humans in an organisation do X vs. Y without subjectivity? significant Human post-hoc explanations of their own behaviour can be grossly inaccurate, with processes creating narratives to support a positive sense of self vs. objective facts. subconscious This can bleed over into AI, e.g. unconscious human datasets and system design. biases in Should we also insist upon another type of XAI: Explainable Actual Intelligence?
Why XAI matters Assault Rifles = Helicopter Adversarial attacks on AI image classifiers can trick systems into predicting an image is X when it is in fact Y. Manipulating a single pixel (unnoticeable to the humane eye) would allow this if a single pixel is the primary explanatory feature by a large margin. With XAI it might be possible to identify one or more overweight features, i.e. features with dominative influence, and alert operators so they can consider redesigning the system if desired. Takeaway If you don t know which features lead to which outcomes: At best: you risk not knowing why the system does better or worse in a given situation and therefore will struggle to systematically improve and monitor its performance. At worst: you risk getting things badly wrong, potentially leading to legal, regulatory, reputational and / or economic damage.
XAI vs. Complexity: Part 1 In general: More complex models = more accurate but less explainable Less complex models = less accurate but more explainable 1 Dimension 2 Dimensions 3 Dimensions In general, complexity is a function of: 1. Dimensionality: the more features, the more dimensions and therefore mathematically trickier to fit a model to the data. feature and output, e.g. y = 2x (linear) vs. y = x2 (non-linear). 3. Monotocity: whether the relationship between an input feature and an output is always in one direction (i.e. always increase or always decrease), in which case it is monotonic 2. Linear vs. Non-Linear:direct proportionality between input
XAI vs. Complexity: Part 2 Putting it all together In the first example we have a linear monotonic two dimensional model, g(x) = 0.8x. In other words for a one unit increase in age (the input feature x), the number of purchases (the output y) increases by 0.8 on average. The model s behaviour is identical across the feature space, i.e. for any age the number of purchases is always predicted as 0.8 x person s age. Thus the model is globally and locally interpretable and therefore easy for us to explain how and why it determines the number of purchases someone will make based on age alone given any age. However, it is less accurate. The model (the line of best fit) does not closely map the relationship of age to number of purchases, resulting in lost profits or wasted marketing! In the second example we have a non-linear non-monotonic two dimensional model, g(x) = f(x) (i.e. the algebraic function is a higher order polynomial). Although more accurate, i.e. the line of best fit more or less exactly maps age to number of purchases, it becomes harder to interpret the model globally i.e. for any given age we can no longer simply say the relationship of age to number of purchases is always 0.8 x person s age instead it will depend on local interpretability, i.e. the particular relationship between age and number of purchases for a particular age or age grouping. As the data demonstrates, the relationship is non-linear and non-monotonic. Imagine adding more features (and thus dimensions), e.g. previous purchasing decisions, interests, health, wealth, socio-economic background, education etc, and you can appreciate how model complexity becomes more accurate, yet less explainable as the mathematical model for relationships of inputs to outputs become harder to map!
Inequality of Explainability Takeaway As a general rule, the more complex / accurate a model, the less explainable it becomes. This DARPA diagram describes how today s available techniques compare in terms of their trade-off between accuracy vs. explainability.
Limits of Explainability Explaining how and why a system does what it does requires a collection of interpretable features that contribute to the decision, e.g. Governing + English Law = English Governing Law . As a general rule more complex models = more accurate = less explainable and vice versa. Data Pre-processing Type of Prediction Many ML applications employ significant data pre-processing to improve accuracy, e.g. Principal Component Analysis ( PCA ). These can obscure the original human meaning of the data. Binary classifiers (e.g. X or Y) are easy to interpret because it s relatively simple to see what data pushes a classifier one way or the other. It is much trickier for multilabel classifiers where the data has no inherent structure, as is the case with image classification (e.g. cat , elephant , dragon , potato , spoon , chinchilla , sausage etc). 1 Complexity Problem Domain Certain problems can t be readily understood by quantifying a handful of individual features. Correlated Features 0 Feature Importance Correlates are routinely dropped or swamped in highly correlated data by the other correlates with greater predictive power. This can hide latent factors, potentially leading to incorrect conclusions. 0 1 Explainability Easy to score individual feature importance, but less easy to score the relative importance of features.
Balancing XAI vs. Performance 5.3 XAI CRITICALITY
Why insist upon XAI? Optimise Retain Maintain Comply Model Performance Control Trust Accountability You can t optimise what you can t understand. XAI enables performance monitoring. This helps ensure the system functions as intended, without unexpected behaviour. AI systems are either used: (a) to decide an outcome; or (b) provide additional data for human interpretation to assist in deciding an outcome. In either case, the system must imbue confidence and instil trust. As with any system, there is always a human ultimately responsible. Grid borrowed from the PWC Explainability report and content paraphrased. We will want to tailor this to our message but I think it s a neat model for discussion Bias detection (in the model and / or the data) is easier if you can understand how and why a the model arrives at its outcomes. XAI helps understand who should be responsible and for what. For instance, if XAI uncovers issues with the dataset, then responsibility might lie with the data curation team (or data provider) but if it relates to the model design it might lie with the engineering team and so on. As AI systems rely on data, if the system uses continuous active learning (e.g. consumer recommendation engines that continually adjust to user interactions over time), this can help monitor training drift, which might cause unexpected outcomes. XAI enables understanding of a model s behaviour, allowing the individuals responsible for the system including any decision makers reliant upon it to identify potential failures or weaknesses and how to avoid them, thereby building trust. Without XAI, accountability is tricky. Decision Making Safety Ethics Regulation XAI can identify how an organisation can make better decisions. XAI can ensure system safety, including: Understanding how and why the system decides what it does can naturally also ensure it remains compliant with ethical considerations, either generic or specific to your particular organisation or need. AI is lightly regulated thus far. Undoubtedly that will change as AI decisions become systemic to decision making processes. Identification of vulnerabilities exploitable by hackers. Identification of the underlying causes for latent defects, which may pose a safety risk, e.g. the treatment of a self-driving car in a particular scenario where its behaviour is undesirable. Identification of vulnerability to training drift, which may also impact safety and security. E.g. If an ML system predicts a 90% chance your employee is going to leave you could offer them a raise but what if the real reason is lack of flexible working? Several industry bodies are developing guidance regarding governance, transparency, explainability, accuracy and data management regarding A.I. systems. Without XAI the system s results may mislead you to the incorrect course of action. With XAI you might be able to say employee X has a 90% chance of leaving because they would benefit from flexible working because they are a primary carer.
But consider XAI use case criticality first Takeaway: importance of XAI is a trade off between (a)technical requirements to ensure functional transparency of the AI system; and (b)the impact of decisions made by the AI system Intention a general need to understand how and why the system makes it predictions in order to ensure it does so according to the intended specifications and requirements, both to identify ways to correct and improve it Importance of: In general, the more impactful a system in terms of the decisions / data for decisions it generates, the more explainable it needs to be. Impact the impact of decisions made by the system, whether revenue or non-revenue based, e.g. the impact of incorrect targeted advert predictions is less than the impact of incorrect cancer diagnosis predictions 1 Control the level of system autonomy, e.g. a system making decisions and automating consequential downstream actions vs. a system that merely makes recommendations to human operators who ultimately decide downstream actions Rate the number of predictions the system is required to make, e.g. 100s vs. millions Rigour the robustness of the system s ability to generalise well and maintain high accuracy for sight unseen data Impact Law & Regulation the legal and regulatory framework within which the system operates , including the likelihood of having to validate or justify decisions the system makes / contributes to before a regulator or court Reputation how the system interacts with the business, compliance, stakeholders, society and the general reputation of your organisation 0 Risk the potential harm due to an adverse outcome resulting from use of the algorithm, including: your executives, sales, operations, technology, societal, ethical, workforce, compliance, legal, regulatory, environmental, health and safety etc 0 1 Explainability
XAI Gap Analysis: Part 1 To ensure responsible use of AI within your business you should define: 1.The criticalityof the system s outputs and their impacts, e.g. is the system deciding outcomes or merely providing data for human decisions? Complexity / Accuracy Criticality Explainability 2.The business, legal, regulatory and contractual scenarios that would or might require you to explain how and why the system produces a given output, i.e. the Necessary Explainability
XAI Gap Analysis: Part 2 1. Work with the vendor (if a bought solution) or your own IT team (if a bought or built solution) to understand the extent to which the proposed / existing system allows users to explain how and whyan output is produced ( System s Explainability ). Ability to Explain 2. Perform a gap analysis of the System s Explainability vs. Necessary Explainability (previous slide). Low High 3. If System s Explainability < Necessary Explainability = negative gap. You should consider customising the solution (if possible) or buying / building something more explainable. Model Complexity 4. If System s Explainability > Necessary Explainability = positive gap. You could consider increasing the complexity of the model to improve its accuracy to the extent it does not reduce Explainability below the Necessary Explainability. Low High 5. In either scenario you will need to work with your IT team (to the extent system is built or bought) and potentially also a vendor (to the extent the system is bought) to understand to what extent it is possible to tweak an existing system, or design a new one, to patch the identified gaps. Likewise you may need to re-engineer the surrounding people and process to reduce or eliminate such gaps. Current Level Required Level 6. The above should be carried out at the outset of any build / buy discussion and throughout the system s implementation to ensure no gap exists at the outset nor emerges over time.
How to do XAI 5.4 XAI TOOLS & TECHNIQUES
XAI Tools & Techniques: Part 1 Local Interpretable Model Agnostic Explanations (LIME) Generative Adversarial Networks (GANs) Shapley Additive Explanations (SHAP) Neural Network Interpreters Sensitivity Analysis Tree Interpreters Marginally alter (peturb) a single input feature and measure the change in model output. Repeat multiple times to build an extensive picture of model behaviour vs. features. Works well for small models with smoothly varying behaviour and well separated features. Captures feature interactions by performing various multi- feature perturbations. Similar to LIME in that it uses a local surrogate model to establish feature importance using Shapley Values, i.e. a feature s importance is its average expected marginal contribution after all possible feature contributions have been considered. Random Forrest algorithms, an ensemble method that trains many similar variations of decision trees and makes decisions based on the majority vote of individual trees, can have a tree interpreter applied to it. This allows for a highly interpretable model with high accuracy that can be understood globally and locally. Contrary to popular belief, there is no inherent barrier to understanding the inner workings of ANNs. The results are measured to fit a (surrogate) linear model to the these results, from which it identifies the importance of features and local feature interactions. However, interpretability of ANNs requires a case by case approach and significantly more expertise than other challenges. What See next slide SHAP provides the best possible feature importance explanation with a model agnostic approach. Works less well with large models and unseparated features. Open source implementations currently exist. Activation Maximisation is one such technique. AM identifies input patterns across each node in the network that combine to maximise predictive accuracy. This makes random forests the go to model for commercial applications. Does not capture interactions between features. Computational expensive: you have to consider all possible feature combinations, which grows exponentially with number of input features. Tree interpreters are also widely open sourced. Model Agnostic Agnostic Agnostic Trees Neural Networks Neural Networks
XAI Tools & Techniques: Part 2 (GANs) Confusingly, data scientists are exploring the use of AI to explain the AI! One such example is the generative adversarial network(aka a GAN ). GANs can be used to identify weaknesses in a model exploitable via an adversarial attack. What is an adversarial attack? For example, in the blurry pictures of a Dog on the right, scientists manipulated specific pixels by changing their colour. Although imperceptible to a human eye, these pixel changes completely altered the response of the neural network classifier (e.g. image = Cat ) despite these images being, and remaining identifiable as Dog , to a human eye. How might a GAN help? Using a GAN you could train two opposing neural networks. The first (in red) the discriminator is trained on real data to classify images of things. The second (in green) the generator is trained on random noise and tries to generate fake images to fool the discriminator. The discriminator scores the generator s fake images between 0 and 1. If the score is too low, the generator adjusts and resubmits a new image to the discriminator. This continues until the generator can create new images that correctly map to the desired output with a high score. How does this aid Explainability? This can help us explain how and why the system labels images the way it does, i.e. by highlighting pixels values that disproportionately influence the classification, and therefore any vulnerabilities in that method exploitable via an adversarial attack undetectable to the human eye.
How to manage other liabilities? What to ask Why ask How do we manage other types of liability? Consider whether the AI solution might give rise to other context specific liability types, e.g. strict product liability arises under the applicable laws or laws and regulations regarding equality and non-discrimination. Consider carefully whether the system and its intended use give rise to any concerns in such circumstances and, if so, how you can mitigate their potential effects. How do we manage legal and regulatory requirements relating to automated decision-making? Under the GDPR, a decision based solely on automated processing, including profiling, which produces legal effects concerning him or her or similarly significantly affects him or her is only permissible in certain conditions: 1. The affected individual must have provided explicit consent, which has a high threshold under the GDPR. 2. The automated decision making is necessary for the performance of the contract, e.g. a credit check against a maintained set of databases. 3. Authorised by law. Provided these conditions are present, the affected individual is entitled to a right of explanation and a right to have human re-evaluation of the decision. You must inform affected individuals about: 1. The fact of automated decision making; 2. The significance of the automated decision making; and 3. How the automated decision making operates, which has been described as a right to explainability (see also section [45] regarding AI explainability). To comply you must provide meaningful information about the logic involved . Certain AI techniques make this challenging to the extent the algorithms and processing of data is opaque. Whilst regulators recognise these challenges, guidance suggests you will need to provide a full description of the data used plus the aims of the processing and counterfactual scenarios as an alternative. How do we manage liability relating to cybersecurity? This has two elements: (a) unauthorised use and (b) unauthorised disclosure. Regarding the former, this might be the hijacking of an autonomous vehicle s control system or the autopilot in an aircraft. The latter would include a data breach, e.g. a leak of personal data from a social media or online retailer used to model customer behaviour. In many jurisdictions the applicable laws impose strict requirements to mitigate such outcomes but also inform information security and data protection authorities and, sometimes the affected data subjects. What safeguards do we have in place or plan to build to prevent and mitigate these circumstances?
Do we require interpretability & explainability? What to ask Why ask Is it possible to interpret and explain the AI solution? Are the machine learning algorithm s results: Auditable: who did what, when and how in the system, e.g. which users provided and / or validated which training samples etc. Interpretable: cause and effect can be observed without necessarily knowing why, i.e. we can observe how changes in an input alter the corresponding output. Explainable: cause and effect can be observed and we know how the two are linked and why, i.e. we understand why changes to an input alter the corresponding output. AKA explainable AI or XAI . What to ask Why ask How critical is XAI in the context of our business case? Why does explainability matter? Knowing how and why inputs produce outputs helps validate and test the system s performance, both at the outset and throughout on-going supervision. In some cases knowing how and why inputs produce outputs can be critical for compliance, e.g. being able to explain why a system denies a loan for person A vs. person B see also sections 31 - 32 on GDPR and the right to explainability regarding automated decision making. How much does it matter? Necessity of explainability depends on the criticality of the output. The leftmost graphic plots illustrative use cases against the criticality of outputs and the corresponding extent of explainability required. At the extremes, irrevocable outcomes are highly critical and require high explainability whereas experiential outcomes are non-critical and necessitate lower
