12th Nov, 2025
9 min read
AI Safety
As a kid in grade school, my teachers often asked me to “show my work” while solving a mathematical problem. Academically, this process still exists and is an effective way to show understanding of a concept one is solving, as well as the thought process behind how one arrives at a conclusion. In Explainable AI or XAI, we call this a white-box model[2], where the inner workings of how an AI arrives at a conclusion is transparent.
But what if I didn’t show my work to my teacher in grade school?
This would have put my results under suspicion of guesswork or cheating. This could also make my teacher doubt my character as a student or lose faith in me altogether. In XAI, a black-box model[1] or an opaque model is one where we cannot see how a model arrived at a conclusion, as is the case. This begs the question: can you really trust what the model is telling you?
For a model to be trustworthy, it needs to be interpretable and explainable, especially in mission-critical application domains, such as banking, finance, healthcare, and law. [1][2]
In his paper, Zachary Lipton[2] expresses that a sufficiently transparent model that evokes trust should have the properties of:
Simulatability: The ability to contemplate a model in its entirety
Decomposability: The ability to comprehend the multiple features or parts of a model
Algorithmic Transparency: The transparency of learning or training of the model
Given the above criteria, even if the model is sufficiently transparent, it should also be explainable or interpretable to be remotely trustworthy to humans. This is where interpretability models, such as LIME or SHAP, come into play, providing human-understandable explanations of individual predictions. For this particular article, we will be focusing on LIME.
What is LIME?
Local Interpretable Model-Agnostic Explanations (LIME) [3] is a post-hoc method used to explain black-box predictions. For a given instance, LIME generates perturbed samples around that instance, obtains the black-box model’s predictions for these samples, and weights them based on their similarity to the original instance. A simple, interpretable surrogate model is then trained on this locally weighted dataset, providing an explanation for the black-box model’s prediction on that specific instance. This is similar to a teacher grading a single exam question by comparing how different students answered that same question, using those variations to understand how each part contributes to the overall grade.
However, LIME does have its drawbacks. While being model-agnostic and versatile, LIME cannot be globally appropriated for the whole model, or even similar black-box models, as its interpretation only accounts for a specific instance.[4] The way datasets are perturbed to derive explanations through LIME also offers a rather shallow perspective, as there is no specified way to decide which data points the explanations apply to[5], and they are often not penetrative enough to be a universal truth for the opaque model. An experiment conducted by Burger C. et al[6] also confirmed LIME's vulnerability against text perturbations by malicious actors, raising further questions about the reliability of its explanations.
Altogether, these limitations highlight that LIME can still only provide specific post-hoc explanations. In a real-world scenario, however, a model’s behaviour is shaped as early as the initial training stage, via Reinforcement Learning.
So, how does AI Learn?
Reinforcement Learning from Human Feedback(RLHF) is one of the most common ways to fine-tune Large-Language Models to better align with human values.[8] A supervised fine-tuned model is provided feedback on its prediction by methods like comparison, scalar ratings, correction, or language feedback, etc., which is then used to train a reward model that guides subsequent reinforcement learning.
There are challenges in obtaining human feedback: [7]
It is difficult to account for a diverse portfolio of feedback, as all humans have different backgrounds, values, beliefs, and morals that can affect how aligned a model is politically, socially, and ethically.
Human evaluators may pursue the wrong goals in training the model altogether, resulting in an inherently misaligned model.
It is difficult to account for the effort required in supervision, as the task itself can be quite complex, or there can be issues ascertaining the evaluator’s bandwidth and availability
It is difficult to prevent malicious actors who can voluntarily add harmful feedback
It is difficult to predict Human error in any stage
Human values are difficult to represent with a reward function
Among many others.
LIME may be able to explain the prediction from a black box; however, if a model is exposed to any level of bias during initial training, the goals it possesses could be inherently misaligned with human expectations. Gonzalez B. et al. [8] express that human evaluators should have adequate representation of people from diverse backgrounds, should be subject to guidance via pluralistic panels to cover all grounds, and should monitor feedback aggregated in an unbiased way to ensure appropriate supervision.
As a UX practitioner, I have often witnessed end users process information through both cognitive bias and heuristics. Framing is a key aspect of how information is presented to end users[9]. For instance, the way feedback is framed to train AI, as well as prompts are created for AI during any phase, can be subject to biased framing, and if not monitored, can lead to skewed AI decision-making. Additionally, since how people think is often different from what they actually do [10], users may appear to trust an explanation or model output but behave differently when interacting with the system.
Conclusion
At present, AI can explain itself in specific scenarios, but we as a people are not privy to its internal machinations, as the language that AI computes on for providing a prediction is alien to us. Furthermore, understanding that AI’s explanations not only stem from inputs during the explanation phase, but start earlier in the pipeline during RLHF training is critical.
End users see AI on a surface level; so if we want to incite trust in any inputs provided by AI, designing AI explanations requires attention not just to technical accuracy, but to how users learn, internalize, and act on the information; balancing conditioned responses, learned understanding, and real-world behavior.
References:
Hassija, V., Chamola, V., Mahapatra, A. et al. Interpreting Black-Box Models: A Review on Explainable Artificial Intelligence. Cogn Comput 16, 45–74 (2024). https://doi.org/10.1007/s12559-023-10179-8
Lipton, Z. C. (2016). The Mythos of Model Interpretability. ArXiv. https://arxiv.org/abs/1606.03490
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016). "Why Should I Trust You?": Explaining the Predictions of Any Classifier. ArXiv. https://arxiv.org/abs/1602.04938
Salih, A.M., Raisi-Estabragh, Z., Galazzo, I.B., Radeva, P., Petersen, S.E., Lekadir, K. and Menegaz, G. (2025), A Perspective on Explainable Artificial Intelligence Methods: SHAP and LIME. Adv. Intell. Syst., 7: 2400304. https://doi.org/10.1002/aisy.202400304
Aysel, H.I.; Cai, X.; Prugel-Bennett, A. Explainable Artificial Intelligence: Advancements and Limitations. Appl. Sci. 2025, 15, 7261. https://doi.org/10.3390/app15137261
Burger, C., Chen, L., & Le, T. (2023). Are Your Explanations Reliable? Investigating the Stability of LIME in Explaining Text Classifiers by Marrying XAI and Adversarial Attack. ArXiv. https://arxiv.org/abs/2305.12351
Casper, S., Davies, X., Shi, C., Gilbert, T. K., Scheurer, J., Rando, J., Freedman, R., Korbak, T., Lindner, D., Freire, P., Wang, T., Marks, S., Segerie, C., Carroll, M., Peng, A., Christoffersen, P., Damani, M., Slocum, S., Anwar, U., . . . Sadigh, D. (2023). Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback. ArXiv. https://arxiv.org/abs/2307.15217
González Barman, K., Lohse, S. & de Regt, H.W. Reinforcement Learning from Human Feedback in LLMs: Whose Culture, Whose Values, Whose Perspectives?. Philos. Technol. 38, 35 (2025). https://doi.org/10.1007/s13347-025-00861-0
Whitenton, K. (2024, July 16). Decision frames: How cognitive biases affect UX practitioners. Nielsen Norman Group. https://www.nngroup.com/articles/decision‑framing‑cognitive‑bias‑ux‑pros/
Laubheimer, P. (2024, March 1). Attitudinal vs. behavioral research in UX. Nielsen Norman Group. https://www.nngroup.com/articles/attitudinal‑behavioral/
