Explainable AI has become a powerful mitigating force that enables modelers, regulators, and laypeople to have more trust and confidence in machine learning models. However, it’s not always easy for data scientists to implement models that are explainable. To help them do so in practice — and raise awareness for some of the pitfalls they may encounter — we held a webinar on the topic that focused on the tradeoffs between white-box and black-box models, fairness and explainability concepts both broadly and in Dataiku, and why Dataiku’s R&D team chose the path they did for explainable AI features included in the platform.
The session was designed to equip data scientists (and data teams at large) with best practices in order to, ultimately, build models in a way that is collaborative and sustainable for the business. But one thing in particular was most memorable to me as I moderated the session. At the end of the webinar, Dataiku’s Christina Hsiao, senior product marketing manager, asked Du Phan, research scientist, what he sees as the value of explainable AI — and his response has stuck with me ever since.
Basically, before Du responded, he shifted his answer to a real-life example of an opaque “system” — the medical doctor. Most people think of a doctor as a kind of black-box system that takes symptoms and test results as input and predicts the diagnosis as output. Without providing information about the way medical tests and evaluations work, the doctor delivers a diagnosis to a patient by explaining high-level indicators that are revealed in the tests.
We don’t really understand what those tests are, yet we trust them for matters of life and death (at least I hope you do). This shows us that not all opaque systems are bad, just the ones that we don’t trust.”
-Du Phan, Research Scientist, Dataiku
Du went on to say that, in his opinion, that is what we want to accomplish with explainable AI. The central idea of explainable AI is about establishing trust from all of the involved parties, such as:
- The data scientist who builds the models wants to make sure that the model conforms to their behavioral expectations.
- The stakeholders who use the model to make decisions want to be assured that the system is making the correct output (without understanding the mathematical details behind the scenes).
- The public, those who are affected directly or indirectly by the predictions of the model, want to ensure that they are not being treated unfairly.
The important point to note is that different individuals will have different expectations about the explainability of a system. It depends on their roles and their understanding of societal and domain standards. Therefore, the decision of whether or not a machine learning system is responsible, fair, and correct should and must always be the result of a discussion between all parties, using all the signals provided by the system.