What Are the Ethical Risks of Your AI Project?

AI systems may cause unintended harms and their increasing adoption leads to additional risks and scrutiny. In this blog post, I present tools project teams can use to identify ethical concerns associated with their AI use cases. I first illustrate why this is so challenging, I then discuss various ethical checklists that were designed for this purpose, and I finally provide some recommendations on how to select such a checklist.

Identifying Potential Ethical Concerns Is Challenging

Let’s start with two real-life examples to show the complexity of such assessment. Imagine first that you manage an online advertising campaign on Facebook to promote STEM careers. You want to reach both men and women so you set the same “maximum bid per click” (i.e., the maximum price you’re willing to pay for one click on the ad) for these groups. But then, something surprising happens: your ad ends up being shown more to men, and up to 45% more for the 35-44 age group. How come? Does the ad-serving algorithm simply reflect actual consumer behavior and maybe the fact that women click less on your ads? Or has the algorithm been contaminated by biased data which made it assume such a behavior?

Through a field test in a similar context, two researchers concluded that neither of these explanations was correct. Women tended to click more on the ads and the disparate number of impressions was uncorrelated with the likely level of institutional bias in the countries covered by the campaign. The real cause was the “economics of ad delivery” — since women are more valuable targets for advertisers, the ad was more likely to be outbid for female consumers.

Let’s say now that you’re a machine learning engineer in charge of developing the recommendation engine of a streaming platform. How do you set the objective for your algorithm? After discussing with various internal stakeholders, it seems clear that you should maximize “engagement”, i.e. the time spent by users on your platform or the number of videos viewed. This metric is directly connected to your platform’s revenues. Thanks to the “wisdom of the crowd,” maximizing engagement also seems a way to promote the best quality content and provide a great user experience. However you soon realize the alarming consequences of this simple choice. Given the “natural pattern of borderline content getting more engagement” (in Mark Zuckerberg’s words), your recommendation engine now makes conspiracy or extremist videos more visible and soon enough, ill-intentioned users discover this trend and create purposely divisive content to generate revenues.

These examples illustrate some of the challenges of anticipating the negative side effects of your AI projects:

• Harms can take many shapes. The first example relates to fairness whereas the second one concerns the social cohesion and the vitality of democratic life.
• Consequences may be indirect or occur only in specific circumstances. For instance, it wasn’t necessarily obvious that extremist or conspiracy videos needed to be specifically taken into consideration to understand the impact of the recommendation engine’s design.
• Technical expertise and domain expertise are both required. A data scientist may be unfamiliar with the subtleties of the online advertising market while an advertising expert may be unable to foresee the implications of technical choices for the algorithms.
• The root cause of the harm may be outside of the machine learning system per se. In the case of the targeted ads, the estimates of the click propensities were unbiased but it’s the use of these estimates in a wider socio-economic context which led to disparate treatment.

Ethical Checklists Have Been Specifically Designed for AI Use Cases

Fortunately, several organizations published ethical checklists to guide practitioners through the identification of the potential ethical concerns of AI use cases. For example:

• DJ Patil, Hilary Mason and Mike Loukides proposed a simple ethical checklist in an article entitled “Of Oaths and Checklists.”
• A team of Microsoft researchers published an “AI Fairness Checklist,” which, as its name suggests, is restricted to fairness and discrimination issues.
• David Leslie of the Turing Institute included a “stakeholder impact assessment” in “Understanding artificial intelligence ethics and safety,” a guide for the responsible design and implementation of AI systems in the public sector.
• The “High-Level Expert Group on Artificial Intelligence” set up by the European Commission published a comprehensive “Assessment List for Trustworthy Artificial Intelligence” along with its “Ethics Guidelines for Trustworthy Artificial Intelligence.”

In a nutshell, these ethical checklists are just lists of questions to structure your assessment of the ethical aspects of your AI projects. They facilitate the communication between project team members and stakeholders, they contribute to harmonizing practices, and they help make and document decisions. Furthermore, their main benefit may simply be their cultural impact throughout your organization: by opening debates about the ethical implications of AI projects, they contribute to raise awareness on this topic and foster a questioning attitude.

So, what questions are included in these ethical checklists? They of course differ from one another but some essential questions are often or always present:

• What are the purpose and high-level objectives of the project? What are the metrics measuring the project’s success?
• What persons will be affected by the project, directly or indirectly, and in normal or anomalous circumstances? How will they be affected?
• What are the potential biases included in the data used to train models?
• How can adversaries abuse the AI system?
• What are the measures foreseen to address the ethical concerns of the project (e.g. to mitigate model bias, to protect personal data, etc.)?
• How will the potential ethical impacts of the AI system be monitored once it is in production? How will anomalies and complaints be addressed?

How to Choose Between These Ethical Checklists?

The ethical checklists share other important features. First, they are broad in scope and illustrate the complexity of the Responsible AI topic, as well as the need to combine various expertises and perspectives. In particular, they generally cover various categories of concerns:

• Fairness (i.e., not performing differently for certain groups in ways considered undesirable)
• Privacy (i.e., protecting personal information)
• Safety and robustness (i.e., functioning reliably and accurately even under harsh conditions)
• Security (i.e., not being abused by malicious actors)
• Other societal impacts (e.g., impacts on jobs, society at large, or democracy)

The ethical checklists also tend to touch upon all stages of the AI project lifecycle: the scoping of potential AI use cases, the exploratory analysis of datasets, the training of machine learning models, their inclusion in a broader IT system and the deployment and operation of such systems. For some of the ethical checklists, the questions are specifically adapted to each of these stages. In any case, the four checklists encourage their users to periodically revisit their answers.

A final commonality between the checklists is that they do not provide an assessment or score that would tell you, for example, whether the identified ethical concerns are significant or whether your mitigation measures are adequate. This is adequate because these complex issues require fine-grained analyses and any kind of mechanical appraisal would probably be too coarse to help.

If you’re now convinced that you should use such an ethical checklist to assess your use cases in a structured manner, the question becomes: which one to choose? There isn’t an obvious answer because none is clearly more appropriate than the others for all situations. With the various ethical checklists, the level of detail, the guidance, but also the “administrative burden” differ. For example, there are more pages in the fourth checklist above than there are questions in the first one! You then need to find the right balance between them, given the severity of the potential impacts of your use cases, the AI maturity of your internal stakeholders, and the culture of your organization, so that these ethical assessments stay meaningful and efficient. To help you with this choice, you can find a more detailed comparison of the four checklists at the end of this blog post.

In any case, don’t hesitate to customize your ethical checklist to better take into account the lessons learned from its use and reflect the needs of your organization. For example, if your organization carries out Data Protection Impact Assessments (DPIA) under article 35 of the GDPR, you should align the two approaches to avoid redundancies or inconsistencies.

Conclusion

Assessing potential ethical concerns for an AI use case is crucial given the significant risks of inadvertent harms. This should be done early in the project lifecycle and regularly reconsidered to take into account the progress made or context changes.

You can use one of the ethical checklists that were designed for this purpose. They’ll give you a head start but it’ll still be up to you and your organization to draw the necessary conclusions. This requires leaders to be aware of the potential ethical implications of AI, as well as thoughtful processes to incorporate technical and business expertise, engage stakeholders, and make difficult decisions. Various practices have recently emerged to enable such deliberation, for example: AI ethics committees, accountability reports, and public notice of existing and proposed uses of AI by public agencies.

More broadly, these ethical checklists should be part of a coherent Responsible AI framework, that would also include other methodological assets, such as ethics guidelines, a template for algorithmic impact assessments, an audit framework, and templates to document datasets and models. If you’re interested in all this, make sure to follow the progress of the “Responsible AI Working Group” of the “Global Partnership on AI.” In December 2020, it will report on the results of its first project which aims at analyzing existing and potential Responsible AI initiatives and “recommend new initiatives and how they could, in practice, be implemented and contribute to promote the responsible development, use and governance of human-centered AI systems.”

Finally, we at Dataiku endeavor to support companies in building and implementing a Responsible AI strategy. We'll keep improving the features of Dataiku DSS related to collaboration, governance and Explainable AI and, if you’re a Dataiku customer and concerned about Responsible AI topics, we’ll be happy to better understand your needs and discuss ways to assist you. Please contact us through your Customer Success Manager!

Many thanks to Winston Maxwell, Astrid Bertrand, Triveni Gandhi, Catie Grasso, Léo Dreyfus-Schmidt, and Sophie Dionnet for their feedback on an earlier version of this blog post.

Bonus: Comparison of the Ethical Checklists