AI has brought significant changes to various industries, but what happens when AI perpetuates inequality and harm? Inclusivity is often overlooked when developing AI systems, which leads to repeated inequalities, particularly for Black end users, with harmful effects when scaled. As data practitioners and end users, we must work to improve our AI systems to prevent such failures.
This is something that I think about everyday and partner with our customers at Dataiku to implement Responsible AI in their AI practice. In this article, we’ll explore key problem areas for non-inclusive AI, share insights from Dr. Brandeis Marshall on how to combat these issues before models are deployed, and discuss how to put these principles into practice in your own AI systems.
Triveni Gandhi and Dr. Brandeis Marshall at the Everyday AI Conference New York
Examples of Non-Inclusive AI
Unfortunately, it’s not hard to find examples of AI that exhibit racism across many industries, such as facial recognition technology, medical algorithms, and language models like ChatGPT. Facial recognition tools have shown to have higher error rates among images of darker-skinned women, leading to a variety of harms, including wrongful arrests and mislabeling. In medical algorithms, using prior healthcare spending as a predictor for current health needs has resulted in Black patients being consistently placed in the same risk category as far healthier white patients.
Even ChatGPT had cases of blatant racism and harmful language after its release, but the number of incidents has gone down as OpenAI further refined it. Because AI is trained on existing data, and anti-Black sentiment is a real and systematic problem in our world, these same beliefs and discrepancies are repeated at scale through these kinds of algorithms.
Bringing Ethics to the Table
Fortunately, a large field of AI ethicists, data scientists, researchers, and developers has emerged in the past six years highlighting the number of issues that emerge from poorly built and unvetted technology. Books like Dr. Safiya Noble’s “Algorithms of Oppression” or Dr. Joy Bulowami’s “Coded Bias” have highlighted the problems of racist AI and brought attention to the issue across the field of data science and machine learning (ML).
Other researchers like Dr. Timnit Gebru and Dr. Margaret Mitchell have proposed methods for assessing data and models before deploying them to production, such as “Datasheets for Datasets” or “ModelCards.” These tools are the first step in ensuring developers are proactively checking for and documenting any potential issues of bias.
Leading the Way on Responsible Tech
Another trailblazer in the space of bringing anti-racist and inclusive perspectives to technology is Dr. Brandeis Marshall, author of “Data Conscience,” which outlines the ways in which anyone can elevate their approach to data analytics, ML, and AI. In her book, Dr. Marshall raises the crucial point that many AI systems are used as single decision points for high stakes outcomes, which is why better organization and oversight from the beginning is important for any team looking to create more effective and fair data products.
I recently spoke to Dr. Marshall at Dataiku’s Everyday AI Conference in NYC about her book and takes on developments in anti-racist technology. She emphasized that the reason we are seeing so many failures in AI today is because many systems were built years ago with biased data and systems. The notorious lack of diversity in tech in its earliest days meant that developers weren’t thinking about how their systems could create harm, especially for people who don’t look like them.
Even now, she argues, “Tech keeps trying to fix future problems without addressing the current ones, which is why Responsible AI is so important.” For example, tech has pushed us towards the Metaverse or open source language models like ChatGPT without first acknowledging ways in which content moderation systems are still underdeveloped and lacking in creating safety for people of color, women, and sexual minorities.
So what can tech do to move forward in a way that promotes more fair and equitable outcomes for everyone? Dr. Marshall outlines three areas to focus on:
1. Increase Diversity in Tech
One of the most immediate ways to create better tech is to include more diverse and varied voices at the table when building and deploying new products. This means hiring more women, people of color, and people from non-traditional backgrounds on all kinds of teams — from business users to developers to implementers.
Dr. Marshall argues diversity is crucial at all stages of the development cycle, which is why “Black and brown people should be employed as designers and developers of data and AI systems, not just as testers and consumers of the end product.” By adding different voices to technology teams, and giving them the space to challenge assumptions, companies can increase the benefits of AI while reducing potential harms before a product is deployed.
2. Blueprints for Data Products
Speaking of harms, Dr. Marshall also recommends companies use blueprints for organizing data products and the associated risks and benefits. Especially in the cases where single decision points can have a huge impact on someone (such as in the previous examples of racist AI), it is important to think through harms before a product goes out the door.
The process of completing a blueprint (or standard) for Responsible AI can act as a measure to ensure developers “stop and think at the computing level” about the products they are building. Blueprints support a socio-technical mindset that bridges real world impacts with data-driven decision making and in the long term also serve as points for accountability when unintended consequences arise.
3. Transparency
Finally, transparency is an important factor in Dr. Marshall’s approach to less racist AI. Whether they are working on new or existing products, data practitioners should be measuring the differing impacts of their work. The simplest way to get started is by visualizing how input data varies across key social groups — such as race, gender, education levels, or even zip code — to assess any underlying disparities before moving forward with a project.
Making the analysis available and a part of regular review during sign-off processes is a core part of ensuring visibility across the development pipeline. The concept of transparency goes all the way to the consumer level as well: It is important for businesses to be clear when they are using AI systems to make decisions about an end user.
It is important for us to acknowledge the limitations and potential harm of AI, yet we should also recognize that there are real and actionable solutions and ways to make AI more equitable and inclusive. And hopefully, with this article, you have some actionable ideas to implement — remember: collaboration, governance, and explainability. It’s an exciting time to see developments in the field of Responsible AI and AI Governance and how they are crucial for creating better outcomes and reducing the potential for bias and discrimination in AI systems. Moreover, the continued efforts of leaders like Dr. Marshall to raise awareness of potential failings in AI can ensure the future of AI is fair, just, and inclusive.